History log of /external/autotest/server/autoserv_parser.py
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
cbeab1213ad41c3631a71c97ec3eddc1aee20049 30-May-2017 Prathmesh Prabhu <pprabhu@chromium.org> [autotest] Respect exclusive --in-lab or --host-attributes arguments

These two arguments to autoserv are not intended to be used together.
The only user of --host-attributes are test_that / test_droid that use
it for tests run outside the context of the lab via
test_runner_utils.perform_run_from_autotest_root.

This CL enforces this mutual exclusion to make way for a future CL that
depends on this property.

+ As we begin to use HostInfoStore for storing the attribute
information, copy the host_attributes into the store.

BUG=chromium:678430
TEST=(1) test_that / test_droid still works.
(2) in-lab hwtests via moblab.
CQ-DEPEND=CL:519569

Change-Id: I0dcb3096a4797143df796855bf3babc155e76c67
Reviewed-on: https://chromium-review.googlesource.com/517771
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
/external/autotest/server/autoserv_parser.py
1bf60eb788365f083d0ee8045a6556f906149dec 02-Dec-2015 Simran Basi <sbasi@google.com> [autotest] autoserv add --lab & --host_attributes arguments

Added two new flags to autoserv.

--lab indicates that autoserv is running in the lab and has
the full Autotest infrastructure at its disposal.

--host_attributes allows host attribute information that is
usually in the database to be retrievable from the command
line arguments.

If --lab is pulled in, autoserv will request the host attributes
from the database at test runtime.

From here this change, then updates the concept of the "machines"
list that test control files receive to now be a list of dicts
that contain the machine hostname and host attributes. This will
enable identifing information the hosts library needs to create
host objects to be available whether or not there is a database
present.

BUG=chromium:564343
TEST=local autoserv runs. Also verified scheduler changes work
via MobLab. waiting on trybot results.
DEPLOY=scheduler

Change-Id: I6021de11317e29e2e6c084d863405910c7d1a71d
Reviewed-on: https://chromium-review.googlesource.com/315230
Commit-Ready: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/server/autoserv_parser.py
c2d99cf743c86fe0519f155ffbcad54df31bd742 18-Nov-2015 J. Richard Barnette <jrbarnette@chromium.org> [autotest] Remove the concept of 'protection level' from repair.

In original Autotest, hosts have a 'protection level' that is
passed to repair jobs in order to limit the specific kinds of repair
that can be done. For the CrOS lab, this concept is largely not
useful; we want a uniform repair procedure determined by the kind of
DUT being repaired, not by a database configuration.

This removes the concept of 'protection level' from the repair flow;
repair steps are now determined only by the class of Host that
handles the operation. The protection level remains in the
database, but the only meaningful value is 'Do not verify'. That
setting on a host will prevent it from running any verify, cleanup,
or repair tasks. This is done for the sake of non-DUT hosts that
exist in the CrOS lab AFE database.

BUG=None
TEST=unit tests, and run repair in a local instance

Change-Id: I1d512eb2b1fe604b70fec00b3a290f5f6b9d9beb
Reviewed-on: https://chromium-review.googlesource.com/312979
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
/external/autotest/server/autoserv_parser.py
d72548622a26b10fb8555d7c43dfb59bf505bd36 07-Aug-2015 Dan Shi <dshi@chromium.org> [autotest] Fix formatting in autoserv argument help strings.

This is the leftover from converting optparser to argparser.

BUG=None
TEST=/usr/local/autotest/server/autoserv --help

Change-Id: I748a55a45240b8d63d9de6b29c9c5b47f2ca4841
Reviewed-on: https://chromium-review.googlesource.com/291458
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
/external/autotest/server/autoserv_parser.py
888cfcaeeffdf302e25b066d9bef9ba535ebbc20 01-Aug-2015 Dan Shi <dshi@chromium.org> [autotest] Update autoserv parser to replace optparse with argparse to support unknown arguments

This fix should avoid lab failures like bug 515339

BUG=chromium:490456
TEST=local run suite jobs, unittest

Change-Id: I06776c080bac2ff438582637a4b93ef279a5ec8a
Reviewed-on: https://chromium-review.googlesource.com/290106
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/server/autoserv_parser.py
70647cafbd061a7754ac304fd9dc067f2b6dbab4 17-Jul-2015 Dan Shi <dshi@chromium.org> [autotest] Save parent job id, build, board and suite info to tko_jobs.

parent job id is passed in through autoserv commandline. autoserv saves the
value to keyval file in results folder. The parser job then reads the parent
job id from keyval file.

build, board and suite info are parsed from job name. The label column in
tko_jobs is essentially the job name. However, that column has a size limit of
100 characters, thus the name could be truncated. This CL parse the actual job
name to get the build, board and suite info and save to tko_jobs table.

BUG=chromium:509770,chromium:509901
TEST=local test
CQ-DEPEND=CL:285026

Change-Id: I06b073b052a9d07ffd36308b1682a7bc12699898
Reviewed-on: https://chromium-review.googlesource.com/286265
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Mungyung Ryu <mkryu@google.com>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/server/autoserv_parser.py
36cfd831af781eed114337efa5b90c103a49b502 10-Oct-2014 Dan Shi <dshi@chromium.org> [autotest] Support updating both firmware and CrOS from run_suite

Add firmware_build option for run_suite to pass in firmware build.
Save the build containing server-side test to job_keyvals.

Design doc:
https://docs.google.com/a/google.com/document/d/115aAHyZaatFDzuKWm61Sj2fImH4NDWikXqifvWKbbW0/edit#

BUG=chromium:270258
TEST=local run_suite:
make sure it works with older build:
./run_suite.py -b veyron_jerry -i trybot-veyron_jerry-paladin/R45-7086.0.0-b8 -p suites -s dummy_server

Call failed as --firmware_ro_build not supported yet.
./run_suite.py -b veyron_jerry -i trybot-veyron_jerry-paladin/R45-7122.0.0-b11 --firmware_build veyron_jerry-firmware/R41-6588.9.0 --test_source_build trybot-veyron_jerry-paladin/R45-7122.0.0-b11 --firmware_ro_build veyron_jerry-firmware/R41-6588.9.0 -p suites -s dummy_server

Call failed as the firmware build does not have test_suites package built:
./run_suite.py -b veyron_jerry -i trybot-veyron_jerry-paladin/R45-7122.0.0-b11 --firmware_build veyron_jerry-firmware/R41-6588.9.0 --test_source_build veyron_jerry-firmware/R41-6588.9.0 -p suites -s dummy_server

make sure it works with new build:
./run_suite.py -b veyron_jerry -i trybot-veyron_jerry-paladin/R45-7122.0.0-b11 --firmware_build veyron_jerry-firmware/R41-6588.9.0 --test_source_build trybot-veyron_jerry-paladin/R45-7122.0.0-b11 -p suites -s dummy_server

DEPLOY=apache,scheduler

Change-Id: I50d23a7e81e4d6224b3483111110f910eb407074
Reviewed-on: https://chromium-review.googlesource.com/272793
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/server/autoserv_parser.py
b669cbdea7f4bfc9cbfcea48d9cf9ce62f0deff5 13-Sep-2013 Dan Shi <dshi@chromium.org> [autotest] Disable install autotest from job_repo_url for test_that

use_packing option for autotest._install method is True by default. When using
test_that to run test, DUT will try to install autotest package from
job_repo_url with server configured in global_config (cautotest). For DUT not
located in lab, there is no job_repo_url existed for such DUT in cautotest, and
therefore autotest will try to copy over the local files. However, when a user
try to use a DUT in the lab to run test_that, autotest package is downloaded
and installed from job_repo_url. That leads to a problem that developers can't
use test_that to verify autotest code.

The fix is to add a new option no_use_packaging to autoserv, and thread the
option to autotest._install. The option is set to False by default, and set
to True in test_that. Also, when this argument is set to True, any package
installation will use AutoservFetcher.

BUG=chromium:265673
TEST=run following command and confirm local autotest code is copied over to
the DUT: test_that -b lumpy 172.22.75.225 dummy_Pass

Change-Id: I365f5f7dc79c8e65cddaa0af77f1f1a1e4b0012a
Reviewed-on: https://chromium-review.googlesource.com/169343
Reviewed-by: Mungyung Ryu <mkryu@google.com>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
Tested-by: Mungyung Ryu <mkryu@google.com>
/external/autotest/server/autoserv_parser.py
ec1d47d85cc83f30631518d8fbb6406036a3ac39 13-Feb-2015 Dan Shi <dshi@chromium.org> [autotest] Add support for scheduler to honor require_ssp attribute in control file

This CL adds changes to pipe require_ssp attribute in control file to autoserv
command. Following are the work flow:

1. The control file parser stores require_ssp attribute value in afe_jobs table.
2. QueueTask compiles command line list, --require-ssp option will be added to
the command line list if following conditions are met:
a. AUTOSERV/enable_ssp_container in global config is True
b. The test is a server-side test
c. require_ssp for the job entry is None or True.
3. When agent_task tries to call run method to run the command, it will check if
there is any drone supporting server-side packaging first. If no drone is found,
the agent task will will run the command in a drone without using server-side
packaging. A warning will be posted in the autoserv log.
4. If a drone without SSP supported is assigned to a test requires SSP, the test
will be run without ssp.

BUG=chromium:453624
TEST=unittest, local test:
set AUTOSERV/enable_ssp_container to True in shadow config;
Create a job for dummy_PassServer in AFE, check require SSP, confirm the job
succeeds but with a warning in the autoserv log.

Create a job for dummy_PassServer_nossp in AFE, uncheck require SSP, confirm
the job passes without warning in the autoserv log.

set AUTOSERV/enable_ssp_container to False in shadow config, restart scheduler.
Create a job for dummy_PassServer in AFE, check require SSP, confirm the job
succeeds without warning in the autoserv log.

also run test_that in local chroot to make sure test_that is not affected.

DEPLOY=apache,scheduler, db migrate must be done before push this CL to prod.

Change-Id: I02f3d137186676ae570e8380d975a1bcd9ffbb94
Reviewed-on: https://chromium-review.googlesource.com/249841
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/server/autoserv_parser.py
df15ec56a94b6f7bad47d1c9d75a57c7b6f7feec 01-Mar-2014 Alex Miller <milleral@chromium.org> [autotest] Remove label dependant special task actions migration code

BUG=chromium:334418
DEPLOY=sync everywhere, then restart scheduler
TEST=special tasks run fine

Change-Id: Ib0c09095ff323ff53defef33509e1642560da0c6
Reviewed-on: https://chromium-review.googlesource.com/188456
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
/external/autotest/server/autoserv_parser.py
667b5f2b48c6096f944a34ca53e769c1031be45f 01-Mar-2014 Alex Miller <milleral@chromium.org> [autotest] Special task control segments take label-dependant actions

Autoserv accepts the labels of a job via the --job-labels argument, and
feeds them into the control segment by injecting the variable
`job_labels` into the global namespace.

cleanup/verify/repair/reset now use the same sort of loop that provision
does to look up additional tests to run that depend on the labels passed
into the control segment.

Note that some care has to be taken here, as we're moving the labels
from --provision to --job-labels, and I'd rather remove a bit of
migration code later than make for a complicated deploy.

BUG=chromium:334418
TEST=special tasks still run fine

Change-Id: I1596b49905f1500e8ce01e81d9aab2a6c7db111f
Reviewed-on: https://chromium-review.googlesource.com/188454
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
/external/autotest/server/autoserv_parser.py
8c1bb42e58dc78740604059b607eb1bc3735587d 14-Sep-2013 Aviv Keshet <akeshet@chromium.org> [autotest] fix autoserv_parser's handling of quoted strings

BUG=chromium:291289
TEST=Ran:
$ test_that -b lumpy --args "a b c d e='f g h'" $host \
dummy_RepeatArgs
Saw the args list logged as:
['a', 'b', 'c', 'd', 'e=f g h']
which is right. Instead of the old behavior
['a', 'b', 'c', 'd', "e='f", 'g', "h'"]
which is dumb.

Change-Id: Ib842229084f92bb069b773dd8d7fe91fc5e3385e
Reviewed-on: https://chromium-review.googlesource.com/169450
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
/external/autotest/server/autoserv_parser.py
6cc20de4cde89b9613d051dc947f68e425f342f3 07-Sep-2013 Fang Deng <fdeng@chromium.org> [autotest] Set default value of --ssh_options to an empty string

This is a better solution to ensure ssh_options is not None.

TEST=Local job runs and ssh commands in the job succeed.
BUG=chromium:286498

Change-Id: I6f05972e476bd1de87623615c157c5db78b06247
Reviewed-on: https://chromium-review.googlesource.com/168492
Tested-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
/external/autotest/server/autoserv_parser.py
c5947faa755945f81537c6c33c322dccacac0ade 04-Sep-2013 Aviv Keshet <akeshet@chromium.org> [autotest] add a --ssh_options option to autoserv

This CL follows the template of
https://chromium-review.googlesource.com/167415

BUG=chromium:285378
TEST=Unit tests pass. Manually verified that specifying
--ssh_options="-F /dev/null -i /dev/null" caused those flags to be
threaded into ssh commands.

Change-Id: Ie0496862d05d42192d55501fc0ed1c3a6274fe79
Reviewed-on: https://chromium-review.googlesource.com/168099
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
/external/autotest/server/autoserv_parser.py
18ee314517d7977c03436f179c202263890938c6 13-Aug-2013 Aviv Keshet <akeshet@chromium.org> [autotest] add an --ssh_verbosity flag to autoserv

This CL adds an autoserv flag to tune the ssh logging verbosity. This is
accomplished by adding a global process-wide GLOBAL_SSH_COMMAND_OPTIONS
variable to site_host, to which autoserv tacks on a '-v' or '-vv' or
'-vvv' string if necessary. This is, admittedly, a little dirty. But the
make_ssh_command does not take a host object as an argument, so to add a
option host-wide to ssh_commands at the host object level would require
some major changes.

BUG=chromium:271621
TEST=Manually ran client/tests/dummy_Pass with various --ssh_verbosity
values, both valid and invalid. Saw that desired ssh verbosity was
realized.

Change-Id: If705c03f9d9d70de89642f42904e2e9fdc40bf61
Reviewed-on: https://gerrit.chromium.org/gerrit/65625
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
/external/autotest/server/autoserv_parser.py
8a91f23cf8a705946a71220dc0c6f06e0987b74a 09-Jul-2013 Christopher Wiley <wiley@chromium.org> autotest: Add flag to autoserv to disable sysinfo step

This saves us time when we don't care about this step.

BUG=chromium:258507
TEST=Ran autoserv directly with --disable_sysinfo and observed
appropriate messages omited/emited.

Change-Id: I57af12a2216a87ecb2c5719591eb1d2497731404
Reviewed-on: https://gerrit.chromium.org/gerrit/61299
Tested-by: Christopher Wiley <wiley@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Christopher Wiley <wiley@chromium.org>
/external/autotest/server/autoserv_parser.py
07e09aff0baf871b33e5479e337e5e3e0523b729 12-Apr-2013 Dan Shi <dshi@chromium.org> [Autotest] merge cleanup and verify

The objective of this CL is to merge cleanup and verify into a single job to
reduce run time of each test. In existing design, by default, a cleanup job is
scheduled after a test is finished, and a verify job is scheduled before a
test is started. By merging these two jobs together, we are seeing the total
run time of these two jobs is reduced from about 47s to 37s, around 10s saving.
That does not include the saving on scheduler to schedule two jobs, which may
take another 5-10s.

The design is to create a new special task, reset, which runs at the beginning
of a job by default. Verify task is changed to not to run by default before a
job starts. Cleanup job will only be run if a job is scheduled to reboot and
any test failed in that job.

BUG=chromium:220679
TEST=tested with run_suite in local machine
DEPLOY=afe,apache,scheduler,change all users' preference on reboot_after to
Never, sql: |update chromeos_autotest_db.afe_users set reboot_after=0|

Change-Id: Ia38baf6b73897b7e09fdf635eadedc752b5eba2f
Reviewed-on: https://gerrit.chromium.org/gerrit/48685
Commit-Queue: Dan Shi <dshi@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/server/autoserv_parser.py
f594c5ee0a769d00b0f0e8a6ebe1c6e19cf53417 04-Jul-2013 Christopher Wiley <wiley@chromium.org> autotest: Allow crash collection to be skipped

Expose this functionality as a flag that defaults to collection crashes
as normal. This makes test runs faster for developers iterating a
test/patch.

BUG=chromium:250366
TEST=Ran a test, observed that collection was skipped.

Change-Id: I61d76ef3618a6bc7e3228fa7ee8bf32dafaec53d
Reviewed-on: https://gerrit.chromium.org/gerrit/60953
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Christopher Wiley <wiley@chromium.org>
Tested-by: Christopher Wiley <wiley@chromium.org>
/external/autotest/server/autoserv_parser.py
cb6f1e2209ae6ea09c78317b114db6e51f6e255b 29-Jun-2013 beeps <beeps@chromium.org> [autotest] Teach autoserv to check a hosts job_repo_url.

Autosev now understands a verify_job_repo_url flag to
mean 'execute a control segment to verify said hosts
contain a valid job_repo_url'. It does this by re-staging
autotest packages at the job_repo_url; If the devserver
embedded in the job_repo_url is unresponsive we get another
devserver, stage autotest on it, and update the job_repo_url.
If the job_repo_url is None we leave it, on the assumption
that the host will go through a re-image before it runs
any tests, which will give it a new job_repo_url.

Pros:
1. If we are unable to stage autotest packages at
job_repo_url the test will fail, soon and fast.
2. Since we perform these actions in the server_job itself
we won't see package installation exceptions later on,
which are just misleading today since we end up
rsyncing client autotest anyway.

Cons:
1. The re-image job will actually try staging the contents
of an autotest build it will surely not need. Something
along these lines is unavoidable in most clean solutions,
though.

TEST=
1. Ran smoke suite, deleted autotest packages on devserver
just before security_Minijail, made sure it passed.
Also deleted the entire build directory.
2. Ran a suite with a bad devserver in the job_repo_url
and confirmed we resolve to a good one, restage
autotest and reset the job_repo_url.
3. Ran a suite after recovery (job_repo_url=None).
4. Invoked autoserv with a list of machines.
5. Checked that cleanup, verify, repair don't get the
verify_job_repo_url flag, and hence, don't re-stage.
Cleanup still passes because it rsyncs client
autotest in the invalid job_repo_url case.
6. Server job fails before running tests if devservers
are down or we're unable to change job_repo_url.
7. BVTs pass

BUG=chromium:255114

Change-Id: I3c5f445962707a0a089f9d755aed4c4d0fdbd8f2
Reviewed-on: https://gerrit.chromium.org/gerrit/60578
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Prashanth Balasubramanian <beeps@chromium.org>
Commit-Queue: Prashanth Balasubramanian <beeps@chromium.org>
/external/autotest/server/autoserv_parser.py
cb79ba7249e62ab26662362908a747fe8b153d93 29-May-2013 Alex Miller <milleral@chromium.org> [autotest] Add provisioning flag to autoserv.

autoserv now accepts a --provision flag that accepts a comma seperated
list of labels that it should provision the targeted machine(s) (via -m)
to. This list gets fed directly into the control segment, which handles
all of the processing of it and provisioning work.

It should also be loudly noted that since we run the provision control
segment as a normal test instead of directly executing the contents, one
cannot specify the -c flag, or else it will try to run the provision
control segment as a client-side test.

This is just some plumbing. It's pretty uneventful.

BUG=chromium:249437
TEST=The following:
1) Ran autoserv --provision cros-version:lumpy-release/R28-3993.0.0 and
the machine was provisioned correctly.
2) Ran autoserv -c --provision and received an error.

Change-Id: I03582a5210d182f171ef725c4e49cc83c1baf793
Reviewed-on: https://gerrit.chromium.org/gerrit/58384
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
/external/autotest/server/autoserv_parser.py
91493c88a1506729a6728660d90cfae591cd52aa 25-Jan-2013 Scott Zawalski <scottz@google.com> Introduce a mechanism for retrying tests at the control file level.

If a test returns unsuccessfully and --retry-test > 0 is specified the test
will be rerun. Options were added to autoserv and are passed through to the
client side job by modifying the base_job.

Tests that fail and then succeed within the --retry-test limits will have a
test keyval set with the number of attempts |test_retries_before_success|.

--retry-test can be utilized immediately in run_remote_test but will need to
be piped through the RPC system and down to the scheduler to provide the proper
--retry-test value from the scheduler's perspective.

Fixed random pylint errors consisting mostly of unused imports.

TEST=unittests
created a randomly failing client and server side test and ran it on devices
BUG=chromium-os:37158

Change-Id: Ibec3935b5f6fd28fc1b6eb7be55de27a571ad777
Reviewed-on: https://gerrit.chromium.org/gerrit/42043
Commit-Queue: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Scott Zawalski <scottz@chromium.org>
/external/autotest/server/autoserv_parser.py
861b2d54aec24228cdb3895dbc40062cb40cb2ad 04-Feb-2011 Eric Li <ericli@chromium.org> Merge remote branch 'cros/upstream' into master

Merged to upstream autotest @4749~@5215.

The entire change list description is too big to enlist here. Please refer to upstream (http://autotest.kernel.org/browser) for more details.

BUG=
TEST=emerged both x86 and arm build.
Tested emerged x86 build bvt against a chromeos device.

Review URL: http://codereview.chromium.org/6246035

Change-Id: I8455f2135c87c321c6efc232e2869dc8f675395e
/external/autotest/server/autoserv_parser.py
e0cbc912c81acfb8844510267b2cefd64d9ae478 11-Mar-2010 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Add support to autoserv for a --control-filename parameter, to allow users to
control where in the results directory autoserv will store the server control
file.

This also changes the archving stage in the scheduler to make use of this
argument, so that the control file from the archving stage is written to
control.archive and does not overwrite the control.srv from the job itself.

Signed-off-by: John Admanski <jadmanski@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@4294 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
4608b005f15444d2ec4601b8274828ad52b5ea51 05-Jan-2010 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Add a new Archiving stage to the scheduler, which runs after Parsing. This stage is responsible for copying results to the results server in a drone setup, a task currently performed directly by the scheduler, and allows for site-specific archiving functionality, replacing the site_parse functionality. It does this by running autoserv with a special control file (scheduler/archive_results.control.srv), which loads and runs code from the new scheduler.archive_results module. The implementation was mostly straightfoward, as the archiving stage is fully analogous to the parser stage. I did make a couple of refactorings:
* factored out the parser throttling code into a common superclass that the ArchiveResultsTask could share
* added some generic flags to Autoserv to duplicate special-case functionality we'd added for the --collect-crashinfo option -- namely, specifying a different pidfile name and specifying that autoserv should allow (and even expect) an existing results directory. in the future, i think it'd be more elegant to make crashinfo collection run using a special control file (as archiving works), rather than a hard-coded command-line option.
* moved call to server_job.init_parser() out of the constructor, since this was an easy source of exceptions that wouldn't get logged.

Note I believe some of the functional test changes slipped into my previous change there, which is why that looks smaller than you'd expect.

Signed-off-by: Steve Howard <showard@google.com>

==== (deleted) //depot/google_vendor_src_branch/autotest/tko/site_parse.py ====


git-svn-id: http://test.kernel.org/svn/autotest/trunk@4070 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
e7d9c605bacd7b1816987994ae18a68c63306a16 02-Jul-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Make the job executiontag available in both the server and client side job
objects as job.tag.

This is useful if your job would like to copy its data off directly to a
results repository on its own from the client machine.

Mostly small changes to pass the data down, though I did some docstring
cleanup near code that I touched which makes the diff larger.

The execution tag is taken from the autoserv -P parameter if supplied
and no explicit --execution-tag parameter is supplied. This prevents
the need to change monitor_db.py to pass yet another autoserv parameter.

Signed-off-by: Gregory Smith <gps@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3359 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
10d8417b8c3576e1ba80ce00b6ae2e9cea826bc0 19-Jun-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> add two logging-related options to autoserv:
--verbose sets the console to DEBUG level output (this is added to autotest client as well)
--no_console_prefix disables the logging prefix entirely on the console

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3301 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
374f3419ad5de9e9c8047928d0851732f0b7c6da 13-May-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Adds a -G to autoserv to be able to set host_group_name in the group keyvals
file that it writes. Useful only if someone has asked autoserv to run the
parser itself.

The scheduler does not need this as it will write that keyvals entry itself.

Adds a unittest for autoserv_parser and fixes a minor bug in --args support.

Signed-off-by: Gregory Smith <gps@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3137 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
012623f89c61d9daad23f9cbca4217a6a15676a1 13-May-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Fix args splitting when we don't have any

Signed-off-by: Martin Bligh <mbligh@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3126 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
a85d4677c9db86d2ad44954b3841456fda2b7f32 13-May-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Consolidate control files to one central one.
Make ktest_nightly use arg passsing instead of creating a custom file

Signed-off-by: Martin Bligh <mbligh@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3125 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
def0c3cccb3b77b2c7e3a90f427cf4a70292dfd9 25-Mar-2009 jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> Add support for running autoserv with a "--collect-crashinfo" flag
that tells autotest to run a job that includes ONLY the crashinfo
collection. This will pull back crashinfo, and if run against a
results directory from a crashed job it will also pull back any
client results that it can find on the remote host.

Risk: Low
Visibility: Adds a new mode to autoserv for just doing crashinfo
collection.

Signed-off-by: John Admanski <jadmanski@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2933 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
f82a182d5751be98b9c385ebdba4faa5b3da8e45 26-Feb-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Have autoserv_parser parse an empty argument list when initialized to
set default values to its configured options. This should fix broken
codepaths that assume they are always reached from autoserv when in fact
in certain cases they are reached from other standalone programs (like
the scheduler).

Signed-off-by: Mihai Rusu <dizzy@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2829 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
a5cb40631a50dcf9235d124613e281f0ffc4d959 17-Feb-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Cleaned up some useless imports and moved autoser_parser actual argument
parsing to be invoked by autoserv on startup instead of automatically on
import (and thus parsing/eating sys.argv from all other standalone
programs that somehow imported autoserv_parser).

Signed-off-by: Mihai Rusu <dizzy@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2801 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
a700772e70548b10721ba709c7ec5194bb6d7597 13-Jan-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Created a client/common_lib/utils/import_site_class function to take
care of site specific importing of classes. Modified code to use it.

Signed-off-by: Mihai Rusu <dizzy@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2626 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
d5ab8c5c44a01b9a9699db574973174570e10eea 03-Dec-2008 jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> Refactor the core pidfile code out of autoserv and into a common lib
module, and add a similar pidfile option to the parser.

Risk: Low
Visibility: Adds an (opt-in) .parser_execute pidfile to the parser.

Signed-off-by: John Admanski <jadmanski@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2538 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
f22fea8f311c4d955b171a2e7869732d6a1067e4 26-Nov-2008 jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> Add an --install-in-tmpdir option to autoserv that allows us to change
the default Autotest client install behaviour so that new clients
are installed in new temporary directories instead of the "standard"
locations. This only affects installs that don't explicitly request
a specific location.

The expected use for this is to allow the use of multiple server jobs
against the same machine without having them collide with one another.

Risk: Low
Visibility: Adds a new flag to autoserv, but the default behaviour
is unchanged.

Signed-off-by: John Admanski <jadmanski@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2510 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
80e1eba601ad04856da1c33dd60cb20035c899fb 19-Nov-2008 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Enable autoserv to operate without a results directory

This is incredibly useful for writing small, non-invasive scripts,
using the power of autoserv

Signed-off-by: Martin Bligh <mbligh@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2444 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
45ae819b2d2a67d0882edafaf1a8f7b95c3fb9d2 05-Nov-2008 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Add a formal cleanup phase to the scheduler flow.
-add a --cleanup or -C option to autoserv, which runs a new control segment, cleanup. this option and control segment obsolete the old -b option and reboot_segment control segment.
-change the RebootTask in the scheduler into a more generic CleanupTask, which calls autoserv --cleanup.
-change the host status "Rebooting" to "Cleaning"



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2377 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
7cd30fd9c5daca002046759e1bf44200803ff1e2 21-Oct-2008 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Enforce use of a directory to put results in from autoserv

Signed-off-by: Martin J. Bligh <mbligh@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@2312 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
b2bea302ba5b469bcfbb093ac5b713153fc80413 24-Jul-2008 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Added the aditional -s flag which specifies that a test is server side.
Will be ignored unless -c is also specified and will then throw an error.

Signed-off-by: Travis Miller <raphtee@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@1898 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
fbc1f0a604bb31dade946ca37cbec1e2d92fbd8f 09-Jul-2008 jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> Add support to job.repair for host protections, specified through a
--host-protection command line parameter. If no parameter is
specified, default to FILESYSTEM_ONLY.

This replaces the existing repair behaviour with the following:
1. If DO_NOT_REPAIR is specified, do nothing.
2. If FILESYTEM_ONLY is specified, use the existing repair
procedure.
3. If NO_PROTECTION is specified, first try the existing repair
process, and if that fails try host.machine_install.

Note that this doesn't yet make changes to the scheduler to pass in
the host protection values stored in the database, it just adds the
command-line support to autoserv.

Risk: Medium
Visibility: Adds a new option to autoserv for specifying host protection levels
during repair operations.

Signed-off-by: John Admanski <jadmanski@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@1791 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
0afbb6369aa5aa9a75ea67dd9e95ec4b21c0c181 06-Jun-2008 jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> Convert all python code to use four-space indents instead of eight-space tabs.

Signed-off-by: John Admanski <jadmanski@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@1658 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
a6f1308334f91fc5b9463cebe34a08d3be733f7c 06-Jun-2008 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Changed the autoserv_parser to simple re-direct to getopts.

Signed-off-by: Travis Miller <raphtee@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@1624 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
a5b607f69c2308081afffb8c9952f33b1d4dbd9b 02-Jun-2008 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Fixed horrible bug in autoserv when given the argument -h

This is the 2nd of a 2 patch set (please apply together).

Signed-off-by: Travis Miller <raphtee@google.com>




git-svn-id: http://test.kernel.org/svn/autotest/trunk@1581 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py
b7dcc7fff7439a9e5b73be2a4266ab78e4db5a21 02-Jun-2008 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Factored parser out into a singleton that can be accessed anywhere. Also allowed the possiblity to
define additional site specific usage info and still have that info displayed via autoserv -h.
Will define site specific usage in future submissions.

Note: This is patch 1 of 2 patches that should be applied together.

Signed-off-by: Travis Miller <raphtee@google.com>




git-svn-id: http://test.kernel.org/svn/autotest/trunk@1580 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/server/autoserv_parser.py