| d48cbcbccb81962ce39819e372acd4af852fc7da |
|
25-Aug-2017 |
Ben Kwa <kenobi@google.com> |
[autotest] Refactor ContainerBucket.
Remove base container management code from ContainerBucket. This change:
- simplifies the ContainerBucket,
- makes it clearer that the base image setup is something that is
performed separately from normal ContainerBucket operation,
- starts the process of decoupling container creation from the
ContainerBucket, which will be necesary to move to a container pool.
BUG=chromium:720219
TEST=sudo python base_image_unittest.py
TEST=sudo python container_bucket_unittest.py
TEST=sudo python container_unittest.py
TEST=sudo python lxc_config_unittest.py
TEST=sudo python lxc_functional_test.py
TEST=sudo python zygote_unittest.py
Change-Id: I719f7b113729c7387ddacaec77c4c63154d52ea0
Reviewed-on: https://chromium-review.googlesource.com/636033
Commit-Ready: Ben Kwa <kenobi@chromium.org>
Tested-by: Ben Kwa <kenobi@chromium.org>
Reviewed-by: Ben Kwa <kenobi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 966db080d632ab2b7c6534e717762173dad3938b |
|
05-Jun-2017 |
Ben Kwa <kenobi@google.com> |
Create a new module for lxc code.
Create a new python module for lxc code. Move lxc.py and related code into that
module. Script-style functionality in site_utils/lxc.py and lxc_cleanup.py was
left as-is.
BUG=chromium:720219
TEST=sudo python site_utils/lxc_functional_test.py -v
2017-06-15 15:29:52,700 All tests passed.
TEST=Install autotest-server code onto moblab, run a test.
Test runs successfully.
Change-Id: I7b8400db64594f4d1268a62838ac379e713738c1
Reviewed-on: https://chromium-review.googlesource.com/538127
Commit-Ready: Ben Kwa <kenobi@chromium.org>
Tested-by: Ben Kwa <kenobi@chromium.org>
Reviewed-by: Ben Kwa <kenobi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 1eba6d21cac341717b678d8e475402bb2ebfde0a |
|
25-May-2017 |
Ben Kwa <kenobi@google.com> |
Use hyphens in container hostnames.
Change from using underscores ('_') to hyphens ('-') in container
hostnames, as underscore is not actually a valid character in hostnames.
BUG=chromium:726131
TEST=sudo python site_utils/lxc_functional_test.py -v
2017-05-25 07:23:30,661 All tests passed.
Change-Id: I604da466ca9ea4ed9be0147ff11f46ac07f38dec
Reviewed-on: https://chromium-review.googlesource.com/515025
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ben Kwa <kenobi@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Jason Kusuma <jkusuma@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| f2c4e3596a2b59848e86f9e94de2a5129326a8cb |
|
24-May-2017 |
Ben Kwa <kenobi@google.com> |
Factor out container cloning.
BUG=chromium:720219
TEST=sudo python site_utils/lxc_functional_test.py -v
2017-05-24 11:27:03,185 All tests passed.
Change-Id: Iaab416c297e6e092ea03a2d230fe2e62d0f8399c
Reviewed-on: https://chromium-review.googlesource.com/514148
Commit-Ready: Ben Kwa <kenobi@chromium.org>
Tested-by: Ben Kwa <kenobi@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| a5b9cc81f62293082ba8afa4310caaa6402f0d14 |
|
20-May-2017 |
Ilja H. Friedel <ihf@chromium.org> |
lxc: tune download retries.
1) Wait a long time between retries (O(server job runtime)).
2) Try not to retry for timeouts. If there was no other error and we did
not download in an hour just give up.
BUG=chromium:723964
TEST=None.
Change-Id: I78dac74e47ffbd6b190abe46eefe52d096db2abd
Reviewed-on: https://chromium-review.googlesource.com/510022
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| 5734df660a52c6c33a03474b61288cc2d09eb6a9 |
|
18-May-2017 |
Ilja H. Friedel <ihf@chromium.org> |
autotest lxc: increase devserver download timeout.
We monitor the timeout on monarch. Increase this to a very high value.
And for sure do not retry that often.
BUG=chromium:723964
TEST=None.
Change-Id: Iad90e4a11e21bdfafc91365680787936b931e4c7
Reviewed-on: https://chromium-review.googlesource.com/508470
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 4c9f349d67a1fa2eb8e1977b6e67bb7d0307eeee |
|
10-May-2017 |
Ilja H. Friedel <ihf@chromium.org> |
[autotest] Fix concurrent download issue in lxc.
If two threads download a file at the same time, one can end up with the
file of the other (which may or may not end up with the correct download
via retry).
BUG=b:38183322
TEST=None.
Change-Id: Ia524af44415e5042ab5c23e8232eed3cb8fd6844
Reviewed-on: https://chromium-review.googlesource.com/501170
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/site_utils/lxc.py
|
| d364662aa4eb1e82174de580ab49a8793eebe857 |
|
18-Apr-2017 |
xixuan <xixuan@chromium.org> |
autotest: add timeout for downloading files for ssp.
Current the retry timeout limit is equal to the timeout for downloading
ssp files, which make retry not retryable. So increase the timeout for retry
to be 3 times of the timeout of download_file.
BUG=chromium:712283
TEST=None
Change-Id: I639cfd24e400e6a4e5ab046dddb0d808205fb721
Reviewed-on: https://chromium-review.googlesource.com/480541
Reviewed-by: Ningning Xia <nxia@chromium.org>
Commit-Queue: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 5e2efb71ffebead22aa4f0744ad843ee79814b43 |
|
07-Feb-2017 |
Dan Shi <dshi@google.com> |
[autotest] Use the metrics_mock object in case chromite is not set up.
BUG=chromium:688166
TEST=unittest
Change-Id: Ic0077cb2dba75a8d820f229060f3f70f507850a1
Reviewed-on: https://chromium-review.googlesource.com/438754
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| 758ebd0b29876e9d2cf6722d2f362a47a51a3dcd |
|
01-Feb-2017 |
Dan Shi <dshi@google.com> |
Do not force to install latest package in container.
For server side test using SSP, installing packages requires to run
apt-get update and other commands that take long time to finish and introduce
unnecessary load on the drone. All these slow down the server side test by
minutes.
This change allows install_packages call to skip installing a package if it's
already installed and user does not force the latest version of the package.
BUG=chromium:660392
TEST=lxc_functional_test.py
Change-Id: I4a8113a06fcec8062807899f7d25dfa4427746a8
Reviewed-on: https://chromium-review.googlesource.com/435463
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| c3fc59925296d3f923dc6c60997b63b2bd2b01e0 |
|
08-Dec-2016 |
Dan Shi <dshi@google.com> |
[autotest] Replace autotest_stats with metrics
BUG=chromium:667171
TEST=unittest
Change-Id: I60686ea5f5d1a7220d780c7d5c9393568170aee7
Reviewed-on: https://chromium-review.googlesource.com/417887
Commit-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 017ac5279707712475db35657ca1adcdfef6c7cf |
|
18-Nov-2016 |
xixuan <xixuan@chromium.org> |
autotest: Fix bugs in fetching packages for ssp.
This CL fix two things:
1. Only use ssh for devserver-related package download, continue using wget for
other server urls, like 'http://storage.googleapis.com'.
2. When using ssh for downloading, first download it as a temporary file, then
mv it to target file with sudo privilege. This is due to 'sudo' is required in
all operations inside container.
BUG=chromium:666414
TEST=Run 'python ./site_utils/lxc_functional_test.py -s -v' on hot.
Run test dummy_PassServer.ssp on hot.
Change-Id: I98307768923809e02b2e559dec9064d67c686563
Reviewed-on: https://chromium-review.googlesource.com/412427
Commit-Queue: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| 4e116826369f95645151f157e07b8256f7602329 |
|
18-Nov-2016 |
xixuan <xixuan@chromium.org> |
autotest: fix bugs for devserver function: download_file
BUG=chromium:666414
TEST=Call lxc.download_extract() in hot, successfully download files.
Change-Id: Ibbc70333c14cbfca0f7e4db3899eb49979b3a726
Reviewed-on: https://chromium-review.googlesource.com/412331
Reviewed-by: Dan Shi <dshi@google.com>
Commit-Queue: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
/external/autotest/site_utils/lxc.py
|
| e97ebd5956237d1a20faa9c0b1ee728d7158a800 |
|
17-Nov-2016 |
Prathmesh Prabhu <pprabhu@chromium.org> |
[Workaround] SSH into devserver when obtaining ssp package.
This is a quick workaround to allow ssp staging to work even when a
devserver is chosen in an incorrect subnet.
BUG=chromium:666414
TEST=test_push passes.
Change-Id: I4db067a203b3fd6745999d744cbc66c72cd7ffdd
Reviewed-on: https://chromium-review.googlesource.com/412384
Reviewed-by: Xixuan Wu <xixuan@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 2f55dc28c5f6ca3f232451ab37f091c872cc50fe |
|
01-Sep-2016 |
Dan Shi <dshi@google.com> |
[autotest] Bug fix in container hostname
CL 376221 introduced a bug in server-side packaging running in Moblab.
In Moblab, the hostname of duts are IP address, and the Moblab functions as a
dhcp server for the duts. For the container to have the same name as the dut
it's testing will confuse the dhcp server.
Also, the container's hostname must start with `test_` to allow DHCP server in
Moblab to set a correct lease time (20min)
This change modify the container name to be a string different from the dut's
hostname. Also, replace dot in the hostname with understore to avoid dns
problem.
BUG=chromium:642783
TEST=in a moblab, sudo python site_utils/lxc_functional_test.py
Change-Id: I9378b4a23f079f54f130e779a723f33c79635dab
Reviewed-on: https://chromium-review.googlesource.com/379613
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| 3be35af753ba283616ff7dd2fef29415aa8f34bf |
|
26-Aug-2016 |
Dan Shi <dshi@google.com> |
[autotest] Set container's hostname to the dut hostname
Update the hostname of the test container to be `dut_name`. Some TradeFed tests
use hostname in test results, which is used to group test results in dashboard.
The default container name is set to be the name of the folder, which is unique
(as it is composed of job id and timestamp. For better result view, the
container's hostname is set to be the dut hostname.
BUG=chromium:637467
TEST=local verify
Change-Id: I1481d88a890020af634a499fb6d469574f9e4504
Reviewed-on: https://chromium-review.googlesource.com/376221
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 9885a1f1b4a386bb381cbf0696162c2ece622aa7 |
|
23-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Add a symlink of gsutil in /usr/bin if it's installed in container
lxc-attach and run command does not run in shell, thus .bashrc is not loaded.
This change creates a symlink in /usr/bin/ for gsutil if it's installed.
BUG=chromium:471312
TEST=local run ssp test, unittest, ssp functional test:
sudo python site_utils/lxc_functional_test.py
Change-Id: I37c4fb1b56fbde4eca8583d33daaa5fefaf5a5db
Reviewed-on: https://chromium-review.googlesource.com/355590
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| db9d053df1c445c91ec6b1c4682145074a3d9532 |
|
22-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Add a new config for server-side packaging to mount a directory
With this change, user can add a config in ssp_deploy_config.json (or
ssp_deploy_shadow_config.json for local overide) to mount a directory in the
host onto a directory inside container. For example:
{
"source": "/usr/local/autotest/results/shared",
"target": "/usr/local/autotest/results/shared",
"mount": true,
"readonly": false,
"force_create": true
}
BUG=chromium:621676
TEST=local run dummy_PassServer with the new config
unittest
Change-Id: I415c22be70d39d29a8a70aabd3d9f1e64a12f2a5
Reviewed-on: https://chromium-review.googlesource.com/355181
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| 2a0f61ef9b1e42552638624f11de240aeaf2c618 |
|
21-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Add an option --name for lxc to set up a new base container
Option --name is added to lxc.py so the script can set up a new base container
before the default base container is updated in global config. This allows lab
to do following in sequence:
1. Use puppet to deploy new base container to all lab servers.
2. After all lab servers are updated, modify the `container_base_name` in shadow
config. With that, new tests will be able to use the new base container.
BUG=chromium:471312
TEST=local run ssp test, unittest, ssp functional test:
sudo python site_utils/lxc_functional_test.py
Change-Id: Ic3b042449789f78cf7ffc37dc59b19ac64a1456c
Reviewed-on: https://chromium-review.googlesource.com/354620
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 0a7e4d61ea0eff2f7786a7bf8819d443073991db |
|
21-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Allow base container to be set in global config
This allows the base container to be updated using shadow config.
Also move the base container folder to gs://abci-ssp bucket, as
chromeos-image-archive bucket deletes files older than 6 months.
BUG=chromium:471312
TEST=local run ssp test, unittest, ssp functional test:
sudo python site_utils/lxc_functional_test.py
Change-Id: Ibc73e439af92f12d774a0b87bdb28d5c3a8f493c
Reviewed-on: https://chromium-review.googlesource.com/354550
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| 1d3ea49f9aa25e778849faba96d423059d142a8b |
|
19-Apr-2016 |
Victor Hsieh <victorhsieh@google.com> |
Tee wget output in download_extract of lxc.py
In crbug.com/604473, we've seen a wget timeout. No null devserver log
is found for that specific request and the cause remains unknown.
Adding the client side log could at least leave some clue for future
error.
TEST=lint check during repo upload
BUG=chromium:604473
Change-Id: Ie0e422e4e574c273b4691019617c69abc1be86eb
Reviewed-on: https://chromium-review.googlesource.com/339491
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Victor Hsieh <victorhsieh@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
| 38fff70fc5257cd1497455e7fbc2742a2bd47ba9 |
|
28-Mar-2016 |
Dan Shi <dshi@google.com> |
[autotest] Save control file before setting up container
This is to prevent test failure caused by missing control file.
Scheduler restart leads to all control files in drone_tmp being deleted.
Due to the long delay of setting up container when running tests with SSP,
this becomes more of an issue.
This change saves the control file to result folder before setting up
container.
BUG=chromium:598379
TEST=local run test, unittest
Change-Id: I10a7715477ce14d3cbba5176292fc9aca19f9f2d
Reviewed-on: https://chromium-review.googlesource.com/335269
Commit-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
/external/autotest/site_utils/lxc.py
|
| afa63872d1b63d8e6cad7de617569c215b03d26c |
|
24-Feb-2016 |
Dan Shi <dshi@google.com> |
[autotest] Change resultdir name inside container to be the same as outside container.
The job folder (resultdir) in drone has the format of job_id-user_name.
However, if a test runs inside the container (using server-side
packaging), its folder name is job_id only. The reason was that we
don't want to pass another piece of information to container.
This causes a problem in crash collection. During log collection, test
job creates a file uncollected_log_file to store the list of files to
collect, in the format of source and target pair, where source is the
origin path of the file, e.g., path to a log file in DUT, and target is
a path to the destination to copy to, e.g.,
/usr/local/autotest/results/226/hardware_StorageQualBase.before
If the log collection was interrupted or autoserv process died before
logs can be collected, another autoserv process will be kicked off by
scheduler to run with --collect-crashinfo. The new autoserv process will
read the content of uncollected_log_file and tries to copy file based on
the source/target pair.
Due to the difference of job folder used in container, crash collection
will fail as the target directory does not exist on the drone (as the
job folder only has job_id, but not in the format of job_id-user_name.
This change passes the full job folder name to setup_test call when
setting up the container to run the test. That way, the result path will
be exactly the same inside and outside of the container.
BUG=chromium:589173
TEST=local run ssp test, unittest
manual run crash collection:
/usr/local/autotest/server/autoserv -p --use-existing-results
--collect-crashinfo 100.96.51.226 -r
'/usr/local/autotest/results/372-debug_user/100.96.51.226'
Change-Id: I4e1abb8c566ca65e8e245e68f29a9b996fe6be32
Reviewed-on: https://chromium-review.googlesource.com/329055
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 3d02fa31fd707a0eb7ec3865cc0f70beaeae166e |
|
05-Nov-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Deploy moblab shadow config to container.
Package install function does not work properly in container running in moblab.
This CL fixed the bug by:
1. copy moblab_config.ini to container if it exists.
2. In install_package_precheck, replace
if not SSP_ENABLED:
with
if not SSP_ENABLED and not utils.is_in_container():
BUG=chromium:551777
TEST=lxc_functional_test.py
Change-Id: I98a0c600a400c6156c9f963b6f62fa89a55d821c
Reviewed-on: https://chromium-review.googlesource.com/310976
Commit-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| a7e0117cdfc8a3efe93c94a21d1aea1fedf0e8df |
|
14-Jul-2015 |
Laurence Goodby <lgoodby@google.com> |
[autotest] increases container networking startup timeout
Server-based tests were frequently failing to run on our moblabs due
to "Could not resolve host: storage.googleapis.com" errors during
container initialization. It seems that it sometimes takes more than 2
minutes for the container's networking to initialize. This CL
increases the timeout.
BUG=chrome-os-partner:42351
TEST=Ran on moblabs.
Change-Id: I02c1f9b55ce8b2678c9abdda59363eee536cd2dd
Reviewed-on: https://chromium-review.googlesource.com/285518
Trybot-Ready: Laurence Goodby <lgoodby@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Laurence Goodby <lgoodby@chromium.org>
Tested-by: Laurence Goodby <lgoodby@chromium.org>
/external/autotest/site_utils/lxc.py
|
| d9094d46b18e76d0951c8f1883b3553525c4ac9e |
|
10-Jul-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a function to install multiple packages
Each package install needs to run apt-get update, which takes extra time to run.
The new function takes in a list of packages and python packages, so `apt-get
update` only needs to run once.
BUG=chromium:508314
TEST=sudo python site_utils/lxc_functional_test.py -v
Change-Id: Ibfcfa24696f6e217820679f5405da67529ef2c45
Reviewed-on: https://chromium-review.googlesource.com/284597
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| af1cc0eac9c060c8a7001464685d9771387507da |
|
08-Jul-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Do not include hostname in timer stats used inside lxc container.
The new timer is used by functions only runs inside container. This is to avoid
creating individual timer stats for each container, which leads to extra disk
space usage.
BUG=None
TEST=unittest
Change-Id: Ic2321b61a3c8ffef3fa5aa98bda101e127a49cd8
Reviewed-on: https://chromium-review.googlesource.com/284262
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 23d3a22552d947e08c348e59f4639f7b6ae7a873 |
|
26-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add retry in install package inside container.
This is to alieviate the flake when running apt-get command.
BUG=chromium:499631
TEST=local lxc_functional_test.py
Change-Id: I0be1d9e305465bb73a5e8e1e5b2f4b57bd234f7d
Reviewed-on: https://chromium-review.googlesource.com/282331
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: mukesh agrawal <quiche@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| ff78f11ff85372a1e2f8ddf8eefbc8b772686148 |
|
12-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Pass current user into container for RPC usage
RPC is called using current OS user if not specified (frontend.py). This causes
an issue for autoserv running inside container, as the OS user is always root.
The root user may fail check_for_acl_violation_hosts if a host has special ACL
that does not allow `everyone`.
This CL add an SSP/user config in global config, lxc related code is updated so
that the OS user of autoserv running in host will be passed into the container
through shadow_config.ini. And frontend will use such user to make RPC.
BUG=chromium:480525,chromium:502897
TEST=local
sudo python site_utils/lxc_functional_test.py -v -s
attach to the container
update SSP/user with a user name (existing one should be root as the process
site_utils/lxc_functional_test.py is executed with sudo)
update frontend.py with code in this CL (this means only newer builds won't
be hit by this bug.)
run some RPC:
import common
from autotest_lib.server.cros.dynamic_suite import frontend_wrappers
_AFE = frontend_wrappers.RetryingAFE()
print _AFE.run('get_stable_version')
Also make sure afe in the host still works.
Change-Id: I9ba924e2652fadffa71d6298f74c5b82e8e26b4e
Reviewed-on: https://chromium-review.googlesource.com/281062
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 1a277ef7800425aad62b284ef0240eee3f54d0a3 |
|
05-Jun-2015 |
Cheng-Yi Chiang <cychiang@chromium.org> |
[autotest] Fix install_package_precheck in lxc
install_package_precheck should return True/False.
BUG=chromium:483063
TEST=run ssp_PackageInstall test on drone.
Change-Id: Ib5834fbf306290efe186903918049461b110e5a2
Reviewed-on: https://chromium-review.googlesource.com/275415
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 3ea0204531322203ab55616b39eca9092eaad23d |
|
06-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add some logging in lxc.py to help troubleshooting package install
BUG=None
TEST=local test
Change-Id: Ic2b1b08f669819134e0402e013cc17400b76ebdc
Reviewed-on: https://chromium-review.googlesource.com/275710
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Roshan Pius <rpius@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 585b7b68efb37226bc2b6cf25e90ea88c208e3c1 |
|
01-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] swap python import order to support pip in moblab
In Moblab, the host's python modules located in
/usr/lib64/python2.7/site-packages is mounted to following folder inside
container: /usr/local/lib/python2.7/dist-packages/. The modules include
an old version of requests module, which is used in autotest
site-packages. For test, the module is only used in
dev_server/symbolicate_dump for requests.call and requests.codes.OK.
When pip is installed inside the container, it installs requests module
with version of 2.2.1 in /usr/lib/python2.7/dist-packages/. The version
is newer than the one used in autotest site-packages, but not the latest
either.
According to /usr/lib/python2.7/site.py, modules in /usr/local/lib are
imported before the ones in /usr/lib. That leads to pip to use the older
version of requests (0.11.2), and it will fail. On the other hand,
requests module 2.2.1 can't be installed in CrOS (refer to CL:265759),
and higher version of requests module can't work with pip.
The only fix to resolve this is to switch the import order, so modules
in /usr/lib can be imported before /usr/local/lib.
BUG=chromium:483371
TEST=local moblab run
sudo python site_utils/lxc_functional_test.py -v
Change-Id: I59ab724ac6bd974da648209248b9f6df3380ae0f
Reviewed-on: https://chromium-review.googlesource.com/274411
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| ca3be48da9dd7882ba282d9e82413b4aad2a8364 |
|
06-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add sudo -n option to avoid password prompt.
Also add a utility function to check if password is needed to run sudo command.
We can add -n option to all sudo command that SSP used. However, once the
property drone.support_ssp is set to False, no SSP code will be executed.
Therefore, there is no need to add -n everywhere.
BUG=chromium:484396
TEST=local run
Start scheduler, confirm no password promt, and drone.support_ssp is False.
Run sudo true, enter password.
Run scheduler again, confirm no password promt, and drone.support_ssp is True.
Change-Id: Ifbd302973db5d443499311e435ec99e32d24cb5b
Reviewed-on: https://chromium-review.googlesource.com/269740
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 0a81428995db06c51af85af271a844e5f62b1576 |
|
20-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Record all args to metadb if job_id is not available.
We are seeing some container creation failed with not job_id recorded.
This change is to record all arguments to help troubleshooting.
BUG=None
TEST=local test
Change-Id: I03f4c31aea2ab01e436fe491d29c767ff272482c
Reviewed-on: https://chromium-review.googlesource.com/272389
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| d3662afdf0d6a45da75d380ce41a8c27c42be82b |
|
07-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a server-side test to validate package install in container.
This will serve as an example about how package installation in container can
be done.
TODO: add this test to autotest ebuild
CQ-DEPEND=CL:269784
BUG=None
TEST=local autotest run
also confirm that test_that run of this test will fail with error:
Install OS package is only supported in server-side packaging.
Change-Id: I439eb528ac5794eb1e3fa2cb25de97dec552f1a4
Reviewed-on: https://chromium-review.googlesource.com/269998
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 5180d55144483c168b166d842be69a56dab353d4 |
|
04-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Retry container clone without using snapshot.
This CL attempts to alieviate the flakeness of lxc-clone in Ganeti instance.
We will evaluate the metadata to confirm the effectiveness, while looking for
the root cause of this issue.
Also add a property rootfs to container. This property will always point to the
folder of the container's rootfs. So the caller doesn't need to check if the
container is created from snapshot clone to determine if or not to use `delta0`
or `rootfs`.
BUG=chromium:484019
TEST=local test
sudo python site_utils/lxc_functional_test.py -v
Change-Id: Ibc04371bfb63e811a5bc14693d678adff6941323
Reviewed-on: https://chromium-review.googlesource.com/269009
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 7836d25b375a791ebc7b3d0d255c8732c4f35be2 |
|
28-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a generic config deployment module to handle configs for container.
For container to run a test, it needs some config files to be able to do
something like:
resolve hostname with dns configured in host.
ssh to dut.
ssh to devserver.
Currently we hard code to copy each file to container, and modify each file for
it to work within container.
This CL provides a generic way to manage a list of deploy config, each config
specifies what file needs to be copied to where and if the existing file should
be overwritten. For container to work in developer's workstation, the special
handling of some configure files is still available. For lab servers, puppet
should be used to push out the config files and shadow deploy config (
ssp_deploy_shadow_config).
BUG=chromium:481706
TEST=local test with and without shadow ssp deploy config
sudo python site_utils/lxc_functional_test.py -v -d chromeos1-dshi1.cros -r 172.17.40.27
also verify in moblab (python package install does not work in moblab yet due
to bug 483371)
Change-Id: I8bbe33a83262b7c746fbe0112cf61cea046931ea
Reviewed-on: https://chromium-review.googlesource.com/267569
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 507fdc40ef300535886bc1ef71d18c1fa697b895 |
|
30-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add two helper functions to install packages.
Server-side test should call these two functions to install required packages.
Ideally, tests will not need to deploy packages to site-packages for test to
run.
BUG=chromium:453614
TEST=site_utils/lxc_functional_test.py
Change-Id: I9ade9e1a804897a32cebac36836367feda13a24b
Reviewed-on: https://chromium-review.googlesource.com/268437
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| e019bd9f7be4abdaeb1aef47998545a1529d4541 |
|
24-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Enable snapshot clone for Ganeti instance using aufs.
Ganeti instances does not support overlayfs due to older kernel. However, it
does support snapshot clone with aufs. Enable this will greatly enhance SSP
performance in Ganeti instances.
BUG=chromium:464834
TEST=verified in server7 with cbf run
http://chromeos-autotest.cbf.corp.google.com/afe/#tab_id=view_job&object_id=10335
Change-Id: I76a7691033214347d5a4e892cd0776bde8c53433
Reviewed-on: https://chromium-review.googlesource.com/267106
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| d68d51c731ef83a45c173497b59a26b2e6870c1b |
|
22-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add autoserv's pid to the container's name
The PID info will be used to determine if a container is orphaned.
BUG=chromium:479383
TEST=sudo python site_utils/lxc_functional_test.py
local run a server side test
Change-Id: I4de2f7f266bcb793cfe1a97e412868aa49d53991
Reviewed-on: https://chromium-review.googlesource.com/266754
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| c4f79fcd5c21dd1d5d917c73582aa08399166d38 |
|
18-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Set ssh loglevel to ERROR for tests running in container.
This change is to avoid logs being flooded with warning `Permanently added
'[hostname]' (RSA) to the list of known hosts.`
BUG=chromium:478364
TEST=local run to confirm log has no such warning message:
http://dshi.mtv/results/589-dshi/172.27.215.232/debug/autoserv.DEBUG
Change-Id: I63c124f96eb7f36a7dca5b9673716beb0ac24fe3
Reviewed-on: https://chromium-review.googlesource.com/266401
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 37bee2292a1c385839e14adf3b2799447d8a98ac |
|
14-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Reduce the container info to collect for better performance
Also report the container usage data to metadb.
BUG=None
TEST=local lxc_functional_test.py, local dummy_server suite, search metadb
http://172.25.61.45:9200/_plugin/elastic-hammer/
query string:
{ "query": {"bool": {"minimum_should_match": 2,
"should": [
{"term": {"_type": "container_run_test"}},
{"range": {"time_recorded":
{"gte": 1419384005.705098,
"lte": 1507282542.893756}}}
]}},
"size": 100}
Change-Id: I739187499a2ec82ffe2a08d24e7d326743380653
Reviewed-on: https://chromium-review.googlesource.com/265532
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| f2a2dd64129cb12158bf065e61d9328ac45b1e2f |
|
10-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update lxc wrapper to work with Ganeti instance
trusty/server with updated kernel does not support lxc-clone with snapshot,
the wrapper (lxc.py) is updated so SSP can work before ticket (t/16003207)
is resolved.
BUG=chromium:453614
TEST=verified in cros-database-bak-test-1
Change-Id: I45208d745dea5614930481b9f159293b1f933cc7
Reviewed-on: https://chromium-review.googlesource.com/265274
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| a958bd386256b2e287f7f023a680f83c4983c1cf |
|
02-Apr-2015 |
Simran Basi <sbasi@google.com> |
lxc: Don't delete the base container if it already exists.
Currently if the base container exists and we don't pass in
force-delete, it will automatically delete the base container.
BUG=None
TEST=panther_moblab build. Reboot's no longer delete the base
container.
Change-Id: I764ccd51df826b5e66fa79cfa06bccbccca8815d
Reviewed-on: https://chromium-review.googlesource.com/263650
Trybot-Ready: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| f35517a77f9a871276396ea57a553f5e6e72cd88 |
|
02-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update ssh config and resolv.conf in container
For container to be able to ssh to duts in lab, it needs the .ssh/conf and
/etc/resolv.conf from the drone. It's not reliable to build the content inside
the base container, as such config may change over the time.
For moblab, it doesn't need ssh/config (it doesn't exist). /etc/resolv.conf
is not needed as well, however, it has no side effect to add such info inside
container.
BUG=chromium:453614
TEST=verify in a test drone to ssh to lab dut using hostname
http://chromeos-autotest.cbf.corp.google.com/afe/#tab_id=view_job&object_id=9685
Change-Id: I4f09a7bdd4e9c8c57dee0e67922c22779aa83797
Reviewed-on: https://chromium-review.googlesource.com/263570
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 387667a8caafacd310b54b5e32e11f9649146126 |
|
24-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update shadow config in container with FQDN of db/afe server name.
container may not be able to resolve hostname like cautotest. The FQDN should be
used in db and afe server name in container's shadow config.
BUG=chromium:453624
TEST=lxc_functional_test. verify with a test drone
Change-Id: I6cf337edc8d36e8fbb9e58cbe5e43e3227b65f10
Reviewed-on: https://chromium-review.googlesource.com/262214
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 5b5f88e9f4a9237221ba17ba10076add151d111f |
|
18-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update shadow config in container with correct db/afe server name.
The original CL 260980 was reverted due to a missing package (netifaces).
This CL is almost the same as the original one except moving import of netifaces
after import common.
Container uses a copy of shadow config from the host. The config file needs to
be updated for autoserv running in container to use:
1. Disable master ssh connection in shadow config, as it is not working
properly in container yet, and produces noise in the log.
2. Update AUTOTEST_WEB/host and SERVER/hostname to be IP of the host
if any is set to localhost or 127.0.0.1.
BUG=chromium:453624
TEST=lxc_functional_test, verify in moblab
http://chromeos1-dshi-moblab1.cros/afe/#tab_id=view_job&object_id=25
Change-Id: Ic836d3f46901117b8f21d655c32b22d9815c328c
Reviewed-on: https://chromium-review.googlesource.com/261575
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| bbe7591943c5d35ff6475c14b0a9c24f5366a418 |
|
19-Mar-2015 |
Simran Basi <sbasi@chromium.org> |
Revert "[autotest] Update shadow config in container with correct db/afe server name."
This reverts commit 309c48f160be2a26429be1456634a73fd89a1a9c.
Change-Id: I7456dc5c920e31f9cc6e019943ea879c1c9e7c28
Reviewed-on: https://chromium-review.googlesource.com/261240
Reviewed-by: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 309c48f160be2a26429be1456634a73fd89a1a9c |
|
18-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update shadow config in container with correct db/afe server name.
Container uses a copy of shadow config from the host. The config file needs to
be updated for autoserv running in container to use:
1. Disable master ssh connection in shadow config, as it is not working
properly in container yet, and produces noise in the log.
2. Update AUTOTEST_WEB/host and SERVER/hostname to be IP of the host
if any is set to localhost or 127.0.0.1.
BUG=chromium:453624
TEST=lxc_functional_test, verify in moblab
http://chromeos1-dshi-moblab1.cros/afe/#tab_id=view_job&object_id=25
Change-Id: I4a487f7b95fc851074680e585553fa739869d900
Reviewed-on: https://chromium-review.googlesource.com/260980
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| cf4d2032ea4bf5af680383f36308d581876bbbb0 |
|
12-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update autoserv to support server-side packaging.
Code change is made so autoserv will honer the --require-ssp arg. When the
argument is specified, autoserv will:
1. Try to locate server-side package based on --image, host attribute
job_repo_url and host's cros-version label.
2. If no server-side package can be staged, the job will be run without ssp.
3. If package can be staged, autoserv will start a container to run the test.
a) parent autoserv process's log will be stored in [results]/wrapper folder.
b) A container will be created from base container, with proper setup.
c) The same autoserv command line (with update on result path and control
etc.), will be executed inside the container.
d) Parent autoserv process will wait for above command to finish and destroy
the container afterwards. If the test job was aborted, container should
also be destroyed.
BUG=chromium:453624
TEST=local setup and run test command:
/usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/16-debug_\
user/$result_dir -m 172.27.215.232 -u udebug_user -l dummy_PassServer -s -P \
16-debug_user/$result_dir -n /usr/local/autotest/results/drone_tmp/attach.2 \
--verify_job_repo_url --require-ssp
create job from afe:
http://dshi.mtv.corp.google.com/afe/#tab_id=view_job&object_id=19
Change-Id: Ida36374dd500cd1fd0b67a86ab8c0198ade4cc36
Reviewed-on: https://chromium-review.googlesource.com/259834
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
| 767dced065c8918b5ff74c1347371a0f0f8d9c69 |
|
01-Feb-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a utility module to interact with LXC binaries.
Add a module (lxc.py) to interact with LXC binaries.
lxc_functional_test.py is a test script to test the module. It will
1. Set up base container in a tmp directory.
2. Create a test container from the base container.
3. Test site-packages and results directory can be mounted and shared between
host and test container.
4. Run autoserv in test mode.
Also add a utility function in client/common_lib/site_utils.py to get the value
of the given argument for the function. It's used in decorator to check function
parameters.
BUG=chromium:453621
CQ-DEPEND=CL:251140
TEST=unittest,
sudo python site_utils/lxc_functional_test.py
sudo python site_utils/lxc.py -s -p /tmp/container_123
sudo python site_utils/lxc.py -s -p /tmp/container_123 -f
sudo python site_utils/lxc.py -p /tmp/container_123 -f
Change-Id: Id7b259c4bc03977974af44d6d88420b220706589
Reviewed-on: https://chromium-review.googlesource.com/247271
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|