966db080d632ab2b7c6534e717762173dad3938b |
|
05-Jun-2017 |
Ben Kwa <kenobi@google.com> |
Create a new module for lxc code. Create a new python module for lxc code. Move lxc.py and related code into that module. Script-style functionality in site_utils/lxc.py and lxc_cleanup.py was left as-is. BUG=chromium:720219 TEST=sudo python site_utils/lxc_functional_test.py -v 2017-06-15 15:29:52,700 All tests passed. TEST=Install autotest-server code onto moblab, run a test. Test runs successfully. Change-Id: I7b8400db64594f4d1268a62838ac379e713738c1 Reviewed-on: https://chromium-review.googlesource.com/538127 Commit-Ready: Ben Kwa <kenobi@chromium.org> Tested-by: Ben Kwa <kenobi@chromium.org> Reviewed-by: Ben Kwa <kenobi@chromium.org>
/external/autotest/site_utils/lxc.py
|
1eba6d21cac341717b678d8e475402bb2ebfde0a |
|
25-May-2017 |
Ben Kwa <kenobi@google.com> |
Use hyphens in container hostnames. Change from using underscores ('_') to hyphens ('-') in container hostnames, as underscore is not actually a valid character in hostnames. BUG=chromium:726131 TEST=sudo python site_utils/lxc_functional_test.py -v 2017-05-25 07:23:30,661 All tests passed. Change-Id: I604da466ca9ea4ed9be0147ff11f46ac07f38dec Reviewed-on: https://chromium-review.googlesource.com/515025 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ben Kwa <kenobi@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Jason Kusuma <jkusuma@chromium.org> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
f2c4e3596a2b59848e86f9e94de2a5129326a8cb |
|
24-May-2017 |
Ben Kwa <kenobi@google.com> |
Factor out container cloning. BUG=chromium:720219 TEST=sudo python site_utils/lxc_functional_test.py -v 2017-05-24 11:27:03,185 All tests passed. Change-Id: Iaab416c297e6e092ea03a2d230fe2e62d0f8399c Reviewed-on: https://chromium-review.googlesource.com/514148 Commit-Ready: Ben Kwa <kenobi@chromium.org> Tested-by: Ben Kwa <kenobi@chromium.org> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
a5b9cc81f62293082ba8afa4310caaa6402f0d14 |
|
20-May-2017 |
Ilja H. Friedel <ihf@chromium.org> |
lxc: tune download retries. 1) Wait a long time between retries (O(server job runtime)). 2) Try not to retry for timeouts. If there was no other error and we did not download in an hour just give up. BUG=chromium:723964 TEST=None. Change-Id: I78dac74e47ffbd6b190abe46eefe52d096db2abd Reviewed-on: https://chromium-review.googlesource.com/510022 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
5734df660a52c6c33a03474b61288cc2d09eb6a9 |
|
18-May-2017 |
Ilja H. Friedel <ihf@chromium.org> |
autotest lxc: increase devserver download timeout. We monitor the timeout on monarch. Increase this to a very high value. And for sure do not retry that often. BUG=chromium:723964 TEST=None. Change-Id: Iad90e4a11e21bdfafc91365680787936b931e4c7 Reviewed-on: https://chromium-review.googlesource.com/508470 Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> Commit-Queue: Aviv Keshet <akeshet@chromium.org>
/external/autotest/site_utils/lxc.py
|
4c9f349d67a1fa2eb8e1977b6e67bb7d0307eeee |
|
10-May-2017 |
Ilja H. Friedel <ihf@chromium.org> |
[autotest] Fix concurrent download issue in lxc. If two threads download a file at the same time, one can end up with the file of the other (which may or may not end up with the correct download via retry). BUG=b:38183322 TEST=None. Change-Id: Ia524af44415e5042ab5c23e8232eed3cb8fd6844 Reviewed-on: https://chromium-review.googlesource.com/501170 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/site_utils/lxc.py
|
d364662aa4eb1e82174de580ab49a8793eebe857 |
|
18-Apr-2017 |
xixuan <xixuan@chromium.org> |
autotest: add timeout for downloading files for ssp. Current the retry timeout limit is equal to the timeout for downloading ssp files, which make retry not retryable. So increase the timeout for retry to be 3 times of the timeout of download_file. BUG=chromium:712283 TEST=None Change-Id: I639cfd24e400e6a4e5ab046dddb0d808205fb721 Reviewed-on: https://chromium-review.googlesource.com/480541 Reviewed-by: Ningning Xia <nxia@chromium.org> Commit-Queue: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org>
/external/autotest/site_utils/lxc.py
|
5e2efb71ffebead22aa4f0744ad843ee79814b43 |
|
07-Feb-2017 |
Dan Shi <dshi@google.com> |
[autotest] Use the metrics_mock object in case chromite is not set up. BUG=chromium:688166 TEST=unittest Change-Id: Ic0077cb2dba75a8d820f229060f3f70f507850a1 Reviewed-on: https://chromium-review.googlesource.com/438754 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
758ebd0b29876e9d2cf6722d2f362a47a51a3dcd |
|
01-Feb-2017 |
Dan Shi <dshi@google.com> |
Do not force to install latest package in container. For server side test using SSP, installing packages requires to run apt-get update and other commands that take long time to finish and introduce unnecessary load on the drone. All these slow down the server side test by minutes. This change allows install_packages call to skip installing a package if it's already installed and user does not force the latest version of the package. BUG=chromium:660392 TEST=lxc_functional_test.py Change-Id: I4a8113a06fcec8062807899f7d25dfa4427746a8 Reviewed-on: https://chromium-review.googlesource.com/435463 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
c3fc59925296d3f923dc6c60997b63b2bd2b01e0 |
|
08-Dec-2016 |
Dan Shi <dshi@google.com> |
[autotest] Replace autotest_stats with metrics BUG=chromium:667171 TEST=unittest Change-Id: I60686ea5f5d1a7220d780c7d5c9393568170aee7 Reviewed-on: https://chromium-review.googlesource.com/417887 Commit-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/site_utils/lxc.py
|
017ac5279707712475db35657ca1adcdfef6c7cf |
|
18-Nov-2016 |
xixuan <xixuan@chromium.org> |
autotest: Fix bugs in fetching packages for ssp. This CL fix two things: 1. Only use ssh for devserver-related package download, continue using wget for other server urls, like 'http://storage.googleapis.com'. 2. When using ssh for downloading, first download it as a temporary file, then mv it to target file with sudo privilege. This is due to 'sudo' is required in all operations inside container. BUG=chromium:666414 TEST=Run 'python ./site_utils/lxc_functional_test.py -s -v' on hot. Run test dummy_PassServer.ssp on hot. Change-Id: I98307768923809e02b2e559dec9064d67c686563 Reviewed-on: https://chromium-review.googlesource.com/412427 Commit-Queue: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
4e116826369f95645151f157e07b8256f7602329 |
|
18-Nov-2016 |
xixuan <xixuan@chromium.org> |
autotest: fix bugs for devserver function: download_file BUG=chromium:666414 TEST=Call lxc.download_extract() in hot, successfully download files. Change-Id: Ibbc70333c14cbfca0f7e4db3899eb49979b3a726 Reviewed-on: https://chromium-review.googlesource.com/412331 Reviewed-by: Dan Shi <dshi@google.com> Commit-Queue: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org>
/external/autotest/site_utils/lxc.py
|
e97ebd5956237d1a20faa9c0b1ee728d7158a800 |
|
17-Nov-2016 |
Prathmesh Prabhu <pprabhu@chromium.org> |
[Workaround] SSH into devserver when obtaining ssp package. This is a quick workaround to allow ssp staging to work even when a devserver is chosen in an incorrect subnet. BUG=chromium:666414 TEST=test_push passes. Change-Id: I4db067a203b3fd6745999d744cbc66c72cd7ffdd Reviewed-on: https://chromium-review.googlesource.com/412384 Reviewed-by: Xixuan Wu <xixuan@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
/external/autotest/site_utils/lxc.py
|
2f55dc28c5f6ca3f232451ab37f091c872cc50fe |
|
01-Sep-2016 |
Dan Shi <dshi@google.com> |
[autotest] Bug fix in container hostname CL 376221 introduced a bug in server-side packaging running in Moblab. In Moblab, the hostname of duts are IP address, and the Moblab functions as a dhcp server for the duts. For the container to have the same name as the dut it's testing will confuse the dhcp server. Also, the container's hostname must start with `test_` to allow DHCP server in Moblab to set a correct lease time (20min) This change modify the container name to be a string different from the dut's hostname. Also, replace dot in the hostname with understore to avoid dns problem. BUG=chromium:642783 TEST=in a moblab, sudo python site_utils/lxc_functional_test.py Change-Id: I9378b4a23f079f54f130e779a723f33c79635dab Reviewed-on: https://chromium-review.googlesource.com/379613 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
3be35af753ba283616ff7dd2fef29415aa8f34bf |
|
26-Aug-2016 |
Dan Shi <dshi@google.com> |
[autotest] Set container's hostname to the dut hostname Update the hostname of the test container to be `dut_name`. Some TradeFed tests use hostname in test results, which is used to group test results in dashboard. The default container name is set to be the name of the folder, which is unique (as it is composed of job id and timestamp. For better result view, the container's hostname is set to be the dut hostname. BUG=chromium:637467 TEST=local verify Change-Id: I1481d88a890020af634a499fb6d469574f9e4504 Reviewed-on: https://chromium-review.googlesource.com/376221 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
9885a1f1b4a386bb381cbf0696162c2ece622aa7 |
|
23-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Add a symlink of gsutil in /usr/bin if it's installed in container lxc-attach and run command does not run in shell, thus .bashrc is not loaded. This change creates a symlink in /usr/bin/ for gsutil if it's installed. BUG=chromium:471312 TEST=local run ssp test, unittest, ssp functional test: sudo python site_utils/lxc_functional_test.py Change-Id: I37c4fb1b56fbde4eca8583d33daaa5fefaf5a5db Reviewed-on: https://chromium-review.googlesource.com/355590 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
db9d053df1c445c91ec6b1c4682145074a3d9532 |
|
22-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Add a new config for server-side packaging to mount a directory With this change, user can add a config in ssp_deploy_config.json (or ssp_deploy_shadow_config.json for local overide) to mount a directory in the host onto a directory inside container. For example: { "source": "/usr/local/autotest/results/shared", "target": "/usr/local/autotest/results/shared", "mount": true, "readonly": false, "force_create": true } BUG=chromium:621676 TEST=local run dummy_PassServer with the new config unittest Change-Id: I415c22be70d39d29a8a70aabd3d9f1e64a12f2a5 Reviewed-on: https://chromium-review.googlesource.com/355181 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
2a0f61ef9b1e42552638624f11de240aeaf2c618 |
|
21-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Add an option --name for lxc to set up a new base container Option --name is added to lxc.py so the script can set up a new base container before the default base container is updated in global config. This allows lab to do following in sequence: 1. Use puppet to deploy new base container to all lab servers. 2. After all lab servers are updated, modify the `container_base_name` in shadow config. With that, new tests will be able to use the new base container. BUG=chromium:471312 TEST=local run ssp test, unittest, ssp functional test: sudo python site_utils/lxc_functional_test.py Change-Id: Ic3b042449789f78cf7ffc37dc59b19ac64a1456c Reviewed-on: https://chromium-review.googlesource.com/354620 Reviewed-by: Aviv Keshet <akeshet@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
0a7e4d61ea0eff2f7786a7bf8819d443073991db |
|
21-Jun-2016 |
Dan Shi <dshi@google.com> |
[autotest] Allow base container to be set in global config This allows the base container to be updated using shadow config. Also move the base container folder to gs://abci-ssp bucket, as chromeos-image-archive bucket deletes files older than 6 months. BUG=chromium:471312 TEST=local run ssp test, unittest, ssp functional test: sudo python site_utils/lxc_functional_test.py Change-Id: Ibc73e439af92f12d774a0b87bdb28d5c3a8f493c Reviewed-on: https://chromium-review.googlesource.com/354550 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> Commit-Queue: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
1d3ea49f9aa25e778849faba96d423059d142a8b |
|
19-Apr-2016 |
Victor Hsieh <victorhsieh@google.com> |
Tee wget output in download_extract of lxc.py In crbug.com/604473, we've seen a wget timeout. No null devserver log is found for that specific request and the cause remains unknown. Adding the client side log could at least leave some clue for future error. TEST=lint check during repo upload BUG=chromium:604473 Change-Id: Ie0e422e4e574c273b4691019617c69abc1be86eb Reviewed-on: https://chromium-review.googlesource.com/339491 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Victor Hsieh <victorhsieh@chromium.org> Reviewed-by: Dan Shi <dshi@google.com>
/external/autotest/site_utils/lxc.py
|
38fff70fc5257cd1497455e7fbc2742a2bd47ba9 |
|
28-Mar-2016 |
Dan Shi <dshi@google.com> |
[autotest] Save control file before setting up container This is to prevent test failure caused by missing control file. Scheduler restart leads to all control files in drone_tmp being deleted. Due to the long delay of setting up container when running tests with SSP, this becomes more of an issue. This change saves the control file to result folder before setting up container. BUG=chromium:598379 TEST=local run test, unittest Change-Id: I10a7715477ce14d3cbba5176292fc9aca19f9f2d Reviewed-on: https://chromium-review.googlesource.com/335269 Commit-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
/external/autotest/site_utils/lxc.py
|
afa63872d1b63d8e6cad7de617569c215b03d26c |
|
24-Feb-2016 |
Dan Shi <dshi@google.com> |
[autotest] Change resultdir name inside container to be the same as outside container. The job folder (resultdir) in drone has the format of job_id-user_name. However, if a test runs inside the container (using server-side packaging), its folder name is job_id only. The reason was that we don't want to pass another piece of information to container. This causes a problem in crash collection. During log collection, test job creates a file uncollected_log_file to store the list of files to collect, in the format of source and target pair, where source is the origin path of the file, e.g., path to a log file in DUT, and target is a path to the destination to copy to, e.g., /usr/local/autotest/results/226/hardware_StorageQualBase.before If the log collection was interrupted or autoserv process died before logs can be collected, another autoserv process will be kicked off by scheduler to run with --collect-crashinfo. The new autoserv process will read the content of uncollected_log_file and tries to copy file based on the source/target pair. Due to the difference of job folder used in container, crash collection will fail as the target directory does not exist on the drone (as the job folder only has job_id, but not in the format of job_id-user_name. This change passes the full job folder name to setup_test call when setting up the container to run the test. That way, the result path will be exactly the same inside and outside of the container. BUG=chromium:589173 TEST=local run ssp test, unittest manual run crash collection: /usr/local/autotest/server/autoserv -p --use-existing-results --collect-crashinfo 100.96.51.226 -r '/usr/local/autotest/results/372-debug_user/100.96.51.226' Change-Id: I4e1abb8c566ca65e8e245e68f29a9b996fe6be32 Reviewed-on: https://chromium-review.googlesource.com/329055 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
3d02fa31fd707a0eb7ec3865cc0f70beaeae166e |
|
05-Nov-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Deploy moblab shadow config to container. Package install function does not work properly in container running in moblab. This CL fixed the bug by: 1. copy moblab_config.ini to container if it exists. 2. In install_package_precheck, replace if not SSP_ENABLED: with if not SSP_ENABLED and not utils.is_in_container(): BUG=chromium:551777 TEST=lxc_functional_test.py Change-Id: I98a0c600a400c6156c9f963b6f62fa89a55d821c Reviewed-on: https://chromium-review.googlesource.com/310976 Commit-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
a7e0117cdfc8a3efe93c94a21d1aea1fedf0e8df |
|
14-Jul-2015 |
Laurence Goodby <lgoodby@google.com> |
[autotest] increases container networking startup timeout Server-based tests were frequently failing to run on our moblabs due to "Could not resolve host: storage.googleapis.com" errors during container initialization. It seems that it sometimes takes more than 2 minutes for the container's networking to initialize. This CL increases the timeout. BUG=chrome-os-partner:42351 TEST=Ran on moblabs. Change-Id: I02c1f9b55ce8b2678c9abdda59363eee536cd2dd Reviewed-on: https://chromium-review.googlesource.com/285518 Trybot-Ready: Laurence Goodby <lgoodby@chromium.org> Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Laurence Goodby <lgoodby@chromium.org> Tested-by: Laurence Goodby <lgoodby@chromium.org>
/external/autotest/site_utils/lxc.py
|
d9094d46b18e76d0951c8f1883b3553525c4ac9e |
|
10-Jul-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a function to install multiple packages Each package install needs to run apt-get update, which takes extra time to run. The new function takes in a list of packages and python packages, so `apt-get update` only needs to run once. BUG=chromium:508314 TEST=sudo python site_utils/lxc_functional_test.py -v Change-Id: Ibfcfa24696f6e217820679f5405da67529ef2c45 Reviewed-on: https://chromium-review.googlesource.com/284597 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
af1cc0eac9c060c8a7001464685d9771387507da |
|
08-Jul-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Do not include hostname in timer stats used inside lxc container. The new timer is used by functions only runs inside container. This is to avoid creating individual timer stats for each container, which leads to extra disk space usage. BUG=None TEST=unittest Change-Id: Ic2321b61a3c8ffef3fa5aa98bda101e127a49cd8 Reviewed-on: https://chromium-review.googlesource.com/284262 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Fang Deng <fdeng@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
23d3a22552d947e08c348e59f4639f7b6ae7a873 |
|
26-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add retry in install package inside container. This is to alieviate the flake when running apt-get command. BUG=chromium:499631 TEST=local lxc_functional_test.py Change-Id: I0be1d9e305465bb73a5e8e1e5b2f4b57bd234f7d Reviewed-on: https://chromium-review.googlesource.com/282331 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: mukesh agrawal <quiche@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
ff78f11ff85372a1e2f8ddf8eefbc8b772686148 |
|
12-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Pass current user into container for RPC usage RPC is called using current OS user if not specified (frontend.py). This causes an issue for autoserv running inside container, as the OS user is always root. The root user may fail check_for_acl_violation_hosts if a host has special ACL that does not allow `everyone`. This CL add an SSP/user config in global config, lxc related code is updated so that the OS user of autoserv running in host will be passed into the container through shadow_config.ini. And frontend will use such user to make RPC. BUG=chromium:480525,chromium:502897 TEST=local sudo python site_utils/lxc_functional_test.py -v -s attach to the container update SSP/user with a user name (existing one should be root as the process site_utils/lxc_functional_test.py is executed with sudo) update frontend.py with code in this CL (this means only newer builds won't be hit by this bug.) run some RPC: import common from autotest_lib.server.cros.dynamic_suite import frontend_wrappers _AFE = frontend_wrappers.RetryingAFE() print _AFE.run('get_stable_version') Also make sure afe in the host still works. Change-Id: I9ba924e2652fadffa71d6298f74c5b82e8e26b4e Reviewed-on: https://chromium-review.googlesource.com/281062 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
1a277ef7800425aad62b284ef0240eee3f54d0a3 |
|
05-Jun-2015 |
Cheng-Yi Chiang <cychiang@chromium.org> |
[autotest] Fix install_package_precheck in lxc install_package_precheck should return True/False. BUG=chromium:483063 TEST=run ssp_PackageInstall test on drone. Change-Id: Ib5834fbf306290efe186903918049461b110e5a2 Reviewed-on: https://chromium-review.googlesource.com/275415 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org> Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
/external/autotest/site_utils/lxc.py
|
3ea0204531322203ab55616b39eca9092eaad23d |
|
06-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add some logging in lxc.py to help troubleshooting package install BUG=None TEST=local test Change-Id: Ic2b1b08f669819134e0402e013cc17400b76ebdc Reviewed-on: https://chromium-review.googlesource.com/275710 Commit-Queue: Dan Shi <dshi@chromium.org> Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Roshan Pius <rpius@chromium.org>
/external/autotest/site_utils/lxc.py
|
585b7b68efb37226bc2b6cf25e90ea88c208e3c1 |
|
01-Jun-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] swap python import order to support pip in moblab In Moblab, the host's python modules located in /usr/lib64/python2.7/site-packages is mounted to following folder inside container: /usr/local/lib/python2.7/dist-packages/. The modules include an old version of requests module, which is used in autotest site-packages. For test, the module is only used in dev_server/symbolicate_dump for requests.call and requests.codes.OK. When pip is installed inside the container, it installs requests module with version of 2.2.1 in /usr/lib/python2.7/dist-packages/. The version is newer than the one used in autotest site-packages, but not the latest either. According to /usr/lib/python2.7/site.py, modules in /usr/local/lib are imported before the ones in /usr/lib. That leads to pip to use the older version of requests (0.11.2), and it will fail. On the other hand, requests module 2.2.1 can't be installed in CrOS (refer to CL:265759), and higher version of requests module can't work with pip. The only fix to resolve this is to switch the import order, so modules in /usr/lib can be imported before /usr/local/lib. BUG=chromium:483371 TEST=local moblab run sudo python site_utils/lxc_functional_test.py -v Change-Id: I59ab724ac6bd974da648209248b9f6df3380ae0f Reviewed-on: https://chromium-review.googlesource.com/274411 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
ca3be48da9dd7882ba282d9e82413b4aad2a8364 |
|
06-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add sudo -n option to avoid password prompt. Also add a utility function to check if password is needed to run sudo command. We can add -n option to all sudo command that SSP used. However, once the property drone.support_ssp is set to False, no SSP code will be executed. Therefore, there is no need to add -n everywhere. BUG=chromium:484396 TEST=local run Start scheduler, confirm no password promt, and drone.support_ssp is False. Run sudo true, enter password. Run scheduler again, confirm no password promt, and drone.support_ssp is True. Change-Id: Ifbd302973db5d443499311e435ec99e32d24cb5b Reviewed-on: https://chromium-review.googlesource.com/269740 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Tested-by: Richard Barnette <jrbarnette@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
0a81428995db06c51af85af271a844e5f62b1576 |
|
20-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Record all args to metadb if job_id is not available. We are seeing some container creation failed with not job_id recorded. This change is to record all arguments to help troubleshooting. BUG=None TEST=local test Change-Id: I03f4c31aea2ab01e436fe491d29c767ff272482c Reviewed-on: https://chromium-review.googlesource.com/272389 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Fang Deng <fdeng@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
d3662afdf0d6a45da75d380ce41a8c27c42be82b |
|
07-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a server-side test to validate package install in container. This will serve as an example about how package installation in container can be done. TODO: add this test to autotest ebuild CQ-DEPEND=CL:269784 BUG=None TEST=local autotest run also confirm that test_that run of this test will fail with error: Install OS package is only supported in server-side packaging. Change-Id: I439eb528ac5794eb1e3fa2cb25de97dec552f1a4 Reviewed-on: https://chromium-review.googlesource.com/269998 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
5180d55144483c168b166d842be69a56dab353d4 |
|
04-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Retry container clone without using snapshot. This CL attempts to alieviate the flakeness of lxc-clone in Ganeti instance. We will evaluate the metadata to confirm the effectiveness, while looking for the root cause of this issue. Also add a property rootfs to container. This property will always point to the folder of the container's rootfs. So the caller doesn't need to check if the container is created from snapshot clone to determine if or not to use `delta0` or `rootfs`. BUG=chromium:484019 TEST=local test sudo python site_utils/lxc_functional_test.py -v Change-Id: Ibc04371bfb63e811a5bc14693d678adff6941323 Reviewed-on: https://chromium-review.googlesource.com/269009 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
7836d25b375a791ebc7b3d0d255c8732c4f35be2 |
|
28-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a generic config deployment module to handle configs for container. For container to run a test, it needs some config files to be able to do something like: resolve hostname with dns configured in host. ssh to dut. ssh to devserver. Currently we hard code to copy each file to container, and modify each file for it to work within container. This CL provides a generic way to manage a list of deploy config, each config specifies what file needs to be copied to where and if the existing file should be overwritten. For container to work in developer's workstation, the special handling of some configure files is still available. For lab servers, puppet should be used to push out the config files and shadow deploy config ( ssp_deploy_shadow_config). BUG=chromium:481706 TEST=local test with and without shadow ssp deploy config sudo python site_utils/lxc_functional_test.py -v -d chromeos1-dshi1.cros -r 172.17.40.27 also verify in moblab (python package install does not work in moblab yet due to bug 483371) Change-Id: I8bbe33a83262b7c746fbe0112cf61cea046931ea Reviewed-on: https://chromium-review.googlesource.com/267569 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
507fdc40ef300535886bc1ef71d18c1fa697b895 |
|
30-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add two helper functions to install packages. Server-side test should call these two functions to install required packages. Ideally, tests will not need to deploy packages to site-packages for test to run. BUG=chromium:453614 TEST=site_utils/lxc_functional_test.py Change-Id: I9ade9e1a804897a32cebac36836367feda13a24b Reviewed-on: https://chromium-review.googlesource.com/268437 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
e019bd9f7be4abdaeb1aef47998545a1529d4541 |
|
24-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Enable snapshot clone for Ganeti instance using aufs. Ganeti instances does not support overlayfs due to older kernel. However, it does support snapshot clone with aufs. Enable this will greatly enhance SSP performance in Ganeti instances. BUG=chromium:464834 TEST=verified in server7 with cbf run http://chromeos-autotest.cbf.corp.google.com/afe/#tab_id=view_job&object_id=10335 Change-Id: I76a7691033214347d5a4e892cd0776bde8c53433 Reviewed-on: https://chromium-review.googlesource.com/267106 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
d68d51c731ef83a45c173497b59a26b2e6870c1b |
|
22-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add autoserv's pid to the container's name The PID info will be used to determine if a container is orphaned. BUG=chromium:479383 TEST=sudo python site_utils/lxc_functional_test.py local run a server side test Change-Id: I4de2f7f266bcb793cfe1a97e412868aa49d53991 Reviewed-on: https://chromium-review.googlesource.com/266754 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
c4f79fcd5c21dd1d5d917c73582aa08399166d38 |
|
18-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Set ssh loglevel to ERROR for tests running in container. This change is to avoid logs being flooded with warning `Permanently added '[hostname]' (RSA) to the list of known hosts.` BUG=chromium:478364 TEST=local run to confirm log has no such warning message: http://dshi.mtv/results/589-dshi/172.27.215.232/debug/autoserv.DEBUG Change-Id: I63c124f96eb7f36a7dca5b9673716beb0ac24fe3 Reviewed-on: https://chromium-review.googlesource.com/266401 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
37bee2292a1c385839e14adf3b2799447d8a98ac |
|
14-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Reduce the container info to collect for better performance Also report the container usage data to metadb. BUG=None TEST=local lxc_functional_test.py, local dummy_server suite, search metadb http://172.25.61.45:9200/_plugin/elastic-hammer/ query string: { "query": {"bool": {"minimum_should_match": 2, "should": [ {"term": {"_type": "container_run_test"}}, {"range": {"time_recorded": {"gte": 1419384005.705098, "lte": 1507282542.893756}}} ]}}, "size": 100} Change-Id: I739187499a2ec82ffe2a08d24e7d326743380653 Reviewed-on: https://chromium-review.googlesource.com/265532 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
f2a2dd64129cb12158bf065e61d9328ac45b1e2f |
|
10-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update lxc wrapper to work with Ganeti instance trusty/server with updated kernel does not support lxc-clone with snapshot, the wrapper (lxc.py) is updated so SSP can work before ticket (t/16003207) is resolved. BUG=chromium:453614 TEST=verified in cros-database-bak-test-1 Change-Id: I45208d745dea5614930481b9f159293b1f933cc7 Reviewed-on: https://chromium-review.googlesource.com/265274 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Fang Deng <fdeng@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
a958bd386256b2e287f7f023a680f83c4983c1cf |
|
02-Apr-2015 |
Simran Basi <sbasi@google.com> |
lxc: Don't delete the base container if it already exists. Currently if the base container exists and we don't pass in force-delete, it will automatically delete the base container. BUG=None TEST=panther_moblab build. Reboot's no longer delete the base container. Change-Id: I764ccd51df826b5e66fa79cfa06bccbccca8815d Reviewed-on: https://chromium-review.googlesource.com/263650 Trybot-Ready: Simran Basi <sbasi@chromium.org> Tested-by: Simran Basi <sbasi@chromium.org> Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
f35517a77f9a871276396ea57a553f5e6e72cd88 |
|
02-Apr-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update ssh config and resolv.conf in container For container to be able to ssh to duts in lab, it needs the .ssh/conf and /etc/resolv.conf from the drone. It's not reliable to build the content inside the base container, as such config may change over the time. For moblab, it doesn't need ssh/config (it doesn't exist). /etc/resolv.conf is not needed as well, however, it has no side effect to add such info inside container. BUG=chromium:453614 TEST=verify in a test drone to ssh to lab dut using hostname http://chromeos-autotest.cbf.corp.google.com/afe/#tab_id=view_job&object_id=9685 Change-Id: I4f09a7bdd4e9c8c57dee0e67922c22779aa83797 Reviewed-on: https://chromium-review.googlesource.com/263570 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
387667a8caafacd310b54b5e32e11f9649146126 |
|
24-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update shadow config in container with FQDN of db/afe server name. container may not be able to resolve hostname like cautotest. The FQDN should be used in db and afe server name in container's shadow config. BUG=chromium:453624 TEST=lxc_functional_test. verify with a test drone Change-Id: I6cf337edc8d36e8fbb9e58cbe5e43e3227b65f10 Reviewed-on: https://chromium-review.googlesource.com/262214 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
5b5f88e9f4a9237221ba17ba10076add151d111f |
|
18-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update shadow config in container with correct db/afe server name. The original CL 260980 was reverted due to a missing package (netifaces). This CL is almost the same as the original one except moving import of netifaces after import common. Container uses a copy of shadow config from the host. The config file needs to be updated for autoserv running in container to use: 1. Disable master ssh connection in shadow config, as it is not working properly in container yet, and produces noise in the log. 2. Update AUTOTEST_WEB/host and SERVER/hostname to be IP of the host if any is set to localhost or 127.0.0.1. BUG=chromium:453624 TEST=lxc_functional_test, verify in moblab http://chromeos1-dshi-moblab1.cros/afe/#tab_id=view_job&object_id=25 Change-Id: Ic836d3f46901117b8f21d655c32b22d9815c328c Reviewed-on: https://chromium-review.googlesource.com/261575 Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Trybot-Ready: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
bbe7591943c5d35ff6475c14b0a9c24f5366a418 |
|
19-Mar-2015 |
Simran Basi <sbasi@chromium.org> |
Revert "[autotest] Update shadow config in container with correct db/afe server name." This reverts commit 309c48f160be2a26429be1456634a73fd89a1a9c. Change-Id: I7456dc5c920e31f9cc6e019943ea879c1c9e7c28 Reviewed-on: https://chromium-review.googlesource.com/261240 Reviewed-by: Simran Basi <sbasi@chromium.org> Tested-by: Simran Basi <sbasi@chromium.org>
/external/autotest/site_utils/lxc.py
|
309c48f160be2a26429be1456634a73fd89a1a9c |
|
18-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update shadow config in container with correct db/afe server name. Container uses a copy of shadow config from the host. The config file needs to be updated for autoserv running in container to use: 1. Disable master ssh connection in shadow config, as it is not working properly in container yet, and produces noise in the log. 2. Update AUTOTEST_WEB/host and SERVER/hostname to be IP of the host if any is set to localhost or 127.0.0.1. BUG=chromium:453624 TEST=lxc_functional_test, verify in moblab http://chromeos1-dshi-moblab1.cros/afe/#tab_id=view_job&object_id=25 Change-Id: I4a487f7b95fc851074680e585553fa739869d900 Reviewed-on: https://chromium-review.googlesource.com/260980 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
cf4d2032ea4bf5af680383f36308d581876bbbb0 |
|
12-Mar-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Update autoserv to support server-side packaging. Code change is made so autoserv will honer the --require-ssp arg. When the argument is specified, autoserv will: 1. Try to locate server-side package based on --image, host attribute job_repo_url and host's cros-version label. 2. If no server-side package can be staged, the job will be run without ssp. 3. If package can be staged, autoserv will start a container to run the test. a) parent autoserv process's log will be stored in [results]/wrapper folder. b) A container will be created from base container, with proper setup. c) The same autoserv command line (with update on result path and control etc.), will be executed inside the container. d) Parent autoserv process will wait for above command to finish and destroy the container afterwards. If the test job was aborted, container should also be destroyed. BUG=chromium:453624 TEST=local setup and run test command: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/16-debug_\ user/$result_dir -m 172.27.215.232 -u udebug_user -l dummy_PassServer -s -P \ 16-debug_user/$result_dir -n /usr/local/autotest/results/drone_tmp/attach.2 \ --verify_job_repo_url --require-ssp create job from afe: http://dshi.mtv.corp.google.com/afe/#tab_id=view_job&object_id=19 Change-Id: Ida36374dd500cd1fd0b67a86ab8c0198ade4cc36 Reviewed-on: https://chromium-review.googlesource.com/259834 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|
767dced065c8918b5ff74c1347371a0f0f8d9c69 |
|
01-Feb-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a utility module to interact with LXC binaries. Add a module (lxc.py) to interact with LXC binaries. lxc_functional_test.py is a test script to test the module. It will 1. Set up base container in a tmp directory. 2. Create a test container from the base container. 3. Test site-packages and results directory can be mounted and shared between host and test container. 4. Run autoserv in test mode. Also add a utility function in client/common_lib/site_utils.py to get the value of the given argument for the function. It's used in decorator to check function parameters. BUG=chromium:453621 CQ-DEPEND=CL:251140 TEST=unittest, sudo python site_utils/lxc_functional_test.py sudo python site_utils/lxc.py -s -p /tmp/container_123 sudo python site_utils/lxc.py -s -p /tmp/container_123 -f sudo python site_utils/lxc.py -p /tmp/container_123 -f Change-Id: Id7b259c4bc03977974af44d6d88420b220706589 Reviewed-on: https://chromium-review.googlesource.com/247271 Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org> Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/lxc.py
|