| a27022cefc4f3e4e498c99609f9b060b16d07dca |
17-Nov-2017 |
Dan Shi <dshi@google.com> |
Typo fix
BUG=None
TEST=None
Change-Id: Icd82973e03fced84797e313c52cde6e6266c75fd
Reviewed-on: https://chromium-review.googlesource.com/776201
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
esult_info.py
|
| 11e350681359d70b15703190c359047aceb4e81b |
03-Nov-2017 |
Dan Shi <dshi@google.com> |
[autotest] Improve performance on result size collection.
1. Don't do a deepcopy for result info of new directories.
2. Delay updating parent result info's size information.
3. Add metrics for the collect_result_sizes call.
BUG=chromium:781066
TEST=unittest
Change-Id: I0b13d8641d1cdfc46c503822f41e25088245f37d
Reviewed-on: https://chromium-review.googlesource.com/753974
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Allen Li <ayatane@chromium.org>
esult_info.py
tils.py
|
| 5e8cde91cd0e40044627bc2241de0ff905e0a7b3 |
31-Oct-2017 |
Dan Shi <dshi@google.com> |
[autotest] Disable recursive calls to update parent result_info's sizes
This allows compiling result summary for a directory with large number of
files to run much faster. Reduce an example of 20k files from 30mins to
12s.
BUG=None
TEST=unittest
Change-Id: I95cd4b3113a7a8686746b33baf60951ef4639079
Reviewed-on: https://chromium-review.googlesource.com/747090
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
esult_info.py
tils.py
|
| cdf50c281416123b59f9ff6d1c76961117177c3c |
18-Oct-2017 |
Dan Shi <dshi@google.com> |
[autotest] Try to throttle non-autotest logs first
This change allows result throttling to throttle non-autotest logs first, and
keep as much autotest logs, e.g., autoserv.DEBUG, as possible for
troubleshooting.
BUG=chromium:774812
TEST=unittest
Change-Id: I5dc79d34589133d08c991692f6c22483f4767948
Reviewed-on: https://chromium-review.googlesource.com/726407
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>
hrink_file_throttler.py
hrottler_lib.py
tils.py
tils_unittest.py
ip_file_throttler.py
|
| 5aaf906ce6dbb255b669457342f7ac59d000001e |
20-Sep-2017 |
Dan Shi <dshi@google.com> |
[autotest] Delete dir_summary json files in parsing
The directory summary files have no debugging value and not human readable.
BUG=None
TEST=unittest, local test.
Change-Id: I05e0032a7d45dbfcba2a5a779b119afba24d59c2
Reviewed-on: https://chromium-review.googlesource.com/675665
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
tils.py
tils_unittest.py
iew.py
|
| a9475ec9bd5391f196cdef49a38be24949ef0a45 |
25-Aug-2017 |
Dan Shi <dshi@google.com> |
[autotest] Do not shrink build info file for ACTS tests
The proper fix is to add the file patterns to
throttler_lib.NON_THROTTLEABLE_FILE_PATTERNS
so no throttler will modify such files.
BUG=b:64797715
TEST=unittest
Change-Id: I4ff10e801facf6e13f3dc6dfc8d7cc7fc65aa3ab
Reviewed-on: https://chromium-review.googlesource.com/636201
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
edupe_file_throttler.py
elete_file_throttler.py
hrink_file_throttler_unittest.py
hrottler_lib.py
|
| 451e8454c7336e3d8d71c4b6b30ed623097a2c91 |
25-Aug-2017 |
Dan Shi <dshi@google.com> |
[autotest] Protect perf.data from being throttled.
*perf.data can be compressed but not deleted. This change prevents that from
happening.
BUG=chromium:716218
TEST=unittest
Change-Id: Ia08bdabe6b251497aa6a402323198d5b8275ab72
Reviewed-on: https://chromium-review.googlesource.com/633853
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>
edupe_file_throttler.py
elete_file_throttler.py
elete_file_throttler_unittest.py
hrink_file_throttler.py
|
| 83a60bbd061ab37e0e6791563644c9efb8064fb3 |
24-Aug-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add a conversion in result info for backwards compatibility
For tests running with server-side packaging, the result tools are from older
branches, which has a different format. This change checks for the format
difference and convert it to the latest if needed.
The conversion can be removed after R62 is in stable channel.
BUG=chromium:716218
TEST=merge summary from older format to new one.
Change-Id: I618c290e1a0231a9c1843be020b5c814e02f969e
Reviewed-on: https://chromium-review.googlesource.com/634266
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
esult_info.py
|
| ac39cc3364754cf7e3f02122b69ea6acf3d4f40d |
23-Aug-2017 |
Dan Shi <dshi@google.com> |
[autotest] Protect ACTS result files from result throttling.
The files are added to NON_THROTTLEABLE_FILE_NAMES, so result throttling won't
affect them.
BUG=b:64797715
TEST=unittest
Change-Id: I464a22af206ad0f586949076758ff02ac69901ff
Reviewed-on: https://chromium-review.googlesource.com/629204
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Benjamin Peake <bpeake@chromium.org>
edupe_file_throttler.py
elete_file_throttler_unittest.py
hrink_file_throttler.py
hrottler_lib.py
|
| 9b179d07ca8eb9c40cb2485cb6a89e2f36e9d4c9 |
21-Aug-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add some exceptions in result throttling
Following files are used by test tracker, so should not be compressed:
BUILD_INFO_*
Following files are used by ACTS tests, so should not be trimmed:
test_run_*
BUG=b:64797715,b:64904537
TEST=local
Change-Id: I1b311cd7d111edc908c759d357a1fc92d098872a
Reviewed-on: https://chromium-review.googlesource.com/624544
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
hrink_file_throttler.py
hrink_file_throttler_unittest.py
ip_file_throttler.py
ip_file_throttler_unittest.py
|
| 5d1066ababfd9e96ade7b0179f8883cdea974e3b |
18-Aug-2017 |
Dan Shi <dshi@google.com> |
[autotest] Fix minor bug in collecting /dev/pstore
/dev/pstore is not writable, so use temp directory to do the collection so
result summary can be collected.
Also add a clean_content option to delete directory content, not the
directory itself.
BUG=chromium:756615
TEST=local run repair
Change-Id: I3e8945b590c3c6d6910ce3dd036f562a1336d472
Reviewed-on: https://chromium-review.googlesource.com/621249
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
tils.py
|
| 67fbe9bccad30b76734051c93231b8794468d1b5 |
17-Aug-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add acts result files to white list.
This change makes sure files used for test tracker are not deleted during result
throttling.
BUG=b:64797715
TEST=unittest
Change-Id: I4c30211ae9ba9046eb14bbceb43a0636e2c30def
Reviewed-on: https://chromium-review.googlesource.com/619388
Reviewed-by: Dan Shi <dshi@google.com>
Commit-Queue: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Trybot-Ready: Dan Shi <dshi@google.com>
elete_file_throttler.py
elete_file_throttler_unittest.py
hrottler_lib.py
|
| 36c4c69318083e7fe12a567b081a5d6f7bcc0404 |
28-Jul-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add a success field in the metrics for result tools.
This allows metrics to be filtered by only successful calls, removing the noise
from offline duts.
BUG=chromium:716218
TEST=local run test
Change-Id: I175f4e907a9734ef3bca3ce54f451ad72a8ff036
Reviewed-on: https://chromium-review.googlesource.com/590951
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>
unner.py
|
| 132c7a9b1075acf9c120a84cd24ea12ef05a9e33 |
26-Jul-2017 |
Dan Shi <dshi@google.com> |
[autotest] Make result_tools run faster
1. Add a module flag to only deploy result_tools to dut once across the test.
This avoid 0.2s for no-op rsync calls to copy result_tools.
2. Add excludes for only deploy py modules to dut.
3. Only update original size at the end of build_from_path. This prevents
recursive updating parent directories' size information, and reduce the
run time to 1/5, for a result with 600 files (more savings for results with
large number of files.)
BUG=chromium:716218
TEST=local run test, check files on the dut.
Change-Id: Iac98ed01c1e372e98d353236f8ba8184a63104e7
Reviewed-on: https://chromium-review.googlesource.com/584864
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
esult_info.py
unner.py
|
| 4f8c0244d0ce557488a7ddbb29f28a2719a2523f |
08-Jul-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add logic in test run and crashcollect to throttle results.
Add throttling call in tko/parse.py to throttle the results created on the
server side.
For special tasks, there is no parse stage, so the final result throttling
is triggered in autoserv (and crashcollect where the code directly calls
host.get_file).
BUG=chromium:716218
TEST=unittest, local run test
Change-Id: I637bbead04f82f8f529d1d0982e3567445901938
Reviewed-on: https://chromium-review.googlesource.com/564140
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
esult_info.py
esult_info_unittest.py
unner.py
tils.py
|
| c958a3d5b7053c6e807ec6ea7967a9271d4e09da |
06-Jul-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add logic in result_tools/utils to throttle results if needed.
The change implements a sequence of throttling strategies to try to reduce result
size to be under the limit.
BUG=chromium:716218
TEST=unittest, local run test
Change-Id: I8a4f7f85b074a8aae1fca600edbd70ed81e6d92d
Reviewed-on: https://chromium-review.googlesource.com/562740
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
unner.py
tils.py
tils_lib.py
tils_unittest.py
|
| 4b691e55808ec03bd55cbf4000adc98d4fb24dcb |
05-Jul-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add two result throttlers (dedupe_file_throttler, zip_file_throttler)
dedupe_file_throttler:
This throttler tries to remove the remove repeated files sharing the same
prefix, for example, screenshots or dumps in the same folder. The dedupe logic
does not compare the file content, instead, it sorts the files with the same
prefix and remove files in the middle part.
zip_file_throttler:
This throttler tries to reduce result size by compress files to tgz file.
BUG=chromium:716218
TEST=unittest
Change-Id: I98993da2ca18823fb4689ac9bea5679646ea2035
Reviewed-on: https://chromium-review.googlesource.com/559849
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
edupe_file_throttler.py
edupe_file_throttler_unittest.py
esult_info.py
ip_file_throttler.py
ip_file_throttler_unittest.py
|
| 1e6dcef33df25a7435c7b111e5a2bf0d86071c9d |
30-Jun-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add a delete_file_throttler to throttle result size
This throttler deletes result files from largest first to reduce result size.
BUG=chromium:716218
TEST=unittest
Change-Id: Ib8a8e9727bed84a1a7143eacab3b70aba9ec1ff9
Reviewed-on: https://chromium-review.googlesource.com/557306
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
elete_file_throttler.py
elete_file_throttler_unittest.py
|
| c45ac69f11a6c334743c9fc7cbafae733e75247b |
20-Jul-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add shrink_file_throttler used to reduce test result size.
shrink_file_throttler tries to remove the middle part of test result files
to reduce the result size.
The logic is not triggered in the utils call yet.
BUG=chromium:716218
TEST=unittest
Change-Id: I7506cdbfeeeaaac0b050cb3ff9acebfde18b5a54
Reviewed-on: https://chromium-review.googlesource.com/580405
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
hrink_file_throttler.py
hrink_file_throttler_unittest.py
hrottler_lib.py
hrottler_lib_unittest.py
|
| 393fc8c903d61bad72ad0ab13d56955ac2888912 |
29-Jun-2017 |
Dan Shi <dshi@google.com> |
[autotest] Add a model to wrap the dictionary of result information.
The wrapper class ResultInfo is added to make the result information model
much cleaner. All result info related properties and methods are consolidated
in a single class.
BUG=chromium:716218
TEST=unittest
Change-Id: Ie40974e7ea9911a813aa7c7879505a2fd5487978
Reviewed-on: https://chromium-review.googlesource.com/579573
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
esult_info.py
esult_info_lib.py
esult_info_unittest.py
tils.py
tils_lib.py
tils_lib_unittest.py
tils_unittest.py
iew.py
|
| ffd5b82912ac09842a2b55dd0706343c231bcf3e |
14-Jul-2017 |
Dan Shi <dshi@google.com> |
[autotest] Report test result sizes to metrics
There are two reporting source:
1. Test jobs report result size information in tko/parse.
2. Special tasks report result size information at the end of autoserv process.
BUG=chromium:716218
TEST=unittest, local run test
Change-Id: I2410f3a3a5a16c1673446633e15c038a5e4ef81f
Reviewed-on: https://chromium-review.googlesource.com/571859
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
tils_lib.py
|
| e2e995d750616022e2d5cfeda5d9eb2bcd78df2d |
27-Jun-2017 |
Dan Shi <dshi@google.com> |
[autotest] Move const and helper methods to shared lib module.
This change has no function changes. It splits some constants and helper
methods to separate modules so others can share.
BUG=chromium:716218
TEST=unittest
Change-Id: Ieb4a4de8cfc825d204c4c2e1247a2e9eeeb0639d
Reviewed-on: https://chromium-review.googlesource.com/548970
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
nittest_lib.py
tils.py
tils_lib.py
tils_unittest.py
iew.py
|
| ad719c19df78d5e28ebb760d87ff1ac08cb0e45d |
28-Jun-2017 |
Allen Li <ayatane@chromium.org> |
Revert "Rename SSHHost.run to SSHHost.run_very_slowly"
This reverts commit 672fb5f8806694d9476f016c0f1094da29120f31.
Reason for revert: Broke android tests: crbug.com/737316
Original change's description:
> Rename SSHHost.run to SSHHost.run_very_slowly
>
> This CL renames several places that calls SSHHost.run directly to SSHHost.run_very_slowly.
> In addition, it warns the caller with a more verbose server stack message now.
>
> BUG=chromium:735653
> TEST=test_that dut graphics_Sanity
> ./utils/unittest_suite.py --debug
> CQ-DEPEND=I2a434782b9b7ed7a3d31289d9925f3c8502a6d9f
>
> Change-Id: Icd22fe9bcc4b5a47320478932194a8b535f4a936
> Reviewed-on: https://chromium-review.googlesource.com/545116
> Commit-Ready: Po-Hsien Wang <pwang@chromium.org>
> Tested-by: Po-Hsien Wang <pwang@chromium.org>
> Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Bug: chromium:735653
Change-Id: If57cd6103d73bc1f0335866d8c130953754df0d9
Reviewed-on: https://chromium-review.googlesource.com/550970
Reviewed-by: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Trybot-Ready: Allen Li <ayatane@chromium.org>
unner.py
|
| 672fb5f8806694d9476f016c0f1094da29120f31 |
22-Jun-2017 |
Po-Hsien Wang <pwang@chromium.org> |
Rename SSHHost.run to SSHHost.run_very_slowly
This CL renames several places that calls SSHHost.run directly to SSHHost.run_very_slowly.
In addition, it warns the caller with a more verbose server stack message now.
BUG=chromium:735653
TEST=test_that dut graphics_Sanity
./utils/unittest_suite.py --debug
CQ-DEPEND=I2a434782b9b7ed7a3d31289d9925f3c8502a6d9f
Change-Id: Icd22fe9bcc4b5a47320478932194a8b535f4a936
Reviewed-on: https://chromium-review.googlesource.com/545116
Commit-Ready: Po-Hsien Wang <pwang@chromium.org>
Tested-by: Po-Hsien Wang <pwang@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
unner.py
|
| 010c0bcc9c7d91c451534d95b12305649627c180 |
22-Jun-2017 |
Dan Shi <dshi@google.com> |
[autotest] Move result_utils to its own package
This change moves result_utils.py to its own package. More code will be added to
enable result throttling. So it's more manageable to put all related files to a
single folder.
This change also push the result tools modules to client before trying to build
result summary. This removes the client side dependency. So all result
throttling logic will be kept on the server side.
BUG=chromium:716218
TEST=unittest, local run dummy tests
Change-Id: Icc3542af99439fd39b1dba36ac45aab6effd1080
Reviewed-on: https://chromium-review.googlesource.com/544049
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Dan Shi <dshi@google.com>
_init__.py
ommon.py
unner.py
tils.py
tils_unittest.py
iew.py
|