| 39164cad07a790017c84482bb5f68768b27313cf |
|
27-Mar-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] only retry tests if they fail with TestFailRetry
For tests that have known flaky failure modes, it is still desirable for
the test to fail (without retries) when an unknown flaky failure occurs.
Therefore, this CL changes the behavior of test retries, so that retires
of a test are only attempted if the test fails with a TestFailRetry
exception. Any other test failure will be treated as a failure
regardless of retries.
This means it is up to the tests in question to catch errors /
exceptions caused by known flaky failure modes, and rethrow those as
TestFailRetry.
For the purposes of testing this change, some changes have been made to
the dummyflake suite.
CQ-DEPEND=Ibaa84f42beac52881cd34351e92474ef1457b15b
BUG=chromium:224372
TEST=added new unit test for test.py
run_remote_tests.sh --remote=<ip> suite:dummyflake;
Suite runs as expected. In particular --
- client/.../dummy_Fail/control.retry_alwaysfail fails immediately even
though the control file uses retries, because the failure is of type
TestFail and hence not retry-able
- The same applies to server/.../dummy_Fail/control.retry_failfast
- client/.../dummy_Fail/control.retry_alwaysflake fails and gets
retried 5 times, failing with a TestRetryFail each time such that it
can be retried
Change-Id: I4d1354cb410856c9de8b720c9f8310cb10d03156
Reviewed-on: https://gerrit.chromium.org/gerrit/46696
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/client/common_lib/test_unittest.py
|
| 91493c88a1506729a6728660d90cfae591cd52aa |
|
25-Jan-2013 |
Scott Zawalski <scottz@google.com> |
Introduce a mechanism for retrying tests at the control file level.
If a test returns unsuccessfully and --retry-test > 0 is specified the test
will be rerun. Options were added to autoserv and are passed through to the
client side job by modifying the base_job.
Tests that fail and then succeed within the --retry-test limits will have a
test keyval set with the number of attempts |test_retries_before_success|.
--retry-test can be utilized immediately in run_remote_test but will need to
be piped through the RPC system and down to the scheduler to provide the proper
--retry-test value from the scheduler's perspective.
Fixed random pylint errors consisting mostly of unused imports.
TEST=unittests
created a randomly failing client and server side test and ran it on devices
BUG=chromium-os:37158
Change-Id: Ibec3935b5f6fd28fc1b6eb7be55de27a571ad777
Reviewed-on: https://gerrit.chromium.org/gerrit/42043
Commit-Queue: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Scott Zawalski <scottz@chromium.org>
/external/autotest/client/common_lib/test_unittest.py
|
| daf6ff0673466abb5b2cf65ca0c5dfe78360a082 |
|
02-Mar-2011 |
Eric Li <ericli@chromium.org> |
Merge remote branch 'autotest-upstream/master' into merge-upstream
rev 5270 and 5271.
BUG=None
TEST=Run BVT test.
Change-Id: I46ee56cea4861d9dd35b98f0a21536122714518c
[PATCH] After_iteration hooks are not executed if test fails.
Base_test's _call_run_once() does not execute the after_iteration hooks
if the test fails. It is important to execute the after_iteration hooks
in the event of a test failure since these hooks may be used to gather
important test logs to help diagnose the failure. This patch
modifies _call_run_once() to execute after_iteration hooks regardless of
the outcome of the test.
A new unit test was added to verify the behavior of _call_run_once(). The
patch was tested using unittest_suite.py to ensure that it did not break
other functionalities. It was also tested with an actual after_iteration
hook that collected test logs and the test was forced to fail.
Risk: High
Visibility: The order of when the after_iteration hooks has been changed
and may affect existing tests.
Bugs: None
Signed-off-by: Thieu Le <thieule@chromium.org>
git-svn-id: svn://test.kernel.org/autotest/trunk@5271 592f7852-d20e-0410-864c-8624ca9c26a4
KVM test: Make unattended setup copy finish.exe to floppies
Fixing yet another bug from the conversion of unattended
install to infrastructure.
Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
Review URL: http://codereview.chromium.org/6598041
/external/autotest/client/common_lib/test_unittest.py
|
| a6082efeb30cd86d09fcf25d55cde15dedd60fce |
|
12-Oct-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add a global mechanism to the job object to allow users to change
the behaviour of profile_only, so that all tests default to
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3818 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| c6bf601d0de97a86cb9d09db08b4ea725fac2003 |
|
02-Oct-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add a WARN message to the status log when a test runs with
profile_only=True but no profilers set up. It's all too easy to make
this mistake (e.g. tweaking a control file set up for profiling runs
to turn off the profilers, but forgetting to remove the profile_only)
and it leads to lots of confusion when the test successfully runs
with zero iterations.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3784 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 6631273af8b88842cbd6202cc4615daf050cc957 |
|
27-Aug-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make a bunch of stuff executable
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3621 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| a9894d0f2364fd7c963d177367004220dd6e6471 |
|
21-Aug-2009 |
jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> |
Back out a python2.4 ref that was accidentally re-inserted.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3588 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| a93fbca94f7ae7db3c4c7d45a881c4a77cddf4df |
|
21-Aug-2009 |
jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> |
Apply a unit test change that was never applied along with the
code changes it was coupled with.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3586 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 7c8ea99fecf140093f2fe3dcd0ab6b5bf65b1bbe |
|
22-Jun-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Not all distros put a symlink in for the python version. However
all should have a /usr/bin/python target. There are still a couple
files that hardcode a /usr/lib/python2.4 path, but that's a harder
problem to solve and outside the scope of this patch.
Signed-off-by: Jesse Keating <jkeating@redhat.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3319 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| f1175bb2edbe3215f4bfa117c6b93d2e090d1d07 |
|
17-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
get rid of some now-obsolete code
* client.common_lib.debug
* abstract_ssh.LoggerFile
Changed {SSHHost,ParamikoHost}.run() default args for std{out,err}_tee to a constant instead of None. This preserves the existing behavior (teeing to the logging module by default), but allows callers to explicitly avoid this teeing if desired, by passing stdout_tee=None.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3293 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 5e703a286f9f112c419a27402b232e78bfd7bdc8 |
|
16-Jun-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
- Initialize the self.iteration variable to 0. When iterations=0 is specified to run_test then self.iteration is left uninitialized and code elsewhere that uses this value gets an attribute error.
- Also increment the self.iteration variable value in the profiling run so that the last iteration results are not overwritten by the profiling run.
Signed-off-by: Ashwin Ganti <aganti@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3287 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 742ae42df0b167f6a29b0f64466fc99401f57018 |
|
13-May-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Exposed an interface for adding before_iteration_hooks and
after_iteration_hooks. These are stored in a list in test
object and are run before and after iteration runs.
Signed-off-by: Travis Miller <raphtee@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3129 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 32cb5b450ce80e5e5871142592973e57f3771ae2 |
|
02-May-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
We now store the failures due to constraints and dont raise
an exception until after postprocess.
Signed-off-by: Travis Miller <raphtee@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3077 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| d4ead172a08004b6059dc2a3546435de2ed42a3f |
|
01-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add missing mock god check playback and fix the test.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3072 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 7af0997bb099f0b0ad331812d6b677bb0c695bd9 |
|
18-Apr-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
This is the second CL relating to a re-implementation of how
constraints on keyvals are evaluated. The code has been moved
into the base_test class itself. This required some changes to the
old tests that had evaluated the constraints themselves.
Signed-off-by: Travis Miller <raphtee@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3012 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 5c1bb2564e01e94f1fccb9272dbd613666db3e63 |
|
25-Mar-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Moved warmup() call from execute() to _exec() before calling on
execute() thus fixing an issue of filtering arguments sent to execute()
based on what run_once/execute accepts but ignoring the fact that
warmup() may have needed some other arguments.
Removed _run_cleanup() and updated users because where it is called
there is no need to translate the exceptions from self.cleanup() (as they
are either ignored or translation happens from the surrounding context).
Defined a _cherry_pick_call() function perform both cherry picking of
arguments and function call. Defined _call_test_function() to wrap a
given function call for test code and translate exceptions if needed.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2934 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 4395bbd96ed53aec5fad87d6c169a47dfdb8c9dd |
|
25-Mar-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added sysinfo support to log pre and post iteration (currently it logs
the contents of /proc/schedstat). Added client.common_lib.test hooks to
call pre and post iteration. Used the hooks in client.bin.test.runtest()
for client tests. Refactored server.test and added wrappers for the
pre/post iteration logging and used them on the new pre/post iteration
hooks.
For each iteration there will be a
resultsdir/sysinfo/iteration.N/schedstat.after and schedstat.before.
If there are tests that override execute() and implement their own
iteration logic they need to be changed over to override run_once()
instead and take advantage of the default execute() that supports the
pre/post iteration hooks and other things.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2929 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 6b97f79e7c71803567e393b92e993705a46480e3 |
|
23-Mar-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Removed code redundancy by moving the
drop_caches/run_once/postprocess_iteration calls
into a function. Possibly fixed bugs because of postprocess_iteration
not being called after run_once iterations from the code handling
"test_length" test time argument.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2923 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| a49c5cb549607a2049c45eed66200e48aae48c3b |
|
26-Feb-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Change the name of the variable to postprocess_profiled_run. Allow it to
be an object attribute too.
Signed-off-by: Jean-Marc Eurin <jmeurin@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2835 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| c931408ae77c801c5188c8e40cab766449442c89 |
|
26-Feb-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add an option (profile_perf) to record performance of profiled runs.
For example:
job.run_test('sleeptest', iterations=0, profile_perf=True)
will record performance of the profiled run.
Signed-off-by: Nikhil Rao <ncrao@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2830 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 4b835b89f2a0bd9aef2df4157fadc34fe2eeb14c |
|
11-Feb-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Adds a profile_only keyword argument to test.execute to be used by
job.run_test() from control files. When specified it behaves identically to
iterations=0 but is much easier to read. Also converts prints in test.execute
into test_log calls and adds a test.execute unit test.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2773 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|
| 234a84f025f03651e1973bfcb84a3059fc0eb373 |
|
20-Nov-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
First (of many) passes at cleaning up the client job exception handling
to do the right thing. Exceptions during test execution == FAIL,
exceptions during other phases == ERROR. Explicitly specified
AutotestError subclasses are passed through.
This fixes a bug where exceptions were ignored when run_cleanup was False.
It adds docstrings for some internal functions and hopefully simplifies
the code a bit.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2483 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/client/common_lib/test_unittest.py
|