| 939f57efdd21c6dd47f92f6b8b6f2221c6911595 |
|
14-Jan-2016 |
Mike Mason <michael.w.mason@intel.com> |
graphics_dEQP: hasty expectation lists used in non-hasty mode
The dEQP autotest mistakenly used the hasty exception lists to exclude
tests even in non-hasty mode, which resulted in some tests not being
run. This was caused by the wrong string being used to filter out
hasty file names in non-hasty mode. This patch fixes that string.
BUG=49397
TEST=test_that <ip_address> graphics_dEQP.gles3.functional
This should run ~41.5K tests, but instead only runs ~22K tests.
The 'dEQP-GLES3.functional.hasty.Flaky' expectation contains ~19K tests.
Change-Id: Ic95aa779428df545b6ee3637a27e45f42af9e029
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/321866
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 0facf6fd666d27bee7965759d71fd47d8a62d46c |
|
09-Dec-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: add control files.
We would like to run tests that fail on a daily schedule.
Run gles3.functional in hasty/sharded mode in preparation for
moving it to bvt-cq one day.
All these files are now created using generate_controlfiles.py.
BUG=chromium:556747
TEST=test_that <samus> graphics_dEQP.gles2.accuracy graphics_dEQP.gles3.accuracy graphics_dEQP.gles2.info graphics_dEQP.gles3.info
Change-Id: I34de4f309d2cb433c1f344a3b43fadc39d32307d
Reviewed-on: https://chromium-review.googlesource.com/317032
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 3d4a32186c4f856be868eaa815c4bfb4e2167b83 |
|
21-Oct-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: warn instead of error.
When we run tests that are known not to be passing they could be
flaky or the failure can depend on previously ran tests. For this
reason just warn if a test that we thought should be failing is
now passing. It does not require immediate attention.
BUG=None.
TEST=None.
Change-Id: I18524da875aef2fee3fe40dd49dd0997cbf4a9c9
Reviewed-on: https://chromium-review.googlesource.com/307924
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 20d15b2e2566840876c1a9f35051403249a98a3a |
|
15-Oct-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: unsupported GPU -> TestNAError.
We want to keep track of these GPUs, in particular in preparation of
moving graphics_dEQP into bvt-cq. And DEPENDENCIES don't work as
they should. (Might still be worth adding DEPENDENCIES in addition to
avoid lab overhead though.)
BUG=chromium:510208
TEST=None.
Change-Id: I6ab344b17e7da103311e384c3e57b98e365f0179
Reviewed-on: https://chromium-review.googlesource.com/305715
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| dd11bd5cf2ae753fe9d229fc37dfb9f72135ffe8 |
|
01-Oct-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: change NotPass to error on pass.
And output how many tests are passing now. This should make reading the
dashboard easier.
BUG=chromium:510208
TEST=Ran on pixel.
Change-Id: I9f7ebe9995d2a3c215a6f5083333b407de9d754e
Reviewed-on: https://chromium-review.googlesource.com/303487
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 94747e735e98a57eec2df7288170bf781983d604 |
|
01-Oct-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: shard gles2.functional test.
We would like to run the test in bvt-cq for which each shard needs to
complete in less than one minute. For this we break it into 10 shards.
BUG=chromium:510208
TEST=Ran all control files on link, checked boundaries.
Change-Id: Ibed146fd9b80693b8bd982b7de3a648a62d33ebe
Reviewed-on: https://chromium-review.googlesource.com/303486
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 5d0958d60980f807a19bc8f1f6d6bef39ad3d3b3 |
|
29-Sep-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: handle hasty only failures.
Even when each individual functional test passes it is possible for
failures to happen when batches of these tests are run in hasty mode.
This prevents from dEQP moving into bvt-cq.
This change adds ability to process_logs.py to parse hasty log files
and generate expectations for these. It also changes graphics_dEQP.py
slightly to honor these hasty failures.
In addition there is some refactoring to run failing tests via
'NotPass' control files. Regularly running NotPass'ing tests allows
for monitoring test fixes and updating expectations based on these.
BUG=chromium:510208
TEST=Ran hasty.Fail and NotPass for testing on link.
Change-Id: I12b61619f56774b8ca5ddfe67c3380bc6ba8ad9e
Reviewed-on: https://chromium-review.googlesource.com/302738
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 85262be07cbd30ea60d911bc915cff8f5f81e462 |
|
17-Sep-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: update Intel whitelists.
Update whitelist so we can move dEQP to bvt-cq after sharding.
Also track pinetrail failure explicitly.
BUG=chromium:510207, chromium:510208, chromium:532691
TEST=Ran on link.
Change-Id: I54c471c791b6b087f297f9835bb6429a99c28226
Reviewed-on: https://chromium-review.googlesource.com/300280
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 294dca534a448c835c2405b3bef98efcbef2ae21 |
|
25-Aug-2015 |
HS Liao <hs.liao@mediatek.com> |
autotest (graphics): fix dEQP log parsing
Use ISO-8859-1 encoding to prevent failed XML parsing of dEQP log.
BUG=chrome-os-partner:42991
test_that -b oak --results_dir=./log <IP> f:.*graphics_dEQP/control.gles2.functional.hasty
Change-Id: Ie360d64dc1d638d5a6973fde12887fead959e9e2
Signed-off-by: HS Liao <hs.liao@mediatek.com>
Reviewed-on: https://chromium-review.googlesource.com/295520
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 42d238fec9cd578bcb1039aeacd0ce6e977241e6 |
|
20-Jun-2015 |
Mike Mason <michael.w.mason@intel.com> |
graphics_dEQP: put hasty logs in same directory as non-hasty logs
Now that hasty mode runs tests in batches of 100, many log files
can still be generated, especially for the functional tests. This
clutters up /tmp unnecessarily and makes log file management more
difficult. This patch puts the hasty logs in the same directory
as non-hasty logs, mitigating these issues.
BUG=None
TEST=test_that --args="hasty=True" <dut_addr> graphics_dEQP.gles3.accuracy
The hasty logs should be in /tmp/dEQP-GLES3.accuracy-logs
Change-Id: Iee79e6700b8bff852faf5a207d7ec4c664896670
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/280417
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Commit-Queue: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 3152d7b9e9fc8f6b724bb426b8af68ba81a0077f |
|
20-Jun-2015 |
Mike Mason <michael.w.mason@intel.com> |
graphics_dEQP: pep8 and pylint cleanup
Fix all pep8 and pylint warnings and errors.
BUG=None
TEST=None
Change-Id: I83bf7f061b45ced91ae6110e01a8d863206798e2
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/280488
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Commit-Queue: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 46217b267744c327703be31204664268832d3e71 |
|
16-Jun-2015 |
Mike Mason <michael.w.mason@intel.com> |
graphics_dEQP: add ability to run individual tests
The filter option previously allowed the running of individual tests,
but no longer does. We need this capability, so this patch adds
a test_names option that works with the filter_args control file.
If test_names is given, the filter option and expectation lists are
ignored. When test_names is not given, the filter option and
expectation lists work as before.
A test name can be a full name or just an initial part. If an
initial part is given, all tests that start with that part will
be run. The option also allows multiple test names in the form
of a comma-separated list. Multiple names must be for the same API
(e.g., dEQP-GLES3, dEQP-GLES2, etc.).
BUG=None
TEST= test_that --args="test_names=dEQP-GLES3.info.version" \
<dut_addr> graphics_dEQP.filter_args
test_that --args="test_names=dEQP-GLES3.accuracy,dEQP-GLES3.stress" \
<dut_addr> graphics_dEQP.filter_args
Change-Id: I215bbef8a983884481c2ca9bad1643f943dadc3e
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/278090
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Frank Henigman <fjhenigman@chromium.org>
Commit-Queue: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 6a0b82bdd5f69d093448934ba0db14bcbfbc0e3d |
|
20-May-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: ignore InternalError.
This error can happen when dEQP misestimates times and they become
negative. We should simply discount them.
Also update blacklist/expectations.
BUG=chromium:488659
TEST=None.
Change-Id: Ica1ef9a12d5c268618c67173eec8315ee8d56fe6
Reviewed-on: https://chromium-review.googlesource.com/272307
Commit-Queue: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 76f809985d8082410e695be0a64a0ab9c54e3937 |
|
15-May-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: make hasty mode more robust.
Instead of running all test cases at once, define a batch size of 100
tests run in groups. That way we can recover more often from test
failures due to state leakage (but it isn't really a fix).
Also add clean up spew to make log files a bit nicer.
Add more flaky tests as harvested from wmatrix/gs.
BUG=chromium:461023
TEST=test_that <link_ip> graphics_dEQP.gles2.functional.hasty
Change-Id: I14717a137acd9d16d1552f732db9dbd8d78060f4
Reviewed-on: https://chromium-review.googlesource.com/271571
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 8af26fa68550b4a04bfe80ece52682caec08a9af |
|
01-May-2015 |
Ilja H. Friedel <ihf@chromium.org> |
deqp: rework test to use expectations.
Instead of using blacklists auto generate expectations from previous
runs using process_logs.py. In particular run only tests by default
that have been shown to be 'Pass'ing reliably.
In addition introduce a way to bootstrap incomplete expectations.
This situation can happen when a GPU family is added, test are new
or have been hanging leading to incomplete logs. Long term we may
consider getting rid of the 'Pass'ing expectations and only use
the bootstrapping.
Furthermore introduce a hasty way of running tests which calls dEQP
binary only once with the list of all tests on stdin. This avoid
the overhead of running each test individually and is quite a bit
faster.
Finally add a script to process_logs.py obtained locally or stored
on autotest Google Storage.
BUG=chromium:461023
TEST=Ran some of the sets on link (ivybridge).
Deleted Pass'ing expectations, watched bootstrapping.
Change-Id: I288071d0e3a36db551c7d7981eb0a3f37dc48229
Reviewed-on: https://chromium-review.googlesource.com/268934
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
Reviewed-by: Michael W Mason <michael.w.mason@intel.com>
Tested-by: Michael W Mason <michael.w.mason@intel.com>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| a8fdc75b9bdf8941b485ad8fe7fa71f9da2e89f6 |
|
25-Apr-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: temporary revert blacklist.
We need log files showing all failures to compile new blacklists.
This change will be reverted next week.
BUG=chromium:461023
TEST=None.
Change-Id: I448dfe22a1f1ee31cc7d101ee296470efc1ddaa1
Reviewed-on: https://chromium-review.googlesource.com/267346
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Dominik Behr <dbehr@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 5858f1e5fdbe558519548db3f595a1175d6c3f66 |
|
21-Apr-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: minor changes.
Add logging information on board/GPU/CPU to simplify postprocessing.
Disable egl and gles31 tests.
BUG=chromium:461023
TEST=pylint
check spew of
test_that 100.96.59.64 graphics_dEQP.gles2.info
Change-Id: Iaefe0cf841b454a79b887ed9d6b017fb487d1f7c
Reviewed-on: https://chromium-review.googlesource.com/266606
Reviewed-by: Dominik Behr <dbehr@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| aac0527e311231a914f4ca11b55135c64a257347 |
|
07-Apr-2015 |
Ilja H. Friedel <ihf@chromium.org> |
graphics_dEQP: don't upload log files to GS.
The log files contain interesting debugging output but are too many/large
for permanent storage. This change moves log files to /tmp and avoids upload.
Furthermore remove control.filter_args from graphics SUITE.
BUG=chromium:461023, chromium:473865
TEST=ran twice
test_that 100.96.59.64 graphics_dEQP.gles3.info
and verified the log directory is in /tmp now.
Change-Id: Ic580acdef5e9107d1407e4b6ad1c231ac0b4d517
Reviewed-on: https://chromium-review.googlesource.com/264149
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|
| 870c962b68e8babda0a03802150613af5d9bd01a |
|
09-Mar-2015 |
Mike Mason <michael.w.mason@intel.com> |
[autotest] Add autotest for DrawElements
dEQP (drawElements Quality Program) is an OpenGL ES testsuite added to
AOSP in the Lollipop release. It is being added to Chrome OS.
This patch adds an autotest for dEQP.
dEQP provides test suites for EGL, GLES2, GLES3 and GLES3.1. The suites
are further broken down into info, accuracy, functional, stress and
performance categories. This patch provides control files for each
suite/category, although we currently only run the GLES3 tests.
In addition, the filter_args control file can be used to specify
tests or test sets using the --args option as shown in the TEST
section below. The filter_args control file runs no tests by default.
Some tests are known to be unsupported, timeout, crash the test process
and/or hang the GPU or CPU on Chrome OS, sometimes leading to reboots.
These tests are listed in a blacklist JSON file as regular expressions
that match one or more tests. graphics_dEQP.py reads and parses the
blacklist file and will not run these tests.
Note that some categories can take a long time to run. For example,
in test runs on auron_paine, gles3.functional took ~110 minutes and
gles3.performance took ~270 minutes. By contrast, gles3.info, accuracy
and stress together took ~20 minutes. Times will vary depending on
board, CPU, kernel, etc.
BUG=chromium:461023
TEST= emerge-link deqp;
cros deploy --board=<board> <remote-addr> deqp
test_that -b <board> <remote-addr> graphics_dEQP.gles3.accuracy
test_that -b <board> --args="filter=dEQP-GLES3.info" <remote-addr> \
graphics_dEQP.filter_args
Depends-Change-Id: Ifa07827370a140f776420808d094156809108b7e
Change-Id: I09bd010f31e00f5ad939faf9224ef3243f31e7e2
Reviewed-on: https://chromium-review.googlesource.com/257651
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Michael W Mason <michael.w.mason@intel.com>
Reviewed-by: Chad Versace <chad.versace@intel.com>
Commit-Queue: Chad Versace <chad.versace@intel.com>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
|