History log of /external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
836e0f1633f6e53ccdf65d72da2acff6d4d666f2 14-Jul-2017 Po-Hsien Wang <pwang@chromium.org> graphics_dEQP.py: fix chromeperf failure records

1) Fix number mismatch from wmatrix & chromeperf
2) Use dedicate function to dispatch _run_tests_invidually & run_tests_hasty

BUG=chromium:742599
TEST=test_that cave graphics_dEQP.vk-master
Before: test_that cave graphics_dEQP.vk-master, cat /tmp/test_that_../.../result-charts.json
"Failures_VK": {
"summary": {
"units": "count",
"type": "scalar",
"value": 134115.0,
"improvement_direction": "down"
}
}

After: test_that cave graphics_dEQP.vk-master, cat /tmp/test_that_../.../result-charts.json
"Failures_VK": {
"summary": {
"units": "count",
"type": "scalar",
"value": 56.0,
"improvement_direction": "down"
}
}

Change-Id: I5390c86efd303cbf57c8bab60ec330ea5d2d4b7a
Reviewed-on: https://chromium-review.googlesource.com/571407
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
b2bc3fb92b98414ea2715439a70050cc0395934a 14-Jun-2017 Po-Hsien Wang <pwang@chromium.org> graphics_dEQP: Report of failures to chromeperf

1) Report of failures to chrome perf as gfx regression
alerts.
2) Modify generate_controlfiles so that control.vk-master is generated
correctly.
3) Modify perf_dashboard_config.json

BUG=chromium:717664
TEST=test_that dut graphics_dEQP.gles2-master.hasty.8

Change-Id: I7db5d5b2e521aecc69eca596d41383c78b4e191f
Reviewed-on: https://chromium-review.googlesource.com/534576
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
7c0bccc268fc4a87fa3b586387a6414ee508299b 24-May-2017 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: add a hand selected bvt list.

This runs on minnie on 15 seconds and exercises the gles2, gles3, gles31
and vk APIs (vk of course is skipped on minnie).

BUG=chromium:725656
TEST=test_that <ip> graphics_dEQP.bvt
graphics_dEQP.bvt [ PASSED ]
graphics_dEQP.bvt Pass 24
graphics_dEQP.bvt Skipped 19

Change-Id: Ib2ee6edec432fa9674c170cae72a0ecb40f80781
Reviewed-on: https://chromium-review.googlesource.com/513508
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Chad Versace <chadversary@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
af271023c914bbf0870ae4d3a866ae647f754ed3 10-May-2017 Po-Hsien Wang <pwang@chromium.org> graphics_dEQP.py: rerun failing test after hasty mode

Tests which is indicated as failure in hasty mode will be
re-runed in individual mode as some tests cause problem
to the next following test and make it fail.

BUG=chromium:712299
TEST=test_that veyron_minnie graphics_dEQP.gles2-master.hasty.9
(original: 1/1415, after rerun: 0/1415)

Change-Id: I43fc844ea7c12bb36b3f59ba5f83a81f8b2945fb
Reviewed-on: https://chromium-review.googlesource.com/502333
Commit-Ready: Po-Hsien Wang <pwang@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
ce4ce2601e62cf5d04ed7eb06da7badda2b21c32 10-May-2017 Po-Hsien Wang <pwang@chromium.org> graphics_dEQP.py: return failing tests in results

Add names of failing tests to a list in the results in _parse_test_results.

BUG=chromium:712299
TEST=test_that gles2-master.hasty.9

Change-Id: I5948ff386d932cb090b86e315dd65efda6c2d733
Reviewed-on: https://chromium-review.googlesource.com/502332
Commit-Ready: Po-Hsien Wang <pwang@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
37f61217c3df4e5ef7688f724fa239598786033f 24-Apr-2017 Po-Hsien Wang <pwang@chromium.org> graphics_dEQP: Fix not exist error for vk-master

1) Check test_names_file to see if the API is supported
2) Delete existing error.TestWarn as they might cause trouble in different system
3) Add a GraphicsApiHelper in graphics_utils.py
4) Change vk-master run frequency

BUG=None
TEST=test_that on veyron_minnie

Change-Id: Ied15ad9ab362e8029e085585d66342d0293889ef
Reviewed-on: https://chromium-review.googlesource.com/486161
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
0a37d8effca38bc9c7b36d5fa218f2561e3b24cc 18-Feb-2017 Chad Versace <chadversary@chromium.org> graphics_dEQP: Prepend /usr/local/$libdir to LD_LIBRARY_PATH

Prepend "/usr/local/lib:/usr/local/lib64" to LD_LIBRARY_PATH for all
dEQP test processes. Fixes the Vulkan CTS.

Currently, no package in the base image requires
media-libs/vulkan-loader. Only media-gfx/deqp does, which is in the test
image. Therefore the build_image scripts install libvulkan.so into
/usr/local.

When in a login shell on Chrome OS, LD_LIBRARY_PATH does contain the
/usr/local libdirs. But processes spawned by autotest do not. Therefore,
deqp worked when ran from a login shell; but it failed to dlopen
libvulkan.so when ran by autotest.

BUG=chromium:689177
TEST=Run `test_that ... graphics_dEQP.vk.info` on sentry. Inspect
client.0.DEBUG to confirm that tests ran.

Change-Id: Ib067acc76db1364328b45ef2c66a6a73a4e12c38
Reviewed-on: https://chromium-review.googlesource.com/444984
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Chad Versace <chadversary@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
360a64ab4f40744e63f467409d9252f39516d0d0 03-Feb-2017 Chad Versace <chadversary@chromium.org> graphics_dEQP: Add Vulkan CTS

Recent versions of dEQP include the Vulkan CTS. Add the following
control files for it:
control.vk-master # The mustpass list from the Android CTS
control.vk-master.hasty
control.vk.api.hasty
control.vk.api.smoke
control.vk.binding_model.hasty
control.vk.glsl.hasty
control.vk.info
control.vk.pipeline.hasty
control.vk.spirv_assembly.hasty

To run these tests, media-gfx/deqp must be built with USE=vulkan.

I tested this on reef (which has an Apollo Lake GPU) with
mesa-17.1.0-r3, and all tests passed in suite
'graphics_dEQP.vk-master.hasty'. I also did a smoke test on sentry
(Skylake GPU) with 'graphics_dEQP.vk.api.smoke'.

BUG=chromium:689177
TEST=$ USE=vulkan emerge-sentry deqp
$ cros deploy $SENTRY_IP deqp mesa vulkan-loader
$ test_that --board=sentry $SENTRY_IP graphics_dEQP.vk.api.smoke
# Confirm 6/6 tests pass.

v4: Fix crash when deqp-vk is not installed.
For each dEQP executable (deqp-egl, deqp-vk, deqp-gles3, etc), the
script chdir'd into the executable's directory before testing if the
executable existed. This made Python unhappy by raising an OSError.

Depends-On: I10233e70cdf2303ace300bef39896e98f2d7636b
Change-Id: I854876d100ad6ece737459661ca2a6b2a05e395c
Reviewed-on: https://chromium-review.googlesource.com/438766
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Chad Versace <chadversary@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
aa2187ee300bff08db48bca4c9f29f2673b88fa7 10-Jan-2017 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: add gpu in reporting to wmatrix.

BUG=None.
TEST=pylint

Change-Id: I8e752b119e5cd4691087e94e7db30763c8877838
Reviewed-on: https://chromium-review.googlesource.com/426371
Commit-Queue: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
8245a0b28b82e76d307657856e7165685444acb2 02-Nov-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: enable on tegra.

But don't fail tests (warning only for now) until we have
new expectations.

BUG=chromium:655820
TEST=Ran on nyan_blaze.

Change-Id: I6b6f74be8e5b4e81d4dd6940234bc1bdd67662bc
Reviewed-on: https://chromium-review.googlesource.com/406629
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
619ee2569463359d155539539f5c0d7b2515ccd6 29-Oct-2016 Ilja H. Friedel <ihf@chromium.org> autotest graphics_*: unify TestStatus.

The goal is for graphics tests to only pass or fail. As much as
possible do not use TestError, TestWarn or TestNA etc. Having
only Pass/Fail will make reading on external dashboards (wmatrix,
sponge) less ambiguous. Also prepend all TestFail messages with
"Failed:" to make reading/understanding messages easier.

BUG=chromium:647465
TEST=pylint

Change-Id: I92160c9def35b72e22af8c19af6275932a63d00b
Reviewed-on: https://chromium-review.googlesource.com/404953
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Dominik Behr <dbehr@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
cbe677a4c77a21b1afba39d7d4552e357694c561 07-Aug-2016 Mike Mason <michael.w.mason@intel.com> graphics_dEQP.py: log message format change causes exception

The following change in a previous commit causes an exception when
formatting the message:

- logging.info('Found gles%d.%d.' % (major, minor))
+ logging.info('Found gles%d.%d.', (major, minor))

This patch removes the parenthesis from 'major, minor' so the format
works.

Change-Id: Ic6d942d1aef97b7d06c241c7fc4546806becca21
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/366981
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
b413556fc5d9ccf1dff1d12ee435334a5d06530f 05-Aug-2016 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: count warnings as passes

Android CTS counts QualityWarning and CompatibilityWarning results as
passes when determining the pass/fail summary. The Chrome OS dEQP
autotest does not. This patch aligns the autotest with CTS so both
calculate summary values the same way.

BUG=None.
TEST=test_that $DUT graphics_dEQP.gles31-master
Observe that warnings are now counted as passes in the fail/pass
summary at the end of the test.

Change-Id: I2393ea6070294cecb16cc78e2e20f2cd070f82c8
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/366872
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
6ee34dbe7e01556c4b6e163df63d958e4d64ae70 01-Jul-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: reduce spew to logs.

The dEQP logs can be on the rather large size. Have less spew recorded
for now.

BUG=None.
TEST=test_that $DUT graphics_dEQP.gles3.info

Change-Id: I50e35cca091ceb5865de675b99db8554af996224
Reviewed-on: https://chromium-review.googlesource.com/357952
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Commit-Queue: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
7beb8026cb255c27ecf622a9de5db0b24fe5ffb2 07-Jun-2016 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: Add start/end times to debug logs

Some dEQP tests can leave the system in a state that causes
subsequent tests to fail. Adding start/end times to the
per-test debug logs can help determine the order tests were
run and which test may have caused other tests to fail.

BUG=None.
TEST=Run graphics_dEQP.py with the debug option and observe
start and end times in the debug logs.

Change-Id: Ie73145b6058f53d85631c5834a416ea82d3e03d7
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/350398
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
39334f4feac6f2b6d4afca7e526f1c510ebce940 29-Apr-2016 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: Debug section should be indented one more level

The "if self._debug" section beginning on line 339 should be indented
one more level so it falls under the "else" clause beginning on
line 315. This debug code only applies to tests that are run, not
skipped tests.

BUG=None.
TEST=Tested on two devices that generate skipped tests (gles3.1 on
link and falco) to ensure the debug path is not followed.
Ran same tests on cyan to ensure the debug section is run
and debug files are generated.

Change-Id: Ibb6a3be4b3207594ff591d7f550fa5433fbd593f
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/341332
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
3e16c8c327f7e8d9514ec1dc5150fd017b1d07b2 25-May-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: switch rogue to pbuffers.

BUG=chrome-os-partner:52663
TEST=None.

Change-Id: I11220f99f3221d4d38e6dc297dd6a94281490f41
Reviewed-on: https://chromium-review.googlesource.com/347240
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Mao Huang <littlecvr@google.com>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
42c5d9dc7ddb0fca45bc9d1801246fbaf99fcc12 25-Apr-2016 Gurchetan Singh <gurchetansingh@chromium.org> graphics_dEQP: Run with pbuffer option on all boards
except Rogue.

There are still some changes needed to enable pbuffers
on Rogue (partner bug 52663). Until then, special case
the board.

BUG=chromium:606472
TEST=Ran dEQP on Cyan and Oak, and verified the desired
options were present.

Change-Id: Id22c3e3981182ec71fc75f00086583a0d6e5a8ca
Reviewed-on: https://chromium-review.googlesource.com/340495
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
4f4e936de19af1ed20a5cbf1b5378bab3bd9c4a1 23-Apr-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: switch to rgba8888d24s8ms0.

We should read configs from the xml files, but that will take a bit
more work. For now use rgba8888d24s8ms0 instead of rgba8888d24s8.

BUG=None.
TEST=None.

Change-Id: I000b78f1edec45d827446675bbc898609cf3566d
Reviewed-on: https://chromium-review.googlesource.com/340363
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
9d61cf98e9a613713182934d0ddf62dbe0948e74 12-Apr-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: raise TestNA instead of WARN.

bvt-cq files automatic bugs on warning, which can happen on machines
which for instance don't support gles31.

BUG=chromium:601323
TEST=None.

Change-Id: I7db8703fa4000f8e81bad18a5fbbc3ff15b6bb74
Reviewed-on: https://chromium-review.googlesource.com/338251
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
d82eb680b395b122c2a50a4852a2703373edf784 05-Apr-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: fix string matching.

gles3 is contained in gles31. Avoid this.

BUG=None.
TEST=Ran on link
bin/autotest tests/graphics_dEQP/control.gles2.info
bin/autotest tests/graphics_dEQP/control.gles3.info
bin/autotest tests/graphics_dEQP/control.gles31.info

Change-Id: Iec23c4e0174898734b617d4d1ad347d6f63c2313
Reviewed-on: https://chromium-review.googlesource.com/337055
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Dominik Behr <dbehr@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
e7c97f899c73c271abcd7eafa73120864e5da604 25-Mar-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: add generic way to skip tests.

Use wflinfo to query gles version before running tests.
Refactor graphics_GLAPICheck.

BUG=None.
TEST=Ran graphics_dEQP.gles{2|3|31}.info on alex, daisy and lars.
Ran graphics_GLAPICheck on alex.

Change-Id: Ifaf1e20240152338439c050a4481d2933def9422
Reviewed-on: https://chromium-review.googlesource.com/334872
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
798f59e7dc114377d5915785b8907518476f0761 17-Mar-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: output valid filter.

We don't want to output "filter = None" even when we specify
tests via file as this is used in post processing expectations.

BUG=None.
TEST=test_that 100.96.58.232 graphics_dEQP.gles2-master
03/16 22:31:45.510 INFO | graphics_dEQP:0441| dEQP test filter = gles2-master

Change-Id: Ia87ada8ea50f2197d5e10ea7945e6db96717d174
Reviewed-on: https://chromium-review.googlesource.com/333155
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
029c6beee5bd404b593860396b3339061feb9a11 15-Mar-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: handle pinetrail gracefully.

Instead of listing anything non-gles2 in the expectations, add custom
code to detect pinetrail and skip tests as needed.

BUG=None.
TEST=Ran locally on DUT.

Change-Id: I986b1bdd922ee3f05269d351123c4549f6830506
Reviewed-on: https://chromium-review.googlesource.com/332647
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
61b4832727f526aeb9d90651d702204c2d572835 24-Feb-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: add cts-6.0_r4 master lists.

These are copied from android-cts-6.0_r4/external/deqp/android/cts/master
and define the tests which need to be passed by CTS.

Also refactor graphics_dEQP to execute these master lists.

BUG=None.
TEST=Ran on lars. Checked logs in /tmp/.
test_that --args="filter=dEQP-GLES2.info" 100.96.58.232 graphics_dEQP.filter_args
test_that --args="test_names=dEQP-GLES2.info.extensions,dEQP-GLES2.info.render_target,dEQP-GLES2.info.renderer" 100.96.58.232 graphics_dEQP.filter_args
test_that 100.96.58.232 graphics_dEQP.gles2.functional.hasty.0
test_that 100.96.58.232 graphics_dEQP.gles2.capability
test_that 100.96.58.232 graphics_dEQP.gles2-master

Change-Id: I1a7c7b58d1d5427137d427c5379cb40ac4bc481a
Reviewed-on: https://chromium-review.googlesource.com/329164
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
b74425356a1232f57a603b8307b595845d0aefbe 09-Mar-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: enable pinetrail.

BUG=None.
TEST=sanity run on alex.

Change-Id: I5d5c0cadef49a1b8b5425989b80da66ca57ddea0
Reviewed-on: https://chromium-review.googlesource.com/331764
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
5f6574479479892e960f8a4332f6fae375cb66bf 08-Mar-2016 Ilja H. Friedel <ihf@chromium.org> autotest: refactor ARM/mali utils.

1) Distinguish between different mali -> mali-t604.
2) Remove temperature TODO, it is long fixed.

BUG=None.
TEST=test_that <daisy> graphics_dEQP.gles2.performance graphics_GLBench.bvt-cq graphics_Idle

Change-Id: I47568cb3369ab213592855430a647fe6366374c1
Reviewed-on: https://chromium-review.googlesource.com/331334
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
cd1caeda9f5fbf3b06ae6b24ba5bec5ed672280a 04-Mar-2016 Stéphane Marchesin <marcheu@chromium.org> deqp: Use RGBA8888 / D24S8 format by default

The surfaceless deqp backend which we use in Chrome OS doesn't specify
a pixel format and runs using an fbo which means that it lets deqp
pick the format.

The format picked for depth is the first in the list of depth formats,
which is GL_DEPTH32F_STENCIL8. Sadly deqp doesn't check that this
format is present (in particular, it is not a requirement for GLES2
implementations, and is only required for GLES3). This means that the
driver rejects the format (rightfully so) and in turn deqp does not
run on these GLES2 stacks.

This patch changes the test to pass -deqp-gl-config-name=rgba8888d24s8
to deqp which forces the use of a D42S8 format, which all GLES
implementations support.

Note that the backends for all the other platforms (android, linux...)
use this format by default already, so by changing this we are making
our deqp testing more comparable with other platforms.

BUG=chromium:532691
TEST=run on alex

Change-Id: I26e432ec7351f636cfe16c265d13de02cd800979
Reviewed-on: https://chromium-review.googlesource.com/330464
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
de00a62a7f9bf266504ca72d8a030880d6928d8f 01-Mar-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: run on mali.

BUG=chromium:543372
TEST=None.

Change-Id: I750d63def76eabda754cc810457086b2ea8299c2
Reviewed-on: https://chromium-review.googlesource.com/329823
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
40196cf868b79919b5d2de58771ceb9b8715adc9 15-Jan-2016 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: add debug option

It is sometimes useful to save additional information about what
happened during a test. Some of this data is currently collected
by autotest, but it is difficult to associate it with a specific
test and to correlate data from multiple sources.

This patch adds a debug option to collect dmesg, stdout and stderr
data for each test. This data is written in JSON format to
<test-name>.debug in the same directory as the individual dEQP
test log files on the DUT.

BUG=None
TEST=test_that --args="debug=True filter=dEQP-GLES3.info" <ip> graphics_dEQP.filter_args
Look for <test-name>.debug files in /tmp/dEQP-GLES3.info-logs
on the DUT.

Change-Id: Icd6d78ac369d288f89f9fb69dcaa60b8e3278c35
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/322225
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
8b2e8bd3ecc4c8ad6aff47a2ea56029f069190b1 17-Feb-2016 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: don't warn if NotPass is empty.

This warning causes too manual checking on wmatrix and has little
value.

BUG=None.
TEST=Ran locally.

Change-Id: Ibc0caafe09548de5bc0568e62e775c621d4c21bf
Reviewed-on: https://chromium-review.googlesource.com/327705
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
939f57efdd21c6dd47f92f6b8b6f2221c6911595 14-Jan-2016 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: hasty expectation lists used in non-hasty mode

The dEQP autotest mistakenly used the hasty exception lists to exclude
tests even in non-hasty mode, which resulted in some tests not being
run. This was caused by the wrong string being used to filter out
hasty file names in non-hasty mode. This patch fixes that string.

BUG=49397
TEST=test_that <ip_address> graphics_dEQP.gles3.functional
This should run ~41.5K tests, but instead only runs ~22K tests.
The 'dEQP-GLES3.functional.hasty.Flaky' expectation contains ~19K tests.

Change-Id: Ic95aa779428df545b6ee3637a27e45f42af9e029
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/321866
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
0facf6fd666d27bee7965759d71fd47d8a62d46c 09-Dec-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: add control files.

We would like to run tests that fail on a daily schedule.
Run gles3.functional in hasty/sharded mode in preparation for
moving it to bvt-cq one day.

All these files are now created using generate_controlfiles.py.

BUG=chromium:556747
TEST=test_that <samus> graphics_dEQP.gles2.accuracy graphics_dEQP.gles3.accuracy graphics_dEQP.gles2.info graphics_dEQP.gles3.info

Change-Id: I34de4f309d2cb433c1f344a3b43fadc39d32307d
Reviewed-on: https://chromium-review.googlesource.com/317032
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
3d4a32186c4f856be868eaa815c4bfb4e2167b83 21-Oct-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: warn instead of error.

When we run tests that are known not to be passing they could be
flaky or the failure can depend on previously ran tests. For this
reason just warn if a test that we thought should be failing is
now passing. It does not require immediate attention.

BUG=None.
TEST=None.

Change-Id: I18524da875aef2fee3fe40dd49dd0997cbf4a9c9
Reviewed-on: https://chromium-review.googlesource.com/307924
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
20d15b2e2566840876c1a9f35051403249a98a3a 15-Oct-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: unsupported GPU -> TestNAError.

We want to keep track of these GPUs, in particular in preparation of
moving graphics_dEQP into bvt-cq. And DEPENDENCIES don't work as
they should. (Might still be worth adding DEPENDENCIES in addition to
avoid lab overhead though.)

BUG=chromium:510208
TEST=None.

Change-Id: I6ab344b17e7da103311e384c3e57b98e365f0179
Reviewed-on: https://chromium-review.googlesource.com/305715
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
dd11bd5cf2ae753fe9d229fc37dfb9f72135ffe8 01-Oct-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: change NotPass to error on pass.

And output how many tests are passing now. This should make reading the
dashboard easier.

BUG=chromium:510208
TEST=Ran on pixel.

Change-Id: I9f7ebe9995d2a3c215a6f5083333b407de9d754e
Reviewed-on: https://chromium-review.googlesource.com/303487
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
94747e735e98a57eec2df7288170bf781983d604 01-Oct-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: shard gles2.functional test.

We would like to run the test in bvt-cq for which each shard needs to
complete in less than one minute. For this we break it into 10 shards.

BUG=chromium:510208
TEST=Ran all control files on link, checked boundaries.

Change-Id: Ibed146fd9b80693b8bd982b7de3a648a62d33ebe
Reviewed-on: https://chromium-review.googlesource.com/303486
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
5d0958d60980f807a19bc8f1f6d6bef39ad3d3b3 29-Sep-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: handle hasty only failures.

Even when each individual functional test passes it is possible for
failures to happen when batches of these tests are run in hasty mode.
This prevents from dEQP moving into bvt-cq.

This change adds ability to process_logs.py to parse hasty log files
and generate expectations for these. It also changes graphics_dEQP.py
slightly to honor these hasty failures.

In addition there is some refactoring to run failing tests via
'NotPass' control files. Regularly running NotPass'ing tests allows
for monitoring test fixes and updating expectations based on these.

BUG=chromium:510208
TEST=Ran hasty.Fail and NotPass for testing on link.

Change-Id: I12b61619f56774b8ca5ddfe67c3380bc6ba8ad9e
Reviewed-on: https://chromium-review.googlesource.com/302738
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
85262be07cbd30ea60d911bc915cff8f5f81e462 17-Sep-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: update Intel whitelists.

Update whitelist so we can move dEQP to bvt-cq after sharding.
Also track pinetrail failure explicitly.

BUG=chromium:510207, chromium:510208, chromium:532691
TEST=Ran on link.

Change-Id: I54c471c791b6b087f297f9835bb6429a99c28226
Reviewed-on: https://chromium-review.googlesource.com/300280
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
294dca534a448c835c2405b3bef98efcbef2ae21 25-Aug-2015 HS Liao <hs.liao@mediatek.com> autotest (graphics): fix dEQP log parsing

Use ISO-8859-1 encoding to prevent failed XML parsing of dEQP log.

BUG=chrome-os-partner:42991
test_that -b oak --results_dir=./log <IP> f:.*graphics_dEQP/control.gles2.functional.hasty

Change-Id: Ie360d64dc1d638d5a6973fde12887fead959e9e2
Signed-off-by: HS Liao <hs.liao@mediatek.com>
Reviewed-on: https://chromium-review.googlesource.com/295520
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
42d238fec9cd578bcb1039aeacd0ce6e977241e6 20-Jun-2015 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: put hasty logs in same directory as non-hasty logs

Now that hasty mode runs tests in batches of 100, many log files
can still be generated, especially for the functional tests. This
clutters up /tmp unnecessarily and makes log file management more
difficult. This patch puts the hasty logs in the same directory
as non-hasty logs, mitigating these issues.

BUG=None
TEST=test_that --args="hasty=True" <dut_addr> graphics_dEQP.gles3.accuracy
The hasty logs should be in /tmp/dEQP-GLES3.accuracy-logs

Change-Id: Iee79e6700b8bff852faf5a207d7ec4c664896670
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/280417
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Commit-Queue: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
3152d7b9e9fc8f6b724bb426b8af68ba81a0077f 20-Jun-2015 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: pep8 and pylint cleanup

Fix all pep8 and pylint warnings and errors.

BUG=None
TEST=None

Change-Id: I83bf7f061b45ced91ae6110e01a8d863206798e2
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/280488
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Commit-Queue: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
46217b267744c327703be31204664268832d3e71 16-Jun-2015 Mike Mason <michael.w.mason@intel.com> graphics_dEQP: add ability to run individual tests

The filter option previously allowed the running of individual tests,
but no longer does. We need this capability, so this patch adds
a test_names option that works with the filter_args control file.
If test_names is given, the filter option and expectation lists are
ignored. When test_names is not given, the filter option and
expectation lists work as before.

A test name can be a full name or just an initial part. If an
initial part is given, all tests that start with that part will
be run. The option also allows multiple test names in the form
of a comma-separated list. Multiple names must be for the same API
(e.g., dEQP-GLES3, dEQP-GLES2, etc.).

BUG=None
TEST= test_that --args="test_names=dEQP-GLES3.info.version" \
<dut_addr> graphics_dEQP.filter_args
test_that --args="test_names=dEQP-GLES3.accuracy,dEQP-GLES3.stress" \
<dut_addr> graphics_dEQP.filter_args

Change-Id: I215bbef8a983884481c2ca9bad1643f943dadc3e
Signed-off-by: Mike Mason <michael.w.mason@intel.com>
Reviewed-on: https://chromium-review.googlesource.com/278090
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Frank Henigman <fjhenigman@chromium.org>
Commit-Queue: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
6a0b82bdd5f69d093448934ba0db14bcbfbc0e3d 20-May-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: ignore InternalError.

This error can happen when dEQP misestimates times and they become
negative. We should simply discount them.

Also update blacklist/expectations.

BUG=chromium:488659
TEST=None.

Change-Id: Ica1ef9a12d5c268618c67173eec8315ee8d56fe6
Reviewed-on: https://chromium-review.googlesource.com/272307
Commit-Queue: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
76f809985d8082410e695be0a64a0ab9c54e3937 15-May-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: make hasty mode more robust.

Instead of running all test cases at once, define a batch size of 100
tests run in groups. That way we can recover more often from test
failures due to state leakage (but it isn't really a fix).

Also add clean up spew to make log files a bit nicer.

Add more flaky tests as harvested from wmatrix/gs.

BUG=chromium:461023
TEST=test_that <link_ip> graphics_dEQP.gles2.functional.hasty

Change-Id: I14717a137acd9d16d1552f732db9dbd8d78060f4
Reviewed-on: https://chromium-review.googlesource.com/271571
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
8af26fa68550b4a04bfe80ece52682caec08a9af 01-May-2015 Ilja H. Friedel <ihf@chromium.org> deqp: rework test to use expectations.

Instead of using blacklists auto generate expectations from previous
runs using process_logs.py. In particular run only tests by default
that have been shown to be 'Pass'ing reliably.

In addition introduce a way to bootstrap incomplete expectations.
This situation can happen when a GPU family is added, test are new
or have been hanging leading to incomplete logs. Long term we may
consider getting rid of the 'Pass'ing expectations and only use
the bootstrapping.

Furthermore introduce a hasty way of running tests which calls dEQP
binary only once with the list of all tests on stdin. This avoid
the overhead of running each test individually and is quite a bit
faster.

Finally add a script to process_logs.py obtained locally or stored
on autotest Google Storage.

BUG=chromium:461023
TEST=Ran some of the sets on link (ivybridge).
Deleted Pass'ing expectations, watched bootstrapping.

Change-Id: I288071d0e3a36db551c7d7981eb0a3f37dc48229
Reviewed-on: https://chromium-review.googlesource.com/268934
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
Reviewed-by: Michael W Mason <michael.w.mason@intel.com>
Tested-by: Michael W Mason <michael.w.mason@intel.com>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
a8fdc75b9bdf8941b485ad8fe7fa71f9da2e89f6 25-Apr-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: temporary revert blacklist.

We need log files showing all failures to compile new blacklists.
This change will be reverted next week.

BUG=chromium:461023
TEST=None.

Change-Id: I448dfe22a1f1ee31cc7d101ee296470efc1ddaa1
Reviewed-on: https://chromium-review.googlesource.com/267346
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Dominik Behr <dbehr@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
5858f1e5fdbe558519548db3f595a1175d6c3f66 21-Apr-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: minor changes.

Add logging information on board/GPU/CPU to simplify postprocessing.
Disable egl and gles31 tests.

BUG=chromium:461023
TEST=pylint
check spew of
test_that 100.96.59.64 graphics_dEQP.gles2.info
Change-Id: Iaefe0cf841b454a79b887ed9d6b017fb487d1f7c
Reviewed-on: https://chromium-review.googlesource.com/266606
Reviewed-by: Dominik Behr <dbehr@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
aac0527e311231a914f4ca11b55135c64a257347 07-Apr-2015 Ilja H. Friedel <ihf@chromium.org> graphics_dEQP: don't upload log files to GS.

The log files contain interesting debugging output but are too many/large
for permanent storage. This change moves log files to /tmp and avoids upload.

Furthermore remove control.filter_args from graphics SUITE.

BUG=chromium:461023, chromium:473865
TEST=ran twice
test_that 100.96.59.64 graphics_dEQP.gles3.info
and verified the log directory is in /tmp now.

Change-Id: Ic580acdef5e9107d1407e4b6ad1c231ac0b4d517
Reviewed-on: https://chromium-review.googlesource.com/264149
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py
870c962b68e8babda0a03802150613af5d9bd01a 09-Mar-2015 Mike Mason <michael.w.mason@intel.com> [autotest] Add autotest for DrawElements

dEQP (drawElements Quality Program) is an OpenGL ES testsuite added to
AOSP in the Lollipop release. It is being added to Chrome OS.
This patch adds an autotest for dEQP.

dEQP provides test suites for EGL, GLES2, GLES3 and GLES3.1. The suites
are further broken down into info, accuracy, functional, stress and
performance categories. This patch provides control files for each
suite/category, although we currently only run the GLES3 tests.
In addition, the filter_args control file can be used to specify
tests or test sets using the --args option as shown in the TEST
section below. The filter_args control file runs no tests by default.

Some tests are known to be unsupported, timeout, crash the test process
and/or hang the GPU or CPU on Chrome OS, sometimes leading to reboots.
These tests are listed in a blacklist JSON file as regular expressions
that match one or more tests. graphics_dEQP.py reads and parses the
blacklist file and will not run these tests.

Note that some categories can take a long time to run. For example,
in test runs on auron_paine, gles3.functional took ~110 minutes and
gles3.performance took ~270 minutes. By contrast, gles3.info, accuracy
and stress together took ~20 minutes. Times will vary depending on
board, CPU, kernel, etc.

BUG=chromium:461023
TEST= emerge-link deqp;
cros deploy --board=<board> <remote-addr> deqp
test_that -b <board> <remote-addr> graphics_dEQP.gles3.accuracy
test_that -b <board> --args="filter=dEQP-GLES3.info" <remote-addr> \
graphics_dEQP.filter_args

Depends-Change-Id: Ifa07827370a140f776420808d094156809108b7e
Change-Id: I09bd010f31e00f5ad939faf9224ef3243f31e7e2
Reviewed-on: https://chromium-review.googlesource.com/257651
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Michael W Mason <michael.w.mason@intel.com>
Reviewed-by: Chad Versace <chad.versace@intel.com>
Commit-Queue: Chad Versace <chad.versace@intel.com>
/external/autotest/client/site_tests/graphics_dEQP/graphics_dEQP.py