| 60a345dd2a6a7c27407153b6c9734d2b63a88c01 |
|
28-Jul-2014 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add MtbSanityValidator
This patch checks two things.
1. Ghost fingers:
The validator checks if there exist ghost fingers before fingers
touch the touch device. This is accomplished by querying the
InputDevice class which syncs with the kernel evdev driver about
the mt slots information.
2. MTB event rules:
The MTB events emitted by a device have to follow some rules.
Otherwise, the events may confuse the state machine in the MTB
parser. As a beginning, this validator checks some simple rules:
a. No TRACKING ID is set to -1 before a slot is assigned a
positive TRACKING ID value.
b. No X, Y, or PRESSURE values could be assigned to a slot
before the slot is assigned a positive TRACKING ID value.
More rules may be added to this validator when needed.
Note that item 1 above is to detect any pre-existing fingers
before the test is conducted, while item 2 is to detect any event
mis-ordering or corruption occurring during the test.
BUG=chrome-os-partner:30965
TEST=All unit tests should pass on a chromebook.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ python tests/run_all_unittests.py
Change-Id: Id85dcb8e54601f5dfd6d77bbf3802fdcb2d74a60
Reviewed-on: https://chromium-review.googlesource.com/210792
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| e3241a0d8761dd21e6ca70ab53e52feea915af49 |
|
28-Jul-2014 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: alphabetic order of raw metrics props by validator names
Reorder the raw metrics properties by validator names.
BUG=chrome-os-partner:30965
TEST=None
Change-Id: Ie2a942f0b07db78c95ad9188de90061779bd5ab1
Reviewed-on: https://chromium-review.googlesource.com/210629
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 3ceb6419c90576ba7df8cee03d4b42d38d1b6d00 |
|
05-Mar-2014 |
Charlie Mooney <charliemooney@chromium.org> |
Adding a drag-latency test to the Touch FW test
A drag latency test has been part of our touch qualification process for
a while now, but has always been run by hand. With Quickstep, this
should be able to be performed automatically as with any other
gesture/validator that we check. The operator will just need to set up
the laser across the pad before running the test.
This CL adds in the robot support for the Quickstep gesture and a new
validator that imports in the latency calculation software from the
touchbot project and computes the drag-latency using the touch data the
firmware collected and pulling the laser data out of the Quickstep sysfs
entries. If there is no Quickstep present, the test will report a
failing dummy value to indicate this, otherwise you would need the
Quickstep device to run the FW test, which is something we don't want to
be a requirement.
BUG=chromium:349174
TEST=Manually ran the test with and without the Quickstep plugged in and
the gesture ran smoothly and the computations worked without crashing
Change-Id: I638da3f837e86d6cf40da3e552c7d56b23e51ee1
Signed-off-by: Charlie Mooney <charliemooney@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/189502
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| bc9dba0f212df1b31ed43032c68f5e07e9ee9974 |
|
01-Mar-2014 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add version info to test logs
When recording gestures files or generating the test summary,
we would like the version info to be prepended to the test logs,
including the standard output, the html report, the pickle log,
and the summary report.
BUG=chromium:315787
TEST=Install the test on a chromebook machine as a first step.
Then, in the chroot of your host:
(cr) $ cd .../site_tests/firmware_TouchMTB
(cr) $ ./version.sh -r ${MACHINE_IP}
On the chromebook, record a few gestures, and then press 'x' to exit.
$ DISPLAY=:0 python main.py -m complete
Now, check the version info in the following files:
$ cd /var/tmp/touch_firmware_test/latest
$ less touch_firmware_report-xxxx
$ less touch_firmware_report-xxxx.html
$ less touch_firmware_report-xxxx.log
The version info should show up in the first few lines look like
test_version: 2014-02-28.06:46:04 5db2d13bf7ff0e56c652213434f581ce6d7a65ee
Change-Id: I549d97574dd44d2cdd7eea7efe0de85a5fd67f66
Reviewed-on: https://chromium-review.googlesource.com/188562
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 4c15c4c2b857b92d9fdc85ee8ccf20e537f0df88 |
|
03-Jan-2014 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: remove spec from descriptions
We have a fw test spec defined here:
https://sites.google.com/a/google.com/chromeos-partners/pages/tech-docs/hardware/touchpad-specification
That should be the one source of truth to avoid having to maintain
a spec in two different places.
BUG=chromium:327448
TEST=Replay logs and generate a summary file.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/link
$ ./summary.sh -d /tmp
Open the summary file, /tmp/summary/summary_xxxx/summary_xxxx.txt.
In the summary table, it shows something like
DrumrollValidator
circle radius (mm) : 0.11 max radius of enclosing circles of tapping points
ReportRateValidator
average time interval (ms) : 6.12 the average of report intervals
max time interval (ms) : 14.48 the max report interval
pct of large intervals (%) : 0.0 pct of intervals larger than expected
Change-Id: I2e79ff29aabeec91cc3a97178e4a5daebb17f277
Reviewed-on: https://chromium-review.googlesource.com/181460
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 7584b3410e81bd7ea282432c2c3c55d9b4a06ea3 |
|
19-Dec-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: display summary table and detailed table
The summary table is used for the spec, and the detailed table for
tracing the root cause or for debugging.
As an example, the top level summary table will show something like
StationaryFingerValidator, StationaryTapValidator
max distance (mm) : 1.45 max distance of any two points from any run
On the other hand, the detailed table will display metrics of the two
related validators separately. The StationaryFingerValidator is used
to measure the pulling effect by another finger on a stationary
finger, while the StationaryTapValidator is used to measure the
wobble of taps/clicks.
StationaryFingerValidator
max distance (mm) : 1.45 max distance of any two points from any run
StationaryTapValidator
max distance (mm) : 0.54 max distance of any two points from any run
Since there may exist distinct root causes of failures about both
validators, the detailed table is useful in showing them separately.
BUG=chromium:330108
TEST=Replay logs and generate a summary file.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/link/
$ ./summary.sh -d /tmp
Open the following summary file, /tmp/summary/summary_xxxx/summary_xxxx.txt
while xxxx represents timestamps. It shows two tables in the summary file.
The first table is "Summary table" which shows something like
...
StationaryFingerValidator StationaryTapValidator
max distance (mm) : 1.45 max distance of any two points
...
The second table is "Detailed table (for debugging)" which shows the
two related validators separately.
...
StationaryFingerValidator
max distance (mm) : 1.45 max distance of any two points
StationaryTapValidator
max distance (mm) : 0.54 max distance of any two points
...
Change-Id: Ib478b39c28c23e49d824be41c57650915ccb569a
Reviewed-on: https://chromium-review.googlesource.com/181094
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| a44a9531f8dbd9c61d1f940397a816d628d4b369 |
|
29-Nov-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add hysteresis validator and its related gesture
A new HysteresisValidator is added to check if there exists a
cursor jump initially. This happens when a finger starts moving,
the cursor keeps at the same position initially. While the finger
moves sufficiently far away, the cursor jumps suddenly to catch
up with the finger and then moves smoothly with the finger thereafter.
A new gesture for performing hysteresis test is also added to draw
lines to various directions with distinct speed.
Since this is a regular line-drawing gesture, the robot is ready
to perform it.
BUG=chromium:326658
TEST=All unit tests, including the new one for hysteresis test,
should pass.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ python tests/run_all_unittests.py
Change-Id: Ib3dcd5fecb87e946e2e2b3665ddd0823d26f07a3
Reviewed-on: https://chromium-review.googlesource.com/179316
Reviewed-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| d1603ecc4264a1c789e0de7e2c584ab83235a068 |
|
21-Nov-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: log details for metrics instead of scores
All the old scores are replaced with the new metrics statistics
in the html report file.
As some of the metrics may not need pass criteria, e.g., the
Linearity Validator, the criteria are only displayed when needed.
BUG=chromium:316291
TEST= Replay the logs:
$ tools/machine_replay.sh -b tests/logs/link
Observe the scores are all replaced with the new metrics statistics
in the html report.
Change-Id: I17c0c7e431544ef30b4bdf1655a204ba3ea8f7a4
Reviewed-on: https://chromium-review.googlesource.com/177741
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| c9cdc9527e1087addd1e21cf58608fd75173df5a |
|
25-Sep-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: show scores and statistics of individual round
The firmware_summary module used to integrate all test data of
the same firmware from different rounds to calculate the statistics.
However, it would be handy to be able to calculate and display
the statistics of individual rounds of the same firmware. As an
example, we may like to examine the consistency of the test results
of the same firmware across different rounds.
We add a new command line option such that when present it treats
every rounds of the same firmware as different ones by appending
a round number to the firmware name.
BUG=chromium:235753
TEST=Replay the lumpy test logs on a chromebook.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Show the scores and metrics of individual round with the
command line option "-i"
$ ./summary.sh -d /tmp -m f -i
Look at the summary report. It displays the scores and metrics
of two individual rounds for fw_11.23, and also two individual
rounds for fw_11.27.
Run all unit tests, and there should be no division by 0 errors.
$ python tests/run_all_unittests.py
Change-Id: I80a657ee35e3952fa4d205f4e19724d8718e416c
Reviewed-on: https://chromium-review.googlesource.com/170592
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 1f0addaf98e671f538602e518ac470f199d298c7 |
|
25-Sep-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: fix division by 0 error in _missed_pct
The _missed_pct lambda function is used compute the missed
percentage of
'(clicks with correct finger IDs, actual clicks)'
In some cases when the number of actual clicks is 0, there are no
missed finger IDs. So just return 0 for this special case to prevent
the devision by 0 error.
BUG=chromium:235753
TEST=None. To be tested together with the next patch.
Change-Id: I97bec0678eac5a6837fe2fa2d4bb0aa35d19446f
Reviewed-on: https://chromium-review.googlesource.com/170591
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 2bef8f1757544a8f9f302d6a30e80c16fc940d18 |
|
22-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: fix descriptions in the summary report
Make summary report more descriptive.
Metrics statistics: fw_11.27 description (lower is better)
---------------------------------------------------------------------
CountPacketsValidator
pct of incorrect cases (%): 0.0 an incorrect case is one where a swipe
has less than 3 packets reported
CountTrackingIDValidator
pct of incorrect cases (%): 6.90 an incorrect case is one where there are
an incorrect number of fingers detected
BUG=chromium:235753
TEST=Replay the logs on link. Observe that the descriptions
are shown like above:
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/lumpy -s
$ ./summary.sh -d /tmp -m f
Change-Id: I6a0fcba8fe2b98f5bd3046e86f45843a2268b1d3
Reviewed-on: https://chromium-review.googlesource.com/66632
Reviewed-by: Andrew de los Reyes <adlr@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 703856c2653c371ddb1196646cdfd85a3a7b1704 |
|
20-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: calculate the final weighted average with missing validators
It is possible that some gestures have been recorded for a firmware
but are missing in the tests for another firmware. The summary module
should be able to handle this situation when calculating the final
weighted average.
Since each validator has an associated weight when calculating the
final weighted average, every firmware maintains its own list of
validators called fw_validators and their corresponding weights.
BUG=chromium:235753
TEST=Replay test logs on a link as follows:
Make a temporary directory.
$ mkdir -p /var/tmp/data
Change directory to the test suite.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Copy the link test data.
$ cp -r tests/logs/lumpy/20130506_030025-fw_11.27-robot_sim/ /var/tmp/data/
Copy the lumpy test data.
$ cp -r tests/logs/link/20130806_221321-fw_1.0.AA-robot/ /var/tmp/data/
Now replay those logs.
$ tools/machine_replay.sh -b /var/tmp/data -s
Generate a summary report.
$ ./summary.sh -d /tmp -m f
Observe the following:
(1) Some scores data are blank in "Test Summary (by gesture)"
(2) Some scores data are blank in "Test Summary (by validator)"
(3) Some metrics data are blank in "Metrics statistics by validator"
(4) It shows the final weighted averages at the end of the report as:
Test Summary (final weighted averages)
--------------------------------------
fw_1.0.AA: 0.910
fw_11.27: 0.839
Change-Id: Ic9c3b5c2a97a7b7836bfddc883a55940f701fe74
Reviewed-on: https://chromium-review.googlesource.com/66819
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| b6bba8d9db954b5adbfb2eb60a220948028b4b6f |
|
26-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric "clicks without finger IDs"
In this patch, we add a new metric to PhysicalClickValidator
"of the n clicks, the % of clicks without a finger ID".
In particular, we check whether the BTN_LEFT event occurs when
the number of tracking (finger) IDs is correct. For example,
if the BTN_LEFT events with values 1 and 0 occur after a finger
has left, it is a failure. The event sequence in such a failure
case may look like
Event: ABS_MT_TRACKING_ID, value 28
Event: ABS_MT_TRACKING_ID, value -1
Event: BTN_LEFT, value 1
Event: BTN_LEFT, value 0
On the other hand, a click case with the correct number of tracking
IDs may look like
Event: ABS_MT_TRACKING_ID, value 28
Event: BTN_LEFT, value 1
Event: ABS_MT_TRACKING_ID, value -1
Event: BTN_LEFT, value 0
BUG=chromium:251193
TEST=Replay logs on a lumpy.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/lumpy/ -s
And then run the summary script to generate a summary report.
$ ./summary.sh -d /tmp -m f
Observe the statistics of the metrics would look like
PhysicalClickValidator
1f-click miss rate (%) : 12.50 the pct of finger IDs w/o a click
1f-click w/o finger IDs (%): 0.00 the pct of clicks w/o correct finger IDs
2f-click miss rate (%) : 75.00 the pct of finger IDs w/o a click
2f-click w/o finger IDs (%): 0.00 the pct of clicks w/o correct finger IDs
The raw metrics values look like
PhysicalClickValidator
1f-click miss rate (%)
** Note: value below represents (actual clicks, expected clicks)
one_finger_physical_click.bottom_side (20130506_032458) : (1, 1)
...
1f-click w/o finger IDs (%)
** Note: value below represents
(clicks with correct finger IDs, actual clicks)
one_finger_physical_click.bottom_left (20130506_032458) : (1, 1)
...
Also all of the unit tests should pass.
$ python tests/run_all_unittests.py
Change-Id: I26b7e7dd6da4456f7b2fae8da7aacdfb5e69b688
Reviewed-on: https://gerrit.chromium.org/gerrit/63588
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| aa7b540459960d887cd7520f4d6a963a33cc7a85 |
|
26-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for PinchValidator
Display the metric for PinchValidator if there are failures.
Otherwise, hide the metric unless a flag "-m a" is specified. The
metric shows the percentage of the incorrect test cases over the total
test cases.
The printing format with failures looks like:
Metrics statistics by validator: fw_12 description
----------------------------------------------------------------------------
PinchValidator
pct of incorrect cases (%) : 12.50 pct of incorrect cases over total cases
The detailed raw metrics values will be printed if a flag "-m f"
is specified. The raw metric value pair represents
(actual relative motion (px), expected relative motion (px))
The format looks like:
PinchValidator
pct of incorrect cases (%)
** Note: value below represents
(actual relative motion (px), expected relative motion (px))
pinch_to_zoom.zoom_in (20130506_032458) : (646.52, 200)
pinch_to_zoom.zoom_out (20130506_032458) : (636.80, 200)
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs.
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with all metrics statistics.
$ ./summary.sh -d /tmp -m a
Generate the summary report with the full raw metrics values on file basis.
$ ./summary.sh -d /tmp -m f
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: I57b867f2d092de2245de2244d50e71375a8e1b40
Reviewed-on: https://gerrit.chromium.org/gerrit/63449
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 8047eb40635b781cdbfd3fa2b33f423b5ff724e1 |
|
25-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for CountPacketsValidator
Display the metric for CountPacketsValidator if there are failures.
Otherwise, hide the metric unless a flag "-m a" is specified. The
metric shows the percentage of the incorrect test cases over the total
test cases.
The printing format with failures looks like:
Metrics statistics by validator: fw_12 description
----------------------------------------------------------------------------
CountPacketsValidator
pct of incorrect cases (%) : 25.00 pct of incorrect cases over total cases
The detailed raw metrics values will be printed if a flag "-m f"
is specified. The raw metric value pair represents
(actual number of packets, expected number of packets)
The format looks like:
CountPacketsValidator
pct of incorrect cases (%)
** Note: value means (actual number of packets, expected number of packets)
one_finger_swipe.bottom_left_to_top_right (20130506_032458) : (2, 3)
one_finger_swipe.bottom_left_to_top_right (20130506_032659) : (4, 3)
We also simplify the metric notes by moving them from firmware_summary
module into raw_metrics_props in the firmware_log module so that the
properties of metrics are maintained in the same structure.
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs.
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with all metrics statistics.
$ ./summary.sh -d /tmp -m a
Generate the summary report with the full raw metrics values on file basis.
$ ./summary.sh -d /tmp -m f
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: Ib3cefee9148167bd4c8d099518f2505d4673c839
Reviewed-on: https://gerrit.chromium.org/gerrit/63332
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 53ee1405e161212a51fcdda64076a1c19ef0b1cb |
|
02-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: import numpy as np
This simple patch just fixes the coding style by using the alias np
to replace n.
BUG=chromium:235753
TEST=None
Change-Id: I8c28907d657f21084175796457f57af278f9726d
Reviewed-on: https://gerrit.chromium.org/gerrit/64317
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 7dbae7827dae0767bffe303708e19483b80a007e |
|
17-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for CountTrackingIDValidator
The metrics of some validators will be hidden by default unless
there are failures. Such validators include CountTrackingIDValidator,
CountPacketsValidator, and PinchValidator.
Display the metric for CountTrackingIDValidator if there are failures.
Otherwise, hide the metric unless a flag "-m a" is specified. The
metric shows the percentage of the incorrect test cases over the total
test cases.
The printing format with failures looks like:
Metrics statistics by validator: fw_12 description
------------------------------------------------------------------------------
CountTrackingIDValidator
pct of incorrect cases (%) : 4.76 pct of incorrect cases over total cases
...
The detailed raw metrics values will be printed if a flag "-m f"
is specified. The raw metric value pair represents
(actual tracking IDs, expected tracking IDs). The format looks like:
CountTrackingIDValidator
pct of incorrect cases (%)
** Note: value below represents (actual tracking IDs, expected tracking IDs)
drag_edge_thumb.left_to_right (20130506_032458) : (5, 1)
one_finger_physical_click.bottom_left (20130506_032458): (1, 1)
...
Note that in the drag_edge_thumb gesture above, the expected number
of tracking IDS (or finger IDs) should be 1. However, 5 tracking IDs
are actually observed due to some firmware problems.
For the one_finger_physical_click gesture, exactly 1 tracking ID is
observed, while the number of expected finger IDs is 1.
We also unify the "-m" (i.e., metrics) flag in summary.sh script.
$ ./summary.sh -m p # show the statistics of the primary metrics
$ ./summary.sh -m a # show the statistics of all metrics, where
# all metrics = the primary metrics + the hidden ones
$ ./summary.sh -m f # show the full raw metrics values on file basis
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs.
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with the metrics statistics.
$ ./summary.sh -d /tmp -m a
Generate the summary report with the full raw metrics values on file basis.
$ ./summary.sh -d /tmp -m f
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: Ie385bfd54bac0a45211fc2a678ad57d9472e4da2
Reviewed-on: https://gerrit.chromium.org/gerrit/62827
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 039967354df47f89616776832ec2a9dd9fea9efb |
|
16-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: print raw metrics values
In the last patch, the statistics of metrics values are printed
as follows:
Metrics statistics by validator : fw_11.23 description
--------------------------------------------------------------------
Linearity(BothEnds)Validator
max error in x (mm) : 9.34 The max err of all samples
...
If a user would like to know which gesture/file causes the max error
above, it would be helpful to print the raw metrics values as follows:
Raw metrics values
--------------------------------------------------------------------
fw_11.23
Linearity(BothEnds)Validator
max error in x (mm)
drag_edge_thumb.left_to_right (20130710_063117) : 3.06
two_fat_fingers_tracking.left_to_right (20130710_063117) : 9.34
...
In this way, the user could observe that the file with
gesture: "two_fat_fingers_tracking",
variation: "left_to_right",
in the folder: "20130710_063117"
causes the "max error in x".
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with short metrics values grouped by validator.
$ ./summary.sh -d /tmp -m
Generate the summary report with detailed metrics values.
$ ./summary.sh -d /tmp --metrics_details
Observe that the raw detailed metrics values are printed this time.
Change-Id: I4d540f71c1c1e91507b9104eff8ac80c14d6654e
Reviewed-on: https://gerrit.chromium.org/gerrit/62329
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 330dcded15b07bf1869bd5d4cbf2bccbc6a14537 |
|
04-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: printing the statistics of metrics
In this patch, we print the statistics of metrics by validators in
the summary report.
We also handle the metrics names in a more unified way. Before,
the same metric name strings, e.g., 'max error in x (mm)', were used
across different modules. Now with a new class MetricNameProps to
keep information about the metric names, the descriptions, and the
stat_func used to calculate the statistics of metrics, the metric
names could be derived consistently by expanding from a template, say
'max error in {} (mm)', with its variations, e.g., ['x', 'y'], to
derive the names: 'max error in x (mm)' and 'max error in y (mm)'.
We also remove the PhysicalClickValidator temporarily which incurs
divide-by-0 error. This validator is not specified in the spec v2.
However, it might be useful to add it back later to make sure that
a tap does not incur physical click.
The metrics in a summary report looks like:
Metrics statistics by validator : fw1.2 description
--------------------------------------------------------------------------------
DrumrollValidator
circle radius (mm) : 1.37 Anything over 2mm is failure
Linearity(BothEnds)Validator
max error in x (mm) : 9.34 The max err of all samples
max error in y (mm) : 6.33 The max err of all samples
rms error in x (mm) : 2.46 The mean of all rms means of all trials
rms error in y (mm) : 1.03 The mean of all rms means of all trials
Linearity(Middle)Validator
max error in x (mm) : 4.91 The max err of all samples
max error in y (mm) : 6.30 The max err of all samples
rms error in x (mm) : 1.27 The mean of all rms means of all trials
rms error in y (mm) : 0.68 The mean of all rms means of all trials
PhysicalClickValidator
1f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
2f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
3f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
4f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
5f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
RangeValidator
left edge not reached (mm): 0.08 Min unreachable distance
ReportRateValidator
average time interval (ms): 12.19 less than 1/60 sec is required
intervals > 1/60 sec (%) : 4.45 0% is required
max time interval (ms) : 57.96 less than 1/60 sec is required
StationaryFingerValidator
max distance (mm) : 3.85 max distance of any two points from
any run
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report
$ ./summary.sh -m -d /tmp
Observe that the metrics are printed as above.
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: If7224354ef866cc3468752caa32feb1439b0b8cc
Reviewed-on: https://gerrit.chromium.org/gerrit/61883
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| a4a163f2153aa8cac79477b51340f7544b9d5cf3 |
|
05-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: update logs for fixing slot problem
In the previous CL (I0c2d62530df9d84e5ec59d9a82fa2147ef05b51e),
we corrected the way of deriving the slot number in a finger path.
Here we update the logs accordingly, and add some unit tests.
BUG=chromium:221864
TEST=Run the unit tests on a lumpy.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ python tests/run_all_unittests.py
All of the unit tests should pass.
Change-Id: I4fc4e5af73857b5256a6713063c8022782f6ce79
Reviewed-on: https://gerrit.chromium.org/gerrit/61020
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 313de5edc48d6af3b8891e69114ddf7d79c8516c |
|
28-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: introduce metrics in the log and summary
In this patch, we introduce the mechanism of adding metrics into the
test result logs, and displaying the metrics in the summary file.
A metric consists of the name, the value, and the unit. When a list
of such metric values are collected, the average, minimal, and
maximal values are calculated.
In particular, we add the metric for DrumrollValidator as the first
step. We also add some drumroll gestures in unit tests for testing
the metrics of DrumrollValidator. Although there is only one single
metric, "circle radius", in DrumrollValidator, it is possible that
there are multiple metrics in a single validator.
Note that a flag "-m" must be added after summary.sh to display the
metrics in the summary file.
BUG=chromium:241092
TEST=Run the following unit tests on lumpy, they should all pass.
$ cd /usr/local/autotest/tests/firmware_TouchMT
$ python tests/run_all_unittests.py
Replay the gesture files on lumpy.
$ tools/machine_replay.sh -b tests/logs/lumpy
Derive the summary file.
$ ./summary.sh -m -d /tmp
$ cat /tmp/summary/summary_*/summary_*.txt
Observe that the metric, circle radius, of Drumroll validator is
displayed as (mean, min, max) tuple for the time being. A more
elaborate format will be developed later.
Metrics (by file)
----------------------------------------------------------------
drumroll.fast
DrumrollValidator
fw_11.23: {'circle_radius_mm': (0.27, 0, 1.18)}
.
.
Note that the gesture files are created for the purpose of running
unit tests, and are not reflecting the actual performance between
the distinct firmware versions.
Now run the summary file without the "-m" flag.
$ ./summary.sh -d /tmp
$ cat /tmp/summary/summary_*/summary_*.txt
There would be no metrics shown in the summary file.
Change-Id: I610b6c4336f012a42af7ed4175e54e902f50683a
Reviewed-on: https://gerrit.chromium.org/gerrit/57311
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| ecc254fc00e6b3504d4f7d87abad937cf73a267d |
|
07-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: refactor firmware_log and firmware_summary modules
Some of the code about test logs was manipulated in firmware_summary
module which made this module become messy. In this patch, we do the
refactoring by moving those code to firmware_log module and create
classes for them. In addition, we also flatten the deep hierarchy of
log dictionaries so that it is much simpler to maintain.
BUG=chromium:238039
TEST=On a chromebook
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Make sure that the following unit test pass.
$ python tests/firmware_summary_unittest.py
Change-Id: Idc5a70ed28745805a4221cc20780c23da0210fb0
Reviewed-on: https://gerrit.chromium.org/gerrit/50320
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|
| 0cbff8bf3666a81b886080f82712943e36dad95f |
|
22-Apr-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: Rename the test suite as firmware_TouchMTB
As the test suite is now applicable to both the touchpad
and the touchscreen, it is proper to rename the test name
from firmware_TouchpadMTB to firmware_TouchMTB.
BUG=chromium:234117
TEST=None. Will be combined with next patch.
Change-Id: I92ecc074023c76f7c2f292b7e22ef960d537819f
Reviewed-on: https://gerrit.chromium.org/gerrit/48781
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_log.py
|