| bc9dba0f212df1b31ed43032c68f5e07e9ee9974 |
|
01-Mar-2014 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add version info to test logs
When recording gestures files or generating the test summary,
we would like the version info to be prepended to the test logs,
including the standard output, the html report, the pickle log,
and the summary report.
BUG=chromium:315787
TEST=Install the test on a chromebook machine as a first step.
Then, in the chroot of your host:
(cr) $ cd .../site_tests/firmware_TouchMTB
(cr) $ ./version.sh -r ${MACHINE_IP}
On the chromebook, record a few gestures, and then press 'x' to exit.
$ DISPLAY=:0 python main.py -m complete
Now, check the version info in the following files:
$ cd /var/tmp/touch_firmware_test/latest
$ less touch_firmware_report-xxxx
$ less touch_firmware_report-xxxx.html
$ less touch_firmware_report-xxxx.log
The version info should show up in the first few lines look like
test_version: 2014-02-28.06:46:04 5db2d13bf7ff0e56c652213434f581ce6d7a65ee
Change-Id: I549d97574dd44d2cdd7eea7efe0de85a5fd67f66
Reviewed-on: https://chromium-review.googlesource.com/188562
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 7584b3410e81bd7ea282432c2c3c55d9b4a06ea3 |
|
19-Dec-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: display summary table and detailed table
The summary table is used for the spec, and the detailed table for
tracing the root cause or for debugging.
As an example, the top level summary table will show something like
StationaryFingerValidator, StationaryTapValidator
max distance (mm) : 1.45 max distance of any two points from any run
On the other hand, the detailed table will display metrics of the two
related validators separately. The StationaryFingerValidator is used
to measure the pulling effect by another finger on a stationary
finger, while the StationaryTapValidator is used to measure the
wobble of taps/clicks.
StationaryFingerValidator
max distance (mm) : 1.45 max distance of any two points from any run
StationaryTapValidator
max distance (mm) : 0.54 max distance of any two points from any run
Since there may exist distinct root causes of failures about both
validators, the detailed table is useful in showing them separately.
BUG=chromium:330108
TEST=Replay logs and generate a summary file.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/link/
$ ./summary.sh -d /tmp
Open the following summary file, /tmp/summary/summary_xxxx/summary_xxxx.txt
while xxxx represents timestamps. It shows two tables in the summary file.
The first table is "Summary table" which shows something like
...
StationaryFingerValidator StationaryTapValidator
max distance (mm) : 1.45 max distance of any two points
...
The second table is "Detailed table (for debugging)" which shows the
two related validators separately.
...
StationaryFingerValidator
max distance (mm) : 1.45 max distance of any two points
StationaryTapValidator
max distance (mm) : 0.54 max distance of any two points
...
Change-Id: Ib478b39c28c23e49d824be41c57650915ccb569a
Reviewed-on: https://chromium-review.googlesource.com/181094
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| b0773e1ff6a735d964b563d494d7a06d19a873ba |
|
21-Nov-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB hide scores behind a flag in summary
This patch hides the old 0.0-1.0 scores behind a flag. The scores
would be kept for a while until they are completely replaced by
the metrics.
BUG=chromium:316291
TEST= Replay the logs and run the summarizer:
$ tools/machine_replay.sh -b tests/logs/link
$ ./summary.sh -d /tmp
Check the summary in /tmp/summary/summary_xxxx/summary_xxxx.txt
Observe that it does not show the scores.
Now run the summarizer with "-s" flag to show scores.
$ ./summary.sh -d /tmp -s
Check the summary in /tmp/summary/summary_xxxx/summary_xxxx.txt
Observe that it does show the scores this time at the end of the file.
Change-Id: I0a5cb75f07ca497839d81cb8ded00eadf3dd8168
Reviewed-on: https://chromium-review.googlesource.com/177743
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 1551829cde6f35f97c603c5a575c9d31e96f1208 |
|
22-Nov-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: display all metrics statistics as default
This patch displays all metrics statistics with raw metrics values
as default.
BUG=chromium:322253
TEST= Replay the logs and run the summarizer:
$ tools/machine_replay.sh -b tests/logs/link
$ ./summary.sh -d /tmp
Check the summary in /tmp/summary/summary_xxxx/summary_xxxx.txt
Observe that it shows the metrics statistics for both
CountPacketsValidator and CountTrackingIDValidator.
Change-Id: Iab506f32a60527f95ebb5ddad53e8e6c5eda4cfb
Reviewed-on: https://chromium-review.googlesource.com/177742
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| c9cdc9527e1087addd1e21cf58608fd75173df5a |
|
25-Sep-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: show scores and statistics of individual round
The firmware_summary module used to integrate all test data of
the same firmware from different rounds to calculate the statistics.
However, it would be handy to be able to calculate and display
the statistics of individual rounds of the same firmware. As an
example, we may like to examine the consistency of the test results
of the same firmware across different rounds.
We add a new command line option such that when present it treats
every rounds of the same firmware as different ones by appending
a round number to the firmware name.
BUG=chromium:235753
TEST=Replay the lumpy test logs on a chromebook.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Show the scores and metrics of individual round with the
command line option "-i"
$ ./summary.sh -d /tmp -m f -i
Look at the summary report. It displays the scores and metrics
of two individual rounds for fw_11.23, and also two individual
rounds for fw_11.27.
Run all unit tests, and there should be no division by 0 errors.
$ python tests/run_all_unittests.py
Change-Id: I80a657ee35e3952fa4d205f4e19724d8718e416c
Reviewed-on: https://chromium-review.googlesource.com/170592
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| defa4829f2c4bcdca038d6b477d7c7d1c2477266 |
|
24-Sep-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: adjust fw string width dynamically
When the firmware name is longer than a default width (10),
the firmware names will be displayed without spaces between
them. In this patch, if a longer fw string width is found,
use it to replace the default width. As an example,
The firmware names stick together:
Metrics statistics by validator : fw_11.23.AAfw_11.27.AA description
--------------------------------------------------------------------
After setting a proper width:
Metrics statistics by validator : fw_11.23.AA fw_11.27.AA description
---------------------------------------------------------------------
BUG=chromium:235753
TEST=None.
Change-Id: Ie34457bbb10ccd9e2c87d63d7ba824d0b64d41f0
Reviewed-on: https://chromium-review.googlesource.com/170590
Reviewed-by: Charles Mooney <charliemooney@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 2bef8f1757544a8f9f302d6a30e80c16fc940d18 |
|
22-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: fix descriptions in the summary report
Make summary report more descriptive.
Metrics statistics: fw_11.27 description (lower is better)
---------------------------------------------------------------------
CountPacketsValidator
pct of incorrect cases (%): 0.0 an incorrect case is one where a swipe
has less than 3 packets reported
CountTrackingIDValidator
pct of incorrect cases (%): 6.90 an incorrect case is one where there are
an incorrect number of fingers detected
BUG=chromium:235753
TEST=Replay the logs on link. Observe that the descriptions
are shown like above:
$ cd /usr/local/autotest/tests/firmware_TouchMTB
$ tools/machine_replay.sh -b tests/logs/lumpy -s
$ ./summary.sh -d /tmp -m f
Change-Id: I6a0fcba8fe2b98f5bd3046e86f45843a2268b1d3
Reviewed-on: https://chromium-review.googlesource.com/66632
Reviewed-by: Andrew de los Reyes <adlr@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 01c5305a76cf57925974566a4928077e275f4079 |
|
20-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: print metrics with missing values
It is possible that some gestures have been recorded for a firmware
but are missing in the tests for another firmware. The summary module
should be able to handle this situation when printing the metrics
statistics. When there are no metrics data, it just leaves proper
spaces.
As an example, there are no drag_edge_thumb gesture files for fw_11.27.
Hence, it just leaves some blank spaces in the summary report when
printing the metrics statistics.
Test Summary (by gesture) : fw_1.0.AA fw_11.27
mean ssd count mean ssd count
-------------------------------------------------------------------------------
drag_edge_thumb
CountTrackingIDValidator: 0.50 0.71 2
BUG=chromium:235753
TEST=None. Will be tested with subsequent patches.
Change-Id: I31311f2d00ce486082c69527b3e23ccfb3daf4f1
Reviewed-on: https://gerrit.chromium.org/gerrit/66818
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| c4752b22ff8275d8c3f63e2d6b93c1ca6aa1daba |
|
20-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: print scores statistics with missing scores data
It is possible that some gestures have been recorded for a firmware
but are missing in the tests for another firmware. The summary module
should be able to handle this situation when printing the scores
statistics. When there are no scores data, it just leaves proper
spaces.
As an example, there are no rapid_taps_20 gesture files for fw_11.27.
Hence, it just leaves some blank spaces in the summary report.
Test Summary (by gesture) : fw_1.0.AA fw_11.27
mean ssd count mean ssd count
----------------------------------------------------------------------
rapid_taps_20
CountTrackingIDValidator: 1.00 0.00 9
BUG=chromium:235753
TEST=None. Will be tested with subsequent patches.
Change-Id: I8d3148a35008ceb6edf61a65c89f77b6c8f66fe8
Reviewed-on: https://gerrit.chromium.org/gerrit/66817
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| aa7b540459960d887cd7520f4d6a963a33cc7a85 |
|
26-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for PinchValidator
Display the metric for PinchValidator if there are failures.
Otherwise, hide the metric unless a flag "-m a" is specified. The
metric shows the percentage of the incorrect test cases over the total
test cases.
The printing format with failures looks like:
Metrics statistics by validator: fw_12 description
----------------------------------------------------------------------------
PinchValidator
pct of incorrect cases (%) : 12.50 pct of incorrect cases over total cases
The detailed raw metrics values will be printed if a flag "-m f"
is specified. The raw metric value pair represents
(actual relative motion (px), expected relative motion (px))
The format looks like:
PinchValidator
pct of incorrect cases (%)
** Note: value below represents
(actual relative motion (px), expected relative motion (px))
pinch_to_zoom.zoom_in (20130506_032458) : (646.52, 200)
pinch_to_zoom.zoom_out (20130506_032458) : (636.80, 200)
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs.
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with all metrics statistics.
$ ./summary.sh -d /tmp -m a
Generate the summary report with the full raw metrics values on file basis.
$ ./summary.sh -d /tmp -m f
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: I57b867f2d092de2245de2244d50e71375a8e1b40
Reviewed-on: https://gerrit.chromium.org/gerrit/63449
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 8047eb40635b781cdbfd3fa2b33f423b5ff724e1 |
|
25-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for CountPacketsValidator
Display the metric for CountPacketsValidator if there are failures.
Otherwise, hide the metric unless a flag "-m a" is specified. The
metric shows the percentage of the incorrect test cases over the total
test cases.
The printing format with failures looks like:
Metrics statistics by validator: fw_12 description
----------------------------------------------------------------------------
CountPacketsValidator
pct of incorrect cases (%) : 25.00 pct of incorrect cases over total cases
The detailed raw metrics values will be printed if a flag "-m f"
is specified. The raw metric value pair represents
(actual number of packets, expected number of packets)
The format looks like:
CountPacketsValidator
pct of incorrect cases (%)
** Note: value means (actual number of packets, expected number of packets)
one_finger_swipe.bottom_left_to_top_right (20130506_032458) : (2, 3)
one_finger_swipe.bottom_left_to_top_right (20130506_032659) : (4, 3)
We also simplify the metric notes by moving them from firmware_summary
module into raw_metrics_props in the firmware_log module so that the
properties of metrics are maintained in the same structure.
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs.
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with all metrics statistics.
$ ./summary.sh -d /tmp -m a
Generate the summary report with the full raw metrics values on file basis.
$ ./summary.sh -d /tmp -m f
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: Ib3cefee9148167bd4c8d099518f2505d4673c839
Reviewed-on: https://gerrit.chromium.org/gerrit/63332
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 7dbae7827dae0767bffe303708e19483b80a007e |
|
17-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for CountTrackingIDValidator
The metrics of some validators will be hidden by default unless
there are failures. Such validators include CountTrackingIDValidator,
CountPacketsValidator, and PinchValidator.
Display the metric for CountTrackingIDValidator if there are failures.
Otherwise, hide the metric unless a flag "-m a" is specified. The
metric shows the percentage of the incorrect test cases over the total
test cases.
The printing format with failures looks like:
Metrics statistics by validator: fw_12 description
------------------------------------------------------------------------------
CountTrackingIDValidator
pct of incorrect cases (%) : 4.76 pct of incorrect cases over total cases
...
The detailed raw metrics values will be printed if a flag "-m f"
is specified. The raw metric value pair represents
(actual tracking IDs, expected tracking IDs). The format looks like:
CountTrackingIDValidator
pct of incorrect cases (%)
** Note: value below represents (actual tracking IDs, expected tracking IDs)
drag_edge_thumb.left_to_right (20130506_032458) : (5, 1)
one_finger_physical_click.bottom_left (20130506_032458): (1, 1)
...
Note that in the drag_edge_thumb gesture above, the expected number
of tracking IDS (or finger IDs) should be 1. However, 5 tracking IDs
are actually observed due to some firmware problems.
For the one_finger_physical_click gesture, exactly 1 tracking ID is
observed, while the number of expected finger IDs is 1.
We also unify the "-m" (i.e., metrics) flag in summary.sh script.
$ ./summary.sh -m p # show the statistics of the primary metrics
$ ./summary.sh -m a # show the statistics of all metrics, where
# all metrics = the primary metrics + the hidden ones
$ ./summary.sh -m f # show the full raw metrics values on file basis
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs.
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with the metrics statistics.
$ ./summary.sh -d /tmp -m a
Generate the summary report with the full raw metrics values on file basis.
$ ./summary.sh -d /tmp -m f
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: Ie385bfd54bac0a45211fc2a678ad57d9472e4da2
Reviewed-on: https://gerrit.chromium.org/gerrit/62827
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 039967354df47f89616776832ec2a9dd9fea9efb |
|
16-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: print raw metrics values
In the last patch, the statistics of metrics values are printed
as follows:
Metrics statistics by validator : fw_11.23 description
--------------------------------------------------------------------
Linearity(BothEnds)Validator
max error in x (mm) : 9.34 The max err of all samples
...
If a user would like to know which gesture/file causes the max error
above, it would be helpful to print the raw metrics values as follows:
Raw metrics values
--------------------------------------------------------------------
fw_11.23
Linearity(BothEnds)Validator
max error in x (mm)
drag_edge_thumb.left_to_right (20130710_063117) : 3.06
two_fat_fingers_tracking.left_to_right (20130710_063117) : 9.34
...
In this way, the user could observe that the file with
gesture: "two_fat_fingers_tracking",
variation: "left_to_right",
in the folder: "20130710_063117"
causes the "max error in x".
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report with short metrics values grouped by validator.
$ ./summary.sh -d /tmp -m
Generate the summary report with detailed metrics values.
$ ./summary.sh -d /tmp --metrics_details
Observe that the raw detailed metrics values are printed this time.
Change-Id: I4d540f71c1c1e91507b9104eff8ac80c14d6654e
Reviewed-on: https://gerrit.chromium.org/gerrit/62329
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 330dcded15b07bf1869bd5d4cbf2bccbc6a14537 |
|
04-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: printing the statistics of metrics
In this patch, we print the statistics of metrics by validators in
the summary report.
We also handle the metrics names in a more unified way. Before,
the same metric name strings, e.g., 'max error in x (mm)', were used
across different modules. Now with a new class MetricNameProps to
keep information about the metric names, the descriptions, and the
stat_func used to calculate the statistics of metrics, the metric
names could be derived consistently by expanding from a template, say
'max error in {} (mm)', with its variations, e.g., ['x', 'y'], to
derive the names: 'max error in x (mm)' and 'max error in y (mm)'.
We also remove the PhysicalClickValidator temporarily which incurs
divide-by-0 error. This validator is not specified in the spec v2.
However, it might be useful to add it back later to make sure that
a tap does not incur physical click.
The metrics in a summary report looks like:
Metrics statistics by validator : fw1.2 description
--------------------------------------------------------------------------------
DrumrollValidator
circle radius (mm) : 1.37 Anything over 2mm is failure
Linearity(BothEnds)Validator
max error in x (mm) : 9.34 The max err of all samples
max error in y (mm) : 6.33 The max err of all samples
rms error in x (mm) : 2.46 The mean of all rms means of all trials
rms error in y (mm) : 1.03 The mean of all rms means of all trials
Linearity(Middle)Validator
max error in x (mm) : 4.91 The max err of all samples
max error in y (mm) : 6.30 The max err of all samples
rms error in x (mm) : 1.27 The mean of all rms means of all trials
rms error in y (mm) : 0.68 The mean of all rms means of all trials
PhysicalClickValidator
1f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
2f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
3f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
4f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
5f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect)
RangeValidator
left edge not reached (mm): 0.08 Min unreachable distance
ReportRateValidator
average time interval (ms): 12.19 less than 1/60 sec is required
intervals > 1/60 sec (%) : 4.45 0% is required
max time interval (ms) : 57.96 less than 1/60 sec is required
StationaryFingerValidator
max distance (mm) : 3.85 max distance of any two points from
any run
BUG=chromium:235753
TEST=Run the command on a chromebook machine to generate a summary report.
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Replay the logs
$ tools/machine_replay.sh -b tests/logs/lumpy -s
Generate the summary report
$ ./summary.sh -m -d /tmp
Observe that the metrics are printed as above.
Also make sure all unit tests pass.
$ python tests/run_all_unittests.py
Change-Id: If7224354ef866cc3468752caa32feb1439b0b8cc
Reviewed-on: https://gerrit.chromium.org/gerrit/61883
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 313de5edc48d6af3b8891e69114ddf7d79c8516c |
|
28-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: introduce metrics in the log and summary
In this patch, we introduce the mechanism of adding metrics into the
test result logs, and displaying the metrics in the summary file.
A metric consists of the name, the value, and the unit. When a list
of such metric values are collected, the average, minimal, and
maximal values are calculated.
In particular, we add the metric for DrumrollValidator as the first
step. We also add some drumroll gestures in unit tests for testing
the metrics of DrumrollValidator. Although there is only one single
metric, "circle radius", in DrumrollValidator, it is possible that
there are multiple metrics in a single validator.
Note that a flag "-m" must be added after summary.sh to display the
metrics in the summary file.
BUG=chromium:241092
TEST=Run the following unit tests on lumpy, they should all pass.
$ cd /usr/local/autotest/tests/firmware_TouchMT
$ python tests/run_all_unittests.py
Replay the gesture files on lumpy.
$ tools/machine_replay.sh -b tests/logs/lumpy
Derive the summary file.
$ ./summary.sh -m -d /tmp
$ cat /tmp/summary/summary_*/summary_*.txt
Observe that the metric, circle radius, of Drumroll validator is
displayed as (mean, min, max) tuple for the time being. A more
elaborate format will be developed later.
Metrics (by file)
----------------------------------------------------------------
drumroll.fast
DrumrollValidator
fw_11.23: {'circle_radius_mm': (0.27, 0, 1.18)}
.
.
Note that the gesture files are created for the purpose of running
unit tests, and are not reflecting the actual performance between
the distinct firmware versions.
Now run the summary file without the "-m" flag.
$ ./summary.sh -d /tmp
$ cat /tmp/summary/summary_*/summary_*.txt
There would be no metrics shown in the summary file.
Change-Id: I610b6c4336f012a42af7ed4175e54e902f50683a
Reviewed-on: https://gerrit.chromium.org/gerrit/57311
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| e9c6a255171607e0467d9025b2a2ed92eb03c87a |
|
03-Jun-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: support the option to display summary metrics
Since there are more options available in the firmware_summary
module, in this patch we add the getopt option support. In
particular, we add an extra option to determine if the summary
metrics would be displayed.
BUG=chromium:241092
TEST=None. This would be tested with the next patch.
Change-Id: Iadbdfac6f6773bc3afb3ff3c8c859f441627bb21
Reviewed-on: https://gerrit.chromium.org/gerrit/57310
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| ecc254fc00e6b3504d4f7d87abad937cf73a267d |
|
07-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: refactor firmware_log and firmware_summary modules
Some of the code about test logs was manipulated in firmware_summary
module which made this module become messy. In this patch, we do the
refactoring by moving those code to firmware_log module and create
classes for them. In addition, we also flatten the deep hierarchy of
log dictionaries so that it is much simpler to maintain.
BUG=chromium:238039
TEST=On a chromebook
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Make sure that the following unit test pass.
$ python tests/firmware_summary_unittest.py
Change-Id: Idc5a70ed28745805a4221cc20780c23da0210fb0
Reviewed-on: https://gerrit.chromium.org/gerrit/50320
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 4fae341b8568845295c491c38613e6e03fe99cb2 |
|
06-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: new logs with raw data for unit tests
Re-design the unit tests for the summary modules for the purpose of a
more systematic coverage on different fw versions, different gestures,
different validators, single/multiple iterations, etc.
The raw data files are also kept so that they could be replayed
anytime when new logs need to be generated due to the change of log
format.
BUG=chromium:238039
TEST=On a chromebook
$ cd /usr/local/autotest/tests/firmware_TouchMTB
Make sure that the following unit test pass.
$ python tests/firmware_summary_unittest.py
Change-Id: Ie42ce42ac36d84bdf9e439cec5a48488f0c8b674
Reviewed-on: https://gerrit.chromium.org/gerrit/50311
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 8c5e275df682ef5b0ad52574ae1e2e31fc724509 |
|
22-Apr-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: Rename the test suite and fix the programs
To finish renaming the test suite, we also need to fix the prompts,
classes, methods, variables, and strings, etc. to reduce confusion.
BUG=chromium:234117
TEST=Install the test suite to a chromebook
(cr) $ ./run_remote_tests.sh --autotest_dir
~/trunk/src/third_party/autotest/files --remote=$MACHINE_IP
firmware_TouchMTB
On the chromebook, conduct the test suite as follows.
$ DISPLAY=:0 python main.py
It should execute normally.
Change-Id: Iaf5d5060bf68e3e0a13913a69227636fba2fbeb6
Reviewed-on: https://gerrit.chromium.org/gerrit/48782
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
| 0cbff8bf3666a81b886080f82712943e36dad95f |
|
22-Apr-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: Rename the test suite as firmware_TouchMTB
As the test suite is now applicable to both the touchpad
and the touchscreen, it is proper to rename the test name
from firmware_TouchpadMTB to firmware_TouchMTB.
BUG=chromium:234117
TEST=None. Will be combined with next patch.
Change-Id: I92ecc074023c76f7c2f292b7e22ef960d537819f
Reviewed-on: https://gerrit.chromium.org/gerrit/48781
Reviewed-by: Charlie Mooney <charliemooney@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|