bc9dba0f212df1b31ed43032c68f5e07e9ee9974 |
|
01-Mar-2014 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add version info to test logs When recording gestures files or generating the test summary, we would like the version info to be prepended to the test logs, including the standard output, the html report, the pickle log, and the summary report. BUG=chromium:315787 TEST=Install the test on a chromebook machine as a first step. Then, in the chroot of your host: (cr) $ cd .../site_tests/firmware_TouchMTB (cr) $ ./version.sh -r ${MACHINE_IP} On the chromebook, record a few gestures, and then press 'x' to exit. $ DISPLAY=:0 python main.py -m complete Now, check the version info in the following files: $ cd /var/tmp/touch_firmware_test/latest $ less touch_firmware_report-xxxx $ less touch_firmware_report-xxxx.html $ less touch_firmware_report-xxxx.log The version info should show up in the first few lines look like test_version: 2014-02-28.06:46:04 5db2d13bf7ff0e56c652213434f581ce6d7a65ee Change-Id: I549d97574dd44d2cdd7eea7efe0de85a5fd67f66 Reviewed-on: https://chromium-review.googlesource.com/188562 Reviewed-by: Charles Mooney <charliemooney@chromium.org> Tested-by: Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
7584b3410e81bd7ea282432c2c3c55d9b4a06ea3 |
|
19-Dec-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: display summary table and detailed table The summary table is used for the spec, and the detailed table for tracing the root cause or for debugging. As an example, the top level summary table will show something like StationaryFingerValidator, StationaryTapValidator max distance (mm) : 1.45 max distance of any two points from any run On the other hand, the detailed table will display metrics of the two related validators separately. The StationaryFingerValidator is used to measure the pulling effect by another finger on a stationary finger, while the StationaryTapValidator is used to measure the wobble of taps/clicks. StationaryFingerValidator max distance (mm) : 1.45 max distance of any two points from any run StationaryTapValidator max distance (mm) : 0.54 max distance of any two points from any run Since there may exist distinct root causes of failures about both validators, the detailed table is useful in showing them separately. BUG=chromium:330108 TEST=Replay logs and generate a summary file. $ cd /usr/local/autotest/tests/firmware_TouchMTB $ tools/machine_replay.sh -b tests/logs/link/ $ ./summary.sh -d /tmp Open the following summary file, /tmp/summary/summary_xxxx/summary_xxxx.txt while xxxx represents timestamps. It shows two tables in the summary file. The first table is "Summary table" which shows something like ... StationaryFingerValidator StationaryTapValidator max distance (mm) : 1.45 max distance of any two points ... The second table is "Detailed table (for debugging)" which shows the two related validators separately. ... StationaryFingerValidator max distance (mm) : 1.45 max distance of any two points StationaryTapValidator max distance (mm) : 0.54 max distance of any two points ... Change-Id: Ib478b39c28c23e49d824be41c57650915ccb569a Reviewed-on: https://chromium-review.googlesource.com/181094 Reviewed-by: Charles Mooney <charliemooney@chromium.org> Commit-Queue: Shyh-In Hwang <josephsih@chromium.org> Tested-by: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
b0773e1ff6a735d964b563d494d7a06d19a873ba |
|
21-Nov-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB hide scores behind a flag in summary This patch hides the old 0.0-1.0 scores behind a flag. The scores would be kept for a while until they are completely replaced by the metrics. BUG=chromium:316291 TEST= Replay the logs and run the summarizer: $ tools/machine_replay.sh -b tests/logs/link $ ./summary.sh -d /tmp Check the summary in /tmp/summary/summary_xxxx/summary_xxxx.txt Observe that it does not show the scores. Now run the summarizer with "-s" flag to show scores. $ ./summary.sh -d /tmp -s Check the summary in /tmp/summary/summary_xxxx/summary_xxxx.txt Observe that it does show the scores this time at the end of the file. Change-Id: I0a5cb75f07ca497839d81cb8ded00eadf3dd8168 Reviewed-on: https://chromium-review.googlesource.com/177743 Reviewed-by: Charles Mooney <charliemooney@chromium.org> Tested-by: Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
1551829cde6f35f97c603c5a575c9d31e96f1208 |
|
22-Nov-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: display all metrics statistics as default This patch displays all metrics statistics with raw metrics values as default. BUG=chromium:322253 TEST= Replay the logs and run the summarizer: $ tools/machine_replay.sh -b tests/logs/link $ ./summary.sh -d /tmp Check the summary in /tmp/summary/summary_xxxx/summary_xxxx.txt Observe that it shows the metrics statistics for both CountPacketsValidator and CountTrackingIDValidator. Change-Id: Iab506f32a60527f95ebb5ddad53e8e6c5eda4cfb Reviewed-on: https://chromium-review.googlesource.com/177742 Tested-by: Shyh-In Hwang <josephsih@chromium.org> Reviewed-by: Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
c9cdc9527e1087addd1e21cf58608fd75173df5a |
|
25-Sep-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: show scores and statistics of individual round The firmware_summary module used to integrate all test data of the same firmware from different rounds to calculate the statistics. However, it would be handy to be able to calculate and display the statistics of individual rounds of the same firmware. As an example, we may like to examine the consistency of the test results of the same firmware across different rounds. We add a new command line option such that when present it treats every rounds of the same firmware as different ones by appending a round number to the firmware name. BUG=chromium:235753 TEST=Replay the lumpy test logs on a chromebook. $ cd /usr/local/autotest/tests/firmware_TouchMTB $ tools/machine_replay.sh -b tests/logs/lumpy -s Show the scores and metrics of individual round with the command line option "-i" $ ./summary.sh -d /tmp -m f -i Look at the summary report. It displays the scores and metrics of two individual rounds for fw_11.23, and also two individual rounds for fw_11.27. Run all unit tests, and there should be no division by 0 errors. $ python tests/run_all_unittests.py Change-Id: I80a657ee35e3952fa4d205f4e19724d8718e416c Reviewed-on: https://chromium-review.googlesource.com/170592 Reviewed-by: Charles Mooney <charliemooney@chromium.org> Tested-by: Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
defa4829f2c4bcdca038d6b477d7c7d1c2477266 |
|
24-Sep-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: adjust fw string width dynamically When the firmware name is longer than a default width (10), the firmware names will be displayed without spaces between them. In this patch, if a longer fw string width is found, use it to replace the default width. As an example, The firmware names stick together: Metrics statistics by validator : fw_11.23.AAfw_11.27.AA description -------------------------------------------------------------------- After setting a proper width: Metrics statistics by validator : fw_11.23.AA fw_11.27.AA description --------------------------------------------------------------------- BUG=chromium:235753 TEST=None. Change-Id: Ie34457bbb10ccd9e2c87d63d7ba824d0b64d41f0 Reviewed-on: https://chromium-review.googlesource.com/170590 Reviewed-by: Charles Mooney <charliemooney@chromium.org> Tested-by: Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
2bef8f1757544a8f9f302d6a30e80c16fc940d18 |
|
22-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: fix descriptions in the summary report Make summary report more descriptive. Metrics statistics: fw_11.27 description (lower is better) --------------------------------------------------------------------- CountPacketsValidator pct of incorrect cases (%): 0.0 an incorrect case is one where a swipe has less than 3 packets reported CountTrackingIDValidator pct of incorrect cases (%): 6.90 an incorrect case is one where there are an incorrect number of fingers detected BUG=chromium:235753 TEST=Replay the logs on link. Observe that the descriptions are shown like above: $ cd /usr/local/autotest/tests/firmware_TouchMTB $ tools/machine_replay.sh -b tests/logs/lumpy -s $ ./summary.sh -d /tmp -m f Change-Id: I6a0fcba8fe2b98f5bd3046e86f45843a2268b1d3 Reviewed-on: https://chromium-review.googlesource.com/66632 Reviewed-by: Andrew de los Reyes <adlr@chromium.org> Tested-by: Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
01c5305a76cf57925974566a4928077e275f4079 |
|
20-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: print metrics with missing values It is possible that some gestures have been recorded for a firmware but are missing in the tests for another firmware. The summary module should be able to handle this situation when printing the metrics statistics. When there are no metrics data, it just leaves proper spaces. As an example, there are no drag_edge_thumb gesture files for fw_11.27. Hence, it just leaves some blank spaces in the summary report when printing the metrics statistics. Test Summary (by gesture) : fw_1.0.AA fw_11.27 mean ssd count mean ssd count ------------------------------------------------------------------------------- drag_edge_thumb CountTrackingIDValidator: 0.50 0.71 2 BUG=chromium:235753 TEST=None. Will be tested with subsequent patches. Change-Id: I31311f2d00ce486082c69527b3e23ccfb3daf4f1 Reviewed-on: https://gerrit.chromium.org/gerrit/66818 Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
c4752b22ff8275d8c3f63e2d6b93c1ca6aa1daba |
|
20-Aug-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: print scores statistics with missing scores data It is possible that some gestures have been recorded for a firmware but are missing in the tests for another firmware. The summary module should be able to handle this situation when printing the scores statistics. When there are no scores data, it just leaves proper spaces. As an example, there are no rapid_taps_20 gesture files for fw_11.27. Hence, it just leaves some blank spaces in the summary report. Test Summary (by gesture) : fw_1.0.AA fw_11.27 mean ssd count mean ssd count ---------------------------------------------------------------------- rapid_taps_20 CountTrackingIDValidator: 1.00 0.00 9 BUG=chromium:235753 TEST=None. Will be tested with subsequent patches. Change-Id: I8d3148a35008ceb6edf61a65c89f77b6c8f66fe8 Reviewed-on: https://gerrit.chromium.org/gerrit/66817 Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
aa7b540459960d887cd7520f4d6a963a33cc7a85 |
|
26-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for PinchValidator Display the metric for PinchValidator if there are failures. Otherwise, hide the metric unless a flag "-m a" is specified. The metric shows the percentage of the incorrect test cases over the total test cases. The printing format with failures looks like: Metrics statistics by validator: fw_12 description ---------------------------------------------------------------------------- PinchValidator pct of incorrect cases (%) : 12.50 pct of incorrect cases over total cases The detailed raw metrics values will be printed if a flag "-m f" is specified. The raw metric value pair represents (actual relative motion (px), expected relative motion (px)) The format looks like: PinchValidator pct of incorrect cases (%) ** Note: value below represents (actual relative motion (px), expected relative motion (px)) pinch_to_zoom.zoom_in (20130506_032458) : (646.52, 200) pinch_to_zoom.zoom_out (20130506_032458) : (636.80, 200) BUG=chromium:235753 TEST=Run the command on a chromebook machine to generate a summary report. $ cd /usr/local/autotest/tests/firmware_TouchMTB Replay the logs. $ tools/machine_replay.sh -b tests/logs/lumpy -s Generate the summary report with all metrics statistics. $ ./summary.sh -d /tmp -m a Generate the summary report with the full raw metrics values on file basis. $ ./summary.sh -d /tmp -m f Also make sure all unit tests pass. $ python tests/run_all_unittests.py Change-Id: I57b867f2d092de2245de2244d50e71375a8e1b40 Reviewed-on: https://gerrit.chromium.org/gerrit/63449 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
8047eb40635b781cdbfd3fa2b33f423b5ff724e1 |
|
25-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for CountPacketsValidator Display the metric for CountPacketsValidator if there are failures. Otherwise, hide the metric unless a flag "-m a" is specified. The metric shows the percentage of the incorrect test cases over the total test cases. The printing format with failures looks like: Metrics statistics by validator: fw_12 description ---------------------------------------------------------------------------- CountPacketsValidator pct of incorrect cases (%) : 25.00 pct of incorrect cases over total cases The detailed raw metrics values will be printed if a flag "-m f" is specified. The raw metric value pair represents (actual number of packets, expected number of packets) The format looks like: CountPacketsValidator pct of incorrect cases (%) ** Note: value means (actual number of packets, expected number of packets) one_finger_swipe.bottom_left_to_top_right (20130506_032458) : (2, 3) one_finger_swipe.bottom_left_to_top_right (20130506_032659) : (4, 3) We also simplify the metric notes by moving them from firmware_summary module into raw_metrics_props in the firmware_log module so that the properties of metrics are maintained in the same structure. BUG=chromium:235753 TEST=Run the command on a chromebook machine to generate a summary report. $ cd /usr/local/autotest/tests/firmware_TouchMTB Replay the logs. $ tools/machine_replay.sh -b tests/logs/lumpy -s Generate the summary report with all metrics statistics. $ ./summary.sh -d /tmp -m a Generate the summary report with the full raw metrics values on file basis. $ ./summary.sh -d /tmp -m f Also make sure all unit tests pass. $ python tests/run_all_unittests.py Change-Id: Ib3cefee9148167bd4c8d099518f2505d4673c839 Reviewed-on: https://gerrit.chromium.org/gerrit/63332 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
7dbae7827dae0767bffe303708e19483b80a007e |
|
17-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: add the metric for CountTrackingIDValidator The metrics of some validators will be hidden by default unless there are failures. Such validators include CountTrackingIDValidator, CountPacketsValidator, and PinchValidator. Display the metric for CountTrackingIDValidator if there are failures. Otherwise, hide the metric unless a flag "-m a" is specified. The metric shows the percentage of the incorrect test cases over the total test cases. The printing format with failures looks like: Metrics statistics by validator: fw_12 description ------------------------------------------------------------------------------ CountTrackingIDValidator pct of incorrect cases (%) : 4.76 pct of incorrect cases over total cases ... The detailed raw metrics values will be printed if a flag "-m f" is specified. The raw metric value pair represents (actual tracking IDs, expected tracking IDs). The format looks like: CountTrackingIDValidator pct of incorrect cases (%) ** Note: value below represents (actual tracking IDs, expected tracking IDs) drag_edge_thumb.left_to_right (20130506_032458) : (5, 1) one_finger_physical_click.bottom_left (20130506_032458): (1, 1) ... Note that in the drag_edge_thumb gesture above, the expected number of tracking IDS (or finger IDs) should be 1. However, 5 tracking IDs are actually observed due to some firmware problems. For the one_finger_physical_click gesture, exactly 1 tracking ID is observed, while the number of expected finger IDs is 1. We also unify the "-m" (i.e., metrics) flag in summary.sh script. $ ./summary.sh -m p # show the statistics of the primary metrics $ ./summary.sh -m a # show the statistics of all metrics, where # all metrics = the primary metrics + the hidden ones $ ./summary.sh -m f # show the full raw metrics values on file basis BUG=chromium:235753 TEST=Run the command on a chromebook machine to generate a summary report. $ cd /usr/local/autotest/tests/firmware_TouchMTB Replay the logs. $ tools/machine_replay.sh -b tests/logs/lumpy -s Generate the summary report with the metrics statistics. $ ./summary.sh -d /tmp -m a Generate the summary report with the full raw metrics values on file basis. $ ./summary.sh -d /tmp -m f Also make sure all unit tests pass. $ python tests/run_all_unittests.py Change-Id: Ie385bfd54bac0a45211fc2a678ad57d9472e4da2 Reviewed-on: https://gerrit.chromium.org/gerrit/62827 Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
039967354df47f89616776832ec2a9dd9fea9efb |
|
16-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: print raw metrics values In the last patch, the statistics of metrics values are printed as follows: Metrics statistics by validator : fw_11.23 description -------------------------------------------------------------------- Linearity(BothEnds)Validator max error in x (mm) : 9.34 The max err of all samples ... If a user would like to know which gesture/file causes the max error above, it would be helpful to print the raw metrics values as follows: Raw metrics values -------------------------------------------------------------------- fw_11.23 Linearity(BothEnds)Validator max error in x (mm) drag_edge_thumb.left_to_right (20130710_063117) : 3.06 two_fat_fingers_tracking.left_to_right (20130710_063117) : 9.34 ... In this way, the user could observe that the file with gesture: "two_fat_fingers_tracking", variation: "left_to_right", in the folder: "20130710_063117" causes the "max error in x". BUG=chromium:235753 TEST=Run the command on a chromebook machine to generate a summary report. $ cd /usr/local/autotest/tests/firmware_TouchMTB Replay the logs $ tools/machine_replay.sh -b tests/logs/lumpy -s Generate the summary report with short metrics values grouped by validator. $ ./summary.sh -d /tmp -m Generate the summary report with detailed metrics values. $ ./summary.sh -d /tmp --metrics_details Observe that the raw detailed metrics values are printed this time. Change-Id: I4d540f71c1c1e91507b9104eff8ac80c14d6654e Reviewed-on: https://gerrit.chromium.org/gerrit/62329 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
330dcded15b07bf1869bd5d4cbf2bccbc6a14537 |
|
04-Jul-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: printing the statistics of metrics In this patch, we print the statistics of metrics by validators in the summary report. We also handle the metrics names in a more unified way. Before, the same metric name strings, e.g., 'max error in x (mm)', were used across different modules. Now with a new class MetricNameProps to keep information about the metric names, the descriptions, and the stat_func used to calculate the statistics of metrics, the metric names could be derived consistently by expanding from a template, say 'max error in {} (mm)', with its variations, e.g., ['x', 'y'], to derive the names: 'max error in x (mm)' and 'max error in y (mm)'. We also remove the PhysicalClickValidator temporarily which incurs divide-by-0 error. This validator is not specified in the spec v2. However, it might be useful to add it back later to make sure that a tap does not incur physical click. The metrics in a summary report looks like: Metrics statistics by validator : fw1.2 description -------------------------------------------------------------------------------- DrumrollValidator circle radius (mm) : 1.37 Anything over 2mm is failure Linearity(BothEnds)Validator max error in x (mm) : 9.34 The max err of all samples max error in y (mm) : 6.33 The max err of all samples rms error in x (mm) : 2.46 The mean of all rms means of all trials rms error in y (mm) : 1.03 The mean of all rms means of all trials Linearity(Middle)Validator max error in x (mm) : 4.91 The max err of all samples max error in y (mm) : 6.30 The max err of all samples rms error in x (mm) : 1.27 The mean of all rms means of all trials rms error in y (mm) : 0.68 The mean of all rms means of all trials PhysicalClickValidator 1f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect) 2f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect) 3f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect) 4f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect) 5f-click miss rate (%) : 0.00 Should be close to 0 (0 is perfect) RangeValidator left edge not reached (mm): 0.08 Min unreachable distance ReportRateValidator average time interval (ms): 12.19 less than 1/60 sec is required intervals > 1/60 sec (%) : 4.45 0% is required max time interval (ms) : 57.96 less than 1/60 sec is required StationaryFingerValidator max distance (mm) : 3.85 max distance of any two points from any run BUG=chromium:235753 TEST=Run the command on a chromebook machine to generate a summary report. $ cd /usr/local/autotest/tests/firmware_TouchMTB Replay the logs $ tools/machine_replay.sh -b tests/logs/lumpy -s Generate the summary report $ ./summary.sh -m -d /tmp Observe that the metrics are printed as above. Also make sure all unit tests pass. $ python tests/run_all_unittests.py Change-Id: If7224354ef866cc3468752caa32feb1439b0b8cc Reviewed-on: https://gerrit.chromium.org/gerrit/61883 Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
313de5edc48d6af3b8891e69114ddf7d79c8516c |
|
28-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: introduce metrics in the log and summary In this patch, we introduce the mechanism of adding metrics into the test result logs, and displaying the metrics in the summary file. A metric consists of the name, the value, and the unit. When a list of such metric values are collected, the average, minimal, and maximal values are calculated. In particular, we add the metric for DrumrollValidator as the first step. We also add some drumroll gestures in unit tests for testing the metrics of DrumrollValidator. Although there is only one single metric, "circle radius", in DrumrollValidator, it is possible that there are multiple metrics in a single validator. Note that a flag "-m" must be added after summary.sh to display the metrics in the summary file. BUG=chromium:241092 TEST=Run the following unit tests on lumpy, they should all pass. $ cd /usr/local/autotest/tests/firmware_TouchMT $ python tests/run_all_unittests.py Replay the gesture files on lumpy. $ tools/machine_replay.sh -b tests/logs/lumpy Derive the summary file. $ ./summary.sh -m -d /tmp $ cat /tmp/summary/summary_*/summary_*.txt Observe that the metric, circle radius, of Drumroll validator is displayed as (mean, min, max) tuple for the time being. A more elaborate format will be developed later. Metrics (by file) ---------------------------------------------------------------- drumroll.fast DrumrollValidator fw_11.23: {'circle_radius_mm': (0.27, 0, 1.18)} . . Note that the gesture files are created for the purpose of running unit tests, and are not reflecting the actual performance between the distinct firmware versions. Now run the summary file without the "-m" flag. $ ./summary.sh -d /tmp $ cat /tmp/summary/summary_*/summary_*.txt There would be no metrics shown in the summary file. Change-Id: I610b6c4336f012a42af7ed4175e54e902f50683a Reviewed-on: https://gerrit.chromium.org/gerrit/57311 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
e9c6a255171607e0467d9025b2a2ed92eb03c87a |
|
03-Jun-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: support the option to display summary metrics Since there are more options available in the firmware_summary module, in this patch we add the getopt option support. In particular, we add an extra option to determine if the summary metrics would be displayed. BUG=chromium:241092 TEST=None. This would be tested with the next patch. Change-Id: Iadbdfac6f6773bc3afb3ff3c8c859f441627bb21 Reviewed-on: https://gerrit.chromium.org/gerrit/57310 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
ecc254fc00e6b3504d4f7d87abad937cf73a267d |
|
07-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: refactor firmware_log and firmware_summary modules Some of the code about test logs was manipulated in firmware_summary module which made this module become messy. In this patch, we do the refactoring by moving those code to firmware_log module and create classes for them. In addition, we also flatten the deep hierarchy of log dictionaries so that it is much simpler to maintain. BUG=chromium:238039 TEST=On a chromebook $ cd /usr/local/autotest/tests/firmware_TouchMTB Make sure that the following unit test pass. $ python tests/firmware_summary_unittest.py Change-Id: Idc5a70ed28745805a4221cc20780c23da0210fb0 Reviewed-on: https://gerrit.chromium.org/gerrit/50320 Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Reviewed-by: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
4fae341b8568845295c491c38613e6e03fe99cb2 |
|
06-May-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: new logs with raw data for unit tests Re-design the unit tests for the summary modules for the purpose of a more systematic coverage on different fw versions, different gestures, different validators, single/multiple iterations, etc. The raw data files are also kept so that they could be replayed anytime when new logs need to be generated due to the change of log format. BUG=chromium:238039 TEST=On a chromebook $ cd /usr/local/autotest/tests/firmware_TouchMTB Make sure that the following unit test pass. $ python tests/firmware_summary_unittest.py Change-Id: Ie42ce42ac36d84bdf9e439cec5a48488f0c8b674 Reviewed-on: https://gerrit.chromium.org/gerrit/50311 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
8c5e275df682ef5b0ad52574ae1e2e31fc724509 |
|
22-Apr-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: Rename the test suite and fix the programs To finish renaming the test suite, we also need to fix the prompts, classes, methods, variables, and strings, etc. to reduce confusion. BUG=chromium:234117 TEST=Install the test suite to a chromebook (cr) $ ./run_remote_tests.sh --autotest_dir ~/trunk/src/third_party/autotest/files --remote=$MACHINE_IP firmware_TouchMTB On the chromebook, conduct the test suite as follows. $ DISPLAY=:0 python main.py It should execute normally. Change-Id: Iaf5d5060bf68e3e0a13913a69227636fba2fbeb6 Reviewed-on: https://gerrit.chromium.org/gerrit/48782 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|
0cbff8bf3666a81b886080f82712943e36dad95f |
|
22-Apr-2013 |
Joseph Hwang <josephsih@chromium.org> |
firmware_TouchMTB: Rename the test suite as firmware_TouchMTB As the test suite is now applicable to both the touchpad and the touchscreen, it is proper to rename the test name from firmware_TouchpadMTB to firmware_TouchMTB. BUG=chromium:234117 TEST=None. Will be combined with next patch. Change-Id: I92ecc074023c76f7c2f292b7e22ef960d537819f Reviewed-on: https://gerrit.chromium.org/gerrit/48781 Reviewed-by: Charlie Mooney <charliemooney@chromium.org> Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org> Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
/external/autotest/client/site_tests/firmware_TouchMTB/firmware_summary.py
|