19066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project#!/usr/bin/python 29066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 39066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project"""Run layout tests using Android emulator and instrumentation. 49066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 59066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project First, you need to get an SD card or sdcard image that has layout tests on it. 69066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Layout tests are in following directory: 771716223201a01a4f4b6fe182ad97720b7d06124Guang Zhu /sdcard/webkit/layout_tests 871716223201a01a4f4b6fe182ad97720b7d06124Guang Zhu For example, /sdcard/webkit/layout_tests/fast 99066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Usage: 119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run all tests under fast/ directory: 129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py, or 139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast 149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run all tests under a sub directory: 169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast/dom 179066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run a single test: 199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast/dom/ 209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project After a merge, if there are changes of layout tests in SD card, you need to 229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project use --refresh-test-list option *once* to re-generate test list on the card. 239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Some other options are: 2571716223201a01a4f4b6fe182ad97720b7d06124Guang Zhu --rebaseline generates expected layout tests results under /sdcard/webkit/expected_result/ 269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --time-out-ms (default is 8000 millis) for each test 279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --adb-options="-e" passes option string to adb 289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --results-directory=..., (default is ./layout-test-results) directory name under which results are stored. 2912077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block --js-engine the JavaScript engine currently in use, determines which set of Android-specific expected results we should use, should be 'jsc' or 'v8' 309066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project""" 319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport logging 339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport optparse 349066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport os 359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport subprocess 369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport sys 379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport time 389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 399066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef CountLineNumber(filename): 409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Compute the number of lines in a given file. 419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project filename: a file name related to the current directory. 449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp = open(os.path.abspath(filename), "r"); 479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project lines = 0 489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for line in fp.readlines(): 499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project lines = lines + 1 509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp.close() 519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return lines 529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef DumpRenderTreeFinished(adb_cmd): 549066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ Check if DumpRenderTree finished running tests 555080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project output: adb_cmd string 589066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 595080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 6071716223201a01a4f4b6fe182ad97720b7d06124Guang Zhu # pull /sdcard/webkit/running_test.txt, if the content is "#DONE", it's done 6171716223201a01a4f4b6fe182ad97720b7d06124Guang Zhu shell_cmd_str = adb_cmd + " shell cat /sdcard/webkit/running_test.txt" 629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_output = subprocess.Popen(shell_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return adb_output.strip() == "#DONE" 649066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 65903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhudef DiffResults(marker, new_results, old_results, diff_results, strip_reason, 66903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu new_count_first=True): 679066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ Given two result files, generate diff and 689066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project write to diff_results file. All arguments are absolute paths 699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project to files. 709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project old_file = open(old_results, "r") 729066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project new_file = open(new_results, "r") 735080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu diff_file = open(diff_results, "a") 749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Read lines from each file 769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ndict = new_file.readlines() 779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project cdict = old_file.readlines() 789066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 799066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Write marker to diff file 809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines(marker + "\n") 819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines("###############\n") 829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Strip reason from result lines 849066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if strip_reason is True: 859066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for i in range(0, len(ndict)): 869066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ndict[i] = ndict[i].split(' ')[0] + "\n" 879066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for i in range(0, len(cdict)): 889066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project cdict[i] = cdict[i].split(' ')[0] + "\n" 899066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 90903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu params = { 91903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu "new": [0, ndict, cdict, "+"], 92903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu "miss": [0, cdict, ndict, "-"] 93903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu } 94903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if new_count_first: 95903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu order = ["new", "miss"] 96903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu else: 97903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu order = ["miss", "new"] 98903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu 99903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu for key in order: 100903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu for line in params[key][1]: 101903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if line not in params[key][2]: 102903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if line[-1] != "\n": 103903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu line += "\n"; 104903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu diff_file.writelines(params[key][3] + line) 105903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu params[key][0] += 1 106903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu 107903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu logging.info(marker + " >>> " + str(params["new"][0]) + " new, " + 108903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu str(params["miss"][0]) + " misses") 1099066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines("\n\n") 1119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project old_file.close() 1139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project new_file.close() 1149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.close() 1159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return 1169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1179066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef CompareResults(ref_dir, results_dir): 1189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Compare results in two directories 1199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 1219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir: the reference directory having layout results as references 1229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir: the results directory 1239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 1249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Comparing results to " + ref_dir) 1259066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1265080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu diff_result = os.path.join(results_dir, "layout_tests_diff.txt") 1279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if os.path.exists(diff_result): 1289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.remove(diff_result) 1299066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 130903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu files=["crashed", "failed", "passed", "nontext"] 1319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for f in files: 1329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project result_file_name = "layout_tests_" + f + ".txt" 1339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project DiffResults(f, os.path.join(results_dir, result_file_name), 1349066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.path.join(ref_dir, result_file_name), diff_result, 135595fbd6ea22708504dc9e24b44fa5eb357a576ecGuang Zhu False, f != "passed") 1369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Detailed diffs are in " + diff_result) 1379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef main(options, args): 1399066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Run the tests. Will call sys.exit when complete. 1405080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 1419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 1429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project options: a dictionary of command line options 1439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project args: a list of sub directories or files to test 1449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 1459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Set up logging format. 1479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project log_level = logging.INFO 1489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.verbose: 1499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project log_level = logging.DEBUG 1509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.basicConfig(level=log_level, 1519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project format='%(message)s') 1529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Include all tests if none are specified. 1549066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not args: 155ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project path = '/'; 1569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project else: 1579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project path = ' '.join(args); 1589066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1599066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_cmd = "adb "; 1609066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.adb_options: 1619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_cmd += options.adb_options 1629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Re-generate the test list if --refresh-test-list is on 1649066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.refresh_test_list: 1659066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Generating test list."); 166ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project generate_test_list_cmd_str = adb_cmd + " shell am instrument -e class com.android.dumprendertree.LayoutTestsAutoTest#generateTestList -e path \"" + path + "\" -w com.android.dumprendertree/.LayoutTestsAutoRunner" 167ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(generate_test_list_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 1689066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if adb_output.find('Process crashed') != -1: 1709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Aborting because cannot generate test list.\n" + adb_output) 1719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1) 1729066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1739066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Running tests") 1759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Count crashed tests. 1779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests = [] 1789066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1796c15f6003a69e664f132342be2252a77e480495bGuang Zhu timeout_ms = '15000' 1809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.time_out_ms: 1819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project timeout_ms = options.time_out_ms 1829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Run test until it's done 1849066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 185ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_prefix = adb_cmd + " shell am instrument" 186ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 187ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix = " -e path \"" + path + "\" -e timeout " + timeout_ms 188ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project if options.rebaseline: 189ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix += " -e rebaseline true" 19012077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block 19112077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block # If the JS engine is not specified on the command line, try reading the 19212077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block # JS_ENGINE environment variable, which is used by the build system in 19312077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block # external/webkit/Android.mk. 19412077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block js_engine = options.js_engine 195dc2711d47d2c05ee1ee1f15131676b53cb3c1d5eBen Murdoch if not js_engine and os.environ.has_key('JS_ENGINE'): 19612077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block js_engine = os.environ['JS_ENGINE'] 19712077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block if js_engine: 19812077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block run_layout_test_cmd_postfix += " -e jsengine " + js_engine 19912077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block 200ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix += " -w com.android.dumprendertree/.LayoutTestsAutoRunner" 201ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 2029066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Call LayoutTestsAutoTest::startLayoutTests. 203ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd = run_layout_test_cmd_prefix + " -e class com.android.dumprendertree.LayoutTestsAutoTest#startLayoutTests" + run_layout_test_cmd_postfix 204ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 205ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(run_layout_test_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 2069066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project while not DumpRenderTreeFinished(adb_cmd): 2079066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Get the running_test.txt 2089066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("DumpRenderTree crashed, output:\n" + adb_output) 2099066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 21071716223201a01a4f4b6fe182ad97720b7d06124Guang Zhu shell_cmd_str = adb_cmd + " shell cat /sdcard/webkit/running_test.txt" 211ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu crashed_test = "" 212ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu while not crashed_test: 213ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu (crashed_test, err) = subprocess.Popen( 214ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu shell_cmd_str, shell=True, stdout=subprocess.PIPE, 215ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu stderr=subprocess.PIPE).communicate() 216ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu crashed_test = crashed_test.strip() 217ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu if not crashed_test: 218ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu logging.error('Cannot get crashed test name, device offline?') 219ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu logging.error('stderr: ' + err) 220ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu logging.error('retrying in 10s...') 221ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu time.sleep(10) 2225080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 2239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(crashed_test + " CRASHED"); 2249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests.append(crashed_test); 2259066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Resuming layout test runner..."); 2279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Call LayoutTestsAutoTest::resumeLayoutTests 228ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd = run_layout_test_cmd_prefix + " -e class com.android.dumprendertree.LayoutTestsAutoTest#resumeLayoutTests" + run_layout_test_cmd_postfix 2299066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 230ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(run_layout_test_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 2319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if adb_output.find('INSTRUMENTATION_FAILED') != -1: 2339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("Error happened : " + adb_output) 2349066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1) 2359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.debug(adb_output); 2379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Done\n"); 2389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2399066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Pull results from /sdcard 2409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir = options.results_directory 2419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not os.path.exists(results_dir): 2429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.makedirs(results_dir) 2439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not os.path.isdir(results_dir): 2449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("Cannot create results dir: " + results_dir); 2459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1); 2469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project result_files = ["/sdcard/layout_tests_passed.txt", 2489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project "/sdcard/layout_tests_failed.txt", 249cf0fd7892b7208ebfa35809b63fc8e4d60e4d466Steve Block "/sdcard/layout_tests_ignored.txt", 2509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project "/sdcard/layout_tests_nontext.txt"] 2515080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu for file in result_files: 2529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project shell_cmd_str = adb_cmd + " pull " + file + " " + results_dir 2539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_output = subprocess.Popen(shell_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 2549066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.debug(adb_output) 2555080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 2569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Create the crash list. 2579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp = open(results_dir + "/layout_tests_crashed.txt", "w"); 2585080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu for crashed_test in crashed_tests: 2595080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu fp.writelines(crashed_test + '\n') 2609066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp.close() 2619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Count the number of tests in each category. 2639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project passed_tests = CountLineNumber(results_dir + "/layout_tests_passed.txt") 2649066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(passed_tests) + " passed") 2659066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project failed_tests = CountLineNumber(results_dir + "/layout_tests_failed.txt") 2669066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(failed_tests) + " failed") 267cf0fd7892b7208ebfa35809b63fc8e4d60e4d466Steve Block ignored_tests = CountLineNumber(results_dir + "/layout_tests_ignored.txt") 268cf0fd7892b7208ebfa35809b63fc8e4d60e4d466Steve Block logging.info(str(ignored_tests) + " ignored results") 2699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests = CountLineNumber(results_dir + "/layout_tests_crashed.txt") 2709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(crashed_tests) + " crashed") 2719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project nontext_tests = CountLineNumber(results_dir + "/layout_tests_nontext.txt") 2729066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(nontext_tests) + " no dumpAsText") 273cf0fd7892b7208ebfa35809b63fc8e4d60e4d466Steve Block logging.info(str(passed_tests + failed_tests + ignored_tests + crashed_tests + nontext_tests) + " TOTAL") 2749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Results are stored under: " + results_dir + "\n") 2769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Comparing results to references to find new fixes and regressions. 2789066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir = os.path.abspath(options.results_directory) 2799066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = options.ref_directory 2809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # if ref_dir is null, cannonify ref_dir to the script dir. 2829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not ref_dir: 2839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project script_self = sys.argv[0] 2849066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project script_dir = os.path.dirname(script_self) 2859066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = os.path.join(script_dir, "results") 2869066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2879066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = os.path.abspath(ref_dir) 2889066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2899066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project CompareResults(ref_dir, results_dir) 2909066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2919066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectif '__main__' == __name__: 2929066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser = optparse.OptionParser() 293ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project option_parser.add_option("", "--rebaseline", action="store_true", 294ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project default=False, 295ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project help="generate expected results for those tests not having one") 2969066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--time-out-ms", 2979066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 2989066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="set the timeout for each test") 2999066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--verbose", action="store_true", 3009066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=False, 3019066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="include debug-level logging") 3029066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--refresh-test-list", action="store_true", 3039066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=False, 3049066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="re-generate test list, it may take some time.") 3059066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--adb-options", 3069066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 3079066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="pass options to adb, such as -d -e, etc"); 3089066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--results-directory", 3099066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default="layout-test-results", 3109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="directory which results are stored.") 3119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--ref-directory", 3129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 3139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project dest="ref_directory", 3149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="directory where reference results are stored.") 31512077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block option_parser.add_option("", "--js-engine", 31612077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block default=None, 31712077e1179b4035ae2b1a44ccf9cd540e14b182eSteve Block help="The JavaScript engine currently in use, which determines which set of Android-specific expected results we should use. Should be 'jsc' or 'v8'."); 3189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 3199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project options, args = option_parser.parse_args(); 3209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project main(options, args) 321