run_layout_tests.py revision ad1e25da74517343b306f5fe72d42b2ccb59ec6e
19066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project#!/usr/bin/python 29066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 39066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project"""Run layout tests using Android emulator and instrumentation. 49066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 59066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project First, you need to get an SD card or sdcard image that has layout tests on it. 69066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Layout tests are in following directory: 79066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project /sdcard/android/layout_tests 89066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project For example, /sdcard/android/layout_tests/fast 99066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Usage: 119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run all tests under fast/ directory: 129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py, or 139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast 149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run all tests under a sub directory: 169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast/dom 179066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run a single test: 199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast/dom/ 209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project After a merge, if there are changes of layout tests in SD card, you need to 229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project use --refresh-test-list option *once* to re-generate test list on the card. 239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Some other options are: 255080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu --rebaseline generates expected layout tests results under /sdcard/android/expected_result/ 269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --time-out-ms (default is 8000 millis) for each test 279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --adb-options="-e" passes option string to adb 289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --results-directory=..., (default is ./layout-test-results) directory name under which results are stored. 299066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project""" 309066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport logging 329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport optparse 339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport os 349066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport subprocess 359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport sys 369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport time 379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef CountLineNumber(filename): 399066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Compute the number of lines in a given file. 409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project filename: a file name related to the current directory. 439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp = open(os.path.abspath(filename), "r"); 469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project lines = 0 479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for line in fp.readlines(): 489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project lines = lines + 1 499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp.close() 509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return lines 519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef DumpRenderTreeFinished(adb_cmd): 539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ Check if DumpRenderTree finished running tests 545080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 559066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project output: adb_cmd string 579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 585080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 59ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project # pull /sdcard/android/running_test.txt, if the content is "#DONE", it's done 60ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project shell_cmd_str = adb_cmd + " shell cat /sdcard/android/running_test.txt" 619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_output = subprocess.Popen(shell_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return adb_output.strip() == "#DONE" 639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 64903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhudef DiffResults(marker, new_results, old_results, diff_results, strip_reason, 65903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu new_count_first=True): 669066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ Given two result files, generate diff and 679066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project write to diff_results file. All arguments are absolute paths 689066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project to files. 699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project old_file = open(old_results, "r") 719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project new_file = open(new_results, "r") 725080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu diff_file = open(diff_results, "a") 739066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Read lines from each file 759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ndict = new_file.readlines() 769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project cdict = old_file.readlines() 779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 789066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Write marker to diff file 799066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines(marker + "\n") 809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines("###############\n") 819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Strip reason from result lines 839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if strip_reason is True: 849066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for i in range(0, len(ndict)): 859066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ndict[i] = ndict[i].split(' ')[0] + "\n" 869066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for i in range(0, len(cdict)): 879066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project cdict[i] = cdict[i].split(' ')[0] + "\n" 889066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 89903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu params = { 90903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu "new": [0, ndict, cdict, "+"], 91903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu "miss": [0, cdict, ndict, "-"] 92903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu } 93903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if new_count_first: 94903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu order = ["new", "miss"] 95903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu else: 96903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu order = ["miss", "new"] 97903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu 98903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu for key in order: 99903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu for line in params[key][1]: 100903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if line not in params[key][2]: 101903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if line[-1] != "\n": 102903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu line += "\n"; 103903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu diff_file.writelines(params[key][3] + line) 104903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu params[key][0] += 1 105903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu 106903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu logging.info(marker + " >>> " + str(params["new"][0]) + " new, " + 107903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu str(params["miss"][0]) + " misses") 1089066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1099066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines("\n\n") 1109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project old_file.close() 1129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project new_file.close() 1139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.close() 1149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return 1159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef CompareResults(ref_dir, results_dir): 1179066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Compare results in two directories 1189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 1209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir: the reference directory having layout results as references 1219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir: the results directory 1229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 1239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Comparing results to " + ref_dir) 1249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1255080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu diff_result = os.path.join(results_dir, "layout_tests_diff.txt") 1269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if os.path.exists(diff_result): 1279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.remove(diff_result) 1289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 129903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu files=["crashed", "failed", "passed", "nontext"] 1309066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for f in files: 1319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project result_file_name = "layout_tests_" + f + ".txt" 1329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project DiffResults(f, os.path.join(results_dir, result_file_name), 1339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.path.join(ref_dir, result_file_name), diff_result, 134595fbd6ea22708504dc9e24b44fa5eb357a576ecGuang Zhu False, f != "passed") 1359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Detailed diffs are in " + diff_result) 1369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef main(options, args): 1389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Run the tests. Will call sys.exit when complete. 1395080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 1409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 1419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project options: a dictionary of command line options 1429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project args: a list of sub directories or files to test 1439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 1449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Set up logging format. 1469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project log_level = logging.INFO 1479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.verbose: 1489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project log_level = logging.DEBUG 1499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.basicConfig(level=log_level, 1509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project format='%(message)s') 1519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Include all tests if none are specified. 1539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not args: 154ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project path = '/'; 1559066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project else: 1569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project path = ' '.join(args); 1579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1589066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_cmd = "adb "; 1599066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.adb_options: 1609066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_cmd += options.adb_options 1619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Re-generate the test list if --refresh-test-list is on 1639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.refresh_test_list: 1649066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Generating test list."); 165ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project generate_test_list_cmd_str = adb_cmd + " shell am instrument -e class com.android.dumprendertree.LayoutTestsAutoTest#generateTestList -e path \"" + path + "\" -w com.android.dumprendertree/.LayoutTestsAutoRunner" 166ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(generate_test_list_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 1679066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1689066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if adb_output.find('Process crashed') != -1: 1699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Aborting because cannot generate test list.\n" + adb_output) 1709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1) 1719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1729066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1739066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Running tests") 1749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Count crashed tests. 1769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests = [] 1779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 17838323a5f9f4a53b0118610ac84a47cb9907d4e4bGuang Zhu timeout_ms = '30000' 1799066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.time_out_ms: 1809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project timeout_ms = options.time_out_ms 1819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Run test until it's done 1839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 184ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_prefix = adb_cmd + " shell am instrument" 185ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 186ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix = " -e path \"" + path + "\" -e timeout " + timeout_ms 187ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project if options.rebaseline: 188ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix += " -e rebaseline true" 189ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix += " -w com.android.dumprendertree/.LayoutTestsAutoRunner" 190ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 1919066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Call LayoutTestsAutoTest::startLayoutTests. 192ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd = run_layout_test_cmd_prefix + " -e class com.android.dumprendertree.LayoutTestsAutoTest#startLayoutTests" + run_layout_test_cmd_postfix 193ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 194ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(run_layout_test_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 1959066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project while not DumpRenderTreeFinished(adb_cmd): 1969066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Get the running_test.txt 1979066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("DumpRenderTree crashed, output:\n" + adb_output) 1989066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 199ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project shell_cmd_str = adb_cmd + " shell cat /sdcard/android/running_test.txt" 200ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu crashed_test = "" 201ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu while not crashed_test: 202ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu (crashed_test, err) = subprocess.Popen( 203ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu shell_cmd_str, shell=True, stdout=subprocess.PIPE, 204ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu stderr=subprocess.PIPE).communicate() 205ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu crashed_test = crashed_test.strip() 206ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu if not crashed_test: 207ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu logging.error('Cannot get crashed test name, device offline?') 208ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu logging.error('stderr: ' + err) 209ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu logging.error('retrying in 10s...') 210ad1e25da74517343b306f5fe72d42b2ccb59ec6eGuang Zhu time.sleep(10) 2115080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 2129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(crashed_test + " CRASHED"); 2139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests.append(crashed_test); 2149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Resuming layout test runner..."); 2169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Call LayoutTestsAutoTest::resumeLayoutTests 217ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd = run_layout_test_cmd_prefix + " -e class com.android.dumprendertree.LayoutTestsAutoTest#resumeLayoutTests" + run_layout_test_cmd_postfix 2189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 219ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(run_layout_test_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 2209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if adb_output.find('INSTRUMENTATION_FAILED') != -1: 2229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("Error happened : " + adb_output) 2239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1) 2249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2259066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.debug(adb_output); 2269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Done\n"); 2279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Pull results from /sdcard 2299066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir = options.results_directory 2309066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not os.path.exists(results_dir): 2319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.makedirs(results_dir) 2329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not os.path.isdir(results_dir): 2339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("Cannot create results dir: " + results_dir); 2349066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1); 2359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project result_files = ["/sdcard/layout_tests_passed.txt", 2379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project "/sdcard/layout_tests_failed.txt", 2389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project "/sdcard/layout_tests_nontext.txt"] 2395080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu for file in result_files: 2409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project shell_cmd_str = adb_cmd + " pull " + file + " " + results_dir 2419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_output = subprocess.Popen(shell_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 2429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.debug(adb_output) 2435080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 2449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Create the crash list. 2459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp = open(results_dir + "/layout_tests_crashed.txt", "w"); 2465080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu for crashed_test in crashed_tests: 2475080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu fp.writelines(crashed_test + '\n') 2489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp.close() 2499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Count the number of tests in each category. 2519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project passed_tests = CountLineNumber(results_dir + "/layout_tests_passed.txt") 2529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(passed_tests) + " passed") 2539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project failed_tests = CountLineNumber(results_dir + "/layout_tests_failed.txt") 2549066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(failed_tests) + " failed") 2559066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests = CountLineNumber(results_dir + "/layout_tests_crashed.txt") 2569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(crashed_tests) + " crashed") 2579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project nontext_tests = CountLineNumber(results_dir + "/layout_tests_nontext.txt") 2589066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(nontext_tests) + " no dumpAsText") 2599066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2609066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Results are stored under: " + results_dir + "\n") 2619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Comparing results to references to find new fixes and regressions. 2639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir = os.path.abspath(options.results_directory) 2649066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = options.ref_directory 2659066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2669066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # if ref_dir is null, cannonify ref_dir to the script dir. 2679066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not ref_dir: 2689066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project script_self = sys.argv[0] 2699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project script_dir = os.path.dirname(script_self) 2709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = os.path.join(script_dir, "results") 2719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2729066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = os.path.abspath(ref_dir) 2739066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project CompareResults(ref_dir, results_dir) 2759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectif '__main__' == __name__: 2779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser = optparse.OptionParser() 278ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project option_parser.add_option("", "--rebaseline", action="store_true", 279ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project default=False, 280ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project help="generate expected results for those tests not having one") 2819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--time-out-ms", 2829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 2839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="set the timeout for each test") 2849066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--verbose", action="store_true", 2859066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=False, 2869066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="include debug-level logging") 2879066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--refresh-test-list", action="store_true", 2889066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=False, 2899066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="re-generate test list, it may take some time.") 2909066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--adb-options", 2919066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 2929066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="pass options to adb, such as -d -e, etc"); 2939066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--results-directory", 2949066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default="layout-test-results", 2959066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="directory which results are stored.") 2969066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--ref-directory", 2979066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 2989066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project dest="ref_directory", 2999066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="directory where reference results are stored.") 3009066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 3019066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project options, args = option_parser.parse_args(); 3029066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project main(options, args) 303