run_layout_tests.py revision 5080b6736e49381aa6b0654067b383b0b67774d6
19066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project#!/usr/bin/python 29066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 39066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project"""Run layout tests using Android emulator and instrumentation. 49066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 59066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project First, you need to get an SD card or sdcard image that has layout tests on it. 69066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Layout tests are in following directory: 79066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project /sdcard/android/layout_tests 89066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project For example, /sdcard/android/layout_tests/fast 99066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Usage: 119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run all tests under fast/ directory: 129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py, or 139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast 149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run all tests under a sub directory: 169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast/dom 179066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Run a single test: 199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project run_layout_tests.py fast/dom/ 209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project After a merge, if there are changes of layout tests in SD card, you need to 229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project use --refresh-test-list option *once* to re-generate test list on the card. 239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Some other options are: 255080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu --rebaseline generates expected layout tests results under /sdcard/android/expected_result/ 269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --time-out-ms (default is 8000 millis) for each test 279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --adb-options="-e" passes option string to adb 289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project --results-directory=..., (default is ./layout-test-results) directory name under which results are stored. 299066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project""" 309066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport logging 329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport optparse 339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport os 349066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport subprocess 359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport sys 369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectimport time 379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef CountLineNumber(filename): 399066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Compute the number of lines in a given file. 409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project filename: a file name related to the current directory. 439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp = open(os.path.abspath(filename), "r"); 469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project lines = 0 479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for line in fp.readlines(): 489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project lines = lines + 1 499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp.close() 509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return lines 519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef DumpRenderTreeFinished(adb_cmd): 539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ Check if DumpRenderTree finished running tests 545080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 559066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project output: adb_cmd string 579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 585080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 59ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project # pull /sdcard/android/running_test.txt, if the content is "#DONE", it's done 60ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project shell_cmd_str = adb_cmd + " shell cat /sdcard/android/running_test.txt" 619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_output = subprocess.Popen(shell_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return adb_output.strip() == "#DONE" 639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 64903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhudef DiffResults(marker, new_results, old_results, diff_results, strip_reason, 65903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu new_count_first=True): 669066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ Given two result files, generate diff and 679066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project write to diff_results file. All arguments are absolute paths 689066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project to files. 699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project old_file = open(old_results, "r") 719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project new_file = open(new_results, "r") 725080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu diff_file = open(diff_results, "a") 739066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Read lines from each file 759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ndict = new_file.readlines() 769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project cdict = old_file.readlines() 779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 789066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Write marker to diff file 799066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines(marker + "\n") 809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines("###############\n") 819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Strip reason from result lines 839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if strip_reason is True: 849066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for i in range(0, len(ndict)): 859066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ndict[i] = ndict[i].split(' ')[0] + "\n" 869066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for i in range(0, len(cdict)): 879066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project cdict[i] = cdict[i].split(' ')[0] + "\n" 889066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 89903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu params = { 90903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu "new": [0, ndict, cdict, "+"], 91903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu "miss": [0, cdict, ndict, "-"] 92903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu } 93903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if new_count_first: 94903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu order = ["new", "miss"] 95903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu else: 96903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu order = ["miss", "new"] 97903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu 98903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu for key in order: 99903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu for line in params[key][1]: 100903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if line not in params[key][2]: 101903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu if line[-1] != "\n": 102903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu line += "\n"; 103903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu diff_file.writelines(params[key][3] + line) 104903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu params[key][0] += 1 105903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu 106903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu logging.info(marker + " >>> " + str(params["new"][0]) + " new, " + 107903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu str(params["miss"][0]) + " misses") 1089066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1099066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.writelines("\n\n") 1109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project old_file.close() 1129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project new_file.close() 1139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project diff_file.close() 1149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project return 1159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef CompareResults(ref_dir, results_dir): 1179066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Compare results in two directories 1189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 1209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir: the reference directory having layout results as references 1219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir: the results directory 1229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 1239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Comparing results to " + ref_dir) 1249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1255080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu diff_result = os.path.join(results_dir, "layout_tests_diff.txt") 1269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if os.path.exists(diff_result): 1279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.remove(diff_result) 1289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 129903f311fdc79b8dc804b63d0e397453d9b0e8d44Guang Zhu files=["crashed", "failed", "passed", "nontext"] 1309066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project for f in files: 1319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project result_file_name = "layout_tests_" + f + ".txt" 1329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project DiffResults(f, os.path.join(results_dir, result_file_name), 1339066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.path.join(ref_dir, result_file_name), diff_result, 134595fbd6ea22708504dc9e24b44fa5eb357a576ecGuang Zhu False, f != "passed") 1359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Detailed diffs are in " + diff_result) 1369066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1379066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectdef main(options, args): 1389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """Run the tests. Will call sys.exit when complete. 1395080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 1409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project Args: 1419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project options: a dictionary of command line options 1429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project args: a list of sub directories or files to test 1439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project """ 1449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Set up logging format. 1469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project log_level = logging.INFO 1479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.verbose: 1489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project log_level = logging.DEBUG 1499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.basicConfig(level=log_level, 1509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project format='%(message)s') 1519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Include all tests if none are specified. 1539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not args: 154ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project path = '/'; 1559066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project else: 1569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project path = ' '.join(args); 1579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1589066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_cmd = "adb "; 1599066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.adb_options: 1609066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_cmd += options.adb_options 1619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Re-generate the test list if --refresh-test-list is on 1639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.refresh_test_list: 1649066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Generating test list."); 165ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project generate_test_list_cmd_str = adb_cmd + " shell am instrument -e class com.android.dumprendertree.LayoutTestsAutoTest#generateTestList -e path \"" + path + "\" -w com.android.dumprendertree/.LayoutTestsAutoRunner" 166ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(generate_test_list_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 1679066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1689066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if adb_output.find('Process crashed') != -1: 1699066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Aborting because cannot generate test list.\n" + adb_output) 1709066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1) 1719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1729066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1739066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Running tests") 1749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Count crashed tests. 1769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests = [] 1779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 178ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project timeout_ms = '5000' 1799066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if options.time_out_ms: 1809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project timeout_ms = options.time_out_ms 1819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 1829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Run test until it's done 1839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 184ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_prefix = adb_cmd + " shell am instrument" 185ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 186ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix = " -e path \"" + path + "\" -e timeout " + timeout_ms 187ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project if options.rebaseline: 188ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix += " -e rebaseline true" 189ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd_postfix += " -w com.android.dumprendertree/.LayoutTestsAutoRunner" 190ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 1919066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Call LayoutTestsAutoTest::startLayoutTests. 192ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd = run_layout_test_cmd_prefix + " -e class com.android.dumprendertree.LayoutTestsAutoTest#startLayoutTests" + run_layout_test_cmd_postfix 193ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project 194ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(run_layout_test_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 1959066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project while not DumpRenderTreeFinished(adb_cmd): 1969066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Get the running_test.txt 1979066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("DumpRenderTree crashed, output:\n" + adb_output) 1989066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 199ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project shell_cmd_str = adb_cmd + " shell cat /sdcard/android/running_test.txt" 2009066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_test = subprocess.Popen(shell_cmd_str, shell=True, stdout=subprocess.PIPE).communicate()[0] 2015080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 2029066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(crashed_test + " CRASHED"); 2039066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests.append(crashed_test); 2049066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2059066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Resuming layout test runner..."); 2069066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Call LayoutTestsAutoTest::resumeLayoutTests 207ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project run_layout_test_cmd = run_layout_test_cmd_prefix + " -e class com.android.dumprendertree.LayoutTestsAutoTest#resumeLayoutTests" + run_layout_test_cmd_postfix 2089066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 209ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project adb_output = subprocess.Popen(run_layout_test_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 2109066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2119066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if adb_output.find('INSTRUMENTATION_FAILED') != -1: 2129066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("Error happened : " + adb_output) 2139066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1) 2149066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2159066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.debug(adb_output); 2169066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Done\n"); 2179066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2189066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Pull results from /sdcard 2199066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir = options.results_directory 2209066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not os.path.exists(results_dir): 2219066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project os.makedirs(results_dir) 2229066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not os.path.isdir(results_dir): 2239066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.error("Cannot create results dir: " + results_dir); 2249066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project sys.exit(1); 2259066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2269066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project result_files = ["/sdcard/layout_tests_passed.txt", 2279066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project "/sdcard/layout_tests_failed.txt", 2289066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project "/sdcard/layout_tests_nontext.txt"] 2295080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu for file in result_files: 2309066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project shell_cmd_str = adb_cmd + " pull " + file + " " + results_dir 2319066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project adb_output = subprocess.Popen(shell_cmd_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0] 2329066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.debug(adb_output) 2335080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu 2349066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Create the crash list. 2359066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp = open(results_dir + "/layout_tests_crashed.txt", "w"); 2365080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu for crashed_test in crashed_tests: 2375080b6736e49381aa6b0654067b383b0b67774d6Guang Zhu fp.writelines(crashed_test + '\n') 2389066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project fp.close() 2399066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2409066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Count the number of tests in each category. 2419066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project passed_tests = CountLineNumber(results_dir + "/layout_tests_passed.txt") 2429066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(passed_tests) + " passed") 2439066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project failed_tests = CountLineNumber(results_dir + "/layout_tests_failed.txt") 2449066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(failed_tests) + " failed") 2459066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project crashed_tests = CountLineNumber(results_dir + "/layout_tests_crashed.txt") 2469066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(crashed_tests) + " crashed") 2479066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project nontext_tests = CountLineNumber(results_dir + "/layout_tests_nontext.txt") 2489066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info(str(nontext_tests) + " no dumpAsText") 2499066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2509066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project logging.info("Results are stored under: " + results_dir + "\n") 2519066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2529066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # Comparing results to references to find new fixes and regressions. 2539066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project results_dir = os.path.abspath(options.results_directory) 2549066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = options.ref_directory 2559066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2569066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project # if ref_dir is null, cannonify ref_dir to the script dir. 2579066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project if not ref_dir: 2589066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project script_self = sys.argv[0] 2599066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project script_dir = os.path.dirname(script_self) 2609066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = os.path.join(script_dir, "results") 2619066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2629066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project ref_dir = os.path.abspath(ref_dir) 2639066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2649066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project CompareResults(ref_dir, results_dir) 2659066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2669066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Projectif '__main__' == __name__: 2679066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser = optparse.OptionParser() 268ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project option_parser.add_option("", "--rebaseline", action="store_true", 269ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project default=False, 270ba87e3e6c985e7175152993b5efcc7dd2f0e1c93The Android Open Source Project help="generate expected results for those tests not having one") 2719066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--time-out-ms", 2729066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 2739066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="set the timeout for each test") 2749066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--verbose", action="store_true", 2759066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=False, 2769066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="include debug-level logging") 2779066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--refresh-test-list", action="store_true", 2789066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=False, 2799066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="re-generate test list, it may take some time.") 2809066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--adb-options", 2819066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 2829066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="pass options to adb, such as -d -e, etc"); 2839066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--results-directory", 2849066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default="layout-test-results", 2859066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="directory which results are stored.") 2869066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project option_parser.add_option("", "--ref-directory", 2879066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project default=None, 2889066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project dest="ref_directory", 2899066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project help="directory where reference results are stored.") 2909066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project 2919066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project options, args = option_parser.parse_args(); 2929066cfe9886ac131c34d59ed0e2d287b0e3c0087The Android Open Source Project main(options, args) 293