gtest_filter_unittest.py revision dd1c93d5709e32713961cfd95fe30489a4ad2d26
1dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter#!/usr/bin/env python 2dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# 3dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Copyright 2005 Google Inc. All Rights Reserved. 4dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# 5dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Redistribution and use in source and binary forms, with or without 6dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# modification, are permitted provided that the following conditions are 7dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# met: 8dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# 9dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# * Redistributions of source code must retain the above copyright 10dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# notice, this list of conditions and the following disclaimer. 11dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# * Redistributions in binary form must reproduce the above 12dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# copyright notice, this list of conditions and the following disclaimer 13dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# in the documentation and/or other materials provided with the 14dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# distribution. 15dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# * Neither the name of Google Inc. nor the names of its 16dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# contributors may be used to endorse or promote products derived from 17dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# this software without specific prior written permission. 18dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# 19dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 31dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter"""Unit test for Google Test test filters. 32dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 33dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterA user can specify which test(s) in a Google Test program to run via either 34dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterthe GTEST_FILTER environment variable or the --gtest_filter flag. 35dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterThis script tests such functionality by invoking 36dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixtergtest_filter_unittest_ (a program written with Google Test) with different 37dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterenvironments and command line flags. 38dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 39dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterNote that test sharding may also influence which tests are filtered. Therefore, 40dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterwe test that here also. 41dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter""" 42dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 43dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter__author__ = 'wan@google.com (Zhanyong Wan)' 44dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 45dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterimport os 46dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterimport re 47dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterimport sets 48dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterimport gtest_test_utils 49dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 50dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Constants. 51dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 52dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterIS_WINDOWS = os.name == 'nt' 53dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 54dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# The environment variable for specifying the test filters. 55dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterFILTER_ENV_VAR = 'GTEST_FILTER' 56dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 57dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# The environment variables for test sharding. 58dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterTOTAL_SHARDS_ENV_VAR = 'GTEST_TOTAL_SHARDS' 59dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterSHARD_INDEX_ENV_VAR = 'GTEST_SHARD_INDEX' 60dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterSHARD_STATUS_FILE_ENV_VAR = 'GTEST_SHARD_STATUS_FILE' 61dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 62dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# The command line flag for specifying the test filters. 63dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterFILTER_FLAG = 'gtest_filter' 64dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 65dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# The command line flag for including disabled tests. 66dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterALSO_RUN_DISABED_TESTS_FLAG = 'gtest_also_run_disabled_tests' 67dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 68dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Command to run the gtest_filter_unittest_ program. 69dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterCOMMAND = gtest_test_utils.GetTestExecutablePath('gtest_filter_unittest_') 70dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 71dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Regex for determining whether parameterized tests are enabled in the binary. 72dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterPARAM_TEST_REGEX = re.compile(r'/ParamTest') 73dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 74dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Regex for parsing test case names from Google Test's output. 75dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterTEST_CASE_REGEX = re.compile(r'^\[\-+\] \d+ tests? from (\w+(/\w+)?)') 76dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 77dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Regex for parsing test names from Google Test's output. 78dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterTEST_REGEX = re.compile(r'^\[\s*RUN\s*\].*\.(\w+(/\w+)?)') 79dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 80dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Full names of all tests in gtest_filter_unittests_. 81dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterPARAM_TESTS = [ 82dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestX/0', 83dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestX/1', 84dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestY/0', 85dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestY/1', 86dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqQ/ParamTest.TestX/0', 87dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqQ/ParamTest.TestX/1', 88dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqQ/ParamTest.TestY/0', 89dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqQ/ParamTest.TestY/1', 90dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ] 91dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 92dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterDISABLED_TESTS = [ 93dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.DISABLED_TestFour', 94dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.DISABLED_TestFive', 95dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.DISABLED_TestC', 96dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'DISABLED_FoobarTest.Test1', 97dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'DISABLED_FoobarTest.DISABLED_Test2', 98dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'DISABLED_FoobarbazTest.TestA', 99dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ] 100dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 101dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# All the non-disabled tests. 102dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken MixterACTIVE_TESTS = [ 103dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Abc', 104dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 105dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 106dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestOne', 107dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestTwo', 108dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestThree', 109dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 110dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 111dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 112dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestB', 113dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 114dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'HasDeathTest.Test1', 115dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'HasDeathTest.Test2', 116dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ] + PARAM_TESTS 117dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 118dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterparam_tests_present = None 119dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 120dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# Utilities. 121dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 122dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 123dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterdef SetEnvVar(env_var, value): 124dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Sets the env variable to 'value'; unsets it when 'value' is None.""" 125dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 126dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if value is not None: 127dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter os.environ[env_var] = value 128dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter elif env_var in os.environ: 129dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter del os.environ[env_var] 130dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 131dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 132dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterdef Run(command): 133dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Runs a test program and returns its exit code and a list of tests run.""" 134dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 135dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter stdout_file = os.popen(command, 'r') 136dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_run = [] 137dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter test_case = '' 138dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter test = '' 139dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter for line in stdout_file: 140dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter match = TEST_CASE_REGEX.match(line) 141dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if match is not None: 142dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter test_case = match.group(1) 143dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter else: 144dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter match = TEST_REGEX.match(line) 145dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if match is not None: 146dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter test = match.group(1) 147dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_run += [test_case + '.' + test] 148dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter exit_code = stdout_file.close() 149dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter return (tests_run, exit_code) 150dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 151dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 152dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterdef InvokeWithModifiedEnv(extra_env, function, *args, **kwargs): 153dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Runs the given function and arguments in a modified environment.""" 154dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter try: 155dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter original_env = os.environ.copy() 156dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter os.environ.update(extra_env) 157dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter return function(*args, **kwargs) 158dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter finally: 159dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter for key in extra_env.iterkeys(): 160dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if key in original_env: 161dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter os.environ[key] = original_env[key] 162dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter else: 163dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter del os.environ[key] 164dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 165dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 166dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterdef RunWithSharding(total_shards, shard_index, command): 167dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Runs a test program shard and returns exit code and a list of tests run.""" 168dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 169dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter extra_env = {SHARD_INDEX_ENV_VAR: str(shard_index), 170dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter TOTAL_SHARDS_ENV_VAR: str(total_shards)} 171dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter return InvokeWithModifiedEnv(extra_env, Run, command) 172dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 173dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter# The unit test. 174dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 175dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 176dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterclass GTestFilterUnitTest(gtest_test_utils.TestCase): 177dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests GTEST_FILTER env variable or --gtest_filter flag to filter tests.""" 178dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 179dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Utilities. 180dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 181dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def AssertSetEqual(self, lhs, rhs): 182dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Asserts that two sets are equal.""" 183dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 184dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter for elem in lhs: 185dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assert_(elem in rhs, '%s in %s' % (elem, rhs)) 186dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 187dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter for elem in rhs: 188dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assert_(elem in lhs, '%s in %s' % (elem, lhs)) 189dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 190dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def AssertPartitionIsValid(self, set_var, list_of_sets): 191dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Asserts that list_of_sets is a valid partition of set_var.""" 192dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 193dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter full_partition = [] 194dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter for slice_var in list_of_sets: 195dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter full_partition.extend(slice_var) 196dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assertEqual(len(set_var), len(full_partition)) 197dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assertEqual(sets.Set(set_var), sets.Set(full_partition)) 198dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 199dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def AdjustForParameterizedTests(self, tests_to_run): 200dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Adjust tests_to_run in case value parameterized tests are disabled.""" 201dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 202dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter global param_tests_present 203dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if not param_tests_present: 204dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter return list(sets.Set(tests_to_run) - sets.Set(PARAM_TESTS)) 205dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter else: 206dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter return tests_to_run 207dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 208dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def RunAndVerify(self, gtest_filter, tests_to_run): 209dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Checks that the binary runs correct set of tests for the given filter.""" 210dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 211dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_to_run = self.AdjustForParameterizedTests(tests_to_run) 212dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 213dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # First, tests using GTEST_FILTER. 214dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 215dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Windows removes empty variables from the environment when passing it 216dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # to a new process. This means it is impossible to pass an empty filter 217dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # into a process using the GTEST_FILTER environment variable. However, 218dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # we can still test the case when the variable is not supplied (i.e., 219dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # gtest_filter is None). 220dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # pylint: disable-msg=C6403 221dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if not IS_WINDOWS or gtest_filter != '': 222dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter SetEnvVar(FILTER_ENV_VAR, gtest_filter) 223dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_run = Run(COMMAND)[0] 224dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter SetEnvVar(FILTER_ENV_VAR, None) 225dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.AssertSetEqual(tests_run, tests_to_run) 226dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # pylint: enable-msg=C6403 227dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 228dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Next, tests using --gtest_filter. 229dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 230dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if gtest_filter is None: 231dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter command = COMMAND 232dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter else: 233dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter command = '%s --%s=%s' % (COMMAND, FILTER_FLAG, gtest_filter) 234dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 235dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_run = Run(command)[0] 236dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.AssertSetEqual(tests_run, tests_to_run) 237dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 238dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def RunAndVerifyWithSharding(self, gtest_filter, total_shards, tests_to_run, 239dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter command=COMMAND, check_exit_0=False): 240dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Checks that binary runs correct tests for the given filter and shard. 241dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 242dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter Runs all shards of gtest_filter_unittest_ with the given filter, and 243dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter verifies that the right set of tests were run. The union of tests run 244dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter on each shard should be identical to tests_to_run, without duplicates. 245dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 246dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter Args: 247dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter gtest_filter: A filter to apply to the tests. 248dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter total_shards: A total number of shards to split test run into. 249dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_to_run: A set of tests expected to run. 250dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter command: A command to invoke the test binary. 251dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter check_exit_0: When set to a true value, make sure that all shards 252dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter return 0. 253dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """ 254dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 255dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_to_run = self.AdjustForParameterizedTests(tests_to_run) 256dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 257dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Windows removes empty variables from the environment when passing it 258dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # to a new process. This means it is impossible to pass an empty filter 259dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # into a process using the GTEST_FILTER environment variable. However, 260dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # we can still test the case when the variable is not supplied (i.e., 261dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # gtest_filter is None). 262dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # pylint: disable-msg=C6403 263dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if not IS_WINDOWS or gtest_filter != '': 264dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter SetEnvVar(FILTER_ENV_VAR, gtest_filter) 265dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter partition = [] 266dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter for i in range(0, total_shards): 267dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter (tests_run, exit_code) = RunWithSharding(total_shards, i, command) 268dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if check_exit_0: 269dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assert_(exit_code is None) 270dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter partition.append(tests_run) 271dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 272dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.AssertPartitionIsValid(tests_to_run, partition) 273dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter SetEnvVar(FILTER_ENV_VAR, None) 274dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # pylint: enable-msg=C6403 275dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 276dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def RunAndVerifyAllowingDisabled(self, gtest_filter, tests_to_run): 277dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Checks that the binary runs correct set of tests for the given filter. 278dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 279dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter Runs gtest_filter_unittest_ with the given filter, and enables 280dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter disabled tests. Verifies that the right set of tests were run. 281dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 282dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter Args: 283dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter gtest_filter: A filter to apply to the tests. 284dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_to_run: A set of tests expected to run. 285dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """ 286dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 287dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_to_run = self.AdjustForParameterizedTests(tests_to_run) 288dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 289dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Construct the command line. 290dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter command = '%s --%s' % (COMMAND, ALSO_RUN_DISABED_TESTS_FLAG) 291dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if gtest_filter is not None: 292dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter command = '%s --%s=%s' % (command, FILTER_FLAG, gtest_filter) 293dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 294dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_run = Run(command)[0] 295dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.AssertSetEqual(tests_run, tests_to_run) 296dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 297dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def setUp(self): 298dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Sets up test case. 299dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 300dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter Determines whether value-parameterized tests are enabled in the binary and 301dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter sets the flags accordingly. 302dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """ 303dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 304dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter global param_tests_present 305dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter if param_tests_present is None: 306dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter param_tests_present = PARAM_TEST_REGEX.search( 307dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter '\n'.join(os.popen(COMMAND, 'r').readlines())) is not None 308dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 309dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testDefaultBehavior(self): 310dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests the behavior of not specifying the filter.""" 311dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 312dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify(None, ACTIVE_TESTS) 313dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 314dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testDefaultBehaviorWithShards(self): 315dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests the behavior without the filter, with sharding enabled.""" 316dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 317dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding(None, 1, ACTIVE_TESTS) 318dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding(None, 2, ACTIVE_TESTS) 319dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding(None, len(ACTIVE_TESTS) - 1, ACTIVE_TESTS) 320dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding(None, len(ACTIVE_TESTS), ACTIVE_TESTS) 321dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding(None, len(ACTIVE_TESTS) + 1, ACTIVE_TESTS) 322dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 323dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testEmptyFilter(self): 324dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests an empty filter.""" 325dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 326dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('', []) 327dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding('', 1, []) 328dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding('', 2, []) 329dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 330dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testBadFilter(self): 331dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests a filter that matches nothing.""" 332dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 333dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('BadFilter', []) 334dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('BadFilter', []) 335dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 336dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testFullName(self): 337dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests filtering by full name.""" 338dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 339dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('FooTest.Xyz', ['FooTest.Xyz']) 340dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('FooTest.Xyz', ['FooTest.Xyz']) 341dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding('FooTest.Xyz', 5, ['FooTest.Xyz']) 342dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 343dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testUniversalFilters(self): 344dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests filters that match everything.""" 345dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 346dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*', ACTIVE_TESTS) 347dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*.*', ACTIVE_TESTS) 348dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding('*.*', len(ACTIVE_TESTS) - 3, ACTIVE_TESTS) 349dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('*', ACTIVE_TESTS + DISABLED_TESTS) 350dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('*.*', ACTIVE_TESTS + DISABLED_TESTS) 351dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 352dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testFilterByTestCase(self): 353dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests filtering by test case name.""" 354dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 355dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('FooTest.*', ['FooTest.Abc', 'FooTest.Xyz']) 356dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 357dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter BAZ_TESTS = ['BazTest.TestOne', 'BazTest.TestA', 'BazTest.TestB'] 358dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('BazTest.*', BAZ_TESTS) 359dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('BazTest.*', 360dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter BAZ_TESTS + ['BazTest.DISABLED_TestC']) 361dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 362dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testFilterByTest(self): 363dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests filtering by test name.""" 364dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 365dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*.TestOne', ['BarTest.TestOne', 'BazTest.TestOne']) 366dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 367dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testFilterDisabledTests(self): 368dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Select only the disabled tests to run.""" 369dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 370dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('DISABLED_FoobarTest.Test1', []) 371dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('DISABLED_FoobarTest.Test1', 372dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ['DISABLED_FoobarTest.Test1']) 373dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 374dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*DISABLED_*', []) 375dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('*DISABLED_*', DISABLED_TESTS) 376dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 377dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*.DISABLED_*', []) 378dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('*.DISABLED_*', [ 379dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.DISABLED_TestFour', 380dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.DISABLED_TestFive', 381dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.DISABLED_TestC', 382dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'DISABLED_FoobarTest.DISABLED_Test2', 383dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 384dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 385dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('DISABLED_*', []) 386dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyAllowingDisabled('DISABLED_*', [ 387dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'DISABLED_FoobarTest.Test1', 388dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'DISABLED_FoobarTest.DISABLED_Test2', 389dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'DISABLED_FoobarbazTest.TestA', 390dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 391dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 392dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testWildcardInTestCaseName(self): 393dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests using wildcard in the test case name.""" 394dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 395dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*a*.*', [ 396dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestOne', 397dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestTwo', 398dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestThree', 399dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 400dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 401dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 402dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestB', 403dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 404dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'HasDeathTest.Test1', 405dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'HasDeathTest.Test2', ] + PARAM_TESTS) 406dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 407dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testWildcardInTestName(self): 408dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests using wildcard in the test name.""" 409dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 410dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*.*A*', ['FooTest.Abc', 'BazTest.TestA']) 411dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 412dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testFilterWithoutDot(self): 413dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests a filter that has no '.' in it.""" 414dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 415dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*z*', [ 416dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 417dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 418dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 419dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 420dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestB', 421dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 422dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 423dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testTwoPatterns(self): 424dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests filters that consist of two patterns.""" 425dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 426dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('Foo*.*:*A*', [ 427dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Abc', 428dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 429dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 430dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 431dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 432dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 433dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # An empty pattern + a non-empty one 434dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify(':*A*', ['FooTest.Abc', 'BazTest.TestA']) 435dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 436dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testThreePatterns(self): 437dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests filters that consist of three patterns.""" 438dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 439dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*oo*:*A*:*One', [ 440dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Abc', 441dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 442dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 443dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestOne', 444dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 445dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 446dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 447dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 448dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 449dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # The 2nd pattern is empty. 450dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*oo*::*One', [ 451dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Abc', 452dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 453dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 454dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestOne', 455dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 456dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 457dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 458dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 459dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # The last 2 patterns are empty. 460dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*oo*::', [ 461dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Abc', 462dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 463dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 464dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 465dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testNegativeFilters(self): 466dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*-HasDeathTest.Test1', [ 467dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Abc', 468dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 469dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 470dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestOne', 471dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestTwo', 472dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestThree', 473dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 474dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 475dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 476dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestB', 477dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 478dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'HasDeathTest.Test2', 479dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ] + PARAM_TESTS) 480dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 481dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*-FooTest.Abc:HasDeathTest.*', [ 482dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'FooTest.Xyz', 483dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 484dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestOne', 485dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestTwo', 486dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestThree', 487dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 488dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 489dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 490dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestB', 491dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ] + PARAM_TESTS) 492dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 493dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('BarTest.*-BarTest.TestOne', [ 494dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestTwo', 495dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestThree', 496dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 497dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 498dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Tests without leading '*'. 499dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('-FooTest.Abc:FooTest.Xyz:HasDeathTest.*', [ 500dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestOne', 501dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestTwo', 502dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BarTest.TestThree', 503dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 504dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestOne', 505dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestA', 506dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'BazTest.TestB', 507dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ] + PARAM_TESTS) 508dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 509dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Value parameterized tests. 510dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*/*', PARAM_TESTS) 511dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 512dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Value parameterized tests filtering by the sequence name. 513dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('SeqP/*', [ 514dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestX/0', 515dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestX/1', 516dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestY/0', 517dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestY/1', 518dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 519dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 520dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter # Value parameterized tests filtering by the test name. 521dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerify('*/0', [ 522dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestX/0', 523dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestY/0', 524dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqQ/ParamTest.TestX/0', 525dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqQ/ParamTest.TestY/0', 526dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ]) 527dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 528dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testFlagOverridesEnvVar(self): 529dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests that the filter flag overrides the filtering env. variable.""" 530dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 531dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter SetEnvVar(FILTER_ENV_VAR, 'Foo*') 532dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter command = '%s --%s=%s' % (COMMAND, FILTER_FLAG, '*One') 533dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter tests_run = Run(command)[0] 534dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter SetEnvVar(FILTER_ENV_VAR, None) 535dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 536dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.AssertSetEqual(tests_run, ['BarTest.TestOne', 'BazTest.TestOne']) 537dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 538dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testShardStatusFileIsCreated(self): 539dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests that the shard file is created if specified in the environment.""" 540dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 541dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter shard_status_file = os.path.join(gtest_test_utils.GetTempDir(), 542dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'shard_status_file') 543dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assert_(not os.path.exists(shard_status_file)) 544dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 545dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter extra_env = {SHARD_STATUS_FILE_ENV_VAR: shard_status_file} 546dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter stdout_file = InvokeWithModifiedEnv(extra_env, os.popen, COMMAND, 'r') 547dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter try: 548dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter stdout_file.readlines() 549dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter finally: 550dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter stdout_file.close() 551dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assert_(os.path.exists(shard_status_file)) 552dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter os.remove(shard_status_file) 553dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 554dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testShardStatusFileIsCreatedWithListTests(self): 555dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests that the shard file is created with --gtest_list_tests.""" 556dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 557dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter shard_status_file = os.path.join(gtest_test_utils.GetTempDir(), 558dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'shard_status_file2') 559dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assert_(not os.path.exists(shard_status_file)) 560dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 561dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter extra_env = {SHARD_STATUS_FILE_ENV_VAR: shard_status_file} 562dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter stdout_file = InvokeWithModifiedEnv(extra_env, os.popen, 563dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter '%s --gtest_list_tests' % COMMAND, 'r') 564dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter try: 565dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter stdout_file.readlines() 566dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter finally: 567dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter stdout_file.close() 568dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.assert_(os.path.exists(shard_status_file)) 569dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter os.remove(shard_status_file) 570dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 571dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter def testShardingWorksWithDeathTests(self): 572dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter """Tests integration with death tests and sharding.""" 573dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter gtest_filter = 'HasDeathTest.*:SeqP/*' 574dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter expected_tests = [ 575dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'HasDeathTest.Test1', 576dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'HasDeathTest.Test2', 577dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 578dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestX/0', 579dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestX/1', 580dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestY/0', 581dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 'SeqP/ParamTest.TestY/1', 582dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter ] 583dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 584dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter for command in (COMMAND + ' --gtest_death_test_style=threadsafe', 585dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter COMMAND + ' --gtest_death_test_style=fast'): 586dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding(gtest_filter, 3, expected_tests, 587dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter check_exit_0=True, command=command) 588dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter self.RunAndVerifyWithSharding(gtest_filter, 5, expected_tests, 589dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter check_exit_0=True, command=command) 590dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter 591dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixterif __name__ == '__main__': 592dd1c93d5709e32713961cfd95fe30489a4ad2d26Ken Mixter gtest_test_utils.Main() 593