doctest.py revision 17111f3b242be06c7ae913f106093891b82d7fee
1# Module doctest. 2# Released to the public domain 16-Jan-2001, 3# by Tim Peters (tim.one@home.com). 4 5# Provided as-is; use at your own risk; no warranty; no promises; enjoy! 6 7"""Module doctest -- a framework for running examples in docstrings. 8 9NORMAL USAGE 10 11In normal use, end each module M with: 12 13def _test(): 14 import doctest, M # replace M with your module's name 15 return doctest.testmod(M) # ditto 16 17if __name__ == "__main__": 18 _test() 19 20Then running the module as a script will cause the examples in the 21docstrings to get executed and verified: 22 23python M.py 24 25This won't display anything unless an example fails, in which case the 26failing example(s) and the cause(s) of the failure(s) are printed to stdout 27(why not stderr? because stderr is a lame hack <0.2 wink>), and the final 28line of output is "Test failed.". 29 30Run it with the -v switch instead: 31 32python M.py -v 33 34and a detailed report of all examples tried is printed to stdout, along 35with assorted summaries at the end. 36 37You can force verbose mode by passing "verbose=1" to testmod, or prohibit 38it by passing "verbose=0". In either of those cases, sys.argv is not 39examined by testmod. 40 41In any case, testmod returns a 2-tuple of ints (f, t), where f is the 42number of docstring examples that failed and t is the total number of 43docstring examples attempted. 44 45 46WHICH DOCSTRINGS ARE EXAMINED? 47 48+ M.__doc__. 49 50+ f.__doc__ for all functions f in M.__dict__.values(), except those 51 with private names and those defined in other modules. 52 53+ C.__doc__ for all classes C in M.__dict__.values(), except those with 54 private names and those defined in other modules. 55 56+ If M.__test__ exists and "is true", it must be a dict, and 57 each entry maps a (string) name to a function object, class object, or 58 string. Function and class object docstrings found from M.__test__ 59 are searched even if the name is private, and strings are searched 60 directly as if they were docstrings. In output, a key K in M.__test__ 61 appears with name 62 <name of M>.__test__.K 63 64Any classes found are recursively searched similarly, to test docstrings in 65their contained methods and nested classes. Private names reached from M's 66globals are skipped, but all names reached from M.__test__ are searched. 67 68By default, a name is considered to be private if it begins with an 69underscore (like "_my_func") but doesn't both begin and end with (at least) 70two underscores (like "__init__"). You can change the default by passing 71your own "isprivate" function to testmod. 72 73If you want to test docstrings in objects with private names too, stuff 74them into an M.__test__ dict, or see ADVANCED USAGE below (e.g., pass your 75own isprivate function to Tester's constructor, or call the rundoc method 76of a Tester instance). 77 78WHAT'S THE EXECUTION CONTEXT? 79 80By default, each time testmod finds a docstring to test, it uses a *copy* 81of M's globals (so that running tests on a module doesn't change the 82module's real globals, and so that one test in M can't leave behind crumbs 83that accidentally allow another test to work). This means examples can 84freely use any names defined at top-level in M. It also means that sloppy 85imports (see above) can cause examples in external docstrings to use 86globals inappropriate for them. 87 88You can force use of your own dict as the execution context by passing 89"globs=your_dict" to testmod instead. Presumably this would be a copy of 90M.__dict__ merged with the globals from other imported modules. 91 92 93WHAT IF I WANT TO TEST A WHOLE PACKAGE? 94 95Piece o' cake, provided the modules do their testing from docstrings. 96Here's the test.py I use for the world's most elaborate Rational/ 97floating-base-conversion pkg (which I'll distribute some day): 98 99from Rational import Cvt 100from Rational import Format 101from Rational import machprec 102from Rational import Rat 103from Rational import Round 104from Rational import utils 105 106modules = (Cvt, 107 Format, 108 machprec, 109 Rat, 110 Round, 111 utils) 112 113def _test(): 114 import doctest 115 import sys 116 verbose = "-v" in sys.argv 117 for mod in modules: 118 doctest.testmod(mod, verbose=verbose, report=0) 119 doctest.master.summarize() 120 121if __name__ == "__main__": 122 _test() 123 124IOW, it just runs testmod on all the pkg modules. testmod remembers the 125names and outcomes (# of failures, # of tries) for each item it's seen, and 126passing "report=0" prevents it from printing a summary in verbose mode. 127Instead, the summary is delayed until all modules have been tested, and 128then "doctest.master.summarize()" forces the summary at the end. 129 130So this is very nice in practice: each module can be tested individually 131with almost no work beyond writing up docstring examples, and collections 132of modules can be tested too as a unit with no more work than the above. 133 134 135WHAT ABOUT EXCEPTIONS? 136 137No problem, as long as the only output generated by the example is the 138traceback itself. For example: 139 140 >>> [1, 2, 3].remove(42) 141 Traceback (most recent call last): 142 File "<stdin>", line 1, in ? 143 ValueError: list.remove(x): x not in list 144 >>> 145 146Note that only the exception type and value are compared (specifically, 147only the last line in the traceback). 148 149 150ADVANCED USAGE 151 152doctest.testmod() captures the testing policy I find most useful most 153often. You may want other policies. 154 155testmod() actually creates a local instance of class doctest.Tester, runs 156appropriate methods of that class, and merges the results into global 157Tester instance doctest.master. 158 159You can create your own instances of doctest.Tester, and so build your own 160policies, or even run methods of doctest.master directly. See 161doctest.Tester.__doc__ for details. 162 163 164SO WHAT DOES A DOCSTRING EXAMPLE LOOK LIKE ALREADY!? 165 166Oh ya. It's easy! In most cases a copy-and-paste of an interactive 167console session works fine -- just make sure the leading whitespace is 168rigidly consistent (you can mix tabs and spaces if you're too lazy to do it 169right, but doctest is not in the business of guessing what you think a tab 170means). 171 172 >>> # comments are ignored 173 >>> x = 12 174 >>> x 175 12 176 >>> if x == 13: 177 ... print "yes" 178 ... else: 179 ... print "no" 180 ... print "NO" 181 ... print "NO!!!" 182 ... 183 no 184 NO 185 NO!!! 186 >>> 187 188Any expected output must immediately follow the final ">>>" or "..." line 189containing the code, and the expected output (if any) extends to the next 190">>>" or all-whitespace line. That's it. 191 192Bummers: 193 194+ Expected output cannot contain an all-whitespace line, since such a line 195 is taken to signal the end of expected output. 196 197+ Output to stdout is captured, but not output to stderr (exception 198 tracebacks are captured via a different means). 199 200+ If you continue a line via backslashing in an interactive session, or for 201 any other reason use a backslash, you need to double the backslash in the 202 docstring version. This is simply because you're in a string, and so the 203 backslash must be escaped for it to survive intact. Like: 204 205>>> if "yes" == \\ 206... "y" + \\ 207... "es": # in the source code you'll see the doubled backslashes 208... print 'yes' 209yes 210 211The starting column doesn't matter: 212 213>>> assert "Easy!" 214 >>> import math 215 >>> math.floor(1.9) 216 1.0 217 218and as many leading whitespace characters are stripped from the expected 219output as appeared in the initial ">>>" line that triggered it. 220 221If you execute this very file, the examples above will be found and 222executed, leading to this output in verbose mode: 223 224Running doctest.__doc__ 225Trying: [1, 2, 3].remove(42) 226Expecting: 227Traceback (most recent call last): 228 File "<stdin>", line 1, in ? 229ValueError: list.remove(x): x not in list 230ok 231Trying: x = 12 232Expecting: nothing 233ok 234Trying: x 235Expecting: 12 236ok 237Trying: 238if x == 13: 239 print "yes" 240else: 241 print "no" 242 print "NO" 243 print "NO!!!" 244Expecting: 245no 246NO 247NO!!! 248ok 249... and a bunch more like that, with this summary at the end: 250 2515 items had no tests: 252 doctest.Tester.__init__ 253 doctest.Tester.run__test__ 254 doctest.Tester.summarize 255 doctest.run_docstring_examples 256 doctest.testmod 25712 items passed all tests: 258 8 tests in doctest 259 6 tests in doctest.Tester 260 10 tests in doctest.Tester.merge 261 14 tests in doctest.Tester.rundict 262 3 tests in doctest.Tester.rundoc 263 3 tests in doctest.Tester.runstring 264 2 tests in doctest.__test__._TestClass 265 2 tests in doctest.__test__._TestClass.__init__ 266 2 tests in doctest.__test__._TestClass.get 267 1 tests in doctest.__test__._TestClass.square 268 2 tests in doctest.__test__.string 269 7 tests in doctest.is_private 27060 tests in 17 items. 27160 passed and 0 failed. 272Test passed. 273""" 274 275__all__ = [ 276 'testmod', 277 'run_docstring_examples', 278 'is_private', 279 'Tester', 280] 281 282import __future__ 283 284import re 285PS1 = ">>>" 286PS2 = "..." 287_isPS1 = re.compile(r"(\s*)" + re.escape(PS1)).match 288_isPS2 = re.compile(r"(\s*)" + re.escape(PS2)).match 289_isEmpty = re.compile(r"\s*$").match 290_isComment = re.compile(r"\s*#").match 291del re 292 293from types import StringTypes as _StringTypes 294 295from inspect import isclass as _isclass 296from inspect import isfunction as _isfunction 297from inspect import ismodule as _ismodule 298from inspect import classify_class_attrs as _classify_class_attrs 299 300# Extract interactive examples from a string. Return a list of triples, 301# (source, outcome, lineno). "source" is the source code, and ends 302# with a newline iff the source spans more than one line. "outcome" is 303# the expected output if any, else an empty string. When not empty, 304# outcome always ends with a newline. "lineno" is the line number, 305# 0-based wrt the start of the string, of the first source line. 306 307def _extract_examples(s): 308 isPS1, isPS2 = _isPS1, _isPS2 309 isEmpty, isComment = _isEmpty, _isComment 310 examples = [] 311 lines = s.split("\n") 312 i, n = 0, len(lines) 313 while i < n: 314 line = lines[i] 315 i = i + 1 316 m = isPS1(line) 317 if m is None: 318 continue 319 j = m.end(0) # beyond the prompt 320 if isEmpty(line, j) or isComment(line, j): 321 # a bare prompt or comment -- not interesting 322 continue 323 lineno = i - 1 324 if line[j] != " ": 325 raise ValueError("line " + `lineno` + " of docstring lacks " 326 "blank after " + PS1 + ": " + line) 327 j = j + 1 328 blanks = m.group(1) 329 nblanks = len(blanks) 330 # suck up this and following PS2 lines 331 source = [] 332 while 1: 333 source.append(line[j:]) 334 line = lines[i] 335 m = isPS2(line) 336 if m: 337 if m.group(1) != blanks: 338 raise ValueError("inconsistent leading whitespace " 339 "in line " + `i` + " of docstring: " + line) 340 i = i + 1 341 else: 342 break 343 if len(source) == 1: 344 source = source[0] 345 else: 346 # get rid of useless null line from trailing empty "..." 347 if source[-1] == "": 348 del source[-1] 349 source = "\n".join(source) + "\n" 350 # suck up response 351 if isPS1(line) or isEmpty(line): 352 expect = "" 353 else: 354 expect = [] 355 while 1: 356 if line[:nblanks] != blanks: 357 raise ValueError("inconsistent leading whitespace " 358 "in line " + `i` + " of docstring: " + line) 359 expect.append(line[nblanks:]) 360 i = i + 1 361 line = lines[i] 362 if isPS1(line) or isEmpty(line): 363 break 364 expect = "\n".join(expect) + "\n" 365 examples.append( (source, expect, lineno) ) 366 return examples 367 368# Capture stdout when running examples. 369 370class _SpoofOut: 371 def __init__(self): 372 self.clear() 373 def write(self, s): 374 self.buf.append(s) 375 def get(self): 376 guts = "".join(self.buf) 377 # If anything at all was written, make sure there's a trailing 378 # newline. There's no way for the expected output to indicate 379 # that a trailing newline is missing. 380 if guts and not guts.endswith("\n"): 381 guts = guts + "\n" 382 return guts 383 def clear(self): 384 self.buf = [] 385 def flush(self): 386 # JPython calls flush 387 pass 388 389# Display some tag-and-msg pairs nicely, keeping the tag and its msg 390# on the same line when that makes sense. 391 392def _tag_out(printer, *tag_msg_pairs): 393 for tag, msg in tag_msg_pairs: 394 printer(tag + ":") 395 msg_has_nl = msg[-1:] == "\n" 396 msg_has_two_nl = msg_has_nl and \ 397 msg.find("\n") < len(msg) - 1 398 if len(tag) + len(msg) < 76 and not msg_has_two_nl: 399 printer(" ") 400 else: 401 printer("\n") 402 printer(msg) 403 if not msg_has_nl: 404 printer("\n") 405 406# Run list of examples, in context globs. "out" can be used to display 407# stuff to "the real" stdout, and fakeout is an instance of _SpoofOut 408# that captures the examples' std output. Return (#failures, #tries). 409 410def _run_examples_inner(out, fakeout, examples, globs, verbose, name, 411 compileflags): 412 import sys, traceback 413 OK, BOOM, FAIL = range(3) 414 NADA = "nothing" 415 stderr = _SpoofOut() 416 failures = 0 417 for source, want, lineno in examples: 418 if verbose: 419 _tag_out(out, ("Trying", source), 420 ("Expecting", want or NADA)) 421 fakeout.clear() 422 try: 423 exec compile(source, "<string>", "single", 424 compileflags, 1) in globs 425 got = fakeout.get() 426 state = OK 427 except: 428 # See whether the exception was expected. 429 if want.find("Traceback (innermost last):\n") == 0 or \ 430 want.find("Traceback (most recent call last):\n") == 0: 431 # Only compare exception type and value - the rest of 432 # the traceback isn't necessary. 433 want = want.split('\n')[-2] + '\n' 434 exc_type, exc_val = sys.exc_info()[:2] 435 got = traceback.format_exception_only(exc_type, exc_val)[-1] 436 state = OK 437 else: 438 # unexpected exception 439 stderr.clear() 440 traceback.print_exc(file=stderr) 441 state = BOOM 442 443 if state == OK: 444 if got == want: 445 if verbose: 446 out("ok\n") 447 continue 448 state = FAIL 449 450 assert state in (FAIL, BOOM) 451 failures = failures + 1 452 out("*" * 65 + "\n") 453 _tag_out(out, ("Failure in example", source)) 454 out("from line #" + `lineno` + " of " + name + "\n") 455 if state == FAIL: 456 _tag_out(out, ("Expected", want or NADA), ("Got", got)) 457 else: 458 assert state == BOOM 459 _tag_out(out, ("Exception raised", stderr.get())) 460 461 return failures, len(examples) 462 463# Get the future-flags associated with the future features that have been 464# imported into globs. 465 466def _extract_future_flags(globs): 467 flags = 0 468 for fname in __future__.all_feature_names: 469 feature = globs.get(fname, None) 470 if feature is getattr(__future__, fname): 471 flags |= feature.compiler_flag 472 return flags 473 474# Run list of examples, in a shallow copy of context (dict) globs. 475# Return (#failures, #tries). 476 477def _run_examples(examples, globs, verbose, name, compileflags): 478 import sys 479 saveout = sys.stdout 480 globs = globs.copy() 481 try: 482 sys.stdout = fakeout = _SpoofOut() 483 x = _run_examples_inner(saveout.write, fakeout, examples, 484 globs, verbose, name, compileflags) 485 finally: 486 sys.stdout = saveout 487 # While Python gc can clean up most cycles on its own, it doesn't 488 # chase frame objects. This is especially irksome when running 489 # generator tests that raise exceptions, because a named generator- 490 # iterator gets an entry in globs, and the generator-iterator 491 # object's frame's traceback info points back to globs. This is 492 # easy to break just by clearing the namespace. This can also 493 # help to break other kinds of cycles, and even for cycles that 494 # gc can break itself it's better to break them ASAP. 495 globs.clear() 496 return x 497 498def run_docstring_examples(f, globs, verbose=0, name="NoName", 499 compileflags=None): 500 """f, globs, verbose=0, name="NoName" -> run examples from f.__doc__. 501 502 Use (a shallow copy of) dict globs as the globals for execution. 503 Return (#failures, #tries). 504 505 If optional arg verbose is true, print stuff even if there are no 506 failures. 507 Use string name in failure msgs. 508 """ 509 510 try: 511 doc = f.__doc__ 512 if not doc: 513 # docstring empty or None 514 return 0, 0 515 # just in case CT invents a doc object that has to be forced 516 # to look like a string <0.9 wink> 517 doc = str(doc) 518 except: 519 return 0, 0 520 521 e = _extract_examples(doc) 522 if not e: 523 return 0, 0 524 if compileflags is None: 525 compileflags = _extract_future_flags(globs) 526 return _run_examples(e, globs, verbose, name, compileflags) 527 528def is_private(prefix, base): 529 """prefix, base -> true iff name prefix + "." + base is "private". 530 531 Prefix may be an empty string, and base does not contain a period. 532 Prefix is ignored (although functions you write conforming to this 533 protocol may make use of it). 534 Return true iff base begins with an (at least one) underscore, but 535 does not both begin and end with (at least) two underscores. 536 537 >>> is_private("a.b", "my_func") 538 0 539 >>> is_private("____", "_my_func") 540 1 541 >>> is_private("someclass", "__init__") 542 0 543 >>> is_private("sometypo", "__init_") 544 1 545 >>> is_private("x.y.z", "_") 546 1 547 >>> is_private("_x.y.z", "__") 548 0 549 >>> is_private("", "") # senseless but consistent 550 0 551 """ 552 553 return base[:1] == "_" and not base[:2] == "__" == base[-2:] 554 555# Determine if a class of function was defined in the given module. 556 557def _from_module(module, object): 558 if _isfunction(object): 559 return module.__dict__ is object.func_globals 560 if _isclass(object): 561 return module.__name__ == object.__module__ 562 raise ValueError("object must be a class or function") 563 564class Tester: 565 """Class Tester -- runs docstring examples and accumulates stats. 566 567In normal use, function doctest.testmod() hides all this from you, 568so use that if you can. Create your own instances of Tester to do 569fancier things. 570 571Methods: 572 runstring(s, name) 573 Search string s for examples to run; use name for logging. 574 Return (#failures, #tries). 575 576 rundoc(object, name=None) 577 Search object.__doc__ for examples to run; use name (or 578 object.__name__) for logging. Return (#failures, #tries). 579 580 rundict(d, name, module=None) 581 Search for examples in docstrings in all of d.values(); use name 582 for logging. Exclude functions and classes not defined in module 583 if specified. Return (#failures, #tries). 584 585 run__test__(d, name) 586 Treat dict d like module.__test__. Return (#failures, #tries). 587 588 summarize(verbose=None) 589 Display summary of testing results, to stdout. Return 590 (#failures, #tries). 591 592 merge(other) 593 Merge in the test results from Tester instance "other". 594 595>>> from doctest import Tester 596>>> t = Tester(globs={'x': 42}, verbose=0) 597>>> t.runstring(r''' 598... >>> x = x * 2 599... >>> print x 600... 42 601... ''', 'XYZ') 602***************************************************************** 603Failure in example: print x 604from line #2 of XYZ 605Expected: 42 606Got: 84 607(1, 2) 608>>> t.runstring(">>> x = x * 2\\n>>> print x\\n84\\n", 'example2') 609(0, 2) 610>>> t.summarize() 611***************************************************************** 6121 items had failures: 613 1 of 2 in XYZ 614***Test Failed*** 1 failures. 615(1, 4) 616>>> t.summarize(verbose=1) 6171 items passed all tests: 618 2 tests in example2 619***************************************************************** 6201 items had failures: 621 1 of 2 in XYZ 6224 tests in 2 items. 6233 passed and 1 failed. 624***Test Failed*** 1 failures. 625(1, 4) 626>>> 627""" 628 629 def __init__(self, mod=None, globs=None, verbose=None, 630 isprivate=None): 631 """mod=None, globs=None, verbose=None, isprivate=None 632 633See doctest.__doc__ for an overview. 634 635Optional keyword arg "mod" is a module, whose globals are used for 636executing examples. If not specified, globs must be specified. 637 638Optional keyword arg "globs" gives a dict to be used as the globals 639when executing examples; if not specified, use the globals from 640module mod. 641 642In either case, a copy of the dict is used for each docstring 643examined. 644 645Optional keyword arg "verbose" prints lots of stuff if true, only 646failures if false; by default, it's true iff "-v" is in sys.argv. 647 648Optional keyword arg "isprivate" specifies a function used to determine 649whether a name is private. The default function is doctest.is_private; 650see its docs for details. 651""" 652 653 if mod is None and globs is None: 654 raise TypeError("Tester.__init__: must specify mod or globs") 655 if mod is not None and not _ismodule(mod): 656 raise TypeError("Tester.__init__: mod must be a module; " + 657 `mod`) 658 if globs is None: 659 globs = mod.__dict__ 660 self.globs = globs 661 662 if verbose is None: 663 import sys 664 verbose = "-v" in sys.argv 665 self.verbose = verbose 666 667 if isprivate is None: 668 isprivate = is_private 669 self.isprivate = isprivate 670 671 self.name2ft = {} # map name to (#failures, #trials) pair 672 673 self.compileflags = _extract_future_flags(globs) 674 675 def runstring(self, s, name): 676 """ 677 s, name -> search string s for examples to run, logging as name. 678 679 Use string name as the key for logging the outcome. 680 Return (#failures, #examples). 681 682 >>> t = Tester(globs={}, verbose=1) 683 >>> test = r''' 684 ... # just an example 685 ... >>> x = 1 + 2 686 ... >>> x 687 ... 3 688 ... ''' 689 >>> t.runstring(test, "Example") 690 Running string Example 691 Trying: x = 1 + 2 692 Expecting: nothing 693 ok 694 Trying: x 695 Expecting: 3 696 ok 697 0 of 2 examples failed in string Example 698 (0, 2) 699 """ 700 701 if self.verbose: 702 print "Running string", name 703 f = t = 0 704 e = _extract_examples(s) 705 if e: 706 f, t = _run_examples(e, self.globs, self.verbose, name, 707 self.compileflags) 708 if self.verbose: 709 print f, "of", t, "examples failed in string", name 710 self.__record_outcome(name, f, t) 711 return f, t 712 713 def rundoc(self, object, name=None): 714 """ 715 object, name=None -> search object.__doc__ for examples to run. 716 717 Use optional string name as the key for logging the outcome; 718 by default use object.__name__. 719 Return (#failures, #examples). 720 If object is a class object, search recursively for method 721 docstrings too. 722 object.__doc__ is examined regardless of name, but if object is 723 a class, whether private names reached from object are searched 724 depends on the constructor's "isprivate" argument. 725 726 >>> t = Tester(globs={}, verbose=0) 727 >>> def _f(): 728 ... '''Trivial docstring example. 729 ... >>> assert 2 == 2 730 ... ''' 731 ... return 32 732 ... 733 >>> t.rundoc(_f) # expect 0 failures in 1 example 734 (0, 1) 735 """ 736 737 if name is None: 738 try: 739 name = object.__name__ 740 except AttributeError: 741 raise ValueError("Tester.rundoc: name must be given " 742 "when object.__name__ doesn't exist; " + `object`) 743 if self.verbose: 744 print "Running", name + ".__doc__" 745 f, t = run_docstring_examples(object, self.globs, self.verbose, name, 746 self.compileflags) 747 if self.verbose: 748 print f, "of", t, "examples failed in", name + ".__doc__" 749 self.__record_outcome(name, f, t) 750 if _isclass(object): 751 # In 2.2, class and static methods complicate life. Build 752 # a dict "that works", by hook or by crook. 753 d = {} 754 for tag, kind, homecls, value in _classify_class_attrs(object): 755 756 if homecls is not object: 757 # Only look at names defined immediately by the class. 758 continue 759 760 elif self.isprivate(name, tag): 761 continue 762 763 elif kind == "method": 764 # value is already a function 765 d[tag] = value 766 767 elif kind == "static method": 768 # value isn't a function, but getattr reveals one 769 d[tag] = getattr(object, tag) 770 771 elif kind == "class method": 772 # Hmm. A classmethod object doesn't seem to reveal 773 # enough. But getattr turns it into a bound method, 774 # and from there .im_func retrieves the underlying 775 # function. 776 d[tag] = getattr(object, tag).im_func 777 778 elif kind == "property": 779 # The methods implementing the property have their 780 # own docstrings -- but the property may have one too. 781 if value.__doc__ is not None: 782 d[tag] = str(value.__doc__) 783 784 elif kind == "data": 785 # Grab nested classes. 786 if _isclass(value): 787 d[tag] = value 788 789 else: 790 raise ValueError("teach doctest about %r" % kind) 791 792 f2, t2 = self.run__test__(d, name) 793 f += f2 794 t += t2 795 796 return f, t 797 798 def rundict(self, d, name, module=None): 799 """ 800 d, name, module=None -> search for docstring examples in d.values(). 801 802 For k, v in d.items() such that v is a function or class, 803 do self.rundoc(v, name + "." + k). Whether this includes 804 objects with private names depends on the constructor's 805 "isprivate" argument. If module is specified, functions and 806 classes that are not defined in module are excluded. 807 Return aggregate (#failures, #examples). 808 809 Build and populate two modules with sample functions to test that 810 exclusion of external functions and classes works. 811 812 >>> import new 813 >>> m1 = new.module('_m1') 814 >>> m2 = new.module('_m2') 815 >>> test_data = \""" 816 ... def _f(): 817 ... '''>>> assert 1 == 1 818 ... ''' 819 ... def g(): 820 ... '''>>> assert 2 != 1 821 ... ''' 822 ... class H: 823 ... '''>>> assert 2 > 1 824 ... ''' 825 ... def bar(self): 826 ... '''>>> assert 1 < 2 827 ... ''' 828 ... \""" 829 >>> exec test_data in m1.__dict__ 830 >>> exec test_data in m2.__dict__ 831 >>> m1.__dict__.update({"f2": m2._f, "g2": m2.g, "h2": m2.H}) 832 833 Tests that objects outside m1 are excluded: 834 835 >>> t = Tester(globs={}, verbose=0) 836 >>> t.rundict(m1.__dict__, "rundict_test", m1) # _f, f2 and g2 and h2 skipped 837 (0, 3) 838 839 Again, but with a custom isprivate function allowing _f: 840 841 >>> t = Tester(globs={}, verbose=0, isprivate=lambda x,y: 0) 842 >>> t.rundict(m1.__dict__, "rundict_test_pvt", m1) # Only f2, g2 and h2 skipped 843 (0, 4) 844 845 And once more, not excluding stuff outside m1: 846 847 >>> t = Tester(globs={}, verbose=0, isprivate=lambda x,y: 0) 848 >>> t.rundict(m1.__dict__, "rundict_test_pvt") # None are skipped. 849 (0, 8) 850 851 The exclusion of objects from outside the designated module is 852 meant to be invoked automagically by testmod. 853 854 >>> testmod(m1) 855 (0, 3) 856 857 """ 858 859 if not hasattr(d, "items"): 860 raise TypeError("Tester.rundict: d must support .items(); " + 861 `d`) 862 f = t = 0 863 # Run the tests by alpha order of names, for consistency in 864 # verbose-mode output. 865 names = d.keys() 866 names.sort() 867 for thisname in names: 868 value = d[thisname] 869 if _isfunction(value) or _isclass(value): 870 if module and not _from_module(module, value): 871 continue 872 f2, t2 = self.__runone(value, name + "." + thisname) 873 f = f + f2 874 t = t + t2 875 return f, t 876 877 def run__test__(self, d, name): 878 """d, name -> Treat dict d like module.__test__. 879 880 Return (#failures, #tries). 881 See testmod.__doc__ for details. 882 """ 883 884 failures = tries = 0 885 prefix = name + "." 886 savepvt = self.isprivate 887 try: 888 self.isprivate = lambda *args: 0 889 # Run the tests by alpha order of names, for consistency in 890 # verbose-mode output. 891 keys = d.keys() 892 keys.sort() 893 for k in keys: 894 v = d[k] 895 thisname = prefix + k 896 if type(v) in _StringTypes: 897 f, t = self.runstring(v, thisname) 898 elif _isfunction(v) or _isclass(v): 899 f, t = self.rundoc(v, thisname) 900 else: 901 raise TypeError("Tester.run__test__: values in " 902 "dict must be strings, functions " 903 "or classes; " + `v`) 904 failures = failures + f 905 tries = tries + t 906 finally: 907 self.isprivate = savepvt 908 return failures, tries 909 910 def summarize(self, verbose=None): 911 """ 912 verbose=None -> summarize results, return (#failures, #tests). 913 914 Print summary of test results to stdout. 915 Optional arg 'verbose' controls how wordy this is. By 916 default, use the verbose setting established by the 917 constructor. 918 """ 919 920 if verbose is None: 921 verbose = self.verbose 922 notests = [] 923 passed = [] 924 failed = [] 925 totalt = totalf = 0 926 for x in self.name2ft.items(): 927 name, (f, t) = x 928 assert f <= t 929 totalt = totalt + t 930 totalf = totalf + f 931 if t == 0: 932 notests.append(name) 933 elif f == 0: 934 passed.append( (name, t) ) 935 else: 936 failed.append(x) 937 if verbose: 938 if notests: 939 print len(notests), "items had no tests:" 940 notests.sort() 941 for thing in notests: 942 print " ", thing 943 if passed: 944 print len(passed), "items passed all tests:" 945 passed.sort() 946 for thing, count in passed: 947 print " %3d tests in %s" % (count, thing) 948 if failed: 949 print "*" * 65 950 print len(failed), "items had failures:" 951 failed.sort() 952 for thing, (f, t) in failed: 953 print " %3d of %3d in %s" % (f, t, thing) 954 if verbose: 955 print totalt, "tests in", len(self.name2ft), "items." 956 print totalt - totalf, "passed and", totalf, "failed." 957 if totalf: 958 print "***Test Failed***", totalf, "failures." 959 elif verbose: 960 print "Test passed." 961 return totalf, totalt 962 963 def merge(self, other): 964 """ 965 other -> merge in test results from the other Tester instance. 966 967 If self and other both have a test result for something 968 with the same name, the (#failures, #tests) results are 969 summed, and a warning is printed to stdout. 970 971 >>> from doctest import Tester 972 >>> t1 = Tester(globs={}, verbose=0) 973 >>> t1.runstring(''' 974 ... >>> x = 12 975 ... >>> print x 976 ... 12 977 ... ''', "t1example") 978 (0, 2) 979 >>> 980 >>> t2 = Tester(globs={}, verbose=0) 981 >>> t2.runstring(''' 982 ... >>> x = 13 983 ... >>> print x 984 ... 13 985 ... ''', "t2example") 986 (0, 2) 987 >>> common = ">>> assert 1 + 2 == 3\\n" 988 >>> t1.runstring(common, "common") 989 (0, 1) 990 >>> t2.runstring(common, "common") 991 (0, 1) 992 >>> t1.merge(t2) 993 *** Tester.merge: 'common' in both testers; summing outcomes. 994 >>> t1.summarize(1) 995 3 items passed all tests: 996 2 tests in common 997 2 tests in t1example 998 2 tests in t2example 999 6 tests in 3 items. 1000 6 passed and 0 failed. 1001 Test passed. 1002 (0, 6) 1003 >>> 1004 """ 1005 1006 d = self.name2ft 1007 for name, (f, t) in other.name2ft.items(): 1008 if d.has_key(name): 1009 print "*** Tester.merge: '" + name + "' in both" \ 1010 " testers; summing outcomes." 1011 f2, t2 = d[name] 1012 f = f + f2 1013 t = t + t2 1014 d[name] = f, t 1015 1016 def __record_outcome(self, name, f, t): 1017 if self.name2ft.has_key(name): 1018 print "*** Warning: '" + name + "' was tested before;", \ 1019 "summing outcomes." 1020 f2, t2 = self.name2ft[name] 1021 f = f + f2 1022 t = t + t2 1023 self.name2ft[name] = f, t 1024 1025 def __runone(self, target, name): 1026 if "." in name: 1027 i = name.rindex(".") 1028 prefix, base = name[:i], name[i+1:] 1029 else: 1030 prefix, base = "", base 1031 if self.isprivate(prefix, base): 1032 return 0, 0 1033 return self.rundoc(target, name) 1034 1035master = None 1036 1037def testmod(m, name=None, globs=None, verbose=None, isprivate=None, 1038 report=1): 1039 """m, name=None, globs=None, verbose=None, isprivate=None, report=1 1040 1041 Test examples in docstrings in functions and classes reachable from 1042 module m, starting with m.__doc__. Private names are skipped. 1043 1044 Also test examples reachable from dict m.__test__ if it exists and is 1045 not None. m.__dict__ maps names to functions, classes and strings; 1046 function and class docstrings are tested even if the name is private; 1047 strings are tested directly, as if they were docstrings. 1048 1049 Return (#failures, #tests). 1050 1051 See doctest.__doc__ for an overview. 1052 1053 Optional keyword arg "name" gives the name of the module; by default 1054 use m.__name__. 1055 1056 Optional keyword arg "globs" gives a dict to be used as the globals 1057 when executing examples; by default, use m.__dict__. A copy of this 1058 dict is actually used for each docstring, so that each docstring's 1059 examples start with a clean slate. 1060 1061 Optional keyword arg "verbose" prints lots of stuff if true, prints 1062 only failures if false; by default, it's true iff "-v" is in sys.argv. 1063 1064 Optional keyword arg "isprivate" specifies a function used to 1065 determine whether a name is private. The default function is 1066 doctest.is_private; see its docs for details. 1067 1068 Optional keyword arg "report" prints a summary at the end when true, 1069 else prints nothing at the end. In verbose mode, the summary is 1070 detailed, else very brief (in fact, empty if all tests passed). 1071 1072 Advanced tomfoolery: testmod runs methods of a local instance of 1073 class doctest.Tester, then merges the results into (or creates) 1074 global Tester instance doctest.master. Methods of doctest.master 1075 can be called directly too, if you want to do something unusual. 1076 Passing report=0 to testmod is especially useful then, to delay 1077 displaying a summary. Invoke doctest.master.summarize(verbose) 1078 when you're done fiddling. 1079 """ 1080 1081 global master 1082 1083 if not _ismodule(m): 1084 raise TypeError("testmod: module required; " + `m`) 1085 if name is None: 1086 name = m.__name__ 1087 tester = Tester(m, globs=globs, verbose=verbose, isprivate=isprivate) 1088 failures, tries = tester.rundoc(m, name) 1089 f, t = tester.rundict(m.__dict__, name, m) 1090 failures = failures + f 1091 tries = tries + t 1092 if hasattr(m, "__test__"): 1093 testdict = m.__test__ 1094 if testdict: 1095 if not hasattr(testdict, "items"): 1096 raise TypeError("testmod: module.__test__ must support " 1097 ".items(); " + `testdict`) 1098 f, t = tester.run__test__(testdict, name + ".__test__") 1099 failures = failures + f 1100 tries = tries + t 1101 if report: 1102 tester.summarize() 1103 if master is None: 1104 master = tester 1105 else: 1106 master.merge(tester) 1107 return failures, tries 1108 1109class _TestClass: 1110 """ 1111 A pointless class, for sanity-checking of docstring testing. 1112 1113 Methods: 1114 square() 1115 get() 1116 1117 >>> _TestClass(13).get() + _TestClass(-12).get() 1118 1 1119 >>> hex(_TestClass(13).square().get()) 1120 '0xa9' 1121 """ 1122 1123 def __init__(self, val): 1124 """val -> _TestClass object with associated value val. 1125 1126 >>> t = _TestClass(123) 1127 >>> print t.get() 1128 123 1129 """ 1130 1131 self.val = val 1132 1133 def square(self): 1134 """square() -> square TestClass's associated value 1135 1136 >>> _TestClass(13).square().get() 1137 169 1138 """ 1139 1140 self.val = self.val ** 2 1141 return self 1142 1143 def get(self): 1144 """get() -> return TestClass's associated value. 1145 1146 >>> x = _TestClass(-42) 1147 >>> print x.get() 1148 -42 1149 """ 1150 1151 return self.val 1152 1153__test__ = {"_TestClass": _TestClass, 1154 "string": r""" 1155 Example of a string object, searched as-is. 1156 >>> x = 1; y = 2 1157 >>> x + y, x * y 1158 (3, 2) 1159 """ 1160 } 1161 1162def _test(): 1163 import doctest 1164 return doctest.testmod(doctest) 1165 1166if __name__ == "__main__": 1167 _test() 1168