pickletools.py revision 90cf212cefd270615a6d7ffa043e1cf3bd9ec9f2
1'''"Executable documentation" for the pickle module. 2 3Extensive comments about the pickle protocols and pickle-machine opcodes 4can be found here. Some functions meant for external use: 5 6genops(pickle) 7 Generate all the opcodes in a pickle, as (opcode, arg, position) triples. 8 9dis(pickle, out=None, memo=None, indentlevel=4) 10 Print a symbolic disassembly of a pickle. 11''' 12 13__all__ = ['dis', 14 'genops', 15 ] 16 17# Other ideas: 18# 19# - A pickle verifier: read a pickle and check it exhaustively for 20# well-formedness. dis() does a lot of this already. 21# 22# - A protocol identifier: examine a pickle and return its protocol number 23# (== the highest .proto attr value among all the opcodes in the pickle). 24# dis() already prints this info at the end. 25# 26# - A pickle optimizer: for example, tuple-building code is sometimes more 27# elaborate than necessary, catering for the possibility that the tuple 28# is recursive. Or lots of times a PUT is generated that's never accessed 29# by a later GET. 30 31 32""" 33"A pickle" is a program for a virtual pickle machine (PM, but more accurately 34called an unpickling machine). It's a sequence of opcodes, interpreted by the 35PM, building an arbitrarily complex Python object. 36 37For the most part, the PM is very simple: there are no looping, testing, or 38conditional instructions, no arithmetic and no function calls. Opcodes are 39executed once each, from first to last, until a STOP opcode is reached. 40 41The PM has two data areas, "the stack" and "the memo". 42 43Many opcodes push Python objects onto the stack; e.g., INT pushes a Python 44integer object on the stack, whose value is gotten from a decimal string 45literal immediately following the INT opcode in the pickle bytestream. Other 46opcodes take Python objects off the stack. The result of unpickling is 47whatever object is left on the stack when the final STOP opcode is executed. 48 49The memo is simply an array of objects, or it can be implemented as a dict 50mapping little integers to objects. The memo serves as the PM's "long term 51memory", and the little integers indexing the memo are akin to variable 52names. Some opcodes pop a stack object into the memo at a given index, 53and others push a memo object at a given index onto the stack again. 54 55At heart, that's all the PM has. Subtleties arise for these reasons: 56 57+ Object identity. Objects can be arbitrarily complex, and subobjects 58 may be shared (for example, the list [a, a] refers to the same object a 59 twice). It can be vital that unpickling recreate an isomorphic object 60 graph, faithfully reproducing sharing. 61 62+ Recursive objects. For example, after "L = []; L.append(L)", L is a 63 list, and L[0] is the same list. This is related to the object identity 64 point, and some sequences of pickle opcodes are subtle in order to 65 get the right result in all cases. 66 67+ Things pickle doesn't know everything about. Examples of things pickle 68 does know everything about are Python's builtin scalar and container 69 types, like ints and tuples. They generally have opcodes dedicated to 70 them. For things like module references and instances of user-defined 71 classes, pickle's knowledge is limited. Historically, many enhancements 72 have been made to the pickle protocol in order to do a better (faster, 73 and/or more compact) job on those. 74 75+ Backward compatibility and micro-optimization. As explained below, 76 pickle opcodes never go away, not even when better ways to do a thing 77 get invented. The repertoire of the PM just keeps growing over time. 78 For example, protocol 0 had two opcodes for building Python integers (INT 79 and LONG), protocol 1 added three more for more-efficient pickling of short 80 integers, and protocol 2 added two more for more-efficient pickling of 81 long integers (before protocol 2, the only ways to pickle a Python long 82 took time quadratic in the number of digits, for both pickling and 83 unpickling). "Opcode bloat" isn't so much a subtlety as a source of 84 wearying complication. 85 86 87Pickle protocols: 88 89For compatibility, the meaning of a pickle opcode never changes. Instead new 90pickle opcodes get added, and each version's unpickler can handle all the 91pickle opcodes in all protocol versions to date. So old pickles continue to 92be readable forever. The pickler can generally be told to restrict itself to 93the subset of opcodes available under previous protocol versions too, so that 94users can create pickles under the current version readable by older 95versions. However, a pickle does not contain its version number embedded 96within it. If an older unpickler tries to read a pickle using a later 97protocol, the result is most likely an exception due to seeing an unknown (in 98the older unpickler) opcode. 99 100The original pickle used what's now called "protocol 0", and what was called 101"text mode" before Python 2.3. The entire pickle bytestream is made up of 102printable 7-bit ASCII characters, plus the newline character, in protocol 0. 103That's why it was called text mode. Protocol 0 is small and elegant, but 104sometimes painfully inefficient. 105 106The second major set of additions is now called "protocol 1", and was called 107"binary mode" before Python 2.3. This added many opcodes with arguments 108consisting of arbitrary bytes, including NUL bytes and unprintable "high bit" 109bytes. Binary mode pickles can be substantially smaller than equivalent 110text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte 111int as 4 bytes following the opcode, which is cheaper to unpickle than the 112(perhaps) 11-character decimal string attached to INT. Protocol 1 also added 113a number of opcodes that operate on many stack elements at once (like APPENDS 114and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE). 115 116The third major set of additions came in Python 2.3, and is called "protocol 1172". This added: 118 119- A better way to pickle instances of new-style classes (NEWOBJ). 120 121- A way for a pickle to identify its protocol (PROTO). 122 123- Time- and space- efficient pickling of long ints (LONG{1,4}). 124 125- Shortcuts for small tuples (TUPLE{1,2,3}}. 126 127- Dedicated opcodes for bools (NEWTRUE, NEWFALSE). 128 129- The "extension registry", a vector of popular objects that can be pushed 130 efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but 131 the registry contents are predefined (there's nothing akin to the memo's 132 PUT). 133 134Another independent change with Python 2.3 is the abandonment of any 135pretense that it might be safe to load pickles received from untrusted 136parties -- no sufficient security analysis has been done to guarantee 137this and there isn't a use case that warrants the expense of such an 138analysis. 139 140To this end, all tests for __safe_for_unpickling__ or for 141copy_reg.safe_constructors are removed from the unpickling code. 142References to these variables in the descriptions below are to be seen 143as describing unpickling in Python 2.2 and before. 144""" 145 146# Meta-rule: Descriptions are stored in instances of descriptor objects, 147# with plain constructors. No meta-language is defined from which 148# descriptors could be constructed. If you want, e.g., XML, write a little 149# program to generate XML from the objects. 150 151############################################################################## 152# Some pickle opcodes have an argument, following the opcode in the 153# bytestream. An argument is of a specific type, described by an instance 154# of ArgumentDescriptor. These are not to be confused with arguments taken 155# off the stack -- ArgumentDescriptor applies only to arguments embedded in 156# the opcode stream, immediately following an opcode. 157 158# Represents the number of bytes consumed by an argument delimited by the 159# next newline character. 160UP_TO_NEWLINE = -1 161 162# Represents the number of bytes consumed by a two-argument opcode where 163# the first argument gives the number of bytes in the second argument. 164TAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int 165TAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int 166 167class ArgumentDescriptor(object): 168 __slots__ = ( 169 # name of descriptor record, also a module global name; a string 170 'name', 171 172 # length of argument, in bytes; an int; UP_TO_NEWLINE and 173 # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length 174 # cases 175 'n', 176 177 # a function taking a file-like object, reading this kind of argument 178 # from the object at the current position, advancing the current 179 # position by n bytes, and returning the value of the argument 180 'reader', 181 182 # human-readable docs for this arg descriptor; a string 183 'doc', 184 ) 185 186 def __init__(self, name, n, reader, doc): 187 assert isinstance(name, str) 188 self.name = name 189 190 assert isinstance(n, int) and (n >= 0 or 191 n in (UP_TO_NEWLINE, 192 TAKEN_FROM_ARGUMENT1, 193 TAKEN_FROM_ARGUMENT4)) 194 self.n = n 195 196 self.reader = reader 197 198 assert isinstance(doc, str) 199 self.doc = doc 200 201from struct import unpack as _unpack 202 203def read_uint1(f): 204 r""" 205 >>> import StringIO 206 >>> read_uint1(StringIO.StringIO('\xff')) 207 255 208 """ 209 210 data = f.read(1) 211 if data: 212 return ord(data) 213 raise ValueError("not enough data in stream to read uint1") 214 215uint1 = ArgumentDescriptor( 216 name='uint1', 217 n=1, 218 reader=read_uint1, 219 doc="One-byte unsigned integer.") 220 221 222def read_uint2(f): 223 r""" 224 >>> import StringIO 225 >>> read_uint2(StringIO.StringIO('\xff\x00')) 226 255 227 >>> read_uint2(StringIO.StringIO('\xff\xff')) 228 65535 229 """ 230 231 data = f.read(2) 232 if len(data) == 2: 233 return _unpack("<H", data)[0] 234 raise ValueError("not enough data in stream to read uint2") 235 236uint2 = ArgumentDescriptor( 237 name='uint2', 238 n=2, 239 reader=read_uint2, 240 doc="Two-byte unsigned integer, little-endian.") 241 242 243def read_int4(f): 244 r""" 245 >>> import StringIO 246 >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00')) 247 255 248 >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31) 249 True 250 """ 251 252 data = f.read(4) 253 if len(data) == 4: 254 return _unpack("<i", data)[0] 255 raise ValueError("not enough data in stream to read int4") 256 257int4 = ArgumentDescriptor( 258 name='int4', 259 n=4, 260 reader=read_int4, 261 doc="Four-byte signed integer, little-endian, 2's complement.") 262 263 264def read_stringnl(f, decode=True, stripquotes=True): 265 r""" 266 >>> import StringIO 267 >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n")) 268 'abcd' 269 270 >>> read_stringnl(StringIO.StringIO("\n")) 271 Traceback (most recent call last): 272 ... 273 ValueError: no string quotes around '' 274 275 >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False) 276 '' 277 278 >>> read_stringnl(StringIO.StringIO("''\n")) 279 '' 280 281 >>> read_stringnl(StringIO.StringIO('"abcd"')) 282 Traceback (most recent call last): 283 ... 284 ValueError: no newline found when trying to read stringnl 285 286 Embedded escapes are undone in the result. 287 >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'")) 288 'a\n\\b\x00c\td' 289 """ 290 291 data = f.readline() 292 if not data.endswith('\n'): 293 raise ValueError("no newline found when trying to read stringnl") 294 data = data[:-1] # lose the newline 295 296 if stripquotes: 297 for q in "'\"": 298 if data.startswith(q): 299 if not data.endswith(q): 300 raise ValueError("strinq quote %r not found at both " 301 "ends of %r" % (q, data)) 302 data = data[1:-1] 303 break 304 else: 305 raise ValueError("no string quotes around %r" % data) 306 307 # I'm not sure when 'string_escape' was added to the std codecs; it's 308 # crazy not to use it if it's there. 309 if decode: 310 data = data.decode('string_escape') 311 return data 312 313stringnl = ArgumentDescriptor( 314 name='stringnl', 315 n=UP_TO_NEWLINE, 316 reader=read_stringnl, 317 doc="""A newline-terminated string. 318 319 This is a repr-style string, with embedded escapes, and 320 bracketing quotes. 321 """) 322 323def read_stringnl_noescape(f): 324 return read_stringnl(f, decode=False, stripquotes=False) 325 326stringnl_noescape = ArgumentDescriptor( 327 name='stringnl_noescape', 328 n=UP_TO_NEWLINE, 329 reader=read_stringnl_noescape, 330 doc="""A newline-terminated string. 331 332 This is a str-style string, without embedded escapes, 333 or bracketing quotes. It should consist solely of 334 printable ASCII characters. 335 """) 336 337def read_stringnl_noescape_pair(f): 338 r""" 339 >>> import StringIO 340 >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk")) 341 'Queue Empty' 342 """ 343 344 return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) 345 346stringnl_noescape_pair = ArgumentDescriptor( 347 name='stringnl_noescape_pair', 348 n=UP_TO_NEWLINE, 349 reader=read_stringnl_noescape_pair, 350 doc="""A pair of newline-terminated strings. 351 352 These are str-style strings, without embedded 353 escapes, or bracketing quotes. They should 354 consist solely of printable ASCII characters. 355 The pair is returned as a single string, with 356 a single blank separating the two strings. 357 """) 358 359def read_string4(f): 360 r""" 361 >>> import StringIO 362 >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc")) 363 '' 364 >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef")) 365 'abc' 366 >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef")) 367 Traceback (most recent call last): 368 ... 369 ValueError: expected 50331648 bytes in a string4, but only 6 remain 370 """ 371 372 n = read_int4(f) 373 if n < 0: 374 raise ValueError("string4 byte count < 0: %d" % n) 375 data = f.read(n) 376 if len(data) == n: 377 return data 378 raise ValueError("expected %d bytes in a string4, but only %d remain" % 379 (n, len(data))) 380 381string4 = ArgumentDescriptor( 382 name="string4", 383 n=TAKEN_FROM_ARGUMENT4, 384 reader=read_string4, 385 doc="""A counted string. 386 387 The first argument is a 4-byte little-endian signed int giving 388 the number of bytes in the string, and the second argument is 389 that many bytes. 390 """) 391 392 393def read_string1(f): 394 r""" 395 >>> import StringIO 396 >>> read_string1(StringIO.StringIO("\x00")) 397 '' 398 >>> read_string1(StringIO.StringIO("\x03abcdef")) 399 'abc' 400 """ 401 402 n = read_uint1(f) 403 assert n >= 0 404 data = f.read(n) 405 if len(data) == n: 406 return data 407 raise ValueError("expected %d bytes in a string1, but only %d remain" % 408 (n, len(data))) 409 410string1 = ArgumentDescriptor( 411 name="string1", 412 n=TAKEN_FROM_ARGUMENT1, 413 reader=read_string1, 414 doc="""A counted string. 415 416 The first argument is a 1-byte unsigned int giving the number 417 of bytes in the string, and the second argument is that many 418 bytes. 419 """) 420 421 422def read_unicodestringnl(f): 423 r""" 424 >>> import StringIO 425 >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk")) 426 u'abc\uabcd' 427 """ 428 429 data = f.readline() 430 if not data.endswith('\n'): 431 raise ValueError("no newline found when trying to read " 432 "unicodestringnl") 433 data = data[:-1] # lose the newline 434 return unicode(data, 'raw-unicode-escape') 435 436unicodestringnl = ArgumentDescriptor( 437 name='unicodestringnl', 438 n=UP_TO_NEWLINE, 439 reader=read_unicodestringnl, 440 doc="""A newline-terminated Unicode string. 441 442 This is raw-unicode-escape encoded, so consists of 443 printable ASCII characters, and may contain embedded 444 escape sequences. 445 """) 446 447def read_unicodestring4(f): 448 r""" 449 >>> import StringIO 450 >>> s = u'abcd\uabcd' 451 >>> enc = s.encode('utf-8') 452 >>> enc 453 'abcd\xea\xaf\x8d' 454 >>> n = chr(len(enc)) + chr(0) * 3 # little-endian 4-byte length 455 >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk')) 456 >>> s == t 457 True 458 459 >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1])) 460 Traceback (most recent call last): 461 ... 462 ValueError: expected 7 bytes in a unicodestring4, but only 6 remain 463 """ 464 465 n = read_int4(f) 466 if n < 0: 467 raise ValueError("unicodestring4 byte count < 0: %d" % n) 468 data = f.read(n) 469 if len(data) == n: 470 return unicode(data, 'utf-8') 471 raise ValueError("expected %d bytes in a unicodestring4, but only %d " 472 "remain" % (n, len(data))) 473 474unicodestring4 = ArgumentDescriptor( 475 name="unicodestring4", 476 n=TAKEN_FROM_ARGUMENT4, 477 reader=read_unicodestring4, 478 doc="""A counted Unicode string. 479 480 The first argument is a 4-byte little-endian signed int 481 giving the number of bytes in the string, and the second 482 argument-- the UTF-8 encoding of the Unicode string -- 483 contains that many bytes. 484 """) 485 486 487def read_decimalnl_short(f): 488 r""" 489 >>> import StringIO 490 >>> read_decimalnl_short(StringIO.StringIO("1234\n56")) 491 1234 492 493 >>> read_decimalnl_short(StringIO.StringIO("1234L\n56")) 494 Traceback (most recent call last): 495 ... 496 ValueError: trailing 'L' not allowed in '1234L' 497 """ 498 499 s = read_stringnl(f, decode=False, stripquotes=False) 500 if s.endswith("L"): 501 raise ValueError("trailing 'L' not allowed in %r" % s) 502 503 # It's not necessarily true that the result fits in a Python short int: 504 # the pickle may have been written on a 64-bit box. There's also a hack 505 # for True and False here. 506 if s == "00": 507 return False 508 elif s == "01": 509 return True 510 511 try: 512 return int(s) 513 except OverflowError: 514 return long(s) 515 516def read_decimalnl_long(f): 517 r""" 518 >>> import StringIO 519 520 >>> read_decimalnl_long(StringIO.StringIO("1234\n56")) 521 Traceback (most recent call last): 522 ... 523 ValueError: trailing 'L' required in '1234' 524 525 Someday the trailing 'L' will probably go away from this output. 526 527 >>> read_decimalnl_long(StringIO.StringIO("1234L\n56")) 528 1234L 529 530 >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\n6")) 531 123456789012345678901234L 532 """ 533 534 s = read_stringnl(f, decode=False, stripquotes=False) 535 if not s.endswith("L"): 536 raise ValueError("trailing 'L' required in %r" % s) 537 return long(s) 538 539 540decimalnl_short = ArgumentDescriptor( 541 name='decimalnl_short', 542 n=UP_TO_NEWLINE, 543 reader=read_decimalnl_short, 544 doc="""A newline-terminated decimal integer literal. 545 546 This never has a trailing 'L', and the integer fit 547 in a short Python int on the box where the pickle 548 was written -- but there's no guarantee it will fit 549 in a short Python int on the box where the pickle 550 is read. 551 """) 552 553decimalnl_long = ArgumentDescriptor( 554 name='decimalnl_long', 555 n=UP_TO_NEWLINE, 556 reader=read_decimalnl_long, 557 doc="""A newline-terminated decimal integer literal. 558 559 This has a trailing 'L', and can represent integers 560 of any size. 561 """) 562 563 564def read_floatnl(f): 565 r""" 566 >>> import StringIO 567 >>> read_floatnl(StringIO.StringIO("-1.25\n6")) 568 -1.25 569 """ 570 s = read_stringnl(f, decode=False, stripquotes=False) 571 return float(s) 572 573floatnl = ArgumentDescriptor( 574 name='floatnl', 575 n=UP_TO_NEWLINE, 576 reader=read_floatnl, 577 doc="""A newline-terminated decimal floating literal. 578 579 In general this requires 17 significant digits for roundtrip 580 identity, and pickling then unpickling infinities, NaNs, and 581 minus zero doesn't work across boxes, or on some boxes even 582 on itself (e.g., Windows can't read the strings it produces 583 for infinities or NaNs). 584 """) 585 586def read_float8(f): 587 r""" 588 >>> import StringIO, struct 589 >>> raw = struct.pack(">d", -1.25) 590 >>> raw 591 '\xbf\xf4\x00\x00\x00\x00\x00\x00' 592 >>> read_float8(StringIO.StringIO(raw + "\n")) 593 -1.25 594 """ 595 596 data = f.read(8) 597 if len(data) == 8: 598 return _unpack(">d", data)[0] 599 raise ValueError("not enough data in stream to read float8") 600 601 602float8 = ArgumentDescriptor( 603 name='float8', 604 n=8, 605 reader=read_float8, 606 doc="""An 8-byte binary representation of a float, big-endian. 607 608 The format is unique to Python, and shared with the struct 609 module (format string '>d') "in theory" (the struct and cPickle 610 implementations don't share the code -- they should). It's 611 strongly related to the IEEE-754 double format, and, in normal 612 cases, is in fact identical to the big-endian 754 double format. 613 On other boxes the dynamic range is limited to that of a 754 614 double, and "add a half and chop" rounding is used to reduce 615 the precision to 53 bits. However, even on a 754 box, 616 infinities, NaNs, and minus zero may not be handled correctly 617 (may not survive roundtrip pickling intact). 618 """) 619 620# Protocol 2 formats 621 622from pickle import decode_long 623 624def read_long1(f): 625 r""" 626 >>> import StringIO 627 >>> read_long1(StringIO.StringIO("\x00")) 628 0L 629 >>> read_long1(StringIO.StringIO("\x02\xff\x00")) 630 255L 631 >>> read_long1(StringIO.StringIO("\x02\xff\x7f")) 632 32767L 633 >>> read_long1(StringIO.StringIO("\x02\x00\xff")) 634 -256L 635 >>> read_long1(StringIO.StringIO("\x02\x00\x80")) 636 -32768L 637 """ 638 639 n = read_uint1(f) 640 data = f.read(n) 641 if len(data) != n: 642 raise ValueError("not enough data in stream to read long1") 643 return decode_long(data) 644 645long1 = ArgumentDescriptor( 646 name="long1", 647 n=TAKEN_FROM_ARGUMENT1, 648 reader=read_long1, 649 doc="""A binary long, little-endian, using 1-byte size. 650 651 This first reads one byte as an unsigned size, then reads that 652 many bytes and interprets them as a little-endian 2's-complement long. 653 If the size is 0, that's taken as a shortcut for the long 0L. 654 """) 655 656def read_long4(f): 657 r""" 658 >>> import StringIO 659 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x00")) 660 255L 661 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x7f")) 662 32767L 663 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\xff")) 664 -256L 665 >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\x80")) 666 -32768L 667 >>> read_long1(StringIO.StringIO("\x00\x00\x00\x00")) 668 0L 669 """ 670 671 n = read_int4(f) 672 if n < 0: 673 raise ValueError("long4 byte count < 0: %d" % n) 674 data = f.read(n) 675 if len(data) != n: 676 raise ValueError("not enough data in stream to read long4") 677 return decode_long(data) 678 679long4 = ArgumentDescriptor( 680 name="long4", 681 n=TAKEN_FROM_ARGUMENT4, 682 reader=read_long4, 683 doc="""A binary representation of a long, little-endian. 684 685 This first reads four bytes as a signed size (but requires the 686 size to be >= 0), then reads that many bytes and interprets them 687 as a little-endian 2's-complement long. If the size is 0, that's taken 688 as a shortcut for the long 0L, although LONG1 should really be used 689 then instead (and in any case where # of bytes < 256). 690 """) 691 692 693############################################################################## 694# Object descriptors. The stack used by the pickle machine holds objects, 695# and in the stack_before and stack_after attributes of OpcodeInfo 696# descriptors we need names to describe the various types of objects that can 697# appear on the stack. 698 699class StackObject(object): 700 __slots__ = ( 701 # name of descriptor record, for info only 702 'name', 703 704 # type of object, or tuple of type objects (meaning the object can 705 # be of any type in the tuple) 706 'obtype', 707 708 # human-readable docs for this kind of stack object; a string 709 'doc', 710 ) 711 712 def __init__(self, name, obtype, doc): 713 assert isinstance(name, str) 714 self.name = name 715 716 assert isinstance(obtype, type) or isinstance(obtype, tuple) 717 if isinstance(obtype, tuple): 718 for contained in obtype: 719 assert isinstance(contained, type) 720 self.obtype = obtype 721 722 assert isinstance(doc, str) 723 self.doc = doc 724 725 def __repr__(self): 726 return self.name 727 728 729pyint = StackObject( 730 name='int', 731 obtype=int, 732 doc="A short (as opposed to long) Python integer object.") 733 734pylong = StackObject( 735 name='long', 736 obtype=long, 737 doc="A long (as opposed to short) Python integer object.") 738 739pyinteger_or_bool = StackObject( 740 name='int_or_bool', 741 obtype=(int, long, bool), 742 doc="A Python integer object (short or long), or " 743 "a Python bool.") 744 745pybool = StackObject( 746 name='bool', 747 obtype=(bool,), 748 doc="A Python bool object.") 749 750pyfloat = StackObject( 751 name='float', 752 obtype=float, 753 doc="A Python float object.") 754 755pystring = StackObject( 756 name='str', 757 obtype=str, 758 doc="A Python string object.") 759 760pyunicode = StackObject( 761 name='unicode', 762 obtype=unicode, 763 doc="A Python Unicode string object.") 764 765pynone = StackObject( 766 name="None", 767 obtype=type(None), 768 doc="The Python None object.") 769 770pytuple = StackObject( 771 name="tuple", 772 obtype=tuple, 773 doc="A Python tuple object.") 774 775pylist = StackObject( 776 name="list", 777 obtype=list, 778 doc="A Python list object.") 779 780pydict = StackObject( 781 name="dict", 782 obtype=dict, 783 doc="A Python dict object.") 784 785anyobject = StackObject( 786 name='any', 787 obtype=object, 788 doc="Any kind of object whatsoever.") 789 790markobject = StackObject( 791 name="mark", 792 obtype=StackObject, 793 doc="""'The mark' is a unique object. 794 795 Opcodes that operate on a variable number of objects 796 generally don't embed the count of objects in the opcode, 797 or pull it off the stack. Instead the MARK opcode is used 798 to push a special marker object on the stack, and then 799 some other opcodes grab all the objects from the top of 800 the stack down to (but not including) the topmost marker 801 object. 802 """) 803 804stackslice = StackObject( 805 name="stackslice", 806 obtype=StackObject, 807 doc="""An object representing a contiguous slice of the stack. 808 809 This is used in conjuction with markobject, to represent all 810 of the stack following the topmost markobject. For example, 811 the POP_MARK opcode changes the stack from 812 813 [..., markobject, stackslice] 814 to 815 [...] 816 817 No matter how many object are on the stack after the topmost 818 markobject, POP_MARK gets rid of all of them (including the 819 topmost markobject too). 820 """) 821 822############################################################################## 823# Descriptors for pickle opcodes. 824 825class OpcodeInfo(object): 826 827 __slots__ = ( 828 # symbolic name of opcode; a string 829 'name', 830 831 # the code used in a bytestream to represent the opcode; a 832 # one-character string 833 'code', 834 835 # If the opcode has an argument embedded in the byte string, an 836 # instance of ArgumentDescriptor specifying its type. Note that 837 # arg.reader(s) can be used to read and decode the argument from 838 # the bytestream s, and arg.doc documents the format of the raw 839 # argument bytes. If the opcode doesn't have an argument embedded 840 # in the bytestream, arg should be None. 841 'arg', 842 843 # what the stack looks like before this opcode runs; a list 844 'stack_before', 845 846 # what the stack looks like after this opcode runs; a list 847 'stack_after', 848 849 # the protocol number in which this opcode was introduced; an int 850 'proto', 851 852 # human-readable docs for this opcode; a string 853 'doc', 854 ) 855 856 def __init__(self, name, code, arg, 857 stack_before, stack_after, proto, doc): 858 assert isinstance(name, str) 859 self.name = name 860 861 assert isinstance(code, str) 862 assert len(code) == 1 863 self.code = code 864 865 assert arg is None or isinstance(arg, ArgumentDescriptor) 866 self.arg = arg 867 868 assert isinstance(stack_before, list) 869 for x in stack_before: 870 assert isinstance(x, StackObject) 871 self.stack_before = stack_before 872 873 assert isinstance(stack_after, list) 874 for x in stack_after: 875 assert isinstance(x, StackObject) 876 self.stack_after = stack_after 877 878 assert isinstance(proto, int) and 0 <= proto <= 2 879 self.proto = proto 880 881 assert isinstance(doc, str) 882 self.doc = doc 883 884I = OpcodeInfo 885opcodes = [ 886 887 # Ways to spell integers. 888 889 I(name='INT', 890 code='I', 891 arg=decimalnl_short, 892 stack_before=[], 893 stack_after=[pyinteger_or_bool], 894 proto=0, 895 doc="""Push an integer or bool. 896 897 The argument is a newline-terminated decimal literal string. 898 899 The intent may have been that this always fit in a short Python int, 900 but INT can be generated in pickles written on a 64-bit box that 901 require a Python long on a 32-bit box. The difference between this 902 and LONG then is that INT skips a trailing 'L', and produces a short 903 int whenever possible. 904 905 Another difference is due to that, when bool was introduced as a 906 distinct type in 2.3, builtin names True and False were also added to 907 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, 908 True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". 909 Leading zeroes are never produced for a genuine integer. The 2.3 910 (and later) unpicklers special-case these and return bool instead; 911 earlier unpicklers ignore the leading "0" and return the int. 912 """), 913 914 I(name='BININT', 915 code='J', 916 arg=int4, 917 stack_before=[], 918 stack_after=[pyint], 919 proto=1, 920 doc="""Push a four-byte signed integer. 921 922 This handles the full range of Python (short) integers on a 32-bit 923 box, directly as binary bytes (1 for the opcode and 4 for the integer). 924 If the integer is non-negative and fits in 1 or 2 bytes, pickling via 925 BININT1 or BININT2 saves space. 926 """), 927 928 I(name='BININT1', 929 code='K', 930 arg=uint1, 931 stack_before=[], 932 stack_after=[pyint], 933 proto=1, 934 doc="""Push a one-byte unsigned integer. 935 936 This is a space optimization for pickling very small non-negative ints, 937 in range(256). 938 """), 939 940 I(name='BININT2', 941 code='M', 942 arg=uint2, 943 stack_before=[], 944 stack_after=[pyint], 945 proto=1, 946 doc="""Push a two-byte unsigned integer. 947 948 This is a space optimization for pickling small positive ints, in 949 range(256, 2**16). Integers in range(256) can also be pickled via 950 BININT2, but BININT1 instead saves a byte. 951 """), 952 953 I(name='LONG', 954 code='L', 955 arg=decimalnl_long, 956 stack_before=[], 957 stack_after=[pylong], 958 proto=0, 959 doc="""Push a long integer. 960 961 The same as INT, except that the literal ends with 'L', and always 962 unpickles to a Python long. There doesn't seem a real purpose to the 963 trailing 'L'. 964 965 Note that LONG takes time quadratic in the number of digits when 966 unpickling (this is simply due to the nature of decimal->binary 967 conversion). Proto 2 added linear-time (in C; still quadratic-time 968 in Python) LONG1 and LONG4 opcodes. 969 """), 970 971 I(name="LONG1", 972 code='\x8a', 973 arg=long1, 974 stack_before=[], 975 stack_after=[pylong], 976 proto=2, 977 doc="""Long integer using one-byte length. 978 979 A more efficient encoding of a Python long; the long1 encoding 980 says it all."""), 981 982 I(name="LONG4", 983 code='\x8b', 984 arg=long4, 985 stack_before=[], 986 stack_after=[pylong], 987 proto=2, 988 doc="""Long integer using found-byte length. 989 990 A more efficient encoding of a Python long; the long4 encoding 991 says it all."""), 992 993 # Ways to spell strings (8-bit, not Unicode). 994 995 I(name='STRING', 996 code='S', 997 arg=stringnl, 998 stack_before=[], 999 stack_after=[pystring], 1000 proto=0, 1001 doc="""Push a Python string object. 1002 1003 The argument is a repr-style string, with bracketing quote characters, 1004 and perhaps embedded escapes. The argument extends until the next 1005 newline character. 1006 """), 1007 1008 I(name='BINSTRING', 1009 code='T', 1010 arg=string4, 1011 stack_before=[], 1012 stack_after=[pystring], 1013 proto=1, 1014 doc="""Push a Python string object. 1015 1016 There are two arguments: the first is a 4-byte little-endian signed int 1017 giving the number of bytes in the string, and the second is that many 1018 bytes, which are taken literally as the string content. 1019 """), 1020 1021 I(name='SHORT_BINSTRING', 1022 code='U', 1023 arg=string1, 1024 stack_before=[], 1025 stack_after=[pystring], 1026 proto=1, 1027 doc="""Push a Python string object. 1028 1029 There are two arguments: the first is a 1-byte unsigned int giving 1030 the number of bytes in the string, and the second is that many bytes, 1031 which are taken literally as the string content. 1032 """), 1033 1034 # Ways to spell None. 1035 1036 I(name='NONE', 1037 code='N', 1038 arg=None, 1039 stack_before=[], 1040 stack_after=[pynone], 1041 proto=0, 1042 doc="Push None on the stack."), 1043 1044 # Ways to spell bools, starting with proto 2. See INT for how this was 1045 # done before proto 2. 1046 1047 I(name='NEWTRUE', 1048 code='\x88', 1049 arg=None, 1050 stack_before=[], 1051 stack_after=[pybool], 1052 proto=2, 1053 doc="""True. 1054 1055 Push True onto the stack."""), 1056 1057 I(name='NEWFALSE', 1058 code='\x89', 1059 arg=None, 1060 stack_before=[], 1061 stack_after=[pybool], 1062 proto=2, 1063 doc="""True. 1064 1065 Push False onto the stack."""), 1066 1067 # Ways to spell Unicode strings. 1068 1069 I(name='UNICODE', 1070 code='V', 1071 arg=unicodestringnl, 1072 stack_before=[], 1073 stack_after=[pyunicode], 1074 proto=0, # this may be pure-text, but it's a later addition 1075 doc="""Push a Python Unicode string object. 1076 1077 The argument is a raw-unicode-escape encoding of a Unicode string, 1078 and so may contain embedded escape sequences. The argument extends 1079 until the next newline character. 1080 """), 1081 1082 I(name='BINUNICODE', 1083 code='X', 1084 arg=unicodestring4, 1085 stack_before=[], 1086 stack_after=[pyunicode], 1087 proto=1, 1088 doc="""Push a Python Unicode string object. 1089 1090 There are two arguments: the first is a 4-byte little-endian signed int 1091 giving the number of bytes in the string. The second is that many 1092 bytes, and is the UTF-8 encoding of the Unicode string. 1093 """), 1094 1095 # Ways to spell floats. 1096 1097 I(name='FLOAT', 1098 code='F', 1099 arg=floatnl, 1100 stack_before=[], 1101 stack_after=[pyfloat], 1102 proto=0, 1103 doc="""Newline-terminated decimal float literal. 1104 1105 The argument is repr(a_float), and in general requires 17 significant 1106 digits for roundtrip conversion to be an identity (this is so for 1107 IEEE-754 double precision values, which is what Python float maps to 1108 on most boxes). 1109 1110 In general, FLOAT cannot be used to transport infinities, NaNs, or 1111 minus zero across boxes (or even on a single box, if the platform C 1112 library can't read the strings it produces for such things -- Windows 1113 is like that), but may do less damage than BINFLOAT on boxes with 1114 greater precision or dynamic range than IEEE-754 double. 1115 """), 1116 1117 I(name='BINFLOAT', 1118 code='G', 1119 arg=float8, 1120 stack_before=[], 1121 stack_after=[pyfloat], 1122 proto=1, 1123 doc="""Float stored in binary form, with 8 bytes of data. 1124 1125 This generally requires less than half the space of FLOAT encoding. 1126 In general, BINFLOAT cannot be used to transport infinities, NaNs, or 1127 minus zero, raises an exception if the exponent exceeds the range of 1128 an IEEE-754 double, and retains no more than 53 bits of precision (if 1129 there are more than that, "add a half and chop" rounding is used to 1130 cut it back to 53 significant bits). 1131 """), 1132 1133 # Ways to build lists. 1134 1135 I(name='EMPTY_LIST', 1136 code=']', 1137 arg=None, 1138 stack_before=[], 1139 stack_after=[pylist], 1140 proto=1, 1141 doc="Push an empty list."), 1142 1143 I(name='APPEND', 1144 code='a', 1145 arg=None, 1146 stack_before=[pylist, anyobject], 1147 stack_after=[pylist], 1148 proto=0, 1149 doc="""Append an object to a list. 1150 1151 Stack before: ... pylist anyobject 1152 Stack after: ... pylist+[anyobject] 1153 1154 although pylist is really extended in-place. 1155 """), 1156 1157 I(name='APPENDS', 1158 code='e', 1159 arg=None, 1160 stack_before=[pylist, markobject, stackslice], 1161 stack_after=[pylist], 1162 proto=1, 1163 doc="""Extend a list by a slice of stack objects. 1164 1165 Stack before: ... pylist markobject stackslice 1166 Stack after: ... pylist+stackslice 1167 1168 although pylist is really extended in-place. 1169 """), 1170 1171 I(name='LIST', 1172 code='l', 1173 arg=None, 1174 stack_before=[markobject, stackslice], 1175 stack_after=[pylist], 1176 proto=0, 1177 doc="""Build a list out of the topmost stack slice, after markobject. 1178 1179 All the stack entries following the topmost markobject are placed into 1180 a single Python list, which single list object replaces all of the 1181 stack from the topmost markobject onward. For example, 1182 1183 Stack before: ... markobject 1 2 3 'abc' 1184 Stack after: ... [1, 2, 3, 'abc'] 1185 """), 1186 1187 # Ways to build tuples. 1188 1189 I(name='EMPTY_TUPLE', 1190 code=')', 1191 arg=None, 1192 stack_before=[], 1193 stack_after=[pytuple], 1194 proto=1, 1195 doc="Push an empty tuple."), 1196 1197 I(name='TUPLE', 1198 code='t', 1199 arg=None, 1200 stack_before=[markobject, stackslice], 1201 stack_after=[pytuple], 1202 proto=0, 1203 doc="""Build a tuple out of the topmost stack slice, after markobject. 1204 1205 All the stack entries following the topmost markobject are placed into 1206 a single Python tuple, which single tuple object replaces all of the 1207 stack from the topmost markobject onward. For example, 1208 1209 Stack before: ... markobject 1 2 3 'abc' 1210 Stack after: ... (1, 2, 3, 'abc') 1211 """), 1212 1213 I(name='TUPLE1', 1214 code='\x85', 1215 arg=None, 1216 stack_before=[anyobject], 1217 stack_after=[pytuple], 1218 proto=2, 1219 doc="""One-tuple. 1220 1221 This code pops one value off the stack and pushes a tuple of 1222 length 1 whose one item is that value back onto it. IOW: 1223 1224 stack[-1] = tuple(stack[-1:]) 1225 """), 1226 1227 I(name='TUPLE2', 1228 code='\x86', 1229 arg=None, 1230 stack_before=[anyobject, anyobject], 1231 stack_after=[pytuple], 1232 proto=2, 1233 doc="""One-tuple. 1234 1235 This code pops two values off the stack and pushes a tuple 1236 of length 2 whose items are those values back onto it. IOW: 1237 1238 stack[-2:] = [tuple(stack[-2:])] 1239 """), 1240 1241 I(name='TUPLE3', 1242 code='\x87', 1243 arg=None, 1244 stack_before=[anyobject, anyobject, anyobject], 1245 stack_after=[pytuple], 1246 proto=2, 1247 doc="""One-tuple. 1248 1249 This code pops three values off the stack and pushes a tuple 1250 of length 3 whose items are those values back onto it. IOW: 1251 1252 stack[-3:] = [tuple(stack[-3:])] 1253 """), 1254 1255 # Ways to build dicts. 1256 1257 I(name='EMPTY_DICT', 1258 code='}', 1259 arg=None, 1260 stack_before=[], 1261 stack_after=[pydict], 1262 proto=1, 1263 doc="Push an empty dict."), 1264 1265 I(name='DICT', 1266 code='d', 1267 arg=None, 1268 stack_before=[markobject, stackslice], 1269 stack_after=[pydict], 1270 proto=0, 1271 doc="""Build a dict out of the topmost stack slice, after markobject. 1272 1273 All the stack entries following the topmost markobject are placed into 1274 a single Python dict, which single dict object replaces all of the 1275 stack from the topmost markobject onward. The stack slice alternates 1276 key, value, key, value, .... For example, 1277 1278 Stack before: ... markobject 1 2 3 'abc' 1279 Stack after: ... {1: 2, 3: 'abc'} 1280 """), 1281 1282 I(name='SETITEM', 1283 code='s', 1284 arg=None, 1285 stack_before=[pydict, anyobject, anyobject], 1286 stack_after=[pydict], 1287 proto=0, 1288 doc="""Add a key+value pair to an existing dict. 1289 1290 Stack before: ... pydict key value 1291 Stack after: ... pydict 1292 1293 where pydict has been modified via pydict[key] = value. 1294 """), 1295 1296 I(name='SETITEMS', 1297 code='u', 1298 arg=None, 1299 stack_before=[pydict, markobject, stackslice], 1300 stack_after=[pydict], 1301 proto=1, 1302 doc="""Add an arbitrary number of key+value pairs to an existing dict. 1303 1304 The slice of the stack following the topmost markobject is taken as 1305 an alternating sequence of keys and values, added to the dict 1306 immediately under the topmost markobject. Everything at and after the 1307 topmost markobject is popped, leaving the mutated dict at the top 1308 of the stack. 1309 1310 Stack before: ... pydict markobject key_1 value_1 ... key_n value_n 1311 Stack after: ... pydict 1312 1313 where pydict has been modified via pydict[key_i] = value_i for i in 1314 1, 2, ..., n, and in that order. 1315 """), 1316 1317 # Stack manipulation. 1318 1319 I(name='POP', 1320 code='0', 1321 arg=None, 1322 stack_before=[anyobject], 1323 stack_after=[], 1324 proto=0, 1325 doc="Discard the top stack item, shrinking the stack by one item."), 1326 1327 I(name='DUP', 1328 code='2', 1329 arg=None, 1330 stack_before=[anyobject], 1331 stack_after=[anyobject, anyobject], 1332 proto=0, 1333 doc="Push the top stack item onto the stack again, duplicating it."), 1334 1335 I(name='MARK', 1336 code='(', 1337 arg=None, 1338 stack_before=[], 1339 stack_after=[markobject], 1340 proto=0, 1341 doc="""Push markobject onto the stack. 1342 1343 markobject is a unique object, used by other opcodes to identify a 1344 region of the stack containing a variable number of objects for them 1345 to work on. See markobject.doc for more detail. 1346 """), 1347 1348 I(name='POP_MARK', 1349 code='1', 1350 arg=None, 1351 stack_before=[markobject, stackslice], 1352 stack_after=[], 1353 proto=0, 1354 doc="""Pop all the stack objects at and above the topmost markobject. 1355 1356 When an opcode using a variable number of stack objects is done, 1357 POP_MARK is used to remove those objects, and to remove the markobject 1358 that delimited their starting position on the stack. 1359 """), 1360 1361 # Memo manipulation. There are really only two operations (get and put), 1362 # each in all-text, "short binary", and "long binary" flavors. 1363 1364 I(name='GET', 1365 code='g', 1366 arg=decimalnl_short, 1367 stack_before=[], 1368 stack_after=[anyobject], 1369 proto=0, 1370 doc="""Read an object from the memo and push it on the stack. 1371 1372 The index of the memo object to push is given by the newline-teriminated 1373 decimal string following. BINGET and LONG_BINGET are space-optimized 1374 versions. 1375 """), 1376 1377 I(name='BINGET', 1378 code='h', 1379 arg=uint1, 1380 stack_before=[], 1381 stack_after=[anyobject], 1382 proto=1, 1383 doc="""Read an object from the memo and push it on the stack. 1384 1385 The index of the memo object to push is given by the 1-byte unsigned 1386 integer following. 1387 """), 1388 1389 I(name='LONG_BINGET', 1390 code='j', 1391 arg=int4, 1392 stack_before=[], 1393 stack_after=[anyobject], 1394 proto=1, 1395 doc="""Read an object from the memo and push it on the stack. 1396 1397 The index of the memo object to push is given by the 4-byte signed 1398 little-endian integer following. 1399 """), 1400 1401 I(name='PUT', 1402 code='p', 1403 arg=decimalnl_short, 1404 stack_before=[], 1405 stack_after=[], 1406 proto=0, 1407 doc="""Store the stack top into the memo. The stack is not popped. 1408 1409 The index of the memo location to write into is given by the newline- 1410 terminated decimal string following. BINPUT and LONG_BINPUT are 1411 space-optimized versions. 1412 """), 1413 1414 I(name='BINPUT', 1415 code='q', 1416 arg=uint1, 1417 stack_before=[], 1418 stack_after=[], 1419 proto=1, 1420 doc="""Store the stack top into the memo. The stack is not popped. 1421 1422 The index of the memo location to write into is given by the 1-byte 1423 unsigned integer following. 1424 """), 1425 1426 I(name='LONG_BINPUT', 1427 code='r', 1428 arg=int4, 1429 stack_before=[], 1430 stack_after=[], 1431 proto=1, 1432 doc="""Store the stack top into the memo. The stack is not popped. 1433 1434 The index of the memo location to write into is given by the 4-byte 1435 signed little-endian integer following. 1436 """), 1437 1438 # Access the extension registry (predefined objects). Akin to the GET 1439 # family. 1440 1441 I(name='EXT1', 1442 code='\x82', 1443 arg=uint1, 1444 stack_before=[], 1445 stack_after=[anyobject], 1446 proto=2, 1447 doc="""Extension code. 1448 1449 This code and the similar EXT2 and EXT4 allow using a registry 1450 of popular objects that are pickled by name, typically classes. 1451 It is envisioned that through a global negotiation and 1452 registration process, third parties can set up a mapping between 1453 ints and object names. 1454 1455 In order to guarantee pickle interchangeability, the extension 1456 code registry ought to be global, although a range of codes may 1457 be reserved for private use. 1458 1459 EXT1 has a 1-byte integer argument. This is used to index into the 1460 extension registry, and the object at that index is pushed on the stack. 1461 """), 1462 1463 I(name='EXT2', 1464 code='\x83', 1465 arg=uint2, 1466 stack_before=[], 1467 stack_after=[anyobject], 1468 proto=2, 1469 doc="""Extension code. 1470 1471 See EXT1. EXT2 has a two-byte integer argument. 1472 """), 1473 1474 I(name='EXT4', 1475 code='\x84', 1476 arg=int4, 1477 stack_before=[], 1478 stack_after=[anyobject], 1479 proto=2, 1480 doc="""Extension code. 1481 1482 See EXT1. EXT4 has a four-byte integer argument. 1483 """), 1484 1485 # Push a class object, or module function, on the stack, via its module 1486 # and name. 1487 1488 I(name='GLOBAL', 1489 code='c', 1490 arg=stringnl_noescape_pair, 1491 stack_before=[], 1492 stack_after=[anyobject], 1493 proto=0, 1494 doc="""Push a global object (module.attr) on the stack. 1495 1496 Two newline-terminated strings follow the GLOBAL opcode. The first is 1497 taken as a module name, and the second as a class name. The class 1498 object module.class is pushed on the stack. More accurately, the 1499 object returned by self.find_class(module, class) is pushed on the 1500 stack, so unpickling subclasses can override this form of lookup. 1501 """), 1502 1503 # Ways to build objects of classes pickle doesn't know about directly 1504 # (user-defined classes). I despair of documenting this accurately 1505 # and comprehensibly -- you really have to read the pickle code to 1506 # find all the special cases. 1507 1508 I(name='REDUCE', 1509 code='R', 1510 arg=None, 1511 stack_before=[anyobject, anyobject], 1512 stack_after=[anyobject], 1513 proto=0, 1514 doc="""Push an object built from a callable and an argument tuple. 1515 1516 The opcode is named to remind of the __reduce__() method. 1517 1518 Stack before: ... callable pytuple 1519 Stack after: ... callable(*pytuple) 1520 1521 The callable and the argument tuple are the first two items returned 1522 by a __reduce__ method. Applying the callable to the argtuple is 1523 supposed to reproduce the original object, or at least get it started. 1524 If the __reduce__ method returns a 3-tuple, the last component is an 1525 argument to be passed to the object's __setstate__, and then the REDUCE 1526 opcode is followed by code to create setstate's argument, and then a 1527 BUILD opcode to apply __setstate__ to that argument. 1528 1529 There are lots of special cases here. The argtuple can be None, in 1530 which case callable.__basicnew__() is called instead to produce the 1531 object to be pushed on the stack. This appears to be a trick unique 1532 to ExtensionClasses, and is deprecated regardless. 1533 1534 If type(callable) is not ClassType, REDUCE complains unless the 1535 callable has been registered with the copy_reg module's 1536 safe_constructors dict, or the callable has a magic 1537 '__safe_for_unpickling__' attribute with a true value. I'm not sure 1538 why it does this, but I've sure seen this complaint often enough when 1539 I didn't want to <wink>. 1540 """), 1541 1542 I(name='BUILD', 1543 code='b', 1544 arg=None, 1545 stack_before=[anyobject, anyobject], 1546 stack_after=[anyobject], 1547 proto=0, 1548 doc="""Finish building an object, via __setstate__ or dict update. 1549 1550 Stack before: ... anyobject argument 1551 Stack after: ... anyobject 1552 1553 where anyobject may have been mutated, as follows: 1554 1555 If the object has a __setstate__ method, 1556 1557 anyobject.__setstate__(argument) 1558 1559 is called. 1560 1561 Else the argument must be a dict, the object must have a __dict__, and 1562 the object is updated via 1563 1564 anyobject.__dict__.update(argument) 1565 1566 This may raise RuntimeError in restricted execution mode (which 1567 disallows access to __dict__ directly); in that case, the object 1568 is updated instead via 1569 1570 for k, v in argument.items(): 1571 anyobject[k] = v 1572 """), 1573 1574 I(name='INST', 1575 code='i', 1576 arg=stringnl_noescape_pair, 1577 stack_before=[markobject, stackslice], 1578 stack_after=[anyobject], 1579 proto=0, 1580 doc="""Build a class instance. 1581 1582 This is the protocol 0 version of protocol 1's OBJ opcode. 1583 INST is followed by two newline-terminated strings, giving a 1584 module and class name, just as for the GLOBAL opcode (and see 1585 GLOBAL for more details about that). self.find_class(module, name) 1586 is used to get a class object. 1587 1588 In addition, all the objects on the stack following the topmost 1589 markobject are gathered into a tuple and popped (along with the 1590 topmost markobject), just as for the TUPLE opcode. 1591 1592 Now it gets complicated. If all of these are true: 1593 1594 + The argtuple is empty (markobject was at the top of the stack 1595 at the start). 1596 1597 + It's an old-style class object (the type of the class object is 1598 ClassType). 1599 1600 + The class object does not have a __getinitargs__ attribute. 1601 1602 then we want to create an old-style class instance without invoking 1603 its __init__() method (pickle has waffled on this over the years; not 1604 calling __init__() is current wisdom). In this case, an instance of 1605 an old-style dummy class is created, and then we try to rebind its 1606 __class__ attribute to the desired class object. If this succeeds, 1607 the new instance object is pushed on the stack, and we're done. In 1608 restricted execution mode it can fail (assignment to __class__ is 1609 disallowed), and I'm not really sure what happens then -- it looks 1610 like the code ends up calling the class object's __init__ anyway, 1611 via falling into the next case. 1612 1613 Else (the argtuple is not empty, it's not an old-style class object, 1614 or the class object does have a __getinitargs__ attribute), the code 1615 first insists that the class object have a __safe_for_unpickling__ 1616 attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, 1617 it doesn't matter whether this attribute has a true or false value, it 1618 only matters whether it exists (XXX this is a bug; cPickle 1619 requires the attribute to be true). If __safe_for_unpickling__ 1620 doesn't exist, UnpicklingError is raised. 1621 1622 Else (the class object does have a __safe_for_unpickling__ attr), 1623 the class object obtained from INST's arguments is applied to the 1624 argtuple obtained from the stack, and the resulting instance object 1625 is pushed on the stack. 1626 1627 NOTE: checks for __safe_for_unpickling__ went away in Python 2.3. 1628 """), 1629 1630 I(name='OBJ', 1631 code='o', 1632 arg=None, 1633 stack_before=[markobject, anyobject, stackslice], 1634 stack_after=[anyobject], 1635 proto=1, 1636 doc="""Build a class instance. 1637 1638 This is the protocol 1 version of protocol 0's INST opcode, and is 1639 very much like it. The major difference is that the class object 1640 is taken off the stack, allowing it to be retrieved from the memo 1641 repeatedly if several instances of the same class are created. This 1642 can be much more efficient (in both time and space) than repeatedly 1643 embedding the module and class names in INST opcodes. 1644 1645 Unlike INST, OBJ takes no arguments from the opcode stream. Instead 1646 the class object is taken off the stack, immediately above the 1647 topmost markobject: 1648 1649 Stack before: ... markobject classobject stackslice 1650 Stack after: ... new_instance_object 1651 1652 As for INST, the remainder of the stack above the markobject is 1653 gathered into an argument tuple, and then the logic seems identical, 1654 except that no __safe_for_unpickling__ check is done (XXX this is 1655 a bug; cPickle does test __safe_for_unpickling__). See INST for 1656 the gory details. 1657 1658 NOTE: In Python 2.3, INST and OBJ are identical except for how they 1659 get the class object. That was always the intent; the implementations 1660 had diverged for accidental reasons. 1661 """), 1662 1663 I(name='NEWOBJ', 1664 code='\x81', 1665 arg=None, 1666 stack_before=[anyobject, anyobject], 1667 stack_after=[anyobject], 1668 proto=2, 1669 doc="""Build an object instance. 1670 1671 The stack before should be thought of as containing a class 1672 object followed by an argument tuple (the tuple being the stack 1673 top). Call these cls and args. They are popped off the stack, 1674 and the value returned by cls.__new__(cls, *args) is pushed back 1675 onto the stack. 1676 """), 1677 1678 # Machine control. 1679 1680 I(name='PROTO', 1681 code='\x80', 1682 arg=uint1, 1683 stack_before=[], 1684 stack_after=[], 1685 proto=2, 1686 doc="""Protocol version indicator. 1687 1688 For protocol 2 and above, a pickle must start with this opcode. 1689 The argument is the protocol version, an int in range(2, 256). 1690 """), 1691 1692 I(name='STOP', 1693 code='.', 1694 arg=None, 1695 stack_before=[anyobject], 1696 stack_after=[], 1697 proto=0, 1698 doc="""Stop the unpickling machine. 1699 1700 Every pickle ends with this opcode. The object at the top of the stack 1701 is popped, and that's the result of unpickling. The stack should be 1702 empty then. 1703 """), 1704 1705 # Ways to deal with persistent IDs. 1706 1707 I(name='PERSID', 1708 code='P', 1709 arg=stringnl_noescape, 1710 stack_before=[], 1711 stack_after=[anyobject], 1712 proto=0, 1713 doc="""Push an object identified by a persistent ID. 1714 1715 The pickle module doesn't define what a persistent ID means. PERSID's 1716 argument is a newline-terminated str-style (no embedded escapes, no 1717 bracketing quote characters) string, which *is* "the persistent ID". 1718 The unpickler passes this string to self.persistent_load(). Whatever 1719 object that returns is pushed on the stack. There is no implementation 1720 of persistent_load() in Python's unpickler: it must be supplied by an 1721 unpickler subclass. 1722 """), 1723 1724 I(name='BINPERSID', 1725 code='Q', 1726 arg=None, 1727 stack_before=[anyobject], 1728 stack_after=[anyobject], 1729 proto=1, 1730 doc="""Push an object identified by a persistent ID. 1731 1732 Like PERSID, except the persistent ID is popped off the stack (instead 1733 of being a string embedded in the opcode bytestream). The persistent 1734 ID is passed to self.persistent_load(), and whatever object that 1735 returns is pushed on the stack. See PERSID for more detail. 1736 """), 1737] 1738del I 1739 1740# Verify uniqueness of .name and .code members. 1741name2i = {} 1742code2i = {} 1743 1744for i, d in enumerate(opcodes): 1745 if d.name in name2i: 1746 raise ValueError("repeated name %r at indices %d and %d" % 1747 (d.name, name2i[d.name], i)) 1748 if d.code in code2i: 1749 raise ValueError("repeated code %r at indices %d and %d" % 1750 (d.code, code2i[d.code], i)) 1751 1752 name2i[d.name] = i 1753 code2i[d.code] = i 1754 1755del name2i, code2i, i, d 1756 1757############################################################################## 1758# Build a code2op dict, mapping opcode characters to OpcodeInfo records. 1759# Also ensure we've got the same stuff as pickle.py, although the 1760# introspection here is dicey. 1761 1762code2op = {} 1763for d in opcodes: 1764 code2op[d.code] = d 1765del d 1766 1767def assure_pickle_consistency(verbose=False): 1768 import pickle, re 1769 1770 copy = code2op.copy() 1771 for name in pickle.__all__: 1772 if not re.match("[A-Z][A-Z0-9_]+$", name): 1773 if verbose: 1774 print "skipping %r: it doesn't look like an opcode name" % name 1775 continue 1776 picklecode = getattr(pickle, name) 1777 if not isinstance(picklecode, str) or len(picklecode) != 1: 1778 if verbose: 1779 print ("skipping %r: value %r doesn't look like a pickle " 1780 "code" % (name, picklecode)) 1781 continue 1782 if picklecode in copy: 1783 if verbose: 1784 print "checking name %r w/ code %r for consistency" % ( 1785 name, picklecode) 1786 d = copy[picklecode] 1787 if d.name != name: 1788 raise ValueError("for pickle code %r, pickle.py uses name %r " 1789 "but we're using name %r" % (picklecode, 1790 name, 1791 d.name)) 1792 # Forget this one. Any left over in copy at the end are a problem 1793 # of a different kind. 1794 del copy[picklecode] 1795 else: 1796 raise ValueError("pickle.py appears to have a pickle opcode with " 1797 "name %r and code %r, but we don't" % 1798 (name, picklecode)) 1799 if copy: 1800 msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"] 1801 for code, d in copy.items(): 1802 msg.append(" name %r with code %r" % (d.name, code)) 1803 raise ValueError("\n".join(msg)) 1804 1805assure_pickle_consistency() 1806del assure_pickle_consistency 1807 1808############################################################################## 1809# A pickle opcode generator. 1810 1811def genops(pickle): 1812 """Generate all the opcodes in a pickle. 1813 1814 'pickle' is a file-like object, or string, containing the pickle. 1815 1816 Each opcode in the pickle is generated, from the current pickle position, 1817 stopping after a STOP opcode is delivered. A triple is generated for 1818 each opcode: 1819 1820 opcode, arg, pos 1821 1822 opcode is an OpcodeInfo record, describing the current opcode. 1823 1824 If the opcode has an argument embedded in the pickle, arg is its decoded 1825 value, as a Python object. If the opcode doesn't have an argument, arg 1826 is None. 1827 1828 If the pickle has a tell() method, pos was the value of pickle.tell() 1829 before reading the current opcode. If the pickle is a string object, 1830 it's wrapped in a StringIO object, and the latter's tell() result is 1831 used. Else (the pickle doesn't have a tell(), and it's not obvious how 1832 to query its current position) pos is None. 1833 """ 1834 1835 import cStringIO as StringIO 1836 1837 if isinstance(pickle, str): 1838 pickle = StringIO.StringIO(pickle) 1839 1840 if hasattr(pickle, "tell"): 1841 getpos = pickle.tell 1842 else: 1843 getpos = lambda: None 1844 1845 while True: 1846 pos = getpos() 1847 code = pickle.read(1) 1848 opcode = code2op.get(code) 1849 if opcode is None: 1850 if code == "": 1851 raise ValueError("pickle exhausted before seeing STOP") 1852 else: 1853 raise ValueError("at position %s, opcode %r unknown" % ( 1854 pos is None and "<unknown>" or pos, 1855 code)) 1856 if opcode.arg is None: 1857 arg = None 1858 else: 1859 arg = opcode.arg.reader(pickle) 1860 yield opcode, arg, pos 1861 if code == '.': 1862 assert opcode.name == 'STOP' 1863 break 1864 1865############################################################################## 1866# A symbolic pickle disassembler. 1867 1868def dis(pickle, out=None, memo=None, indentlevel=4): 1869 """Produce a symbolic disassembly of a pickle. 1870 1871 'pickle' is a file-like object, or string, containing a (at least one) 1872 pickle. The pickle is disassembled from the current position, through 1873 the first STOP opcode encountered. 1874 1875 Optional arg 'out' is a file-like object to which the disassembly is 1876 printed. It defaults to sys.stdout. 1877 1878 Optional arg 'memo' is a Python dict, used as the pickle's memo. It 1879 may be mutated by dis(), if the pickle contains PUT or BINPUT opcodes. 1880 Passing the same memo object to another dis() call then allows disassembly 1881 to proceed across multiple pickles that were all created by the same 1882 pickler with the same memo. Ordinarily you don't need to worry about this. 1883 1884 Optional arg indentlevel is the number of blanks by which to indent 1885 a new MARK level. It defaults to 4. 1886 1887 In addition to printing the disassembly, some sanity checks are made: 1888 1889 + All embedded opcode arguments "make sense". 1890 1891 + Explicit and implicit pop operations have enough items on the stack. 1892 1893 + When an opcode implicitly refers to a markobject, a markobject is 1894 actually on the stack. 1895 1896 + A memo entry isn't referenced before it's defined. 1897 1898 + The markobject isn't stored in the memo. 1899 1900 + A memo entry isn't redefined. 1901 """ 1902 1903 # Most of the hair here is for sanity checks, but most of it is needed 1904 # anyway to detect when a protocol 0 POP takes a MARK off the stack 1905 # (which in turn is needed to indent MARK blocks correctly). 1906 1907 stack = [] # crude emulation of unpickler stack 1908 if memo is None: 1909 memo = {} # crude emulation of unpicker memo 1910 maxproto = -1 # max protocol number seen 1911 markstack = [] # bytecode positions of MARK opcodes 1912 indentchunk = ' ' * indentlevel 1913 errormsg = None 1914 for opcode, arg, pos in genops(pickle): 1915 if pos is not None: 1916 print >> out, "%5d:" % pos, 1917 1918 line = "%-4s %s%s" % (repr(opcode.code)[1:-1], 1919 indentchunk * len(markstack), 1920 opcode.name) 1921 1922 maxproto = max(maxproto, opcode.proto) 1923 before = opcode.stack_before # don't mutate 1924 after = opcode.stack_after # don't mutate 1925 numtopop = len(before) 1926 1927 # See whether a MARK should be popped. 1928 markmsg = None 1929 if markobject in before or (opcode.name == "POP" and 1930 stack and 1931 stack[-1] is markobject): 1932 assert markobject not in after 1933 if __debug__: 1934 if markobject in before: 1935 assert before[-1] is stackslice 1936 if markstack: 1937 markpos = markstack.pop() 1938 if markpos is None: 1939 markmsg = "(MARK at unknown opcode offset)" 1940 else: 1941 markmsg = "(MARK at %d)" % markpos 1942 # Pop everything at and after the topmost markobject. 1943 while stack[-1] is not markobject: 1944 stack.pop() 1945 stack.pop() 1946 # Stop later code from popping too much. 1947 try: 1948 numtopop = before.index(markobject) 1949 except ValueError: 1950 assert opcode.name == "POP" 1951 numtopop = 0 1952 else: 1953 errormsg = markmsg = "no MARK exists on stack" 1954 1955 # Check for correct memo usage. 1956 if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"): 1957 assert arg is not None 1958 if arg in memo: 1959 errormsg = "memo key %r already defined" % arg 1960 elif not stack: 1961 errormsg = "stack is empty -- can't store into memo" 1962 elif stack[-1] is markobject: 1963 errormsg = "can't store markobject in the memo" 1964 else: 1965 memo[arg] = stack[-1] 1966 1967 elif opcode.name in ("GET", "BINGET", "LONG_BINGET"): 1968 if arg in memo: 1969 assert len(after) == 1 1970 after = [memo[arg]] # for better stack emulation 1971 else: 1972 errormsg = "memo key %r has never been stored into" % arg 1973 1974 if arg is not None or markmsg: 1975 # make a mild effort to align arguments 1976 line += ' ' * (10 - len(opcode.name)) 1977 if arg is not None: 1978 line += ' ' + repr(arg) 1979 if markmsg: 1980 line += ' ' + markmsg 1981 print >> out, line 1982 1983 if errormsg: 1984 # Note that we delayed complaining until the offending opcode 1985 # was printed. 1986 raise ValueError(errormsg) 1987 1988 # Emulate the stack effects. 1989 if len(stack) < numtopop: 1990 raise ValueError("tries to pop %d items from stack with " 1991 "only %d items" % (numtopop, len(stack))) 1992 if numtopop: 1993 del stack[-numtopop:] 1994 if markobject in after: 1995 assert markobject not in before 1996 markstack.append(pos) 1997 1998 stack.extend(after) 1999 2000 print >> out, "highest protocol among opcodes =", maxproto 2001 if stack: 2002 raise ValueError("stack not empty after STOP: %r" % stack) 2003 2004_dis_test = r""" 2005>>> import pickle 2006>>> x = [1, 2, (3, 4), {'abc': u"def"}] 2007>>> pkl = pickle.dumps(x, 0) 2008>>> dis(pkl) 2009 0: ( MARK 2010 1: l LIST (MARK at 0) 2011 2: p PUT 0 2012 5: I INT 1 2013 8: a APPEND 2014 9: I INT 2 2015 12: a APPEND 2016 13: ( MARK 2017 14: I INT 3 2018 17: I INT 4 2019 20: t TUPLE (MARK at 13) 2020 21: p PUT 1 2021 24: a APPEND 2022 25: ( MARK 2023 26: d DICT (MARK at 25) 2024 27: p PUT 2 2025 30: S STRING 'abc' 2026 37: p PUT 3 2027 40: V UNICODE u'def' 2028 45: p PUT 4 2029 48: s SETITEM 2030 49: a APPEND 2031 50: . STOP 2032highest protocol among opcodes = 0 2033 2034Try again with a "binary" pickle. 2035 2036>>> pkl = pickle.dumps(x, 1) 2037>>> dis(pkl) 2038 0: ] EMPTY_LIST 2039 1: q BINPUT 0 2040 3: ( MARK 2041 4: K BININT1 1 2042 6: K BININT1 2 2043 8: ( MARK 2044 9: K BININT1 3 2045 11: K BININT1 4 2046 13: t TUPLE (MARK at 8) 2047 14: q BINPUT 1 2048 16: } EMPTY_DICT 2049 17: q BINPUT 2 2050 19: U SHORT_BINSTRING 'abc' 2051 24: q BINPUT 3 2052 26: X BINUNICODE u'def' 2053 34: q BINPUT 4 2054 36: s SETITEM 2055 37: e APPENDS (MARK at 3) 2056 38: . STOP 2057highest protocol among opcodes = 1 2058 2059Exercise the INST/OBJ/BUILD family. 2060 2061>>> import random 2062>>> dis(pickle.dumps(random.random, 0)) 2063 0: c GLOBAL 'random random' 2064 15: p PUT 0 2065 18: . STOP 2066highest protocol among opcodes = 0 2067 2068>>> x = [pickle.PicklingError()] * 2 2069>>> dis(pickle.dumps(x, 0)) 2070 0: ( MARK 2071 1: l LIST (MARK at 0) 2072 2: p PUT 0 2073 5: ( MARK 2074 6: i INST 'pickle PicklingError' (MARK at 5) 2075 28: p PUT 1 2076 31: ( MARK 2077 32: d DICT (MARK at 31) 2078 33: p PUT 2 2079 36: S STRING 'args' 2080 44: p PUT 3 2081 47: ( MARK 2082 48: t TUPLE (MARK at 47) 2083 49: s SETITEM 2084 50: b BUILD 2085 51: a APPEND 2086 52: g GET 1 2087 55: a APPEND 2088 56: . STOP 2089highest protocol among opcodes = 0 2090 2091>>> dis(pickle.dumps(x, 1)) 2092 0: ] EMPTY_LIST 2093 1: q BINPUT 0 2094 3: ( MARK 2095 4: ( MARK 2096 5: c GLOBAL 'pickle PicklingError' 2097 27: q BINPUT 1 2098 29: o OBJ (MARK at 4) 2099 30: q BINPUT 2 2100 32: } EMPTY_DICT 2101 33: q BINPUT 3 2102 35: U SHORT_BINSTRING 'args' 2103 41: q BINPUT 4 2104 43: ) EMPTY_TUPLE 2105 44: s SETITEM 2106 45: b BUILD 2107 46: h BINGET 2 2108 48: e APPENDS (MARK at 3) 2109 49: . STOP 2110highest protocol among opcodes = 1 2111 2112Try "the canonical" recursive-object test. 2113 2114>>> L = [] 2115>>> T = L, 2116>>> L.append(T) 2117>>> L[0] is T 2118True 2119>>> T[0] is L 2120True 2121>>> L[0][0] is L 2122True 2123>>> T[0][0] is T 2124True 2125>>> dis(pickle.dumps(L, 0)) 2126 0: ( MARK 2127 1: l LIST (MARK at 0) 2128 2: p PUT 0 2129 5: ( MARK 2130 6: g GET 0 2131 9: t TUPLE (MARK at 5) 2132 10: p PUT 1 2133 13: a APPEND 2134 14: . STOP 2135highest protocol among opcodes = 0 2136 2137>>> dis(pickle.dumps(L, 1)) 2138 0: ] EMPTY_LIST 2139 1: q BINPUT 0 2140 3: ( MARK 2141 4: h BINGET 0 2142 6: t TUPLE (MARK at 3) 2143 7: q BINPUT 1 2144 9: a APPEND 2145 10: . STOP 2146highest protocol among opcodes = 1 2147 2148Note that, in the protocol 0 pickle of the recursive tuple, the disassembler 2149has to emulate the stack in order to realize that the POP opcode at 16 gets 2150rid of the MARK at 0. 2151 2152>>> dis(pickle.dumps(T, 0)) 2153 0: ( MARK 2154 1: ( MARK 2155 2: l LIST (MARK at 1) 2156 3: p PUT 0 2157 6: ( MARK 2158 7: g GET 0 2159 10: t TUPLE (MARK at 6) 2160 11: p PUT 1 2161 14: a APPEND 2162 15: 0 POP 2163 16: 0 POP (MARK at 0) 2164 17: g GET 1 2165 20: . STOP 2166highest protocol among opcodes = 0 2167 2168>>> dis(pickle.dumps(T, 1)) 2169 0: ( MARK 2170 1: ] EMPTY_LIST 2171 2: q BINPUT 0 2172 4: ( MARK 2173 5: h BINGET 0 2174 7: t TUPLE (MARK at 4) 2175 8: q BINPUT 1 2176 10: a APPEND 2177 11: 1 POP_MARK (MARK at 0) 2178 12: h BINGET 1 2179 14: . STOP 2180highest protocol among opcodes = 1 2181 2182Try protocol 2. 2183 2184>>> dis(pickle.dumps(L, 2)) 2185 0: \x80 PROTO 2 2186 2: ] EMPTY_LIST 2187 3: q BINPUT 0 2188 5: h BINGET 0 2189 7: \x85 TUPLE1 2190 8: q BINPUT 1 2191 10: a APPEND 2192 11: . STOP 2193highest protocol among opcodes = 2 2194 2195>>> dis(pickle.dumps(T, 2)) 2196 0: \x80 PROTO 2 2197 2: ] EMPTY_LIST 2198 3: q BINPUT 0 2199 5: h BINGET 0 2200 7: \x85 TUPLE1 2201 8: q BINPUT 1 2202 10: a APPEND 2203 11: 0 POP 2204 12: h BINGET 1 2205 14: . STOP 2206highest protocol among opcodes = 2 2207""" 2208 2209_memo_test = r""" 2210>>> import pickle 2211>>> from StringIO import StringIO 2212>>> f = StringIO() 2213>>> p = pickle.Pickler(f, 2) 2214>>> x = [1, 2, 3] 2215>>> p.dump(x) 2216>>> p.dump(x) 2217>>> f.seek(0) 2218>>> memo = {} 2219>>> dis(f, memo=memo) 2220 0: \x80 PROTO 2 2221 2: ] EMPTY_LIST 2222 3: q BINPUT 0 2223 5: ( MARK 2224 6: K BININT1 1 2225 8: K BININT1 2 2226 10: K BININT1 3 2227 12: e APPENDS (MARK at 5) 2228 13: . STOP 2229highest protocol among opcodes = 2 2230>>> dis(f, memo=memo) 2231 14: \x80 PROTO 2 2232 16: h BINGET 0 2233 18: . STOP 2234highest protocol among opcodes = 2 2235""" 2236 2237__test__ = {'disassembler_test': _dis_test, 2238 'disassembler_memo_test': _memo_test, 2239 } 2240 2241def _test(): 2242 import doctest 2243 return doctest.testmod() 2244 2245if __name__ == "__main__": 2246 _test() 2247