pickletools.py revision 929d3840589cf8159520e4082467d7cab6d5a57f
1'''"Executable documentation" for the pickle module. 2 3Extensive comments about the pickle protocols and pickle-machine opcodes 4can be found here. Some functions meant for external use: 5 6genops(pickle) 7 Generate all the opcodes in a pickle, as (opcode, arg, position) triples. 8 9dis(pickle, out=None, memo=None, indentlevel=4) 10 Print a symbolic disassembly of a pickle. 11''' 12 13import codecs 14import pickle 15import re 16 17__all__ = ['dis', 'genops', 'optimize'] 18 19bytes_types = pickle.bytes_types 20 21# Other ideas: 22# 23# - A pickle verifier: read a pickle and check it exhaustively for 24# well-formedness. dis() does a lot of this already. 25# 26# - A protocol identifier: examine a pickle and return its protocol number 27# (== the highest .proto attr value among all the opcodes in the pickle). 28# dis() already prints this info at the end. 29# 30# - A pickle optimizer: for example, tuple-building code is sometimes more 31# elaborate than necessary, catering for the possibility that the tuple 32# is recursive. Or lots of times a PUT is generated that's never accessed 33# by a later GET. 34 35 36""" 37"A pickle" is a program for a virtual pickle machine (PM, but more accurately 38called an unpickling machine). It's a sequence of opcodes, interpreted by the 39PM, building an arbitrarily complex Python object. 40 41For the most part, the PM is very simple: there are no looping, testing, or 42conditional instructions, no arithmetic and no function calls. Opcodes are 43executed once each, from first to last, until a STOP opcode is reached. 44 45The PM has two data areas, "the stack" and "the memo". 46 47Many opcodes push Python objects onto the stack; e.g., INT pushes a Python 48integer object on the stack, whose value is gotten from a decimal string 49literal immediately following the INT opcode in the pickle bytestream. Other 50opcodes take Python objects off the stack. The result of unpickling is 51whatever object is left on the stack when the final STOP opcode is executed. 52 53The memo is simply an array of objects, or it can be implemented as a dict 54mapping little integers to objects. The memo serves as the PM's "long term 55memory", and the little integers indexing the memo are akin to variable 56names. Some opcodes pop a stack object into the memo at a given index, 57and others push a memo object at a given index onto the stack again. 58 59At heart, that's all the PM has. Subtleties arise for these reasons: 60 61+ Object identity. Objects can be arbitrarily complex, and subobjects 62 may be shared (for example, the list [a, a] refers to the same object a 63 twice). It can be vital that unpickling recreate an isomorphic object 64 graph, faithfully reproducing sharing. 65 66+ Recursive objects. For example, after "L = []; L.append(L)", L is a 67 list, and L[0] is the same list. This is related to the object identity 68 point, and some sequences of pickle opcodes are subtle in order to 69 get the right result in all cases. 70 71+ Things pickle doesn't know everything about. Examples of things pickle 72 does know everything about are Python's builtin scalar and container 73 types, like ints and tuples. They generally have opcodes dedicated to 74 them. For things like module references and instances of user-defined 75 classes, pickle's knowledge is limited. Historically, many enhancements 76 have been made to the pickle protocol in order to do a better (faster, 77 and/or more compact) job on those. 78 79+ Backward compatibility and micro-optimization. As explained below, 80 pickle opcodes never go away, not even when better ways to do a thing 81 get invented. The repertoire of the PM just keeps growing over time. 82 For example, protocol 0 had two opcodes for building Python integers (INT 83 and LONG), protocol 1 added three more for more-efficient pickling of short 84 integers, and protocol 2 added two more for more-efficient pickling of 85 long integers (before protocol 2, the only ways to pickle a Python long 86 took time quadratic in the number of digits, for both pickling and 87 unpickling). "Opcode bloat" isn't so much a subtlety as a source of 88 wearying complication. 89 90 91Pickle protocols: 92 93For compatibility, the meaning of a pickle opcode never changes. Instead new 94pickle opcodes get added, and each version's unpickler can handle all the 95pickle opcodes in all protocol versions to date. So old pickles continue to 96be readable forever. The pickler can generally be told to restrict itself to 97the subset of opcodes available under previous protocol versions too, so that 98users can create pickles under the current version readable by older 99versions. However, a pickle does not contain its version number embedded 100within it. If an older unpickler tries to read a pickle using a later 101protocol, the result is most likely an exception due to seeing an unknown (in 102the older unpickler) opcode. 103 104The original pickle used what's now called "protocol 0", and what was called 105"text mode" before Python 2.3. The entire pickle bytestream is made up of 106printable 7-bit ASCII characters, plus the newline character, in protocol 0. 107That's why it was called text mode. Protocol 0 is small and elegant, but 108sometimes painfully inefficient. 109 110The second major set of additions is now called "protocol 1", and was called 111"binary mode" before Python 2.3. This added many opcodes with arguments 112consisting of arbitrary bytes, including NUL bytes and unprintable "high bit" 113bytes. Binary mode pickles can be substantially smaller than equivalent 114text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte 115int as 4 bytes following the opcode, which is cheaper to unpickle than the 116(perhaps) 11-character decimal string attached to INT. Protocol 1 also added 117a number of opcodes that operate on many stack elements at once (like APPENDS 118and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE). 119 120The third major set of additions came in Python 2.3, and is called "protocol 1212". This added: 122 123- A better way to pickle instances of new-style classes (NEWOBJ). 124 125- A way for a pickle to identify its protocol (PROTO). 126 127- Time- and space- efficient pickling of long ints (LONG{1,4}). 128 129- Shortcuts for small tuples (TUPLE{1,2,3}}. 130 131- Dedicated opcodes for bools (NEWTRUE, NEWFALSE). 132 133- The "extension registry", a vector of popular objects that can be pushed 134 efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but 135 the registry contents are predefined (there's nothing akin to the memo's 136 PUT). 137 138Another independent change with Python 2.3 is the abandonment of any 139pretense that it might be safe to load pickles received from untrusted 140parties -- no sufficient security analysis has been done to guarantee 141this and there isn't a use case that warrants the expense of such an 142analysis. 143 144To this end, all tests for __safe_for_unpickling__ or for 145copyreg.safe_constructors are removed from the unpickling code. 146References to these variables in the descriptions below are to be seen 147as describing unpickling in Python 2.2 and before. 148""" 149 150# Meta-rule: Descriptions are stored in instances of descriptor objects, 151# with plain constructors. No meta-language is defined from which 152# descriptors could be constructed. If you want, e.g., XML, write a little 153# program to generate XML from the objects. 154 155############################################################################## 156# Some pickle opcodes have an argument, following the opcode in the 157# bytestream. An argument is of a specific type, described by an instance 158# of ArgumentDescriptor. These are not to be confused with arguments taken 159# off the stack -- ArgumentDescriptor applies only to arguments embedded in 160# the opcode stream, immediately following an opcode. 161 162# Represents the number of bytes consumed by an argument delimited by the 163# next newline character. 164UP_TO_NEWLINE = -1 165 166# Represents the number of bytes consumed by a two-argument opcode where 167# the first argument gives the number of bytes in the second argument. 168TAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int 169TAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int 170 171class ArgumentDescriptor(object): 172 __slots__ = ( 173 # name of descriptor record, also a module global name; a string 174 'name', 175 176 # length of argument, in bytes; an int; UP_TO_NEWLINE and 177 # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length 178 # cases 179 'n', 180 181 # a function taking a file-like object, reading this kind of argument 182 # from the object at the current position, advancing the current 183 # position by n bytes, and returning the value of the argument 184 'reader', 185 186 # human-readable docs for this arg descriptor; a string 187 'doc', 188 ) 189 190 def __init__(self, name, n, reader, doc): 191 assert isinstance(name, str) 192 self.name = name 193 194 assert isinstance(n, int) and (n >= 0 or 195 n in (UP_TO_NEWLINE, 196 TAKEN_FROM_ARGUMENT1, 197 TAKEN_FROM_ARGUMENT4)) 198 self.n = n 199 200 self.reader = reader 201 202 assert isinstance(doc, str) 203 self.doc = doc 204 205from struct import unpack as _unpack 206 207def read_uint1(f): 208 r""" 209 >>> import io 210 >>> read_uint1(io.BytesIO(b'\xff')) 211 255 212 """ 213 214 data = f.read(1) 215 if data: 216 return data[0] 217 raise ValueError("not enough data in stream to read uint1") 218 219uint1 = ArgumentDescriptor( 220 name='uint1', 221 n=1, 222 reader=read_uint1, 223 doc="One-byte unsigned integer.") 224 225 226def read_uint2(f): 227 r""" 228 >>> import io 229 >>> read_uint2(io.BytesIO(b'\xff\x00')) 230 255 231 >>> read_uint2(io.BytesIO(b'\xff\xff')) 232 65535 233 """ 234 235 data = f.read(2) 236 if len(data) == 2: 237 return _unpack("<H", data)[0] 238 raise ValueError("not enough data in stream to read uint2") 239 240uint2 = ArgumentDescriptor( 241 name='uint2', 242 n=2, 243 reader=read_uint2, 244 doc="Two-byte unsigned integer, little-endian.") 245 246 247def read_int4(f): 248 r""" 249 >>> import io 250 >>> read_int4(io.BytesIO(b'\xff\x00\x00\x00')) 251 255 252 >>> read_int4(io.BytesIO(b'\x00\x00\x00\x80')) == -(2**31) 253 True 254 """ 255 256 data = f.read(4) 257 if len(data) == 4: 258 return _unpack("<i", data)[0] 259 raise ValueError("not enough data in stream to read int4") 260 261int4 = ArgumentDescriptor( 262 name='int4', 263 n=4, 264 reader=read_int4, 265 doc="Four-byte signed integer, little-endian, 2's complement.") 266 267 268def read_stringnl(f, decode=True, stripquotes=True): 269 r""" 270 >>> import io 271 >>> read_stringnl(io.BytesIO(b"'abcd'\nefg\n")) 272 'abcd' 273 274 >>> read_stringnl(io.BytesIO(b"\n")) 275 Traceback (most recent call last): 276 ... 277 ValueError: no string quotes around b'' 278 279 >>> read_stringnl(io.BytesIO(b"\n"), stripquotes=False) 280 '' 281 282 >>> read_stringnl(io.BytesIO(b"''\n")) 283 '' 284 285 >>> read_stringnl(io.BytesIO(b'"abcd"')) 286 Traceback (most recent call last): 287 ... 288 ValueError: no newline found when trying to read stringnl 289 290 Embedded escapes are undone in the result. 291 >>> read_stringnl(io.BytesIO(br"'a\n\\b\x00c\td'" + b"\n'e'")) 292 'a\n\\b\x00c\td' 293 """ 294 295 data = f.readline() 296 if not data.endswith(b'\n'): 297 raise ValueError("no newline found when trying to read stringnl") 298 data = data[:-1] # lose the newline 299 300 if stripquotes: 301 for q in (b'"', b"'"): 302 if data.startswith(q): 303 if not data.endswith(q): 304 raise ValueError("strinq quote %r not found at both " 305 "ends of %r" % (q, data)) 306 data = data[1:-1] 307 break 308 else: 309 raise ValueError("no string quotes around %r" % data) 310 311 if decode: 312 data = codecs.escape_decode(data)[0].decode("ascii") 313 return data 314 315stringnl = ArgumentDescriptor( 316 name='stringnl', 317 n=UP_TO_NEWLINE, 318 reader=read_stringnl, 319 doc="""A newline-terminated string. 320 321 This is a repr-style string, with embedded escapes, and 322 bracketing quotes. 323 """) 324 325def read_stringnl_noescape(f): 326 return read_stringnl(f, stripquotes=False) 327 328stringnl_noescape = ArgumentDescriptor( 329 name='stringnl_noescape', 330 n=UP_TO_NEWLINE, 331 reader=read_stringnl_noescape, 332 doc="""A newline-terminated string. 333 334 This is a str-style string, without embedded escapes, 335 or bracketing quotes. It should consist solely of 336 printable ASCII characters. 337 """) 338 339def read_stringnl_noescape_pair(f): 340 r""" 341 >>> import io 342 >>> read_stringnl_noescape_pair(io.BytesIO(b"Queue\nEmpty\njunk")) 343 'Queue Empty' 344 """ 345 346 return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) 347 348stringnl_noescape_pair = ArgumentDescriptor( 349 name='stringnl_noescape_pair', 350 n=UP_TO_NEWLINE, 351 reader=read_stringnl_noescape_pair, 352 doc="""A pair of newline-terminated strings. 353 354 These are str-style strings, without embedded 355 escapes, or bracketing quotes. They should 356 consist solely of printable ASCII characters. 357 The pair is returned as a single string, with 358 a single blank separating the two strings. 359 """) 360 361def read_string4(f): 362 r""" 363 >>> import io 364 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x00abc")) 365 '' 366 >>> read_string4(io.BytesIO(b"\x03\x00\x00\x00abcdef")) 367 'abc' 368 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x03abcdef")) 369 Traceback (most recent call last): 370 ... 371 ValueError: expected 50331648 bytes in a string4, but only 6 remain 372 """ 373 374 n = read_int4(f) 375 if n < 0: 376 raise ValueError("string4 byte count < 0: %d" % n) 377 data = f.read(n) 378 if len(data) == n: 379 return data.decode("latin-1") 380 raise ValueError("expected %d bytes in a string4, but only %d remain" % 381 (n, len(data))) 382 383string4 = ArgumentDescriptor( 384 name="string4", 385 n=TAKEN_FROM_ARGUMENT4, 386 reader=read_string4, 387 doc="""A counted string. 388 389 The first argument is a 4-byte little-endian signed int giving 390 the number of bytes in the string, and the second argument is 391 that many bytes. 392 """) 393 394 395def read_string1(f): 396 r""" 397 >>> import io 398 >>> read_string1(io.BytesIO(b"\x00")) 399 '' 400 >>> read_string1(io.BytesIO(b"\x03abcdef")) 401 'abc' 402 """ 403 404 n = read_uint1(f) 405 assert n >= 0 406 data = f.read(n) 407 if len(data) == n: 408 return data.decode("latin-1") 409 raise ValueError("expected %d bytes in a string1, but only %d remain" % 410 (n, len(data))) 411 412string1 = ArgumentDescriptor( 413 name="string1", 414 n=TAKEN_FROM_ARGUMENT1, 415 reader=read_string1, 416 doc="""A counted string. 417 418 The first argument is a 1-byte unsigned int giving the number 419 of bytes in the string, and the second argument is that many 420 bytes. 421 """) 422 423 424def read_unicodestringnl(f): 425 r""" 426 >>> import io 427 >>> read_unicodestringnl(io.BytesIO(b"abc\\uabcd\njunk")) == 'abc\uabcd' 428 True 429 """ 430 431 data = f.readline() 432 if not data.endswith(b'\n'): 433 raise ValueError("no newline found when trying to read " 434 "unicodestringnl") 435 data = data[:-1] # lose the newline 436 return str(data, 'raw-unicode-escape') 437 438unicodestringnl = ArgumentDescriptor( 439 name='unicodestringnl', 440 n=UP_TO_NEWLINE, 441 reader=read_unicodestringnl, 442 doc="""A newline-terminated Unicode string. 443 444 This is raw-unicode-escape encoded, so consists of 445 printable ASCII characters, and may contain embedded 446 escape sequences. 447 """) 448 449def read_unicodestring4(f): 450 r""" 451 >>> import io 452 >>> s = 'abcd\uabcd' 453 >>> enc = s.encode('utf-8') 454 >>> enc 455 b'abcd\xea\xaf\x8d' 456 >>> n = bytes([len(enc), 0, 0, 0]) # little-endian 4-byte length 457 >>> t = read_unicodestring4(io.BytesIO(n + enc + b'junk')) 458 >>> s == t 459 True 460 461 >>> read_unicodestring4(io.BytesIO(n + enc[:-1])) 462 Traceback (most recent call last): 463 ... 464 ValueError: expected 7 bytes in a unicodestring4, but only 6 remain 465 """ 466 467 n = read_int4(f) 468 if n < 0: 469 raise ValueError("unicodestring4 byte count < 0: %d" % n) 470 data = f.read(n) 471 if len(data) == n: 472 return str(data, 'utf-8', 'surrogatepass') 473 raise ValueError("expected %d bytes in a unicodestring4, but only %d " 474 "remain" % (n, len(data))) 475 476unicodestring4 = ArgumentDescriptor( 477 name="unicodestring4", 478 n=TAKEN_FROM_ARGUMENT4, 479 reader=read_unicodestring4, 480 doc="""A counted Unicode string. 481 482 The first argument is a 4-byte little-endian signed int 483 giving the number of bytes in the string, and the second 484 argument-- the UTF-8 encoding of the Unicode string -- 485 contains that many bytes. 486 """) 487 488 489def read_decimalnl_short(f): 490 r""" 491 >>> import io 492 >>> read_decimalnl_short(io.BytesIO(b"1234\n56")) 493 1234 494 495 >>> read_decimalnl_short(io.BytesIO(b"1234L\n56")) 496 Traceback (most recent call last): 497 ... 498 ValueError: trailing 'L' not allowed in b'1234L' 499 """ 500 501 s = read_stringnl(f, decode=False, stripquotes=False) 502 if s.endswith(b"L"): 503 raise ValueError("trailing 'L' not allowed in %r" % s) 504 505 # It's not necessarily true that the result fits in a Python short int: 506 # the pickle may have been written on a 64-bit box. There's also a hack 507 # for True and False here. 508 if s == b"00": 509 return False 510 elif s == b"01": 511 return True 512 513 try: 514 return int(s) 515 except OverflowError: 516 return int(s) 517 518def read_decimalnl_long(f): 519 r""" 520 >>> import io 521 522 >>> read_decimalnl_long(io.BytesIO(b"1234L\n56")) 523 1234 524 525 >>> read_decimalnl_long(io.BytesIO(b"123456789012345678901234L\n6")) 526 123456789012345678901234 527 """ 528 529 s = read_stringnl(f, decode=False, stripquotes=False) 530 if s[-1:] == b'L': 531 s = s[:-1] 532 return int(s) 533 534 535decimalnl_short = ArgumentDescriptor( 536 name='decimalnl_short', 537 n=UP_TO_NEWLINE, 538 reader=read_decimalnl_short, 539 doc="""A newline-terminated decimal integer literal. 540 541 This never has a trailing 'L', and the integer fit 542 in a short Python int on the box where the pickle 543 was written -- but there's no guarantee it will fit 544 in a short Python int on the box where the pickle 545 is read. 546 """) 547 548decimalnl_long = ArgumentDescriptor( 549 name='decimalnl_long', 550 n=UP_TO_NEWLINE, 551 reader=read_decimalnl_long, 552 doc="""A newline-terminated decimal integer literal. 553 554 This has a trailing 'L', and can represent integers 555 of any size. 556 """) 557 558 559def read_floatnl(f): 560 r""" 561 >>> import io 562 >>> read_floatnl(io.BytesIO(b"-1.25\n6")) 563 -1.25 564 """ 565 s = read_stringnl(f, decode=False, stripquotes=False) 566 return float(s) 567 568floatnl = ArgumentDescriptor( 569 name='floatnl', 570 n=UP_TO_NEWLINE, 571 reader=read_floatnl, 572 doc="""A newline-terminated decimal floating literal. 573 574 In general this requires 17 significant digits for roundtrip 575 identity, and pickling then unpickling infinities, NaNs, and 576 minus zero doesn't work across boxes, or on some boxes even 577 on itself (e.g., Windows can't read the strings it produces 578 for infinities or NaNs). 579 """) 580 581def read_float8(f): 582 r""" 583 >>> import io, struct 584 >>> raw = struct.pack(">d", -1.25) 585 >>> raw 586 b'\xbf\xf4\x00\x00\x00\x00\x00\x00' 587 >>> read_float8(io.BytesIO(raw + b"\n")) 588 -1.25 589 """ 590 591 data = f.read(8) 592 if len(data) == 8: 593 return _unpack(">d", data)[0] 594 raise ValueError("not enough data in stream to read float8") 595 596 597float8 = ArgumentDescriptor( 598 name='float8', 599 n=8, 600 reader=read_float8, 601 doc="""An 8-byte binary representation of a float, big-endian. 602 603 The format is unique to Python, and shared with the struct 604 module (format string '>d') "in theory" (the struct and pickle 605 implementations don't share the code -- they should). It's 606 strongly related to the IEEE-754 double format, and, in normal 607 cases, is in fact identical to the big-endian 754 double format. 608 On other boxes the dynamic range is limited to that of a 754 609 double, and "add a half and chop" rounding is used to reduce 610 the precision to 53 bits. However, even on a 754 box, 611 infinities, NaNs, and minus zero may not be handled correctly 612 (may not survive roundtrip pickling intact). 613 """) 614 615# Protocol 2 formats 616 617from pickle import decode_long 618 619def read_long1(f): 620 r""" 621 >>> import io 622 >>> read_long1(io.BytesIO(b"\x00")) 623 0 624 >>> read_long1(io.BytesIO(b"\x02\xff\x00")) 625 255 626 >>> read_long1(io.BytesIO(b"\x02\xff\x7f")) 627 32767 628 >>> read_long1(io.BytesIO(b"\x02\x00\xff")) 629 -256 630 >>> read_long1(io.BytesIO(b"\x02\x00\x80")) 631 -32768 632 """ 633 634 n = read_uint1(f) 635 data = f.read(n) 636 if len(data) != n: 637 raise ValueError("not enough data in stream to read long1") 638 return decode_long(data) 639 640long1 = ArgumentDescriptor( 641 name="long1", 642 n=TAKEN_FROM_ARGUMENT1, 643 reader=read_long1, 644 doc="""A binary long, little-endian, using 1-byte size. 645 646 This first reads one byte as an unsigned size, then reads that 647 many bytes and interprets them as a little-endian 2's-complement long. 648 If the size is 0, that's taken as a shortcut for the long 0L. 649 """) 650 651def read_long4(f): 652 r""" 653 >>> import io 654 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x00")) 655 255 656 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x7f")) 657 32767 658 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\xff")) 659 -256 660 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\x80")) 661 -32768 662 >>> read_long1(io.BytesIO(b"\x00\x00\x00\x00")) 663 0 664 """ 665 666 n = read_int4(f) 667 if n < 0: 668 raise ValueError("long4 byte count < 0: %d" % n) 669 data = f.read(n) 670 if len(data) != n: 671 raise ValueError("not enough data in stream to read long4") 672 return decode_long(data) 673 674long4 = ArgumentDescriptor( 675 name="long4", 676 n=TAKEN_FROM_ARGUMENT4, 677 reader=read_long4, 678 doc="""A binary representation of a long, little-endian. 679 680 This first reads four bytes as a signed size (but requires the 681 size to be >= 0), then reads that many bytes and interprets them 682 as a little-endian 2's-complement long. If the size is 0, that's taken 683 as a shortcut for the int 0, although LONG1 should really be used 684 then instead (and in any case where # of bytes < 256). 685 """) 686 687 688############################################################################## 689# Object descriptors. The stack used by the pickle machine holds objects, 690# and in the stack_before and stack_after attributes of OpcodeInfo 691# descriptors we need names to describe the various types of objects that can 692# appear on the stack. 693 694class StackObject(object): 695 __slots__ = ( 696 # name of descriptor record, for info only 697 'name', 698 699 # type of object, or tuple of type objects (meaning the object can 700 # be of any type in the tuple) 701 'obtype', 702 703 # human-readable docs for this kind of stack object; a string 704 'doc', 705 ) 706 707 def __init__(self, name, obtype, doc): 708 assert isinstance(name, str) 709 self.name = name 710 711 assert isinstance(obtype, type) or isinstance(obtype, tuple) 712 if isinstance(obtype, tuple): 713 for contained in obtype: 714 assert isinstance(contained, type) 715 self.obtype = obtype 716 717 assert isinstance(doc, str) 718 self.doc = doc 719 720 def __repr__(self): 721 return self.name 722 723 724pyint = StackObject( 725 name='int', 726 obtype=int, 727 doc="A short (as opposed to long) Python integer object.") 728 729pylong = StackObject( 730 name='long', 731 obtype=int, 732 doc="A long (as opposed to short) Python integer object.") 733 734pyinteger_or_bool = StackObject( 735 name='int_or_bool', 736 obtype=(int, int, bool), 737 doc="A Python integer object (short or long), or " 738 "a Python bool.") 739 740pybool = StackObject( 741 name='bool', 742 obtype=(bool,), 743 doc="A Python bool object.") 744 745pyfloat = StackObject( 746 name='float', 747 obtype=float, 748 doc="A Python float object.") 749 750pystring = StackObject( 751 name='string', 752 obtype=bytes, 753 doc="A Python (8-bit) string object.") 754 755pybytes = StackObject( 756 name='bytes', 757 obtype=bytes, 758 doc="A Python bytes object.") 759 760pyunicode = StackObject( 761 name='str', 762 obtype=str, 763 doc="A Python (Unicode) string object.") 764 765pynone = StackObject( 766 name="None", 767 obtype=type(None), 768 doc="The Python None object.") 769 770pytuple = StackObject( 771 name="tuple", 772 obtype=tuple, 773 doc="A Python tuple object.") 774 775pylist = StackObject( 776 name="list", 777 obtype=list, 778 doc="A Python list object.") 779 780pydict = StackObject( 781 name="dict", 782 obtype=dict, 783 doc="A Python dict object.") 784 785anyobject = StackObject( 786 name='any', 787 obtype=object, 788 doc="Any kind of object whatsoever.") 789 790markobject = StackObject( 791 name="mark", 792 obtype=StackObject, 793 doc="""'The mark' is a unique object. 794 795 Opcodes that operate on a variable number of objects 796 generally don't embed the count of objects in the opcode, 797 or pull it off the stack. Instead the MARK opcode is used 798 to push a special marker object on the stack, and then 799 some other opcodes grab all the objects from the top of 800 the stack down to (but not including) the topmost marker 801 object. 802 """) 803 804stackslice = StackObject( 805 name="stackslice", 806 obtype=StackObject, 807 doc="""An object representing a contiguous slice of the stack. 808 809 This is used in conjuction with markobject, to represent all 810 of the stack following the topmost markobject. For example, 811 the POP_MARK opcode changes the stack from 812 813 [..., markobject, stackslice] 814 to 815 [...] 816 817 No matter how many object are on the stack after the topmost 818 markobject, POP_MARK gets rid of all of them (including the 819 topmost markobject too). 820 """) 821 822############################################################################## 823# Descriptors for pickle opcodes. 824 825class OpcodeInfo(object): 826 827 __slots__ = ( 828 # symbolic name of opcode; a string 829 'name', 830 831 # the code used in a bytestream to represent the opcode; a 832 # one-character string 833 'code', 834 835 # If the opcode has an argument embedded in the byte string, an 836 # instance of ArgumentDescriptor specifying its type. Note that 837 # arg.reader(s) can be used to read and decode the argument from 838 # the bytestream s, and arg.doc documents the format of the raw 839 # argument bytes. If the opcode doesn't have an argument embedded 840 # in the bytestream, arg should be None. 841 'arg', 842 843 # what the stack looks like before this opcode runs; a list 844 'stack_before', 845 846 # what the stack looks like after this opcode runs; a list 847 'stack_after', 848 849 # the protocol number in which this opcode was introduced; an int 850 'proto', 851 852 # human-readable docs for this opcode; a string 853 'doc', 854 ) 855 856 def __init__(self, name, code, arg, 857 stack_before, stack_after, proto, doc): 858 assert isinstance(name, str) 859 self.name = name 860 861 assert isinstance(code, str) 862 assert len(code) == 1 863 self.code = code 864 865 assert arg is None or isinstance(arg, ArgumentDescriptor) 866 self.arg = arg 867 868 assert isinstance(stack_before, list) 869 for x in stack_before: 870 assert isinstance(x, StackObject) 871 self.stack_before = stack_before 872 873 assert isinstance(stack_after, list) 874 for x in stack_after: 875 assert isinstance(x, StackObject) 876 self.stack_after = stack_after 877 878 assert isinstance(proto, int) and 0 <= proto <= 3 879 self.proto = proto 880 881 assert isinstance(doc, str) 882 self.doc = doc 883 884I = OpcodeInfo 885opcodes = [ 886 887 # Ways to spell integers. 888 889 I(name='INT', 890 code='I', 891 arg=decimalnl_short, 892 stack_before=[], 893 stack_after=[pyinteger_or_bool], 894 proto=0, 895 doc="""Push an integer or bool. 896 897 The argument is a newline-terminated decimal literal string. 898 899 The intent may have been that this always fit in a short Python int, 900 but INT can be generated in pickles written on a 64-bit box that 901 require a Python long on a 32-bit box. The difference between this 902 and LONG then is that INT skips a trailing 'L', and produces a short 903 int whenever possible. 904 905 Another difference is due to that, when bool was introduced as a 906 distinct type in 2.3, builtin names True and False were also added to 907 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, 908 True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". 909 Leading zeroes are never produced for a genuine integer. The 2.3 910 (and later) unpicklers special-case these and return bool instead; 911 earlier unpicklers ignore the leading "0" and return the int. 912 """), 913 914 I(name='BININT', 915 code='J', 916 arg=int4, 917 stack_before=[], 918 stack_after=[pyint], 919 proto=1, 920 doc="""Push a four-byte signed integer. 921 922 This handles the full range of Python (short) integers on a 32-bit 923 box, directly as binary bytes (1 for the opcode and 4 for the integer). 924 If the integer is non-negative and fits in 1 or 2 bytes, pickling via 925 BININT1 or BININT2 saves space. 926 """), 927 928 I(name='BININT1', 929 code='K', 930 arg=uint1, 931 stack_before=[], 932 stack_after=[pyint], 933 proto=1, 934 doc="""Push a one-byte unsigned integer. 935 936 This is a space optimization for pickling very small non-negative ints, 937 in range(256). 938 """), 939 940 I(name='BININT2', 941 code='M', 942 arg=uint2, 943 stack_before=[], 944 stack_after=[pyint], 945 proto=1, 946 doc="""Push a two-byte unsigned integer. 947 948 This is a space optimization for pickling small positive ints, in 949 range(256, 2**16). Integers in range(256) can also be pickled via 950 BININT2, but BININT1 instead saves a byte. 951 """), 952 953 I(name='LONG', 954 code='L', 955 arg=decimalnl_long, 956 stack_before=[], 957 stack_after=[pylong], 958 proto=0, 959 doc="""Push a long integer. 960 961 The same as INT, except that the literal ends with 'L', and always 962 unpickles to a Python long. There doesn't seem a real purpose to the 963 trailing 'L'. 964 965 Note that LONG takes time quadratic in the number of digits when 966 unpickling (this is simply due to the nature of decimal->binary 967 conversion). Proto 2 added linear-time (in C; still quadratic-time 968 in Python) LONG1 and LONG4 opcodes. 969 """), 970 971 I(name="LONG1", 972 code='\x8a', 973 arg=long1, 974 stack_before=[], 975 stack_after=[pylong], 976 proto=2, 977 doc="""Long integer using one-byte length. 978 979 A more efficient encoding of a Python long; the long1 encoding 980 says it all."""), 981 982 I(name="LONG4", 983 code='\x8b', 984 arg=long4, 985 stack_before=[], 986 stack_after=[pylong], 987 proto=2, 988 doc="""Long integer using found-byte length. 989 990 A more efficient encoding of a Python long; the long4 encoding 991 says it all."""), 992 993 # Ways to spell strings (8-bit, not Unicode). 994 995 I(name='STRING', 996 code='S', 997 arg=stringnl, 998 stack_before=[], 999 stack_after=[pystring], 1000 proto=0, 1001 doc="""Push a Python string object. 1002 1003 The argument is a repr-style string, with bracketing quote characters, 1004 and perhaps embedded escapes. The argument extends until the next 1005 newline character. (Actually, they are decoded into a str instance 1006 using the encoding given to the Unpickler constructor. or the default, 1007 'ASCII'.) 1008 """), 1009 1010 I(name='BINSTRING', 1011 code='T', 1012 arg=string4, 1013 stack_before=[], 1014 stack_after=[pystring], 1015 proto=1, 1016 doc="""Push a Python string object. 1017 1018 There are two arguments: the first is a 4-byte little-endian signed int 1019 giving the number of bytes in the string, and the second is that many 1020 bytes, which are taken literally as the string content. (Actually, 1021 they are decoded into a str instance using the encoding given to the 1022 Unpickler constructor. or the default, 'ASCII'.) 1023 """), 1024 1025 I(name='SHORT_BINSTRING', 1026 code='U', 1027 arg=string1, 1028 stack_before=[], 1029 stack_after=[pystring], 1030 proto=1, 1031 doc="""Push a Python string object. 1032 1033 There are two arguments: the first is a 1-byte unsigned int giving 1034 the number of bytes in the string, and the second is that many bytes, 1035 which are taken literally as the string content. (Actually, they 1036 are decoded into a str instance using the encoding given to the 1037 Unpickler constructor. or the default, 'ASCII'.) 1038 """), 1039 1040 # Bytes (protocol 3 only; older protocols don't support bytes at all) 1041 1042 I(name='BINBYTES', 1043 code='B', 1044 arg=string4, 1045 stack_before=[], 1046 stack_after=[pybytes], 1047 proto=3, 1048 doc="""Push a Python bytes object. 1049 1050 There are two arguments: the first is a 4-byte little-endian signed int 1051 giving the number of bytes in the string, and the second is that many 1052 bytes, which are taken literally as the bytes content. 1053 """), 1054 1055 I(name='SHORT_BINBYTES', 1056 code='C', 1057 arg=string1, 1058 stack_before=[], 1059 stack_after=[pybytes], 1060 proto=3, 1061 doc="""Push a Python string object. 1062 1063 There are two arguments: the first is a 1-byte unsigned int giving 1064 the number of bytes in the string, and the second is that many bytes, 1065 which are taken literally as the string content. 1066 """), 1067 1068 # Ways to spell None. 1069 1070 I(name='NONE', 1071 code='N', 1072 arg=None, 1073 stack_before=[], 1074 stack_after=[pynone], 1075 proto=0, 1076 doc="Push None on the stack."), 1077 1078 # Ways to spell bools, starting with proto 2. See INT for how this was 1079 # done before proto 2. 1080 1081 I(name='NEWTRUE', 1082 code='\x88', 1083 arg=None, 1084 stack_before=[], 1085 stack_after=[pybool], 1086 proto=2, 1087 doc="""True. 1088 1089 Push True onto the stack."""), 1090 1091 I(name='NEWFALSE', 1092 code='\x89', 1093 arg=None, 1094 stack_before=[], 1095 stack_after=[pybool], 1096 proto=2, 1097 doc="""True. 1098 1099 Push False onto the stack."""), 1100 1101 # Ways to spell Unicode strings. 1102 1103 I(name='UNICODE', 1104 code='V', 1105 arg=unicodestringnl, 1106 stack_before=[], 1107 stack_after=[pyunicode], 1108 proto=0, # this may be pure-text, but it's a later addition 1109 doc="""Push a Python Unicode string object. 1110 1111 The argument is a raw-unicode-escape encoding of a Unicode string, 1112 and so may contain embedded escape sequences. The argument extends 1113 until the next newline character. 1114 """), 1115 1116 I(name='BINUNICODE', 1117 code='X', 1118 arg=unicodestring4, 1119 stack_before=[], 1120 stack_after=[pyunicode], 1121 proto=1, 1122 doc="""Push a Python Unicode string object. 1123 1124 There are two arguments: the first is a 4-byte little-endian signed int 1125 giving the number of bytes in the string. The second is that many 1126 bytes, and is the UTF-8 encoding of the Unicode string. 1127 """), 1128 1129 # Ways to spell floats. 1130 1131 I(name='FLOAT', 1132 code='F', 1133 arg=floatnl, 1134 stack_before=[], 1135 stack_after=[pyfloat], 1136 proto=0, 1137 doc="""Newline-terminated decimal float literal. 1138 1139 The argument is repr(a_float), and in general requires 17 significant 1140 digits for roundtrip conversion to be an identity (this is so for 1141 IEEE-754 double precision values, which is what Python float maps to 1142 on most boxes). 1143 1144 In general, FLOAT cannot be used to transport infinities, NaNs, or 1145 minus zero across boxes (or even on a single box, if the platform C 1146 library can't read the strings it produces for such things -- Windows 1147 is like that), but may do less damage than BINFLOAT on boxes with 1148 greater precision or dynamic range than IEEE-754 double. 1149 """), 1150 1151 I(name='BINFLOAT', 1152 code='G', 1153 arg=float8, 1154 stack_before=[], 1155 stack_after=[pyfloat], 1156 proto=1, 1157 doc="""Float stored in binary form, with 8 bytes of data. 1158 1159 This generally requires less than half the space of FLOAT encoding. 1160 In general, BINFLOAT cannot be used to transport infinities, NaNs, or 1161 minus zero, raises an exception if the exponent exceeds the range of 1162 an IEEE-754 double, and retains no more than 53 bits of precision (if 1163 there are more than that, "add a half and chop" rounding is used to 1164 cut it back to 53 significant bits). 1165 """), 1166 1167 # Ways to build lists. 1168 1169 I(name='EMPTY_LIST', 1170 code=']', 1171 arg=None, 1172 stack_before=[], 1173 stack_after=[pylist], 1174 proto=1, 1175 doc="Push an empty list."), 1176 1177 I(name='APPEND', 1178 code='a', 1179 arg=None, 1180 stack_before=[pylist, anyobject], 1181 stack_after=[pylist], 1182 proto=0, 1183 doc="""Append an object to a list. 1184 1185 Stack before: ... pylist anyobject 1186 Stack after: ... pylist+[anyobject] 1187 1188 although pylist is really extended in-place. 1189 """), 1190 1191 I(name='APPENDS', 1192 code='e', 1193 arg=None, 1194 stack_before=[pylist, markobject, stackslice], 1195 stack_after=[pylist], 1196 proto=1, 1197 doc="""Extend a list by a slice of stack objects. 1198 1199 Stack before: ... pylist markobject stackslice 1200 Stack after: ... pylist+stackslice 1201 1202 although pylist is really extended in-place. 1203 """), 1204 1205 I(name='LIST', 1206 code='l', 1207 arg=None, 1208 stack_before=[markobject, stackslice], 1209 stack_after=[pylist], 1210 proto=0, 1211 doc="""Build a list out of the topmost stack slice, after markobject. 1212 1213 All the stack entries following the topmost markobject are placed into 1214 a single Python list, which single list object replaces all of the 1215 stack from the topmost markobject onward. For example, 1216 1217 Stack before: ... markobject 1 2 3 'abc' 1218 Stack after: ... [1, 2, 3, 'abc'] 1219 """), 1220 1221 # Ways to build tuples. 1222 1223 I(name='EMPTY_TUPLE', 1224 code=')', 1225 arg=None, 1226 stack_before=[], 1227 stack_after=[pytuple], 1228 proto=1, 1229 doc="Push an empty tuple."), 1230 1231 I(name='TUPLE', 1232 code='t', 1233 arg=None, 1234 stack_before=[markobject, stackslice], 1235 stack_after=[pytuple], 1236 proto=0, 1237 doc="""Build a tuple out of the topmost stack slice, after markobject. 1238 1239 All the stack entries following the topmost markobject are placed into 1240 a single Python tuple, which single tuple object replaces all of the 1241 stack from the topmost markobject onward. For example, 1242 1243 Stack before: ... markobject 1 2 3 'abc' 1244 Stack after: ... (1, 2, 3, 'abc') 1245 """), 1246 1247 I(name='TUPLE1', 1248 code='\x85', 1249 arg=None, 1250 stack_before=[anyobject], 1251 stack_after=[pytuple], 1252 proto=2, 1253 doc="""Build a one-tuple out of the topmost item on the stack. 1254 1255 This code pops one value off the stack and pushes a tuple of 1256 length 1 whose one item is that value back onto it. In other 1257 words: 1258 1259 stack[-1] = tuple(stack[-1:]) 1260 """), 1261 1262 I(name='TUPLE2', 1263 code='\x86', 1264 arg=None, 1265 stack_before=[anyobject, anyobject], 1266 stack_after=[pytuple], 1267 proto=2, 1268 doc="""Build a two-tuple out of the top two items on the stack. 1269 1270 This code pops two values off the stack and pushes a tuple of 1271 length 2 whose items are those values back onto it. In other 1272 words: 1273 1274 stack[-2:] = [tuple(stack[-2:])] 1275 """), 1276 1277 I(name='TUPLE3', 1278 code='\x87', 1279 arg=None, 1280 stack_before=[anyobject, anyobject, anyobject], 1281 stack_after=[pytuple], 1282 proto=2, 1283 doc="""Build a three-tuple out of the top three items on the stack. 1284 1285 This code pops three values off the stack and pushes a tuple of 1286 length 3 whose items are those values back onto it. In other 1287 words: 1288 1289 stack[-3:] = [tuple(stack[-3:])] 1290 """), 1291 1292 # Ways to build dicts. 1293 1294 I(name='EMPTY_DICT', 1295 code='}', 1296 arg=None, 1297 stack_before=[], 1298 stack_after=[pydict], 1299 proto=1, 1300 doc="Push an empty dict."), 1301 1302 I(name='DICT', 1303 code='d', 1304 arg=None, 1305 stack_before=[markobject, stackslice], 1306 stack_after=[pydict], 1307 proto=0, 1308 doc="""Build a dict out of the topmost stack slice, after markobject. 1309 1310 All the stack entries following the topmost markobject are placed into 1311 a single Python dict, which single dict object replaces all of the 1312 stack from the topmost markobject onward. The stack slice alternates 1313 key, value, key, value, .... For example, 1314 1315 Stack before: ... markobject 1 2 3 'abc' 1316 Stack after: ... {1: 2, 3: 'abc'} 1317 """), 1318 1319 I(name='SETITEM', 1320 code='s', 1321 arg=None, 1322 stack_before=[pydict, anyobject, anyobject], 1323 stack_after=[pydict], 1324 proto=0, 1325 doc="""Add a key+value pair to an existing dict. 1326 1327 Stack before: ... pydict key value 1328 Stack after: ... pydict 1329 1330 where pydict has been modified via pydict[key] = value. 1331 """), 1332 1333 I(name='SETITEMS', 1334 code='u', 1335 arg=None, 1336 stack_before=[pydict, markobject, stackslice], 1337 stack_after=[pydict], 1338 proto=1, 1339 doc="""Add an arbitrary number of key+value pairs to an existing dict. 1340 1341 The slice of the stack following the topmost markobject is taken as 1342 an alternating sequence of keys and values, added to the dict 1343 immediately under the topmost markobject. Everything at and after the 1344 topmost markobject is popped, leaving the mutated dict at the top 1345 of the stack. 1346 1347 Stack before: ... pydict markobject key_1 value_1 ... key_n value_n 1348 Stack after: ... pydict 1349 1350 where pydict has been modified via pydict[key_i] = value_i for i in 1351 1, 2, ..., n, and in that order. 1352 """), 1353 1354 # Stack manipulation. 1355 1356 I(name='POP', 1357 code='0', 1358 arg=None, 1359 stack_before=[anyobject], 1360 stack_after=[], 1361 proto=0, 1362 doc="Discard the top stack item, shrinking the stack by one item."), 1363 1364 I(name='DUP', 1365 code='2', 1366 arg=None, 1367 stack_before=[anyobject], 1368 stack_after=[anyobject, anyobject], 1369 proto=0, 1370 doc="Push the top stack item onto the stack again, duplicating it."), 1371 1372 I(name='MARK', 1373 code='(', 1374 arg=None, 1375 stack_before=[], 1376 stack_after=[markobject], 1377 proto=0, 1378 doc="""Push markobject onto the stack. 1379 1380 markobject is a unique object, used by other opcodes to identify a 1381 region of the stack containing a variable number of objects for them 1382 to work on. See markobject.doc for more detail. 1383 """), 1384 1385 I(name='POP_MARK', 1386 code='1', 1387 arg=None, 1388 stack_before=[markobject, stackslice], 1389 stack_after=[], 1390 proto=1, 1391 doc="""Pop all the stack objects at and above the topmost markobject. 1392 1393 When an opcode using a variable number of stack objects is done, 1394 POP_MARK is used to remove those objects, and to remove the markobject 1395 that delimited their starting position on the stack. 1396 """), 1397 1398 # Memo manipulation. There are really only two operations (get and put), 1399 # each in all-text, "short binary", and "long binary" flavors. 1400 1401 I(name='GET', 1402 code='g', 1403 arg=decimalnl_short, 1404 stack_before=[], 1405 stack_after=[anyobject], 1406 proto=0, 1407 doc="""Read an object from the memo and push it on the stack. 1408 1409 The index of the memo object to push is given by the newline-teriminated 1410 decimal string following. BINGET and LONG_BINGET are space-optimized 1411 versions. 1412 """), 1413 1414 I(name='BINGET', 1415 code='h', 1416 arg=uint1, 1417 stack_before=[], 1418 stack_after=[anyobject], 1419 proto=1, 1420 doc="""Read an object from the memo and push it on the stack. 1421 1422 The index of the memo object to push is given by the 1-byte unsigned 1423 integer following. 1424 """), 1425 1426 I(name='LONG_BINGET', 1427 code='j', 1428 arg=int4, 1429 stack_before=[], 1430 stack_after=[anyobject], 1431 proto=1, 1432 doc="""Read an object from the memo and push it on the stack. 1433 1434 The index of the memo object to push is given by the 4-byte signed 1435 little-endian integer following. 1436 """), 1437 1438 I(name='PUT', 1439 code='p', 1440 arg=decimalnl_short, 1441 stack_before=[], 1442 stack_after=[], 1443 proto=0, 1444 doc="""Store the stack top into the memo. The stack is not popped. 1445 1446 The index of the memo location to write into is given by the newline- 1447 terminated decimal string following. BINPUT and LONG_BINPUT are 1448 space-optimized versions. 1449 """), 1450 1451 I(name='BINPUT', 1452 code='q', 1453 arg=uint1, 1454 stack_before=[], 1455 stack_after=[], 1456 proto=1, 1457 doc="""Store the stack top into the memo. The stack is not popped. 1458 1459 The index of the memo location to write into is given by the 1-byte 1460 unsigned integer following. 1461 """), 1462 1463 I(name='LONG_BINPUT', 1464 code='r', 1465 arg=int4, 1466 stack_before=[], 1467 stack_after=[], 1468 proto=1, 1469 doc="""Store the stack top into the memo. The stack is not popped. 1470 1471 The index of the memo location to write into is given by the 4-byte 1472 signed little-endian integer following. 1473 """), 1474 1475 # Access the extension registry (predefined objects). Akin to the GET 1476 # family. 1477 1478 I(name='EXT1', 1479 code='\x82', 1480 arg=uint1, 1481 stack_before=[], 1482 stack_after=[anyobject], 1483 proto=2, 1484 doc="""Extension code. 1485 1486 This code and the similar EXT2 and EXT4 allow using a registry 1487 of popular objects that are pickled by name, typically classes. 1488 It is envisioned that through a global negotiation and 1489 registration process, third parties can set up a mapping between 1490 ints and object names. 1491 1492 In order to guarantee pickle interchangeability, the extension 1493 code registry ought to be global, although a range of codes may 1494 be reserved for private use. 1495 1496 EXT1 has a 1-byte integer argument. This is used to index into the 1497 extension registry, and the object at that index is pushed on the stack. 1498 """), 1499 1500 I(name='EXT2', 1501 code='\x83', 1502 arg=uint2, 1503 stack_before=[], 1504 stack_after=[anyobject], 1505 proto=2, 1506 doc="""Extension code. 1507 1508 See EXT1. EXT2 has a two-byte integer argument. 1509 """), 1510 1511 I(name='EXT4', 1512 code='\x84', 1513 arg=int4, 1514 stack_before=[], 1515 stack_after=[anyobject], 1516 proto=2, 1517 doc="""Extension code. 1518 1519 See EXT1. EXT4 has a four-byte integer argument. 1520 """), 1521 1522 # Push a class object, or module function, on the stack, via its module 1523 # and name. 1524 1525 I(name='GLOBAL', 1526 code='c', 1527 arg=stringnl_noescape_pair, 1528 stack_before=[], 1529 stack_after=[anyobject], 1530 proto=0, 1531 doc="""Push a global object (module.attr) on the stack. 1532 1533 Two newline-terminated strings follow the GLOBAL opcode. The first is 1534 taken as a module name, and the second as a class name. The class 1535 object module.class is pushed on the stack. More accurately, the 1536 object returned by self.find_class(module, class) is pushed on the 1537 stack, so unpickling subclasses can override this form of lookup. 1538 """), 1539 1540 # Ways to build objects of classes pickle doesn't know about directly 1541 # (user-defined classes). I despair of documenting this accurately 1542 # and comprehensibly -- you really have to read the pickle code to 1543 # find all the special cases. 1544 1545 I(name='REDUCE', 1546 code='R', 1547 arg=None, 1548 stack_before=[anyobject, anyobject], 1549 stack_after=[anyobject], 1550 proto=0, 1551 doc="""Push an object built from a callable and an argument tuple. 1552 1553 The opcode is named to remind of the __reduce__() method. 1554 1555 Stack before: ... callable pytuple 1556 Stack after: ... callable(*pytuple) 1557 1558 The callable and the argument tuple are the first two items returned 1559 by a __reduce__ method. Applying the callable to the argtuple is 1560 supposed to reproduce the original object, or at least get it started. 1561 If the __reduce__ method returns a 3-tuple, the last component is an 1562 argument to be passed to the object's __setstate__, and then the REDUCE 1563 opcode is followed by code to create setstate's argument, and then a 1564 BUILD opcode to apply __setstate__ to that argument. 1565 1566 If not isinstance(callable, type), REDUCE complains unless the 1567 callable has been registered with the copyreg module's 1568 safe_constructors dict, or the callable has a magic 1569 '__safe_for_unpickling__' attribute with a true value. I'm not sure 1570 why it does this, but I've sure seen this complaint often enough when 1571 I didn't want to <wink>. 1572 """), 1573 1574 I(name='BUILD', 1575 code='b', 1576 arg=None, 1577 stack_before=[anyobject, anyobject], 1578 stack_after=[anyobject], 1579 proto=0, 1580 doc="""Finish building an object, via __setstate__ or dict update. 1581 1582 Stack before: ... anyobject argument 1583 Stack after: ... anyobject 1584 1585 where anyobject may have been mutated, as follows: 1586 1587 If the object has a __setstate__ method, 1588 1589 anyobject.__setstate__(argument) 1590 1591 is called. 1592 1593 Else the argument must be a dict, the object must have a __dict__, and 1594 the object is updated via 1595 1596 anyobject.__dict__.update(argument) 1597 """), 1598 1599 I(name='INST', 1600 code='i', 1601 arg=stringnl_noescape_pair, 1602 stack_before=[markobject, stackslice], 1603 stack_after=[anyobject], 1604 proto=0, 1605 doc="""Build a class instance. 1606 1607 This is the protocol 0 version of protocol 1's OBJ opcode. 1608 INST is followed by two newline-terminated strings, giving a 1609 module and class name, just as for the GLOBAL opcode (and see 1610 GLOBAL for more details about that). self.find_class(module, name) 1611 is used to get a class object. 1612 1613 In addition, all the objects on the stack following the topmost 1614 markobject are gathered into a tuple and popped (along with the 1615 topmost markobject), just as for the TUPLE opcode. 1616 1617 Now it gets complicated. If all of these are true: 1618 1619 + The argtuple is empty (markobject was at the top of the stack 1620 at the start). 1621 1622 + The class object does not have a __getinitargs__ attribute. 1623 1624 then we want to create an old-style class instance without invoking 1625 its __init__() method (pickle has waffled on this over the years; not 1626 calling __init__() is current wisdom). In this case, an instance of 1627 an old-style dummy class is created, and then we try to rebind its 1628 __class__ attribute to the desired class object. If this succeeds, 1629 the new instance object is pushed on the stack, and we're done. 1630 1631 Else (the argtuple is not empty, it's not an old-style class object, 1632 or the class object does have a __getinitargs__ attribute), the code 1633 first insists that the class object have a __safe_for_unpickling__ 1634 attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, 1635 it doesn't matter whether this attribute has a true or false value, it 1636 only matters whether it exists (XXX this is a bug). If 1637 __safe_for_unpickling__ doesn't exist, UnpicklingError is raised. 1638 1639 Else (the class object does have a __safe_for_unpickling__ attr), 1640 the class object obtained from INST's arguments is applied to the 1641 argtuple obtained from the stack, and the resulting instance object 1642 is pushed on the stack. 1643 1644 NOTE: checks for __safe_for_unpickling__ went away in Python 2.3. 1645 """), 1646 1647 I(name='OBJ', 1648 code='o', 1649 arg=None, 1650 stack_before=[markobject, anyobject, stackslice], 1651 stack_after=[anyobject], 1652 proto=1, 1653 doc="""Build a class instance. 1654 1655 This is the protocol 1 version of protocol 0's INST opcode, and is 1656 very much like it. The major difference is that the class object 1657 is taken off the stack, allowing it to be retrieved from the memo 1658 repeatedly if several instances of the same class are created. This 1659 can be much more efficient (in both time and space) than repeatedly 1660 embedding the module and class names in INST opcodes. 1661 1662 Unlike INST, OBJ takes no arguments from the opcode stream. Instead 1663 the class object is taken off the stack, immediately above the 1664 topmost markobject: 1665 1666 Stack before: ... markobject classobject stackslice 1667 Stack after: ... new_instance_object 1668 1669 As for INST, the remainder of the stack above the markobject is 1670 gathered into an argument tuple, and then the logic seems identical, 1671 except that no __safe_for_unpickling__ check is done (XXX this is 1672 a bug). See INST for the gory details. 1673 1674 NOTE: In Python 2.3, INST and OBJ are identical except for how they 1675 get the class object. That was always the intent; the implementations 1676 had diverged for accidental reasons. 1677 """), 1678 1679 I(name='NEWOBJ', 1680 code='\x81', 1681 arg=None, 1682 stack_before=[anyobject, anyobject], 1683 stack_after=[anyobject], 1684 proto=2, 1685 doc="""Build an object instance. 1686 1687 The stack before should be thought of as containing a class 1688 object followed by an argument tuple (the tuple being the stack 1689 top). Call these cls and args. They are popped off the stack, 1690 and the value returned by cls.__new__(cls, *args) is pushed back 1691 onto the stack. 1692 """), 1693 1694 # Machine control. 1695 1696 I(name='PROTO', 1697 code='\x80', 1698 arg=uint1, 1699 stack_before=[], 1700 stack_after=[], 1701 proto=2, 1702 doc="""Protocol version indicator. 1703 1704 For protocol 2 and above, a pickle must start with this opcode. 1705 The argument is the protocol version, an int in range(2, 256). 1706 """), 1707 1708 I(name='STOP', 1709 code='.', 1710 arg=None, 1711 stack_before=[anyobject], 1712 stack_after=[], 1713 proto=0, 1714 doc="""Stop the unpickling machine. 1715 1716 Every pickle ends with this opcode. The object at the top of the stack 1717 is popped, and that's the result of unpickling. The stack should be 1718 empty then. 1719 """), 1720 1721 # Ways to deal with persistent IDs. 1722 1723 I(name='PERSID', 1724 code='P', 1725 arg=stringnl_noescape, 1726 stack_before=[], 1727 stack_after=[anyobject], 1728 proto=0, 1729 doc="""Push an object identified by a persistent ID. 1730 1731 The pickle module doesn't define what a persistent ID means. PERSID's 1732 argument is a newline-terminated str-style (no embedded escapes, no 1733 bracketing quote characters) string, which *is* "the persistent ID". 1734 The unpickler passes this string to self.persistent_load(). Whatever 1735 object that returns is pushed on the stack. There is no implementation 1736 of persistent_load() in Python's unpickler: it must be supplied by an 1737 unpickler subclass. 1738 """), 1739 1740 I(name='BINPERSID', 1741 code='Q', 1742 arg=None, 1743 stack_before=[anyobject], 1744 stack_after=[anyobject], 1745 proto=1, 1746 doc="""Push an object identified by a persistent ID. 1747 1748 Like PERSID, except the persistent ID is popped off the stack (instead 1749 of being a string embedded in the opcode bytestream). The persistent 1750 ID is passed to self.persistent_load(), and whatever object that 1751 returns is pushed on the stack. See PERSID for more detail. 1752 """), 1753] 1754del I 1755 1756# Verify uniqueness of .name and .code members. 1757name2i = {} 1758code2i = {} 1759 1760for i, d in enumerate(opcodes): 1761 if d.name in name2i: 1762 raise ValueError("repeated name %r at indices %d and %d" % 1763 (d.name, name2i[d.name], i)) 1764 if d.code in code2i: 1765 raise ValueError("repeated code %r at indices %d and %d" % 1766 (d.code, code2i[d.code], i)) 1767 1768 name2i[d.name] = i 1769 code2i[d.code] = i 1770 1771del name2i, code2i, i, d 1772 1773############################################################################## 1774# Build a code2op dict, mapping opcode characters to OpcodeInfo records. 1775# Also ensure we've got the same stuff as pickle.py, although the 1776# introspection here is dicey. 1777 1778code2op = {} 1779for d in opcodes: 1780 code2op[d.code] = d 1781del d 1782 1783def assure_pickle_consistency(verbose=False): 1784 1785 copy = code2op.copy() 1786 for name in pickle.__all__: 1787 if not re.match("[A-Z][A-Z0-9_]+$", name): 1788 if verbose: 1789 print("skipping %r: it doesn't look like an opcode name" % name) 1790 continue 1791 picklecode = getattr(pickle, name) 1792 if not isinstance(picklecode, bytes) or len(picklecode) != 1: 1793 if verbose: 1794 print(("skipping %r: value %r doesn't look like a pickle " 1795 "code" % (name, picklecode))) 1796 continue 1797 picklecode = picklecode.decode("latin-1") 1798 if picklecode in copy: 1799 if verbose: 1800 print("checking name %r w/ code %r for consistency" % ( 1801 name, picklecode)) 1802 d = copy[picklecode] 1803 if d.name != name: 1804 raise ValueError("for pickle code %r, pickle.py uses name %r " 1805 "but we're using name %r" % (picklecode, 1806 name, 1807 d.name)) 1808 # Forget this one. Any left over in copy at the end are a problem 1809 # of a different kind. 1810 del copy[picklecode] 1811 else: 1812 raise ValueError("pickle.py appears to have a pickle opcode with " 1813 "name %r and code %r, but we don't" % 1814 (name, picklecode)) 1815 if copy: 1816 msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"] 1817 for code, d in copy.items(): 1818 msg.append(" name %r with code %r" % (d.name, code)) 1819 raise ValueError("\n".join(msg)) 1820 1821assure_pickle_consistency() 1822del assure_pickle_consistency 1823 1824############################################################################## 1825# A pickle opcode generator. 1826 1827def genops(pickle): 1828 """Generate all the opcodes in a pickle. 1829 1830 'pickle' is a file-like object, or string, containing the pickle. 1831 1832 Each opcode in the pickle is generated, from the current pickle position, 1833 stopping after a STOP opcode is delivered. A triple is generated for 1834 each opcode: 1835 1836 opcode, arg, pos 1837 1838 opcode is an OpcodeInfo record, describing the current opcode. 1839 1840 If the opcode has an argument embedded in the pickle, arg is its decoded 1841 value, as a Python object. If the opcode doesn't have an argument, arg 1842 is None. 1843 1844 If the pickle has a tell() method, pos was the value of pickle.tell() 1845 before reading the current opcode. If the pickle is a bytes object, 1846 it's wrapped in a BytesIO object, and the latter's tell() result is 1847 used. Else (the pickle doesn't have a tell(), and it's not obvious how 1848 to query its current position) pos is None. 1849 """ 1850 1851 if isinstance(pickle, bytes_types): 1852 import io 1853 pickle = io.BytesIO(pickle) 1854 1855 if hasattr(pickle, "tell"): 1856 getpos = pickle.tell 1857 else: 1858 getpos = lambda: None 1859 1860 while True: 1861 pos = getpos() 1862 code = pickle.read(1) 1863 opcode = code2op.get(code.decode("latin-1")) 1864 if opcode is None: 1865 if code == b"": 1866 raise ValueError("pickle exhausted before seeing STOP") 1867 else: 1868 raise ValueError("at position %s, opcode %r unknown" % ( 1869 pos is None and "<unknown>" or pos, 1870 code)) 1871 if opcode.arg is None: 1872 arg = None 1873 else: 1874 arg = opcode.arg.reader(pickle) 1875 yield opcode, arg, pos 1876 if code == b'.': 1877 assert opcode.name == 'STOP' 1878 break 1879 1880############################################################################## 1881# A pickle optimizer. 1882 1883def optimize(p): 1884 'Optimize a pickle string by removing unused PUT opcodes' 1885 gets = set() # set of args used by a GET opcode 1886 puts = [] # (arg, startpos, stoppos) for the PUT opcodes 1887 prevpos = None # set to pos if previous opcode was a PUT 1888 for opcode, arg, pos in genops(p): 1889 if prevpos is not None: 1890 puts.append((prevarg, prevpos, pos)) 1891 prevpos = None 1892 if 'PUT' in opcode.name: 1893 prevarg, prevpos = arg, pos 1894 elif 'GET' in opcode.name: 1895 gets.add(arg) 1896 1897 # Copy the pickle string except for PUTS without a corresponding GET 1898 s = [] 1899 i = 0 1900 for arg, start, stop in puts: 1901 j = stop if (arg in gets) else start 1902 s.append(p[i:j]) 1903 i = stop 1904 s.append(p[i:]) 1905 return b''.join(s) 1906 1907############################################################################## 1908# A symbolic pickle disassembler. 1909 1910def dis(pickle, out=None, memo=None, indentlevel=4, annotate=0): 1911 """Produce a symbolic disassembly of a pickle. 1912 1913 'pickle' is a file-like object, or string, containing a (at least one) 1914 pickle. The pickle is disassembled from the current position, through 1915 the first STOP opcode encountered. 1916 1917 Optional arg 'out' is a file-like object to which the disassembly is 1918 printed. It defaults to sys.stdout. 1919 1920 Optional arg 'memo' is a Python dict, used as the pickle's memo. It 1921 may be mutated by dis(), if the pickle contains PUT or BINPUT opcodes. 1922 Passing the same memo object to another dis() call then allows disassembly 1923 to proceed across multiple pickles that were all created by the same 1924 pickler with the same memo. Ordinarily you don't need to worry about this. 1925 1926 Optional arg 'indentlevel' is the number of blanks by which to indent 1927 a new MARK level. It defaults to 4. 1928 1929 Optional arg 'annotate' if nonzero instructs dis() to add short 1930 description of the opcode on each line of disassembled output. 1931 The value given to 'annotate' must be an integer and is used as a 1932 hint for the column where annotation should start. The default 1933 value is 0, meaning no annotations. 1934 1935 In addition to printing the disassembly, some sanity checks are made: 1936 1937 + All embedded opcode arguments "make sense". 1938 1939 + Explicit and implicit pop operations have enough items on the stack. 1940 1941 + When an opcode implicitly refers to a markobject, a markobject is 1942 actually on the stack. 1943 1944 + A memo entry isn't referenced before it's defined. 1945 1946 + The markobject isn't stored in the memo. 1947 1948 + A memo entry isn't redefined. 1949 """ 1950 1951 # Most of the hair here is for sanity checks, but most of it is needed 1952 # anyway to detect when a protocol 0 POP takes a MARK off the stack 1953 # (which in turn is needed to indent MARK blocks correctly). 1954 1955 stack = [] # crude emulation of unpickler stack 1956 if memo is None: 1957 memo = {} # crude emulation of unpicker memo 1958 maxproto = -1 # max protocol number seen 1959 markstack = [] # bytecode positions of MARK opcodes 1960 indentchunk = ' ' * indentlevel 1961 errormsg = None 1962 annocol = annotate # columnt hint for annotations 1963 for opcode, arg, pos in genops(pickle): 1964 if pos is not None: 1965 print("%5d:" % pos, end=' ', file=out) 1966 1967 line = "%-4s %s%s" % (repr(opcode.code)[1:-1], 1968 indentchunk * len(markstack), 1969 opcode.name) 1970 1971 maxproto = max(maxproto, opcode.proto) 1972 before = opcode.stack_before # don't mutate 1973 after = opcode.stack_after # don't mutate 1974 numtopop = len(before) 1975 1976 # See whether a MARK should be popped. 1977 markmsg = None 1978 if markobject in before or (opcode.name == "POP" and 1979 stack and 1980 stack[-1] is markobject): 1981 assert markobject not in after 1982 if __debug__: 1983 if markobject in before: 1984 assert before[-1] is stackslice 1985 if markstack: 1986 markpos = markstack.pop() 1987 if markpos is None: 1988 markmsg = "(MARK at unknown opcode offset)" 1989 else: 1990 markmsg = "(MARK at %d)" % markpos 1991 # Pop everything at and after the topmost markobject. 1992 while stack[-1] is not markobject: 1993 stack.pop() 1994 stack.pop() 1995 # Stop later code from popping too much. 1996 try: 1997 numtopop = before.index(markobject) 1998 except ValueError: 1999 assert opcode.name == "POP" 2000 numtopop = 0 2001 else: 2002 errormsg = markmsg = "no MARK exists on stack" 2003 2004 # Check for correct memo usage. 2005 if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"): 2006 assert arg is not None 2007 if arg in memo: 2008 errormsg = "memo key %r already defined" % arg 2009 elif not stack: 2010 errormsg = "stack is empty -- can't store into memo" 2011 elif stack[-1] is markobject: 2012 errormsg = "can't store markobject in the memo" 2013 else: 2014 memo[arg] = stack[-1] 2015 2016 elif opcode.name in ("GET", "BINGET", "LONG_BINGET"): 2017 if arg in memo: 2018 assert len(after) == 1 2019 after = [memo[arg]] # for better stack emulation 2020 else: 2021 errormsg = "memo key %r has never been stored into" % arg 2022 2023 if arg is not None or markmsg: 2024 # make a mild effort to align arguments 2025 line += ' ' * (10 - len(opcode.name)) 2026 if arg is not None: 2027 line += ' ' + repr(arg) 2028 if markmsg: 2029 line += ' ' + markmsg 2030 if annotate: 2031 line += ' ' * (annocol - len(line)) 2032 # make a mild effort to align annotations 2033 annocol = len(line) 2034 if annocol > 50: 2035 annocol = annotate 2036 line += ' ' + opcode.doc.split('\n', 1)[0] 2037 print(line, file=out) 2038 2039 if errormsg: 2040 # Note that we delayed complaining until the offending opcode 2041 # was printed. 2042 raise ValueError(errormsg) 2043 2044 # Emulate the stack effects. 2045 if len(stack) < numtopop: 2046 raise ValueError("tries to pop %d items from stack with " 2047 "only %d items" % (numtopop, len(stack))) 2048 if numtopop: 2049 del stack[-numtopop:] 2050 if markobject in after: 2051 assert markobject not in before 2052 markstack.append(pos) 2053 2054 stack.extend(after) 2055 2056 print("highest protocol among opcodes =", maxproto, file=out) 2057 if stack: 2058 raise ValueError("stack not empty after STOP: %r" % stack) 2059 2060# For use in the doctest, simply as an example of a class to pickle. 2061class _Example: 2062 def __init__(self, value): 2063 self.value = value 2064 2065_dis_test = r""" 2066>>> import pickle 2067>>> x = [1, 2, (3, 4), {b'abc': "def"}] 2068>>> pkl0 = pickle.dumps(x, 0) 2069>>> dis(pkl0) 2070 0: ( MARK 2071 1: l LIST (MARK at 0) 2072 2: p PUT 0 2073 5: L LONG 1 2074 9: a APPEND 2075 10: L LONG 2 2076 14: a APPEND 2077 15: ( MARK 2078 16: L LONG 3 2079 20: L LONG 4 2080 24: t TUPLE (MARK at 15) 2081 25: p PUT 1 2082 28: a APPEND 2083 29: ( MARK 2084 30: d DICT (MARK at 29) 2085 31: p PUT 2 2086 34: c GLOBAL '__builtin__ bytes' 2087 53: p PUT 3 2088 56: ( MARK 2089 57: ( MARK 2090 58: l LIST (MARK at 57) 2091 59: p PUT 4 2092 62: L LONG 97 2093 67: a APPEND 2094 68: L LONG 98 2095 73: a APPEND 2096 74: L LONG 99 2097 79: a APPEND 2098 80: t TUPLE (MARK at 56) 2099 81: p PUT 5 2100 84: R REDUCE 2101 85: p PUT 6 2102 88: V UNICODE 'def' 2103 93: p PUT 7 2104 96: s SETITEM 2105 97: a APPEND 2106 98: . STOP 2107highest protocol among opcodes = 0 2108 2109Try again with a "binary" pickle. 2110 2111>>> pkl1 = pickle.dumps(x, 1) 2112>>> dis(pkl1) 2113 0: ] EMPTY_LIST 2114 1: q BINPUT 0 2115 3: ( MARK 2116 4: K BININT1 1 2117 6: K BININT1 2 2118 8: ( MARK 2119 9: K BININT1 3 2120 11: K BININT1 4 2121 13: t TUPLE (MARK at 8) 2122 14: q BINPUT 1 2123 16: } EMPTY_DICT 2124 17: q BINPUT 2 2125 19: c GLOBAL '__builtin__ bytes' 2126 38: q BINPUT 3 2127 40: ( MARK 2128 41: ] EMPTY_LIST 2129 42: q BINPUT 4 2130 44: ( MARK 2131 45: K BININT1 97 2132 47: K BININT1 98 2133 49: K BININT1 99 2134 51: e APPENDS (MARK at 44) 2135 52: t TUPLE (MARK at 40) 2136 53: q BINPUT 5 2137 55: R REDUCE 2138 56: q BINPUT 6 2139 58: X BINUNICODE 'def' 2140 66: q BINPUT 7 2141 68: s SETITEM 2142 69: e APPENDS (MARK at 3) 2143 70: . STOP 2144highest protocol among opcodes = 1 2145 2146Exercise the INST/OBJ/BUILD family. 2147 2148>>> import pickletools 2149>>> dis(pickle.dumps(pickletools.dis, 0)) 2150 0: c GLOBAL 'pickletools dis' 2151 17: p PUT 0 2152 20: . STOP 2153highest protocol among opcodes = 0 2154 2155>>> from pickletools import _Example 2156>>> x = [_Example(42)] * 2 2157>>> dis(pickle.dumps(x, 0)) 2158 0: ( MARK 2159 1: l LIST (MARK at 0) 2160 2: p PUT 0 2161 5: c GLOBAL 'copy_reg _reconstructor' 2162 30: p PUT 1 2163 33: ( MARK 2164 34: c GLOBAL 'pickletools _Example' 2165 56: p PUT 2 2166 59: c GLOBAL '__builtin__ object' 2167 79: p PUT 3 2168 82: N NONE 2169 83: t TUPLE (MARK at 33) 2170 84: p PUT 4 2171 87: R REDUCE 2172 88: p PUT 5 2173 91: ( MARK 2174 92: d DICT (MARK at 91) 2175 93: p PUT 6 2176 96: V UNICODE 'value' 2177 103: p PUT 7 2178 106: L LONG 42 2179 111: s SETITEM 2180 112: b BUILD 2181 113: a APPEND 2182 114: g GET 5 2183 117: a APPEND 2184 118: . STOP 2185highest protocol among opcodes = 0 2186 2187>>> dis(pickle.dumps(x, 1)) 2188 0: ] EMPTY_LIST 2189 1: q BINPUT 0 2190 3: ( MARK 2191 4: c GLOBAL 'copy_reg _reconstructor' 2192 29: q BINPUT 1 2193 31: ( MARK 2194 32: c GLOBAL 'pickletools _Example' 2195 54: q BINPUT 2 2196 56: c GLOBAL '__builtin__ object' 2197 76: q BINPUT 3 2198 78: N NONE 2199 79: t TUPLE (MARK at 31) 2200 80: q BINPUT 4 2201 82: R REDUCE 2202 83: q BINPUT 5 2203 85: } EMPTY_DICT 2204 86: q BINPUT 6 2205 88: X BINUNICODE 'value' 2206 98: q BINPUT 7 2207 100: K BININT1 42 2208 102: s SETITEM 2209 103: b BUILD 2210 104: h BINGET 5 2211 106: e APPENDS (MARK at 3) 2212 107: . STOP 2213highest protocol among opcodes = 1 2214 2215Try "the canonical" recursive-object test. 2216 2217>>> L = [] 2218>>> T = L, 2219>>> L.append(T) 2220>>> L[0] is T 2221True 2222>>> T[0] is L 2223True 2224>>> L[0][0] is L 2225True 2226>>> T[0][0] is T 2227True 2228>>> dis(pickle.dumps(L, 0)) 2229 0: ( MARK 2230 1: l LIST (MARK at 0) 2231 2: p PUT 0 2232 5: ( MARK 2233 6: g GET 0 2234 9: t TUPLE (MARK at 5) 2235 10: p PUT 1 2236 13: a APPEND 2237 14: . STOP 2238highest protocol among opcodes = 0 2239 2240>>> dis(pickle.dumps(L, 1)) 2241 0: ] EMPTY_LIST 2242 1: q BINPUT 0 2243 3: ( MARK 2244 4: h BINGET 0 2245 6: t TUPLE (MARK at 3) 2246 7: q BINPUT 1 2247 9: a APPEND 2248 10: . STOP 2249highest protocol among opcodes = 1 2250 2251Note that, in the protocol 0 pickle of the recursive tuple, the disassembler 2252has to emulate the stack in order to realize that the POP opcode at 16 gets 2253rid of the MARK at 0. 2254 2255>>> dis(pickle.dumps(T, 0)) 2256 0: ( MARK 2257 1: ( MARK 2258 2: l LIST (MARK at 1) 2259 3: p PUT 0 2260 6: ( MARK 2261 7: g GET 0 2262 10: t TUPLE (MARK at 6) 2263 11: p PUT 1 2264 14: a APPEND 2265 15: 0 POP 2266 16: 0 POP (MARK at 0) 2267 17: g GET 1 2268 20: . STOP 2269highest protocol among opcodes = 0 2270 2271>>> dis(pickle.dumps(T, 1)) 2272 0: ( MARK 2273 1: ] EMPTY_LIST 2274 2: q BINPUT 0 2275 4: ( MARK 2276 5: h BINGET 0 2277 7: t TUPLE (MARK at 4) 2278 8: q BINPUT 1 2279 10: a APPEND 2280 11: 1 POP_MARK (MARK at 0) 2281 12: h BINGET 1 2282 14: . STOP 2283highest protocol among opcodes = 1 2284 2285Try protocol 2. 2286 2287>>> dis(pickle.dumps(L, 2)) 2288 0: \x80 PROTO 2 2289 2: ] EMPTY_LIST 2290 3: q BINPUT 0 2291 5: h BINGET 0 2292 7: \x85 TUPLE1 2293 8: q BINPUT 1 2294 10: a APPEND 2295 11: . STOP 2296highest protocol among opcodes = 2 2297 2298>>> dis(pickle.dumps(T, 2)) 2299 0: \x80 PROTO 2 2300 2: ] EMPTY_LIST 2301 3: q BINPUT 0 2302 5: h BINGET 0 2303 7: \x85 TUPLE1 2304 8: q BINPUT 1 2305 10: a APPEND 2306 11: 0 POP 2307 12: h BINGET 1 2308 14: . STOP 2309highest protocol among opcodes = 2 2310 2311Try protocol 3 with annotations: 2312 2313>>> dis(pickle.dumps(T, 3), annotate=1) 2314 0: \x80 PROTO 3 Protocol version indicator. 2315 2: ] EMPTY_LIST Push an empty list. 2316 3: q BINPUT 0 Store the stack top into the memo. The stack is not popped. 2317 5: h BINGET 0 Read an object from the memo and push it on the stack. 2318 7: \x85 TUPLE1 Build a one-tuple out of the topmost item on the stack. 2319 8: q BINPUT 1 Store the stack top into the memo. The stack is not popped. 2320 10: a APPEND Append an object to a list. 2321 11: 0 POP Discard the top stack item, shrinking the stack by one item. 2322 12: h BINGET 1 Read an object from the memo and push it on the stack. 2323 14: . STOP Stop the unpickling machine. 2324highest protocol among opcodes = 2 2325 2326""" 2327 2328_memo_test = r""" 2329>>> import pickle 2330>>> import io 2331>>> f = io.BytesIO() 2332>>> p = pickle.Pickler(f, 2) 2333>>> x = [1, 2, 3] 2334>>> p.dump(x) 2335>>> p.dump(x) 2336>>> f.seek(0) 23370 2338>>> memo = {} 2339>>> dis(f, memo=memo) 2340 0: \x80 PROTO 2 2341 2: ] EMPTY_LIST 2342 3: q BINPUT 0 2343 5: ( MARK 2344 6: K BININT1 1 2345 8: K BININT1 2 2346 10: K BININT1 3 2347 12: e APPENDS (MARK at 5) 2348 13: . STOP 2349highest protocol among opcodes = 2 2350>>> dis(f, memo=memo) 2351 14: \x80 PROTO 2 2352 16: h BINGET 0 2353 18: . STOP 2354highest protocol among opcodes = 2 2355""" 2356 2357__test__ = {'disassembler_test': _dis_test, 2358 'disassembler_memo_test': _memo_test, 2359 } 2360 2361def _test(): 2362 import doctest 2363 return doctest.testmod() 2364 2365if __name__ == "__main__": 2366 import sys, argparse 2367 parser = argparse.ArgumentParser( 2368 description='disassemble one or more pickle files') 2369 parser.add_argument( 2370 'pickle_file', type=argparse.FileType('br'), 2371 nargs='*', help='the pickle file') 2372 parser.add_argument( 2373 '-o', '--output', default=sys.stdout, type=argparse.FileType('w'), 2374 help='the file where the output should be written') 2375 parser.add_argument( 2376 '-m', '--memo', action='store_true', 2377 help='preserve memo between disassemblies') 2378 parser.add_argument( 2379 '-l', '--indentlevel', default=4, type=int, 2380 help='the number of blanks by which to indent a new MARK level') 2381 parser.add_argument( 2382 '-a', '--annotate', action='store_true', 2383 help='annotate each line with a short opcode description') 2384 parser.add_argument( 2385 '-p', '--preamble', default="==> {name} <==", 2386 help='if more than one pickle file is specified, print this before' 2387 ' each disassembly') 2388 parser.add_argument( 2389 '-t', '--test', action='store_true', 2390 help='run self-test suite') 2391 parser.add_argument( 2392 '-v', action='store_true', 2393 help='run verbosely; only affects self-test run') 2394 args = parser.parse_args() 2395 if args.test: 2396 _test() 2397 else: 2398 annotate = 30 if args.annotate else 0 2399 if not args.pickle_file: 2400 parser.print_help() 2401 elif len(args.pickle_file) == 1: 2402 dis(args.pickle_file[0], args.output, None, 2403 args.indentlevel, annotate) 2404 else: 2405 memo = {} if args.memo else None 2406 for f in args.pickle_file: 2407 preamble = args.preamble.format(name=f.name) 2408 args.output.write(preamble + '\n') 2409 dis(f, args.output, memo, args.indentlevel, annotate) 2410