pickletools.py revision 2bb96f593a067dafc7f02fef659ff22994b83399
1'''"Executable documentation" for the pickle module. 2 3Extensive comments about the pickle protocols and pickle-machine opcodes 4can be found here. Some functions meant for external use: 5 6genops(pickle) 7 Generate all the opcodes in a pickle, as (opcode, arg, position) triples. 8 9dis(pickle, out=None, memo=None, indentlevel=4) 10 Print a symbolic disassembly of a pickle. 11''' 12 13import codecs 14import pickle 15import re 16 17__all__ = ['dis', 'genops', 'optimize'] 18 19bytes_types = pickle.bytes_types 20 21# Other ideas: 22# 23# - A pickle verifier: read a pickle and check it exhaustively for 24# well-formedness. dis() does a lot of this already. 25# 26# - A protocol identifier: examine a pickle and return its protocol number 27# (== the highest .proto attr value among all the opcodes in the pickle). 28# dis() already prints this info at the end. 29# 30# - A pickle optimizer: for example, tuple-building code is sometimes more 31# elaborate than necessary, catering for the possibility that the tuple 32# is recursive. Or lots of times a PUT is generated that's never accessed 33# by a later GET. 34 35 36""" 37"A pickle" is a program for a virtual pickle machine (PM, but more accurately 38called an unpickling machine). It's a sequence of opcodes, interpreted by the 39PM, building an arbitrarily complex Python object. 40 41For the most part, the PM is very simple: there are no looping, testing, or 42conditional instructions, no arithmetic and no function calls. Opcodes are 43executed once each, from first to last, until a STOP opcode is reached. 44 45The PM has two data areas, "the stack" and "the memo". 46 47Many opcodes push Python objects onto the stack; e.g., INT pushes a Python 48integer object on the stack, whose value is gotten from a decimal string 49literal immediately following the INT opcode in the pickle bytestream. Other 50opcodes take Python objects off the stack. The result of unpickling is 51whatever object is left on the stack when the final STOP opcode is executed. 52 53The memo is simply an array of objects, or it can be implemented as a dict 54mapping little integers to objects. The memo serves as the PM's "long term 55memory", and the little integers indexing the memo are akin to variable 56names. Some opcodes pop a stack object into the memo at a given index, 57and others push a memo object at a given index onto the stack again. 58 59At heart, that's all the PM has. Subtleties arise for these reasons: 60 61+ Object identity. Objects can be arbitrarily complex, and subobjects 62 may be shared (for example, the list [a, a] refers to the same object a 63 twice). It can be vital that unpickling recreate an isomorphic object 64 graph, faithfully reproducing sharing. 65 66+ Recursive objects. For example, after "L = []; L.append(L)", L is a 67 list, and L[0] is the same list. This is related to the object identity 68 point, and some sequences of pickle opcodes are subtle in order to 69 get the right result in all cases. 70 71+ Things pickle doesn't know everything about. Examples of things pickle 72 does know everything about are Python's builtin scalar and container 73 types, like ints and tuples. They generally have opcodes dedicated to 74 them. For things like module references and instances of user-defined 75 classes, pickle's knowledge is limited. Historically, many enhancements 76 have been made to the pickle protocol in order to do a better (faster, 77 and/or more compact) job on those. 78 79+ Backward compatibility and micro-optimization. As explained below, 80 pickle opcodes never go away, not even when better ways to do a thing 81 get invented. The repertoire of the PM just keeps growing over time. 82 For example, protocol 0 had two opcodes for building Python integers (INT 83 and LONG), protocol 1 added three more for more-efficient pickling of short 84 integers, and protocol 2 added two more for more-efficient pickling of 85 long integers (before protocol 2, the only ways to pickle a Python long 86 took time quadratic in the number of digits, for both pickling and 87 unpickling). "Opcode bloat" isn't so much a subtlety as a source of 88 wearying complication. 89 90 91Pickle protocols: 92 93For compatibility, the meaning of a pickle opcode never changes. Instead new 94pickle opcodes get added, and each version's unpickler can handle all the 95pickle opcodes in all protocol versions to date. So old pickles continue to 96be readable forever. The pickler can generally be told to restrict itself to 97the subset of opcodes available under previous protocol versions too, so that 98users can create pickles under the current version readable by older 99versions. However, a pickle does not contain its version number embedded 100within it. If an older unpickler tries to read a pickle using a later 101protocol, the result is most likely an exception due to seeing an unknown (in 102the older unpickler) opcode. 103 104The original pickle used what's now called "protocol 0", and what was called 105"text mode" before Python 2.3. The entire pickle bytestream is made up of 106printable 7-bit ASCII characters, plus the newline character, in protocol 0. 107That's why it was called text mode. Protocol 0 is small and elegant, but 108sometimes painfully inefficient. 109 110The second major set of additions is now called "protocol 1", and was called 111"binary mode" before Python 2.3. This added many opcodes with arguments 112consisting of arbitrary bytes, including NUL bytes and unprintable "high bit" 113bytes. Binary mode pickles can be substantially smaller than equivalent 114text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte 115int as 4 bytes following the opcode, which is cheaper to unpickle than the 116(perhaps) 11-character decimal string attached to INT. Protocol 1 also added 117a number of opcodes that operate on many stack elements at once (like APPENDS 118and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE). 119 120The third major set of additions came in Python 2.3, and is called "protocol 1212". This added: 122 123- A better way to pickle instances of new-style classes (NEWOBJ). 124 125- A way for a pickle to identify its protocol (PROTO). 126 127- Time- and space- efficient pickling of long ints (LONG{1,4}). 128 129- Shortcuts for small tuples (TUPLE{1,2,3}}. 130 131- Dedicated opcodes for bools (NEWTRUE, NEWFALSE). 132 133- The "extension registry", a vector of popular objects that can be pushed 134 efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but 135 the registry contents are predefined (there's nothing akin to the memo's 136 PUT). 137 138Another independent change with Python 2.3 is the abandonment of any 139pretense that it might be safe to load pickles received from untrusted 140parties -- no sufficient security analysis has been done to guarantee 141this and there isn't a use case that warrants the expense of such an 142analysis. 143 144To this end, all tests for __safe_for_unpickling__ or for 145copyreg.safe_constructors are removed from the unpickling code. 146References to these variables in the descriptions below are to be seen 147as describing unpickling in Python 2.2 and before. 148""" 149 150# Meta-rule: Descriptions are stored in instances of descriptor objects, 151# with plain constructors. No meta-language is defined from which 152# descriptors could be constructed. If you want, e.g., XML, write a little 153# program to generate XML from the objects. 154 155############################################################################## 156# Some pickle opcodes have an argument, following the opcode in the 157# bytestream. An argument is of a specific type, described by an instance 158# of ArgumentDescriptor. These are not to be confused with arguments taken 159# off the stack -- ArgumentDescriptor applies only to arguments embedded in 160# the opcode stream, immediately following an opcode. 161 162# Represents the number of bytes consumed by an argument delimited by the 163# next newline character. 164UP_TO_NEWLINE = -1 165 166# Represents the number of bytes consumed by a two-argument opcode where 167# the first argument gives the number of bytes in the second argument. 168TAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int 169TAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int 170 171class ArgumentDescriptor(object): 172 __slots__ = ( 173 # name of descriptor record, also a module global name; a string 174 'name', 175 176 # length of argument, in bytes; an int; UP_TO_NEWLINE and 177 # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length 178 # cases 179 'n', 180 181 # a function taking a file-like object, reading this kind of argument 182 # from the object at the current position, advancing the current 183 # position by n bytes, and returning the value of the argument 184 'reader', 185 186 # human-readable docs for this arg descriptor; a string 187 'doc', 188 ) 189 190 def __init__(self, name, n, reader, doc): 191 assert isinstance(name, str) 192 self.name = name 193 194 assert isinstance(n, int) and (n >= 0 or 195 n in (UP_TO_NEWLINE, 196 TAKEN_FROM_ARGUMENT1, 197 TAKEN_FROM_ARGUMENT4)) 198 self.n = n 199 200 self.reader = reader 201 202 assert isinstance(doc, str) 203 self.doc = doc 204 205from struct import unpack as _unpack 206 207def read_uint1(f): 208 r""" 209 >>> import io 210 >>> read_uint1(io.BytesIO(b'\xff')) 211 255 212 """ 213 214 data = f.read(1) 215 if data: 216 return data[0] 217 raise ValueError("not enough data in stream to read uint1") 218 219uint1 = ArgumentDescriptor( 220 name='uint1', 221 n=1, 222 reader=read_uint1, 223 doc="One-byte unsigned integer.") 224 225 226def read_uint2(f): 227 r""" 228 >>> import io 229 >>> read_uint2(io.BytesIO(b'\xff\x00')) 230 255 231 >>> read_uint2(io.BytesIO(b'\xff\xff')) 232 65535 233 """ 234 235 data = f.read(2) 236 if len(data) == 2: 237 return _unpack("<H", data)[0] 238 raise ValueError("not enough data in stream to read uint2") 239 240uint2 = ArgumentDescriptor( 241 name='uint2', 242 n=2, 243 reader=read_uint2, 244 doc="Two-byte unsigned integer, little-endian.") 245 246 247def read_int4(f): 248 r""" 249 >>> import io 250 >>> read_int4(io.BytesIO(b'\xff\x00\x00\x00')) 251 255 252 >>> read_int4(io.BytesIO(b'\x00\x00\x00\x80')) == -(2**31) 253 True 254 """ 255 256 data = f.read(4) 257 if len(data) == 4: 258 return _unpack("<i", data)[0] 259 raise ValueError("not enough data in stream to read int4") 260 261int4 = ArgumentDescriptor( 262 name='int4', 263 n=4, 264 reader=read_int4, 265 doc="Four-byte signed integer, little-endian, 2's complement.") 266 267 268def read_stringnl(f, decode=True, stripquotes=True): 269 r""" 270 >>> import io 271 >>> read_stringnl(io.BytesIO(b"'abcd'\nefg\n")) 272 'abcd' 273 274 >>> read_stringnl(io.BytesIO(b"\n")) 275 Traceback (most recent call last): 276 ... 277 ValueError: no string quotes around b'' 278 279 >>> read_stringnl(io.BytesIO(b"\n"), stripquotes=False) 280 '' 281 282 >>> read_stringnl(io.BytesIO(b"''\n")) 283 '' 284 285 >>> read_stringnl(io.BytesIO(b'"abcd"')) 286 Traceback (most recent call last): 287 ... 288 ValueError: no newline found when trying to read stringnl 289 290 Embedded escapes are undone in the result. 291 >>> read_stringnl(io.BytesIO(br"'a\n\\b\x00c\td'" + b"\n'e'")) 292 'a\n\\b\x00c\td' 293 """ 294 295 data = f.readline() 296 if not data.endswith(b'\n'): 297 raise ValueError("no newline found when trying to read stringnl") 298 data = data[:-1] # lose the newline 299 300 if stripquotes: 301 for q in (b'"', b"'"): 302 if data.startswith(q): 303 if not data.endswith(q): 304 raise ValueError("strinq quote %r not found at both " 305 "ends of %r" % (q, data)) 306 data = data[1:-1] 307 break 308 else: 309 raise ValueError("no string quotes around %r" % data) 310 311 if decode: 312 data = codecs.escape_decode(data)[0].decode("ascii") 313 return data 314 315stringnl = ArgumentDescriptor( 316 name='stringnl', 317 n=UP_TO_NEWLINE, 318 reader=read_stringnl, 319 doc="""A newline-terminated string. 320 321 This is a repr-style string, with embedded escapes, and 322 bracketing quotes. 323 """) 324 325def read_stringnl_noescape(f): 326 return read_stringnl(f, stripquotes=False) 327 328stringnl_noescape = ArgumentDescriptor( 329 name='stringnl_noescape', 330 n=UP_TO_NEWLINE, 331 reader=read_stringnl_noescape, 332 doc="""A newline-terminated string. 333 334 This is a str-style string, without embedded escapes, 335 or bracketing quotes. It should consist solely of 336 printable ASCII characters. 337 """) 338 339def read_stringnl_noescape_pair(f): 340 r""" 341 >>> import io 342 >>> read_stringnl_noescape_pair(io.BytesIO(b"Queue\nEmpty\njunk")) 343 'Queue Empty' 344 """ 345 346 return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) 347 348stringnl_noescape_pair = ArgumentDescriptor( 349 name='stringnl_noescape_pair', 350 n=UP_TO_NEWLINE, 351 reader=read_stringnl_noescape_pair, 352 doc="""A pair of newline-terminated strings. 353 354 These are str-style strings, without embedded 355 escapes, or bracketing quotes. They should 356 consist solely of printable ASCII characters. 357 The pair is returned as a single string, with 358 a single blank separating the two strings. 359 """) 360 361def read_string4(f): 362 r""" 363 >>> import io 364 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x00abc")) 365 '' 366 >>> read_string4(io.BytesIO(b"\x03\x00\x00\x00abcdef")) 367 'abc' 368 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x03abcdef")) 369 Traceback (most recent call last): 370 ... 371 ValueError: expected 50331648 bytes in a string4, but only 6 remain 372 """ 373 374 n = read_int4(f) 375 if n < 0: 376 raise ValueError("string4 byte count < 0: %d" % n) 377 data = f.read(n) 378 if len(data) == n: 379 return data.decode("latin-1") 380 raise ValueError("expected %d bytes in a string4, but only %d remain" % 381 (n, len(data))) 382 383string4 = ArgumentDescriptor( 384 name="string4", 385 n=TAKEN_FROM_ARGUMENT4, 386 reader=read_string4, 387 doc="""A counted string. 388 389 The first argument is a 4-byte little-endian signed int giving 390 the number of bytes in the string, and the second argument is 391 that many bytes. 392 """) 393 394 395def read_string1(f): 396 r""" 397 >>> import io 398 >>> read_string1(io.BytesIO(b"\x00")) 399 '' 400 >>> read_string1(io.BytesIO(b"\x03abcdef")) 401 'abc' 402 """ 403 404 n = read_uint1(f) 405 assert n >= 0 406 data = f.read(n) 407 if len(data) == n: 408 return data.decode("latin-1") 409 raise ValueError("expected %d bytes in a string1, but only %d remain" % 410 (n, len(data))) 411 412string1 = ArgumentDescriptor( 413 name="string1", 414 n=TAKEN_FROM_ARGUMENT1, 415 reader=read_string1, 416 doc="""A counted string. 417 418 The first argument is a 1-byte unsigned int giving the number 419 of bytes in the string, and the second argument is that many 420 bytes. 421 """) 422 423 424def read_unicodestringnl(f): 425 r""" 426 >>> import io 427 >>> read_unicodestringnl(io.BytesIO(b"abc\\uabcd\njunk")) == 'abc\uabcd' 428 True 429 """ 430 431 data = f.readline() 432 if not data.endswith(b'\n'): 433 raise ValueError("no newline found when trying to read " 434 "unicodestringnl") 435 data = data[:-1] # lose the newline 436 return str(data, 'raw-unicode-escape') 437 438unicodestringnl = ArgumentDescriptor( 439 name='unicodestringnl', 440 n=UP_TO_NEWLINE, 441 reader=read_unicodestringnl, 442 doc="""A newline-terminated Unicode string. 443 444 This is raw-unicode-escape encoded, so consists of 445 printable ASCII characters, and may contain embedded 446 escape sequences. 447 """) 448 449def read_unicodestring4(f): 450 r""" 451 >>> import io 452 >>> s = 'abcd\uabcd' 453 >>> enc = s.encode('utf-8') 454 >>> enc 455 b'abcd\xea\xaf\x8d' 456 >>> n = bytes([len(enc), 0, 0, 0]) # little-endian 4-byte length 457 >>> t = read_unicodestring4(io.BytesIO(n + enc + b'junk')) 458 >>> s == t 459 True 460 461 >>> read_unicodestring4(io.BytesIO(n + enc[:-1])) 462 Traceback (most recent call last): 463 ... 464 ValueError: expected 7 bytes in a unicodestring4, but only 6 remain 465 """ 466 467 n = read_int4(f) 468 if n < 0: 469 raise ValueError("unicodestring4 byte count < 0: %d" % n) 470 data = f.read(n) 471 if len(data) == n: 472 return str(data, 'utf-8', 'surrogatepass') 473 raise ValueError("expected %d bytes in a unicodestring4, but only %d " 474 "remain" % (n, len(data))) 475 476unicodestring4 = ArgumentDescriptor( 477 name="unicodestring4", 478 n=TAKEN_FROM_ARGUMENT4, 479 reader=read_unicodestring4, 480 doc="""A counted Unicode string. 481 482 The first argument is a 4-byte little-endian signed int 483 giving the number of bytes in the string, and the second 484 argument-- the UTF-8 encoding of the Unicode string -- 485 contains that many bytes. 486 """) 487 488 489def read_decimalnl_short(f): 490 r""" 491 >>> import io 492 >>> read_decimalnl_short(io.BytesIO(b"1234\n56")) 493 1234 494 495 >>> read_decimalnl_short(io.BytesIO(b"1234L\n56")) 496 Traceback (most recent call last): 497 ... 498 ValueError: trailing 'L' not allowed in b'1234L' 499 """ 500 501 s = read_stringnl(f, decode=False, stripquotes=False) 502 if s.endswith(b"L"): 503 raise ValueError("trailing 'L' not allowed in %r" % s) 504 505 # It's not necessarily true that the result fits in a Python short int: 506 # the pickle may have been written on a 64-bit box. There's also a hack 507 # for True and False here. 508 if s == b"00": 509 return False 510 elif s == b"01": 511 return True 512 513 return int(s) 514 515def read_decimalnl_long(f): 516 r""" 517 >>> import io 518 519 >>> read_decimalnl_long(io.BytesIO(b"1234L\n56")) 520 1234 521 522 >>> read_decimalnl_long(io.BytesIO(b"123456789012345678901234L\n6")) 523 123456789012345678901234 524 """ 525 526 s = read_stringnl(f, decode=False, stripquotes=False) 527 if s[-1:] == b'L': 528 s = s[:-1] 529 return int(s) 530 531 532decimalnl_short = ArgumentDescriptor( 533 name='decimalnl_short', 534 n=UP_TO_NEWLINE, 535 reader=read_decimalnl_short, 536 doc="""A newline-terminated decimal integer literal. 537 538 This never has a trailing 'L', and the integer fit 539 in a short Python int on the box where the pickle 540 was written -- but there's no guarantee it will fit 541 in a short Python int on the box where the pickle 542 is read. 543 """) 544 545decimalnl_long = ArgumentDescriptor( 546 name='decimalnl_long', 547 n=UP_TO_NEWLINE, 548 reader=read_decimalnl_long, 549 doc="""A newline-terminated decimal integer literal. 550 551 This has a trailing 'L', and can represent integers 552 of any size. 553 """) 554 555 556def read_floatnl(f): 557 r""" 558 >>> import io 559 >>> read_floatnl(io.BytesIO(b"-1.25\n6")) 560 -1.25 561 """ 562 s = read_stringnl(f, decode=False, stripquotes=False) 563 return float(s) 564 565floatnl = ArgumentDescriptor( 566 name='floatnl', 567 n=UP_TO_NEWLINE, 568 reader=read_floatnl, 569 doc="""A newline-terminated decimal floating literal. 570 571 In general this requires 17 significant digits for roundtrip 572 identity, and pickling then unpickling infinities, NaNs, and 573 minus zero doesn't work across boxes, or on some boxes even 574 on itself (e.g., Windows can't read the strings it produces 575 for infinities or NaNs). 576 """) 577 578def read_float8(f): 579 r""" 580 >>> import io, struct 581 >>> raw = struct.pack(">d", -1.25) 582 >>> raw 583 b'\xbf\xf4\x00\x00\x00\x00\x00\x00' 584 >>> read_float8(io.BytesIO(raw + b"\n")) 585 -1.25 586 """ 587 588 data = f.read(8) 589 if len(data) == 8: 590 return _unpack(">d", data)[0] 591 raise ValueError("not enough data in stream to read float8") 592 593 594float8 = ArgumentDescriptor( 595 name='float8', 596 n=8, 597 reader=read_float8, 598 doc="""An 8-byte binary representation of a float, big-endian. 599 600 The format is unique to Python, and shared with the struct 601 module (format string '>d') "in theory" (the struct and pickle 602 implementations don't share the code -- they should). It's 603 strongly related to the IEEE-754 double format, and, in normal 604 cases, is in fact identical to the big-endian 754 double format. 605 On other boxes the dynamic range is limited to that of a 754 606 double, and "add a half and chop" rounding is used to reduce 607 the precision to 53 bits. However, even on a 754 box, 608 infinities, NaNs, and minus zero may not be handled correctly 609 (may not survive roundtrip pickling intact). 610 """) 611 612# Protocol 2 formats 613 614from pickle import decode_long 615 616def read_long1(f): 617 r""" 618 >>> import io 619 >>> read_long1(io.BytesIO(b"\x00")) 620 0 621 >>> read_long1(io.BytesIO(b"\x02\xff\x00")) 622 255 623 >>> read_long1(io.BytesIO(b"\x02\xff\x7f")) 624 32767 625 >>> read_long1(io.BytesIO(b"\x02\x00\xff")) 626 -256 627 >>> read_long1(io.BytesIO(b"\x02\x00\x80")) 628 -32768 629 """ 630 631 n = read_uint1(f) 632 data = f.read(n) 633 if len(data) != n: 634 raise ValueError("not enough data in stream to read long1") 635 return decode_long(data) 636 637long1 = ArgumentDescriptor( 638 name="long1", 639 n=TAKEN_FROM_ARGUMENT1, 640 reader=read_long1, 641 doc="""A binary long, little-endian, using 1-byte size. 642 643 This first reads one byte as an unsigned size, then reads that 644 many bytes and interprets them as a little-endian 2's-complement long. 645 If the size is 0, that's taken as a shortcut for the long 0L. 646 """) 647 648def read_long4(f): 649 r""" 650 >>> import io 651 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x00")) 652 255 653 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x7f")) 654 32767 655 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\xff")) 656 -256 657 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\x80")) 658 -32768 659 >>> read_long1(io.BytesIO(b"\x00\x00\x00\x00")) 660 0 661 """ 662 663 n = read_int4(f) 664 if n < 0: 665 raise ValueError("long4 byte count < 0: %d" % n) 666 data = f.read(n) 667 if len(data) != n: 668 raise ValueError("not enough data in stream to read long4") 669 return decode_long(data) 670 671long4 = ArgumentDescriptor( 672 name="long4", 673 n=TAKEN_FROM_ARGUMENT4, 674 reader=read_long4, 675 doc="""A binary representation of a long, little-endian. 676 677 This first reads four bytes as a signed size (but requires the 678 size to be >= 0), then reads that many bytes and interprets them 679 as a little-endian 2's-complement long. If the size is 0, that's taken 680 as a shortcut for the int 0, although LONG1 should really be used 681 then instead (and in any case where # of bytes < 256). 682 """) 683 684 685############################################################################## 686# Object descriptors. The stack used by the pickle machine holds objects, 687# and in the stack_before and stack_after attributes of OpcodeInfo 688# descriptors we need names to describe the various types of objects that can 689# appear on the stack. 690 691class StackObject(object): 692 __slots__ = ( 693 # name of descriptor record, for info only 694 'name', 695 696 # type of object, or tuple of type objects (meaning the object can 697 # be of any type in the tuple) 698 'obtype', 699 700 # human-readable docs for this kind of stack object; a string 701 'doc', 702 ) 703 704 def __init__(self, name, obtype, doc): 705 assert isinstance(name, str) 706 self.name = name 707 708 assert isinstance(obtype, type) or isinstance(obtype, tuple) 709 if isinstance(obtype, tuple): 710 for contained in obtype: 711 assert isinstance(contained, type) 712 self.obtype = obtype 713 714 assert isinstance(doc, str) 715 self.doc = doc 716 717 def __repr__(self): 718 return self.name 719 720 721pyint = StackObject( 722 name='int', 723 obtype=int, 724 doc="A short (as opposed to long) Python integer object.") 725 726pylong = StackObject( 727 name='long', 728 obtype=int, 729 doc="A long (as opposed to short) Python integer object.") 730 731pyinteger_or_bool = StackObject( 732 name='int_or_bool', 733 obtype=(int, bool), 734 doc="A Python integer object (short or long), or " 735 "a Python bool.") 736 737pybool = StackObject( 738 name='bool', 739 obtype=(bool,), 740 doc="A Python bool object.") 741 742pyfloat = StackObject( 743 name='float', 744 obtype=float, 745 doc="A Python float object.") 746 747pystring = StackObject( 748 name='string', 749 obtype=bytes, 750 doc="A Python (8-bit) string object.") 751 752pybytes = StackObject( 753 name='bytes', 754 obtype=bytes, 755 doc="A Python bytes object.") 756 757pyunicode = StackObject( 758 name='str', 759 obtype=str, 760 doc="A Python (Unicode) string object.") 761 762pynone = StackObject( 763 name="None", 764 obtype=type(None), 765 doc="The Python None object.") 766 767pytuple = StackObject( 768 name="tuple", 769 obtype=tuple, 770 doc="A Python tuple object.") 771 772pylist = StackObject( 773 name="list", 774 obtype=list, 775 doc="A Python list object.") 776 777pydict = StackObject( 778 name="dict", 779 obtype=dict, 780 doc="A Python dict object.") 781 782anyobject = StackObject( 783 name='any', 784 obtype=object, 785 doc="Any kind of object whatsoever.") 786 787markobject = StackObject( 788 name="mark", 789 obtype=StackObject, 790 doc="""'The mark' is a unique object. 791 792 Opcodes that operate on a variable number of objects 793 generally don't embed the count of objects in the opcode, 794 or pull it off the stack. Instead the MARK opcode is used 795 to push a special marker object on the stack, and then 796 some other opcodes grab all the objects from the top of 797 the stack down to (but not including) the topmost marker 798 object. 799 """) 800 801stackslice = StackObject( 802 name="stackslice", 803 obtype=StackObject, 804 doc="""An object representing a contiguous slice of the stack. 805 806 This is used in conjuction with markobject, to represent all 807 of the stack following the topmost markobject. For example, 808 the POP_MARK opcode changes the stack from 809 810 [..., markobject, stackslice] 811 to 812 [...] 813 814 No matter how many object are on the stack after the topmost 815 markobject, POP_MARK gets rid of all of them (including the 816 topmost markobject too). 817 """) 818 819############################################################################## 820# Descriptors for pickle opcodes. 821 822class OpcodeInfo(object): 823 824 __slots__ = ( 825 # symbolic name of opcode; a string 826 'name', 827 828 # the code used in a bytestream to represent the opcode; a 829 # one-character string 830 'code', 831 832 # If the opcode has an argument embedded in the byte string, an 833 # instance of ArgumentDescriptor specifying its type. Note that 834 # arg.reader(s) can be used to read and decode the argument from 835 # the bytestream s, and arg.doc documents the format of the raw 836 # argument bytes. If the opcode doesn't have an argument embedded 837 # in the bytestream, arg should be None. 838 'arg', 839 840 # what the stack looks like before this opcode runs; a list 841 'stack_before', 842 843 # what the stack looks like after this opcode runs; a list 844 'stack_after', 845 846 # the protocol number in which this opcode was introduced; an int 847 'proto', 848 849 # human-readable docs for this opcode; a string 850 'doc', 851 ) 852 853 def __init__(self, name, code, arg, 854 stack_before, stack_after, proto, doc): 855 assert isinstance(name, str) 856 self.name = name 857 858 assert isinstance(code, str) 859 assert len(code) == 1 860 self.code = code 861 862 assert arg is None or isinstance(arg, ArgumentDescriptor) 863 self.arg = arg 864 865 assert isinstance(stack_before, list) 866 for x in stack_before: 867 assert isinstance(x, StackObject) 868 self.stack_before = stack_before 869 870 assert isinstance(stack_after, list) 871 for x in stack_after: 872 assert isinstance(x, StackObject) 873 self.stack_after = stack_after 874 875 assert isinstance(proto, int) and 0 <= proto <= 3 876 self.proto = proto 877 878 assert isinstance(doc, str) 879 self.doc = doc 880 881I = OpcodeInfo 882opcodes = [ 883 884 # Ways to spell integers. 885 886 I(name='INT', 887 code='I', 888 arg=decimalnl_short, 889 stack_before=[], 890 stack_after=[pyinteger_or_bool], 891 proto=0, 892 doc="""Push an integer or bool. 893 894 The argument is a newline-terminated decimal literal string. 895 896 The intent may have been that this always fit in a short Python int, 897 but INT can be generated in pickles written on a 64-bit box that 898 require a Python long on a 32-bit box. The difference between this 899 and LONG then is that INT skips a trailing 'L', and produces a short 900 int whenever possible. 901 902 Another difference is due to that, when bool was introduced as a 903 distinct type in 2.3, builtin names True and False were also added to 904 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, 905 True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". 906 Leading zeroes are never produced for a genuine integer. The 2.3 907 (and later) unpicklers special-case these and return bool instead; 908 earlier unpicklers ignore the leading "0" and return the int. 909 """), 910 911 I(name='BININT', 912 code='J', 913 arg=int4, 914 stack_before=[], 915 stack_after=[pyint], 916 proto=1, 917 doc="""Push a four-byte signed integer. 918 919 This handles the full range of Python (short) integers on a 32-bit 920 box, directly as binary bytes (1 for the opcode and 4 for the integer). 921 If the integer is non-negative and fits in 1 or 2 bytes, pickling via 922 BININT1 or BININT2 saves space. 923 """), 924 925 I(name='BININT1', 926 code='K', 927 arg=uint1, 928 stack_before=[], 929 stack_after=[pyint], 930 proto=1, 931 doc="""Push a one-byte unsigned integer. 932 933 This is a space optimization for pickling very small non-negative ints, 934 in range(256). 935 """), 936 937 I(name='BININT2', 938 code='M', 939 arg=uint2, 940 stack_before=[], 941 stack_after=[pyint], 942 proto=1, 943 doc="""Push a two-byte unsigned integer. 944 945 This is a space optimization for pickling small positive ints, in 946 range(256, 2**16). Integers in range(256) can also be pickled via 947 BININT2, but BININT1 instead saves a byte. 948 """), 949 950 I(name='LONG', 951 code='L', 952 arg=decimalnl_long, 953 stack_before=[], 954 stack_after=[pylong], 955 proto=0, 956 doc="""Push a long integer. 957 958 The same as INT, except that the literal ends with 'L', and always 959 unpickles to a Python long. There doesn't seem a real purpose to the 960 trailing 'L'. 961 962 Note that LONG takes time quadratic in the number of digits when 963 unpickling (this is simply due to the nature of decimal->binary 964 conversion). Proto 2 added linear-time (in C; still quadratic-time 965 in Python) LONG1 and LONG4 opcodes. 966 """), 967 968 I(name="LONG1", 969 code='\x8a', 970 arg=long1, 971 stack_before=[], 972 stack_after=[pylong], 973 proto=2, 974 doc="""Long integer using one-byte length. 975 976 A more efficient encoding of a Python long; the long1 encoding 977 says it all."""), 978 979 I(name="LONG4", 980 code='\x8b', 981 arg=long4, 982 stack_before=[], 983 stack_after=[pylong], 984 proto=2, 985 doc="""Long integer using found-byte length. 986 987 A more efficient encoding of a Python long; the long4 encoding 988 says it all."""), 989 990 # Ways to spell strings (8-bit, not Unicode). 991 992 I(name='STRING', 993 code='S', 994 arg=stringnl, 995 stack_before=[], 996 stack_after=[pystring], 997 proto=0, 998 doc="""Push a Python string object. 999 1000 The argument is a repr-style string, with bracketing quote characters, 1001 and perhaps embedded escapes. The argument extends until the next 1002 newline character. (Actually, they are decoded into a str instance 1003 using the encoding given to the Unpickler constructor. or the default, 1004 'ASCII'.) 1005 """), 1006 1007 I(name='BINSTRING', 1008 code='T', 1009 arg=string4, 1010 stack_before=[], 1011 stack_after=[pystring], 1012 proto=1, 1013 doc="""Push a Python string object. 1014 1015 There are two arguments: the first is a 4-byte little-endian signed int 1016 giving the number of bytes in the string, and the second is that many 1017 bytes, which are taken literally as the string content. (Actually, 1018 they are decoded into a str instance using the encoding given to the 1019 Unpickler constructor. or the default, 'ASCII'.) 1020 """), 1021 1022 I(name='SHORT_BINSTRING', 1023 code='U', 1024 arg=string1, 1025 stack_before=[], 1026 stack_after=[pystring], 1027 proto=1, 1028 doc="""Push a Python string object. 1029 1030 There are two arguments: the first is a 1-byte unsigned int giving 1031 the number of bytes in the string, and the second is that many bytes, 1032 which are taken literally as the string content. (Actually, they 1033 are decoded into a str instance using the encoding given to the 1034 Unpickler constructor. or the default, 'ASCII'.) 1035 """), 1036 1037 # Bytes (protocol 3 only; older protocols don't support bytes at all) 1038 1039 I(name='BINBYTES', 1040 code='B', 1041 arg=string4, 1042 stack_before=[], 1043 stack_after=[pybytes], 1044 proto=3, 1045 doc="""Push a Python bytes object. 1046 1047 There are two arguments: the first is a 4-byte little-endian signed int 1048 giving the number of bytes in the string, and the second is that many 1049 bytes, which are taken literally as the bytes content. 1050 """), 1051 1052 I(name='SHORT_BINBYTES', 1053 code='C', 1054 arg=string1, 1055 stack_before=[], 1056 stack_after=[pybytes], 1057 proto=3, 1058 doc="""Push a Python string object. 1059 1060 There are two arguments: the first is a 1-byte unsigned int giving 1061 the number of bytes in the string, and the second is that many bytes, 1062 which are taken literally as the string content. 1063 """), 1064 1065 # Ways to spell None. 1066 1067 I(name='NONE', 1068 code='N', 1069 arg=None, 1070 stack_before=[], 1071 stack_after=[pynone], 1072 proto=0, 1073 doc="Push None on the stack."), 1074 1075 # Ways to spell bools, starting with proto 2. See INT for how this was 1076 # done before proto 2. 1077 1078 I(name='NEWTRUE', 1079 code='\x88', 1080 arg=None, 1081 stack_before=[], 1082 stack_after=[pybool], 1083 proto=2, 1084 doc="""True. 1085 1086 Push True onto the stack."""), 1087 1088 I(name='NEWFALSE', 1089 code='\x89', 1090 arg=None, 1091 stack_before=[], 1092 stack_after=[pybool], 1093 proto=2, 1094 doc="""True. 1095 1096 Push False onto the stack."""), 1097 1098 # Ways to spell Unicode strings. 1099 1100 I(name='UNICODE', 1101 code='V', 1102 arg=unicodestringnl, 1103 stack_before=[], 1104 stack_after=[pyunicode], 1105 proto=0, # this may be pure-text, but it's a later addition 1106 doc="""Push a Python Unicode string object. 1107 1108 The argument is a raw-unicode-escape encoding of a Unicode string, 1109 and so may contain embedded escape sequences. The argument extends 1110 until the next newline character. 1111 """), 1112 1113 I(name='BINUNICODE', 1114 code='X', 1115 arg=unicodestring4, 1116 stack_before=[], 1117 stack_after=[pyunicode], 1118 proto=1, 1119 doc="""Push a Python Unicode string object. 1120 1121 There are two arguments: the first is a 4-byte little-endian signed int 1122 giving the number of bytes in the string. The second is that many 1123 bytes, and is the UTF-8 encoding of the Unicode string. 1124 """), 1125 1126 # Ways to spell floats. 1127 1128 I(name='FLOAT', 1129 code='F', 1130 arg=floatnl, 1131 stack_before=[], 1132 stack_after=[pyfloat], 1133 proto=0, 1134 doc="""Newline-terminated decimal float literal. 1135 1136 The argument is repr(a_float), and in general requires 17 significant 1137 digits for roundtrip conversion to be an identity (this is so for 1138 IEEE-754 double precision values, which is what Python float maps to 1139 on most boxes). 1140 1141 In general, FLOAT cannot be used to transport infinities, NaNs, or 1142 minus zero across boxes (or even on a single box, if the platform C 1143 library can't read the strings it produces for such things -- Windows 1144 is like that), but may do less damage than BINFLOAT on boxes with 1145 greater precision or dynamic range than IEEE-754 double. 1146 """), 1147 1148 I(name='BINFLOAT', 1149 code='G', 1150 arg=float8, 1151 stack_before=[], 1152 stack_after=[pyfloat], 1153 proto=1, 1154 doc="""Float stored in binary form, with 8 bytes of data. 1155 1156 This generally requires less than half the space of FLOAT encoding. 1157 In general, BINFLOAT cannot be used to transport infinities, NaNs, or 1158 minus zero, raises an exception if the exponent exceeds the range of 1159 an IEEE-754 double, and retains no more than 53 bits of precision (if 1160 there are more than that, "add a half and chop" rounding is used to 1161 cut it back to 53 significant bits). 1162 """), 1163 1164 # Ways to build lists. 1165 1166 I(name='EMPTY_LIST', 1167 code=']', 1168 arg=None, 1169 stack_before=[], 1170 stack_after=[pylist], 1171 proto=1, 1172 doc="Push an empty list."), 1173 1174 I(name='APPEND', 1175 code='a', 1176 arg=None, 1177 stack_before=[pylist, anyobject], 1178 stack_after=[pylist], 1179 proto=0, 1180 doc="""Append an object to a list. 1181 1182 Stack before: ... pylist anyobject 1183 Stack after: ... pylist+[anyobject] 1184 1185 although pylist is really extended in-place. 1186 """), 1187 1188 I(name='APPENDS', 1189 code='e', 1190 arg=None, 1191 stack_before=[pylist, markobject, stackslice], 1192 stack_after=[pylist], 1193 proto=1, 1194 doc="""Extend a list by a slice of stack objects. 1195 1196 Stack before: ... pylist markobject stackslice 1197 Stack after: ... pylist+stackslice 1198 1199 although pylist is really extended in-place. 1200 """), 1201 1202 I(name='LIST', 1203 code='l', 1204 arg=None, 1205 stack_before=[markobject, stackslice], 1206 stack_after=[pylist], 1207 proto=0, 1208 doc="""Build a list out of the topmost stack slice, after markobject. 1209 1210 All the stack entries following the topmost markobject are placed into 1211 a single Python list, which single list object replaces all of the 1212 stack from the topmost markobject onward. For example, 1213 1214 Stack before: ... markobject 1 2 3 'abc' 1215 Stack after: ... [1, 2, 3, 'abc'] 1216 """), 1217 1218 # Ways to build tuples. 1219 1220 I(name='EMPTY_TUPLE', 1221 code=')', 1222 arg=None, 1223 stack_before=[], 1224 stack_after=[pytuple], 1225 proto=1, 1226 doc="Push an empty tuple."), 1227 1228 I(name='TUPLE', 1229 code='t', 1230 arg=None, 1231 stack_before=[markobject, stackslice], 1232 stack_after=[pytuple], 1233 proto=0, 1234 doc="""Build a tuple out of the topmost stack slice, after markobject. 1235 1236 All the stack entries following the topmost markobject are placed into 1237 a single Python tuple, which single tuple object replaces all of the 1238 stack from the topmost markobject onward. For example, 1239 1240 Stack before: ... markobject 1 2 3 'abc' 1241 Stack after: ... (1, 2, 3, 'abc') 1242 """), 1243 1244 I(name='TUPLE1', 1245 code='\x85', 1246 arg=None, 1247 stack_before=[anyobject], 1248 stack_after=[pytuple], 1249 proto=2, 1250 doc="""Build a one-tuple out of the topmost item on the stack. 1251 1252 This code pops one value off the stack and pushes a tuple of 1253 length 1 whose one item is that value back onto it. In other 1254 words: 1255 1256 stack[-1] = tuple(stack[-1:]) 1257 """), 1258 1259 I(name='TUPLE2', 1260 code='\x86', 1261 arg=None, 1262 stack_before=[anyobject, anyobject], 1263 stack_after=[pytuple], 1264 proto=2, 1265 doc="""Build a two-tuple out of the top two items on the stack. 1266 1267 This code pops two values off the stack and pushes a tuple of 1268 length 2 whose items are those values back onto it. In other 1269 words: 1270 1271 stack[-2:] = [tuple(stack[-2:])] 1272 """), 1273 1274 I(name='TUPLE3', 1275 code='\x87', 1276 arg=None, 1277 stack_before=[anyobject, anyobject, anyobject], 1278 stack_after=[pytuple], 1279 proto=2, 1280 doc="""Build a three-tuple out of the top three items on the stack. 1281 1282 This code pops three values off the stack and pushes a tuple of 1283 length 3 whose items are those values back onto it. In other 1284 words: 1285 1286 stack[-3:] = [tuple(stack[-3:])] 1287 """), 1288 1289 # Ways to build dicts. 1290 1291 I(name='EMPTY_DICT', 1292 code='}', 1293 arg=None, 1294 stack_before=[], 1295 stack_after=[pydict], 1296 proto=1, 1297 doc="Push an empty dict."), 1298 1299 I(name='DICT', 1300 code='d', 1301 arg=None, 1302 stack_before=[markobject, stackslice], 1303 stack_after=[pydict], 1304 proto=0, 1305 doc="""Build a dict out of the topmost stack slice, after markobject. 1306 1307 All the stack entries following the topmost markobject are placed into 1308 a single Python dict, which single dict object replaces all of the 1309 stack from the topmost markobject onward. The stack slice alternates 1310 key, value, key, value, .... For example, 1311 1312 Stack before: ... markobject 1 2 3 'abc' 1313 Stack after: ... {1: 2, 3: 'abc'} 1314 """), 1315 1316 I(name='SETITEM', 1317 code='s', 1318 arg=None, 1319 stack_before=[pydict, anyobject, anyobject], 1320 stack_after=[pydict], 1321 proto=0, 1322 doc="""Add a key+value pair to an existing dict. 1323 1324 Stack before: ... pydict key value 1325 Stack after: ... pydict 1326 1327 where pydict has been modified via pydict[key] = value. 1328 """), 1329 1330 I(name='SETITEMS', 1331 code='u', 1332 arg=None, 1333 stack_before=[pydict, markobject, stackslice], 1334 stack_after=[pydict], 1335 proto=1, 1336 doc="""Add an arbitrary number of key+value pairs to an existing dict. 1337 1338 The slice of the stack following the topmost markobject is taken as 1339 an alternating sequence of keys and values, added to the dict 1340 immediately under the topmost markobject. Everything at and after the 1341 topmost markobject is popped, leaving the mutated dict at the top 1342 of the stack. 1343 1344 Stack before: ... pydict markobject key_1 value_1 ... key_n value_n 1345 Stack after: ... pydict 1346 1347 where pydict has been modified via pydict[key_i] = value_i for i in 1348 1, 2, ..., n, and in that order. 1349 """), 1350 1351 # Stack manipulation. 1352 1353 I(name='POP', 1354 code='0', 1355 arg=None, 1356 stack_before=[anyobject], 1357 stack_after=[], 1358 proto=0, 1359 doc="Discard the top stack item, shrinking the stack by one item."), 1360 1361 I(name='DUP', 1362 code='2', 1363 arg=None, 1364 stack_before=[anyobject], 1365 stack_after=[anyobject, anyobject], 1366 proto=0, 1367 doc="Push the top stack item onto the stack again, duplicating it."), 1368 1369 I(name='MARK', 1370 code='(', 1371 arg=None, 1372 stack_before=[], 1373 stack_after=[markobject], 1374 proto=0, 1375 doc="""Push markobject onto the stack. 1376 1377 markobject is a unique object, used by other opcodes to identify a 1378 region of the stack containing a variable number of objects for them 1379 to work on. See markobject.doc for more detail. 1380 """), 1381 1382 I(name='POP_MARK', 1383 code='1', 1384 arg=None, 1385 stack_before=[markobject, stackslice], 1386 stack_after=[], 1387 proto=1, 1388 doc="""Pop all the stack objects at and above the topmost markobject. 1389 1390 When an opcode using a variable number of stack objects is done, 1391 POP_MARK is used to remove those objects, and to remove the markobject 1392 that delimited their starting position on the stack. 1393 """), 1394 1395 # Memo manipulation. There are really only two operations (get and put), 1396 # each in all-text, "short binary", and "long binary" flavors. 1397 1398 I(name='GET', 1399 code='g', 1400 arg=decimalnl_short, 1401 stack_before=[], 1402 stack_after=[anyobject], 1403 proto=0, 1404 doc="""Read an object from the memo and push it on the stack. 1405 1406 The index of the memo object to push is given by the newline-terminated 1407 decimal string following. BINGET and LONG_BINGET are space-optimized 1408 versions. 1409 """), 1410 1411 I(name='BINGET', 1412 code='h', 1413 arg=uint1, 1414 stack_before=[], 1415 stack_after=[anyobject], 1416 proto=1, 1417 doc="""Read an object from the memo and push it on the stack. 1418 1419 The index of the memo object to push is given by the 1-byte unsigned 1420 integer following. 1421 """), 1422 1423 I(name='LONG_BINGET', 1424 code='j', 1425 arg=int4, 1426 stack_before=[], 1427 stack_after=[anyobject], 1428 proto=1, 1429 doc="""Read an object from the memo and push it on the stack. 1430 1431 The index of the memo object to push is given by the 4-byte signed 1432 little-endian integer following. 1433 """), 1434 1435 I(name='PUT', 1436 code='p', 1437 arg=decimalnl_short, 1438 stack_before=[], 1439 stack_after=[], 1440 proto=0, 1441 doc="""Store the stack top into the memo. The stack is not popped. 1442 1443 The index of the memo location to write into is given by the newline- 1444 terminated decimal string following. BINPUT and LONG_BINPUT are 1445 space-optimized versions. 1446 """), 1447 1448 I(name='BINPUT', 1449 code='q', 1450 arg=uint1, 1451 stack_before=[], 1452 stack_after=[], 1453 proto=1, 1454 doc="""Store the stack top into the memo. The stack is not popped. 1455 1456 The index of the memo location to write into is given by the 1-byte 1457 unsigned integer following. 1458 """), 1459 1460 I(name='LONG_BINPUT', 1461 code='r', 1462 arg=int4, 1463 stack_before=[], 1464 stack_after=[], 1465 proto=1, 1466 doc="""Store the stack top into the memo. The stack is not popped. 1467 1468 The index of the memo location to write into is given by the 4-byte 1469 signed little-endian integer following. 1470 """), 1471 1472 # Access the extension registry (predefined objects). Akin to the GET 1473 # family. 1474 1475 I(name='EXT1', 1476 code='\x82', 1477 arg=uint1, 1478 stack_before=[], 1479 stack_after=[anyobject], 1480 proto=2, 1481 doc="""Extension code. 1482 1483 This code and the similar EXT2 and EXT4 allow using a registry 1484 of popular objects that are pickled by name, typically classes. 1485 It is envisioned that through a global negotiation and 1486 registration process, third parties can set up a mapping between 1487 ints and object names. 1488 1489 In order to guarantee pickle interchangeability, the extension 1490 code registry ought to be global, although a range of codes may 1491 be reserved for private use. 1492 1493 EXT1 has a 1-byte integer argument. This is used to index into the 1494 extension registry, and the object at that index is pushed on the stack. 1495 """), 1496 1497 I(name='EXT2', 1498 code='\x83', 1499 arg=uint2, 1500 stack_before=[], 1501 stack_after=[anyobject], 1502 proto=2, 1503 doc="""Extension code. 1504 1505 See EXT1. EXT2 has a two-byte integer argument. 1506 """), 1507 1508 I(name='EXT4', 1509 code='\x84', 1510 arg=int4, 1511 stack_before=[], 1512 stack_after=[anyobject], 1513 proto=2, 1514 doc="""Extension code. 1515 1516 See EXT1. EXT4 has a four-byte integer argument. 1517 """), 1518 1519 # Push a class object, or module function, on the stack, via its module 1520 # and name. 1521 1522 I(name='GLOBAL', 1523 code='c', 1524 arg=stringnl_noescape_pair, 1525 stack_before=[], 1526 stack_after=[anyobject], 1527 proto=0, 1528 doc="""Push a global object (module.attr) on the stack. 1529 1530 Two newline-terminated strings follow the GLOBAL opcode. The first is 1531 taken as a module name, and the second as a class name. The class 1532 object module.class is pushed on the stack. More accurately, the 1533 object returned by self.find_class(module, class) is pushed on the 1534 stack, so unpickling subclasses can override this form of lookup. 1535 """), 1536 1537 # Ways to build objects of classes pickle doesn't know about directly 1538 # (user-defined classes). I despair of documenting this accurately 1539 # and comprehensibly -- you really have to read the pickle code to 1540 # find all the special cases. 1541 1542 I(name='REDUCE', 1543 code='R', 1544 arg=None, 1545 stack_before=[anyobject, anyobject], 1546 stack_after=[anyobject], 1547 proto=0, 1548 doc="""Push an object built from a callable and an argument tuple. 1549 1550 The opcode is named to remind of the __reduce__() method. 1551 1552 Stack before: ... callable pytuple 1553 Stack after: ... callable(*pytuple) 1554 1555 The callable and the argument tuple are the first two items returned 1556 by a __reduce__ method. Applying the callable to the argtuple is 1557 supposed to reproduce the original object, or at least get it started. 1558 If the __reduce__ method returns a 3-tuple, the last component is an 1559 argument to be passed to the object's __setstate__, and then the REDUCE 1560 opcode is followed by code to create setstate's argument, and then a 1561 BUILD opcode to apply __setstate__ to that argument. 1562 1563 If not isinstance(callable, type), REDUCE complains unless the 1564 callable has been registered with the copyreg module's 1565 safe_constructors dict, or the callable has a magic 1566 '__safe_for_unpickling__' attribute with a true value. I'm not sure 1567 why it does this, but I've sure seen this complaint often enough when 1568 I didn't want to <wink>. 1569 """), 1570 1571 I(name='BUILD', 1572 code='b', 1573 arg=None, 1574 stack_before=[anyobject, anyobject], 1575 stack_after=[anyobject], 1576 proto=0, 1577 doc="""Finish building an object, via __setstate__ or dict update. 1578 1579 Stack before: ... anyobject argument 1580 Stack after: ... anyobject 1581 1582 where anyobject may have been mutated, as follows: 1583 1584 If the object has a __setstate__ method, 1585 1586 anyobject.__setstate__(argument) 1587 1588 is called. 1589 1590 Else the argument must be a dict, the object must have a __dict__, and 1591 the object is updated via 1592 1593 anyobject.__dict__.update(argument) 1594 """), 1595 1596 I(name='INST', 1597 code='i', 1598 arg=stringnl_noescape_pair, 1599 stack_before=[markobject, stackslice], 1600 stack_after=[anyobject], 1601 proto=0, 1602 doc="""Build a class instance. 1603 1604 This is the protocol 0 version of protocol 1's OBJ opcode. 1605 INST is followed by two newline-terminated strings, giving a 1606 module and class name, just as for the GLOBAL opcode (and see 1607 GLOBAL for more details about that). self.find_class(module, name) 1608 is used to get a class object. 1609 1610 In addition, all the objects on the stack following the topmost 1611 markobject are gathered into a tuple and popped (along with the 1612 topmost markobject), just as for the TUPLE opcode. 1613 1614 Now it gets complicated. If all of these are true: 1615 1616 + The argtuple is empty (markobject was at the top of the stack 1617 at the start). 1618 1619 + The class object does not have a __getinitargs__ attribute. 1620 1621 then we want to create an old-style class instance without invoking 1622 its __init__() method (pickle has waffled on this over the years; not 1623 calling __init__() is current wisdom). In this case, an instance of 1624 an old-style dummy class is created, and then we try to rebind its 1625 __class__ attribute to the desired class object. If this succeeds, 1626 the new instance object is pushed on the stack, and we're done. 1627 1628 Else (the argtuple is not empty, it's not an old-style class object, 1629 or the class object does have a __getinitargs__ attribute), the code 1630 first insists that the class object have a __safe_for_unpickling__ 1631 attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, 1632 it doesn't matter whether this attribute has a true or false value, it 1633 only matters whether it exists (XXX this is a bug). If 1634 __safe_for_unpickling__ doesn't exist, UnpicklingError is raised. 1635 1636 Else (the class object does have a __safe_for_unpickling__ attr), 1637 the class object obtained from INST's arguments is applied to the 1638 argtuple obtained from the stack, and the resulting instance object 1639 is pushed on the stack. 1640 1641 NOTE: checks for __safe_for_unpickling__ went away in Python 2.3. 1642 """), 1643 1644 I(name='OBJ', 1645 code='o', 1646 arg=None, 1647 stack_before=[markobject, anyobject, stackslice], 1648 stack_after=[anyobject], 1649 proto=1, 1650 doc="""Build a class instance. 1651 1652 This is the protocol 1 version of protocol 0's INST opcode, and is 1653 very much like it. The major difference is that the class object 1654 is taken off the stack, allowing it to be retrieved from the memo 1655 repeatedly if several instances of the same class are created. This 1656 can be much more efficient (in both time and space) than repeatedly 1657 embedding the module and class names in INST opcodes. 1658 1659 Unlike INST, OBJ takes no arguments from the opcode stream. Instead 1660 the class object is taken off the stack, immediately above the 1661 topmost markobject: 1662 1663 Stack before: ... markobject classobject stackslice 1664 Stack after: ... new_instance_object 1665 1666 As for INST, the remainder of the stack above the markobject is 1667 gathered into an argument tuple, and then the logic seems identical, 1668 except that no __safe_for_unpickling__ check is done (XXX this is 1669 a bug). See INST for the gory details. 1670 1671 NOTE: In Python 2.3, INST and OBJ are identical except for how they 1672 get the class object. That was always the intent; the implementations 1673 had diverged for accidental reasons. 1674 """), 1675 1676 I(name='NEWOBJ', 1677 code='\x81', 1678 arg=None, 1679 stack_before=[anyobject, anyobject], 1680 stack_after=[anyobject], 1681 proto=2, 1682 doc="""Build an object instance. 1683 1684 The stack before should be thought of as containing a class 1685 object followed by an argument tuple (the tuple being the stack 1686 top). Call these cls and args. They are popped off the stack, 1687 and the value returned by cls.__new__(cls, *args) is pushed back 1688 onto the stack. 1689 """), 1690 1691 # Machine control. 1692 1693 I(name='PROTO', 1694 code='\x80', 1695 arg=uint1, 1696 stack_before=[], 1697 stack_after=[], 1698 proto=2, 1699 doc="""Protocol version indicator. 1700 1701 For protocol 2 and above, a pickle must start with this opcode. 1702 The argument is the protocol version, an int in range(2, 256). 1703 """), 1704 1705 I(name='STOP', 1706 code='.', 1707 arg=None, 1708 stack_before=[anyobject], 1709 stack_after=[], 1710 proto=0, 1711 doc="""Stop the unpickling machine. 1712 1713 Every pickle ends with this opcode. The object at the top of the stack 1714 is popped, and that's the result of unpickling. The stack should be 1715 empty then. 1716 """), 1717 1718 # Ways to deal with persistent IDs. 1719 1720 I(name='PERSID', 1721 code='P', 1722 arg=stringnl_noescape, 1723 stack_before=[], 1724 stack_after=[anyobject], 1725 proto=0, 1726 doc="""Push an object identified by a persistent ID. 1727 1728 The pickle module doesn't define what a persistent ID means. PERSID's 1729 argument is a newline-terminated str-style (no embedded escapes, no 1730 bracketing quote characters) string, which *is* "the persistent ID". 1731 The unpickler passes this string to self.persistent_load(). Whatever 1732 object that returns is pushed on the stack. There is no implementation 1733 of persistent_load() in Python's unpickler: it must be supplied by an 1734 unpickler subclass. 1735 """), 1736 1737 I(name='BINPERSID', 1738 code='Q', 1739 arg=None, 1740 stack_before=[anyobject], 1741 stack_after=[anyobject], 1742 proto=1, 1743 doc="""Push an object identified by a persistent ID. 1744 1745 Like PERSID, except the persistent ID is popped off the stack (instead 1746 of being a string embedded in the opcode bytestream). The persistent 1747 ID is passed to self.persistent_load(), and whatever object that 1748 returns is pushed on the stack. See PERSID for more detail. 1749 """), 1750] 1751del I 1752 1753# Verify uniqueness of .name and .code members. 1754name2i = {} 1755code2i = {} 1756 1757for i, d in enumerate(opcodes): 1758 if d.name in name2i: 1759 raise ValueError("repeated name %r at indices %d and %d" % 1760 (d.name, name2i[d.name], i)) 1761 if d.code in code2i: 1762 raise ValueError("repeated code %r at indices %d and %d" % 1763 (d.code, code2i[d.code], i)) 1764 1765 name2i[d.name] = i 1766 code2i[d.code] = i 1767 1768del name2i, code2i, i, d 1769 1770############################################################################## 1771# Build a code2op dict, mapping opcode characters to OpcodeInfo records. 1772# Also ensure we've got the same stuff as pickle.py, although the 1773# introspection here is dicey. 1774 1775code2op = {} 1776for d in opcodes: 1777 code2op[d.code] = d 1778del d 1779 1780def assure_pickle_consistency(verbose=False): 1781 1782 copy = code2op.copy() 1783 for name in pickle.__all__: 1784 if not re.match("[A-Z][A-Z0-9_]+$", name): 1785 if verbose: 1786 print("skipping %r: it doesn't look like an opcode name" % name) 1787 continue 1788 picklecode = getattr(pickle, name) 1789 if not isinstance(picklecode, bytes) or len(picklecode) != 1: 1790 if verbose: 1791 print(("skipping %r: value %r doesn't look like a pickle " 1792 "code" % (name, picklecode))) 1793 continue 1794 picklecode = picklecode.decode("latin-1") 1795 if picklecode in copy: 1796 if verbose: 1797 print("checking name %r w/ code %r for consistency" % ( 1798 name, picklecode)) 1799 d = copy[picklecode] 1800 if d.name != name: 1801 raise ValueError("for pickle code %r, pickle.py uses name %r " 1802 "but we're using name %r" % (picklecode, 1803 name, 1804 d.name)) 1805 # Forget this one. Any left over in copy at the end are a problem 1806 # of a different kind. 1807 del copy[picklecode] 1808 else: 1809 raise ValueError("pickle.py appears to have a pickle opcode with " 1810 "name %r and code %r, but we don't" % 1811 (name, picklecode)) 1812 if copy: 1813 msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"] 1814 for code, d in copy.items(): 1815 msg.append(" name %r with code %r" % (d.name, code)) 1816 raise ValueError("\n".join(msg)) 1817 1818assure_pickle_consistency() 1819del assure_pickle_consistency 1820 1821############################################################################## 1822# A pickle opcode generator. 1823 1824def genops(pickle): 1825 """Generate all the opcodes in a pickle. 1826 1827 'pickle' is a file-like object, or string, containing the pickle. 1828 1829 Each opcode in the pickle is generated, from the current pickle position, 1830 stopping after a STOP opcode is delivered. A triple is generated for 1831 each opcode: 1832 1833 opcode, arg, pos 1834 1835 opcode is an OpcodeInfo record, describing the current opcode. 1836 1837 If the opcode has an argument embedded in the pickle, arg is its decoded 1838 value, as a Python object. If the opcode doesn't have an argument, arg 1839 is None. 1840 1841 If the pickle has a tell() method, pos was the value of pickle.tell() 1842 before reading the current opcode. If the pickle is a bytes object, 1843 it's wrapped in a BytesIO object, and the latter's tell() result is 1844 used. Else (the pickle doesn't have a tell(), and it's not obvious how 1845 to query its current position) pos is None. 1846 """ 1847 1848 if isinstance(pickle, bytes_types): 1849 import io 1850 pickle = io.BytesIO(pickle) 1851 1852 if hasattr(pickle, "tell"): 1853 getpos = pickle.tell 1854 else: 1855 getpos = lambda: None 1856 1857 while True: 1858 pos = getpos() 1859 code = pickle.read(1) 1860 opcode = code2op.get(code.decode("latin-1")) 1861 if opcode is None: 1862 if code == b"": 1863 raise ValueError("pickle exhausted before seeing STOP") 1864 else: 1865 raise ValueError("at position %s, opcode %r unknown" % ( 1866 pos is None and "<unknown>" or pos, 1867 code)) 1868 if opcode.arg is None: 1869 arg = None 1870 else: 1871 arg = opcode.arg.reader(pickle) 1872 yield opcode, arg, pos 1873 if code == b'.': 1874 assert opcode.name == 'STOP' 1875 break 1876 1877############################################################################## 1878# A pickle optimizer. 1879 1880def optimize(p): 1881 'Optimize a pickle string by removing unused PUT opcodes' 1882 gets = set() # set of args used by a GET opcode 1883 puts = [] # (arg, startpos, stoppos) for the PUT opcodes 1884 prevpos = None # set to pos if previous opcode was a PUT 1885 for opcode, arg, pos in genops(p): 1886 if prevpos is not None: 1887 puts.append((prevarg, prevpos, pos)) 1888 prevpos = None 1889 if 'PUT' in opcode.name: 1890 prevarg, prevpos = arg, pos 1891 elif 'GET' in opcode.name: 1892 gets.add(arg) 1893 1894 # Copy the pickle string except for PUTS without a corresponding GET 1895 s = [] 1896 i = 0 1897 for arg, start, stop in puts: 1898 j = stop if (arg in gets) else start 1899 s.append(p[i:j]) 1900 i = stop 1901 s.append(p[i:]) 1902 return b''.join(s) 1903 1904############################################################################## 1905# A symbolic pickle disassembler. 1906 1907def dis(pickle, out=None, memo=None, indentlevel=4, annotate=0): 1908 """Produce a symbolic disassembly of a pickle. 1909 1910 'pickle' is a file-like object, or string, containing a (at least one) 1911 pickle. The pickle is disassembled from the current position, through 1912 the first STOP opcode encountered. 1913 1914 Optional arg 'out' is a file-like object to which the disassembly is 1915 printed. It defaults to sys.stdout. 1916 1917 Optional arg 'memo' is a Python dict, used as the pickle's memo. It 1918 may be mutated by dis(), if the pickle contains PUT or BINPUT opcodes. 1919 Passing the same memo object to another dis() call then allows disassembly 1920 to proceed across multiple pickles that were all created by the same 1921 pickler with the same memo. Ordinarily you don't need to worry about this. 1922 1923 Optional arg 'indentlevel' is the number of blanks by which to indent 1924 a new MARK level. It defaults to 4. 1925 1926 Optional arg 'annotate' if nonzero instructs dis() to add short 1927 description of the opcode on each line of disassembled output. 1928 The value given to 'annotate' must be an integer and is used as a 1929 hint for the column where annotation should start. The default 1930 value is 0, meaning no annotations. 1931 1932 In addition to printing the disassembly, some sanity checks are made: 1933 1934 + All embedded opcode arguments "make sense". 1935 1936 + Explicit and implicit pop operations have enough items on the stack. 1937 1938 + When an opcode implicitly refers to a markobject, a markobject is 1939 actually on the stack. 1940 1941 + A memo entry isn't referenced before it's defined. 1942 1943 + The markobject isn't stored in the memo. 1944 1945 + A memo entry isn't redefined. 1946 """ 1947 1948 # Most of the hair here is for sanity checks, but most of it is needed 1949 # anyway to detect when a protocol 0 POP takes a MARK off the stack 1950 # (which in turn is needed to indent MARK blocks correctly). 1951 1952 stack = [] # crude emulation of unpickler stack 1953 if memo is None: 1954 memo = {} # crude emulation of unpicker memo 1955 maxproto = -1 # max protocol number seen 1956 markstack = [] # bytecode positions of MARK opcodes 1957 indentchunk = ' ' * indentlevel 1958 errormsg = None 1959 annocol = annotate # columnt hint for annotations 1960 for opcode, arg, pos in genops(pickle): 1961 if pos is not None: 1962 print("%5d:" % pos, end=' ', file=out) 1963 1964 line = "%-4s %s%s" % (repr(opcode.code)[1:-1], 1965 indentchunk * len(markstack), 1966 opcode.name) 1967 1968 maxproto = max(maxproto, opcode.proto) 1969 before = opcode.stack_before # don't mutate 1970 after = opcode.stack_after # don't mutate 1971 numtopop = len(before) 1972 1973 # See whether a MARK should be popped. 1974 markmsg = None 1975 if markobject in before or (opcode.name == "POP" and 1976 stack and 1977 stack[-1] is markobject): 1978 assert markobject not in after 1979 if __debug__: 1980 if markobject in before: 1981 assert before[-1] is stackslice 1982 if markstack: 1983 markpos = markstack.pop() 1984 if markpos is None: 1985 markmsg = "(MARK at unknown opcode offset)" 1986 else: 1987 markmsg = "(MARK at %d)" % markpos 1988 # Pop everything at and after the topmost markobject. 1989 while stack[-1] is not markobject: 1990 stack.pop() 1991 stack.pop() 1992 # Stop later code from popping too much. 1993 try: 1994 numtopop = before.index(markobject) 1995 except ValueError: 1996 assert opcode.name == "POP" 1997 numtopop = 0 1998 else: 1999 errormsg = markmsg = "no MARK exists on stack" 2000 2001 # Check for correct memo usage. 2002 if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"): 2003 assert arg is not None 2004 if arg in memo: 2005 errormsg = "memo key %r already defined" % arg 2006 elif not stack: 2007 errormsg = "stack is empty -- can't store into memo" 2008 elif stack[-1] is markobject: 2009 errormsg = "can't store markobject in the memo" 2010 else: 2011 memo[arg] = stack[-1] 2012 2013 elif opcode.name in ("GET", "BINGET", "LONG_BINGET"): 2014 if arg in memo: 2015 assert len(after) == 1 2016 after = [memo[arg]] # for better stack emulation 2017 else: 2018 errormsg = "memo key %r has never been stored into" % arg 2019 2020 if arg is not None or markmsg: 2021 # make a mild effort to align arguments 2022 line += ' ' * (10 - len(opcode.name)) 2023 if arg is not None: 2024 line += ' ' + repr(arg) 2025 if markmsg: 2026 line += ' ' + markmsg 2027 if annotate: 2028 line += ' ' * (annocol - len(line)) 2029 # make a mild effort to align annotations 2030 annocol = len(line) 2031 if annocol > 50: 2032 annocol = annotate 2033 line += ' ' + opcode.doc.split('\n', 1)[0] 2034 print(line, file=out) 2035 2036 if errormsg: 2037 # Note that we delayed complaining until the offending opcode 2038 # was printed. 2039 raise ValueError(errormsg) 2040 2041 # Emulate the stack effects. 2042 if len(stack) < numtopop: 2043 raise ValueError("tries to pop %d items from stack with " 2044 "only %d items" % (numtopop, len(stack))) 2045 if numtopop: 2046 del stack[-numtopop:] 2047 if markobject in after: 2048 assert markobject not in before 2049 markstack.append(pos) 2050 2051 stack.extend(after) 2052 2053 print("highest protocol among opcodes =", maxproto, file=out) 2054 if stack: 2055 raise ValueError("stack not empty after STOP: %r" % stack) 2056 2057# For use in the doctest, simply as an example of a class to pickle. 2058class _Example: 2059 def __init__(self, value): 2060 self.value = value 2061 2062_dis_test = r""" 2063>>> import pickle 2064>>> x = [1, 2, (3, 4), {b'abc': "def"}] 2065>>> pkl0 = pickle.dumps(x, 0) 2066>>> dis(pkl0) 2067 0: ( MARK 2068 1: l LIST (MARK at 0) 2069 2: p PUT 0 2070 5: L LONG 1 2071 9: a APPEND 2072 10: L LONG 2 2073 14: a APPEND 2074 15: ( MARK 2075 16: L LONG 3 2076 20: L LONG 4 2077 24: t TUPLE (MARK at 15) 2078 25: p PUT 1 2079 28: a APPEND 2080 29: ( MARK 2081 30: d DICT (MARK at 29) 2082 31: p PUT 2 2083 34: c GLOBAL '__builtin__ bytes' 2084 53: p PUT 3 2085 56: ( MARK 2086 57: ( MARK 2087 58: l LIST (MARK at 57) 2088 59: p PUT 4 2089 62: L LONG 97 2090 67: a APPEND 2091 68: L LONG 98 2092 73: a APPEND 2093 74: L LONG 99 2094 79: a APPEND 2095 80: t TUPLE (MARK at 56) 2096 81: p PUT 5 2097 84: R REDUCE 2098 85: p PUT 6 2099 88: V UNICODE 'def' 2100 93: p PUT 7 2101 96: s SETITEM 2102 97: a APPEND 2103 98: . STOP 2104highest protocol among opcodes = 0 2105 2106Try again with a "binary" pickle. 2107 2108>>> pkl1 = pickle.dumps(x, 1) 2109>>> dis(pkl1) 2110 0: ] EMPTY_LIST 2111 1: q BINPUT 0 2112 3: ( MARK 2113 4: K BININT1 1 2114 6: K BININT1 2 2115 8: ( MARK 2116 9: K BININT1 3 2117 11: K BININT1 4 2118 13: t TUPLE (MARK at 8) 2119 14: q BINPUT 1 2120 16: } EMPTY_DICT 2121 17: q BINPUT 2 2122 19: c GLOBAL '__builtin__ bytes' 2123 38: q BINPUT 3 2124 40: ( MARK 2125 41: ] EMPTY_LIST 2126 42: q BINPUT 4 2127 44: ( MARK 2128 45: K BININT1 97 2129 47: K BININT1 98 2130 49: K BININT1 99 2131 51: e APPENDS (MARK at 44) 2132 52: t TUPLE (MARK at 40) 2133 53: q BINPUT 5 2134 55: R REDUCE 2135 56: q BINPUT 6 2136 58: X BINUNICODE 'def' 2137 66: q BINPUT 7 2138 68: s SETITEM 2139 69: e APPENDS (MARK at 3) 2140 70: . STOP 2141highest protocol among opcodes = 1 2142 2143Exercise the INST/OBJ/BUILD family. 2144 2145>>> import pickletools 2146>>> dis(pickle.dumps(pickletools.dis, 0)) 2147 0: c GLOBAL 'pickletools dis' 2148 17: p PUT 0 2149 20: . STOP 2150highest protocol among opcodes = 0 2151 2152>>> from pickletools import _Example 2153>>> x = [_Example(42)] * 2 2154>>> dis(pickle.dumps(x, 0)) 2155 0: ( MARK 2156 1: l LIST (MARK at 0) 2157 2: p PUT 0 2158 5: c GLOBAL 'copy_reg _reconstructor' 2159 30: p PUT 1 2160 33: ( MARK 2161 34: c GLOBAL 'pickletools _Example' 2162 56: p PUT 2 2163 59: c GLOBAL '__builtin__ object' 2164 79: p PUT 3 2165 82: N NONE 2166 83: t TUPLE (MARK at 33) 2167 84: p PUT 4 2168 87: R REDUCE 2169 88: p PUT 5 2170 91: ( MARK 2171 92: d DICT (MARK at 91) 2172 93: p PUT 6 2173 96: V UNICODE 'value' 2174 103: p PUT 7 2175 106: L LONG 42 2176 111: s SETITEM 2177 112: b BUILD 2178 113: a APPEND 2179 114: g GET 5 2180 117: a APPEND 2181 118: . STOP 2182highest protocol among opcodes = 0 2183 2184>>> dis(pickle.dumps(x, 1)) 2185 0: ] EMPTY_LIST 2186 1: q BINPUT 0 2187 3: ( MARK 2188 4: c GLOBAL 'copy_reg _reconstructor' 2189 29: q BINPUT 1 2190 31: ( MARK 2191 32: c GLOBAL 'pickletools _Example' 2192 54: q BINPUT 2 2193 56: c GLOBAL '__builtin__ object' 2194 76: q BINPUT 3 2195 78: N NONE 2196 79: t TUPLE (MARK at 31) 2197 80: q BINPUT 4 2198 82: R REDUCE 2199 83: q BINPUT 5 2200 85: } EMPTY_DICT 2201 86: q BINPUT 6 2202 88: X BINUNICODE 'value' 2203 98: q BINPUT 7 2204 100: K BININT1 42 2205 102: s SETITEM 2206 103: b BUILD 2207 104: h BINGET 5 2208 106: e APPENDS (MARK at 3) 2209 107: . STOP 2210highest protocol among opcodes = 1 2211 2212Try "the canonical" recursive-object test. 2213 2214>>> L = [] 2215>>> T = L, 2216>>> L.append(T) 2217>>> L[0] is T 2218True 2219>>> T[0] is L 2220True 2221>>> L[0][0] is L 2222True 2223>>> T[0][0] is T 2224True 2225>>> dis(pickle.dumps(L, 0)) 2226 0: ( MARK 2227 1: l LIST (MARK at 0) 2228 2: p PUT 0 2229 5: ( MARK 2230 6: g GET 0 2231 9: t TUPLE (MARK at 5) 2232 10: p PUT 1 2233 13: a APPEND 2234 14: . STOP 2235highest protocol among opcodes = 0 2236 2237>>> dis(pickle.dumps(L, 1)) 2238 0: ] EMPTY_LIST 2239 1: q BINPUT 0 2240 3: ( MARK 2241 4: h BINGET 0 2242 6: t TUPLE (MARK at 3) 2243 7: q BINPUT 1 2244 9: a APPEND 2245 10: . STOP 2246highest protocol among opcodes = 1 2247 2248Note that, in the protocol 0 pickle of the recursive tuple, the disassembler 2249has to emulate the stack in order to realize that the POP opcode at 16 gets 2250rid of the MARK at 0. 2251 2252>>> dis(pickle.dumps(T, 0)) 2253 0: ( MARK 2254 1: ( MARK 2255 2: l LIST (MARK at 1) 2256 3: p PUT 0 2257 6: ( MARK 2258 7: g GET 0 2259 10: t TUPLE (MARK at 6) 2260 11: p PUT 1 2261 14: a APPEND 2262 15: 0 POP 2263 16: 0 POP (MARK at 0) 2264 17: g GET 1 2265 20: . STOP 2266highest protocol among opcodes = 0 2267 2268>>> dis(pickle.dumps(T, 1)) 2269 0: ( MARK 2270 1: ] EMPTY_LIST 2271 2: q BINPUT 0 2272 4: ( MARK 2273 5: h BINGET 0 2274 7: t TUPLE (MARK at 4) 2275 8: q BINPUT 1 2276 10: a APPEND 2277 11: 1 POP_MARK (MARK at 0) 2278 12: h BINGET 1 2279 14: . STOP 2280highest protocol among opcodes = 1 2281 2282Try protocol 2. 2283 2284>>> dis(pickle.dumps(L, 2)) 2285 0: \x80 PROTO 2 2286 2: ] EMPTY_LIST 2287 3: q BINPUT 0 2288 5: h BINGET 0 2289 7: \x85 TUPLE1 2290 8: q BINPUT 1 2291 10: a APPEND 2292 11: . STOP 2293highest protocol among opcodes = 2 2294 2295>>> dis(pickle.dumps(T, 2)) 2296 0: \x80 PROTO 2 2297 2: ] EMPTY_LIST 2298 3: q BINPUT 0 2299 5: h BINGET 0 2300 7: \x85 TUPLE1 2301 8: q BINPUT 1 2302 10: a APPEND 2303 11: 0 POP 2304 12: h BINGET 1 2305 14: . STOP 2306highest protocol among opcodes = 2 2307 2308Try protocol 3 with annotations: 2309 2310>>> dis(pickle.dumps(T, 3), annotate=1) 2311 0: \x80 PROTO 3 Protocol version indicator. 2312 2: ] EMPTY_LIST Push an empty list. 2313 3: q BINPUT 0 Store the stack top into the memo. The stack is not popped. 2314 5: h BINGET 0 Read an object from the memo and push it on the stack. 2315 7: \x85 TUPLE1 Build a one-tuple out of the topmost item on the stack. 2316 8: q BINPUT 1 Store the stack top into the memo. The stack is not popped. 2317 10: a APPEND Append an object to a list. 2318 11: 0 POP Discard the top stack item, shrinking the stack by one item. 2319 12: h BINGET 1 Read an object from the memo and push it on the stack. 2320 14: . STOP Stop the unpickling machine. 2321highest protocol among opcodes = 2 2322 2323""" 2324 2325_memo_test = r""" 2326>>> import pickle 2327>>> import io 2328>>> f = io.BytesIO() 2329>>> p = pickle.Pickler(f, 2) 2330>>> x = [1, 2, 3] 2331>>> p.dump(x) 2332>>> p.dump(x) 2333>>> f.seek(0) 23340 2335>>> memo = {} 2336>>> dis(f, memo=memo) 2337 0: \x80 PROTO 2 2338 2: ] EMPTY_LIST 2339 3: q BINPUT 0 2340 5: ( MARK 2341 6: K BININT1 1 2342 8: K BININT1 2 2343 10: K BININT1 3 2344 12: e APPENDS (MARK at 5) 2345 13: . STOP 2346highest protocol among opcodes = 2 2347>>> dis(f, memo=memo) 2348 14: \x80 PROTO 2 2349 16: h BINGET 0 2350 18: . STOP 2351highest protocol among opcodes = 2 2352""" 2353 2354__test__ = {'disassembler_test': _dis_test, 2355 'disassembler_memo_test': _memo_test, 2356 } 2357 2358def _test(): 2359 import doctest 2360 return doctest.testmod() 2361 2362if __name__ == "__main__": 2363 import sys, argparse 2364 parser = argparse.ArgumentParser( 2365 description='disassemble one or more pickle files') 2366 parser.add_argument( 2367 'pickle_file', type=argparse.FileType('br'), 2368 nargs='*', help='the pickle file') 2369 parser.add_argument( 2370 '-o', '--output', default=sys.stdout, type=argparse.FileType('w'), 2371 help='the file where the output should be written') 2372 parser.add_argument( 2373 '-m', '--memo', action='store_true', 2374 help='preserve memo between disassemblies') 2375 parser.add_argument( 2376 '-l', '--indentlevel', default=4, type=int, 2377 help='the number of blanks by which to indent a new MARK level') 2378 parser.add_argument( 2379 '-a', '--annotate', action='store_true', 2380 help='annotate each line with a short opcode description') 2381 parser.add_argument( 2382 '-p', '--preamble', default="==> {name} <==", 2383 help='if more than one pickle file is specified, print this before' 2384 ' each disassembly') 2385 parser.add_argument( 2386 '-t', '--test', action='store_true', 2387 help='run self-test suite') 2388 parser.add_argument( 2389 '-v', action='store_true', 2390 help='run verbosely; only affects self-test run') 2391 args = parser.parse_args() 2392 if args.test: 2393 _test() 2394 else: 2395 annotate = 30 if args.annotate else 0 2396 if not args.pickle_file: 2397 parser.print_help() 2398 elif len(args.pickle_file) == 1: 2399 dis(args.pickle_file[0], args.output, None, 2400 args.indentlevel, annotate) 2401 else: 2402 memo = {} if args.memo else None 2403 for f in args.pickle_file: 2404 preamble = args.preamble.format(name=f.name) 2405 args.output.write(preamble + '\n') 2406 dis(f, args.output, memo, args.indentlevel, annotate) 2407