pickletools.py revision 43277d64eb5e008e59009bd03887d66c134445ef
15445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanaro'''"Executable documentation" for the pickle module. 28ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 38ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersExtensive comments about the pickle protocols and pickle-machine opcodes 48ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscan be found here. Some functions meant for external use: 58ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 68ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersgenops(pickle) 78ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Generate all the opcodes in a pickle, as (opcode, arg, position) triples. 88ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 98ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdis(pickle, out=None, indentlevel=4) 108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Print a symbolic disassembly of a pickle. 115445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanaro''' 128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Other ideas: 148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# 158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# - A pickle verifier: read a pickle and check it exhaustively for 16c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters# well-formedness. dis() does a lot of this already. 178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# 188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# - A protocol identifier: examine a pickle and return its protocol number 198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# (== the highest .proto attr value among all the opcodes in the pickle). 20c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters# dis() already prints this info at the end. 218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# 228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# - A pickle optimizer: for example, tuple-building code is sometimes more 238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# elaborate than necessary, catering for the possibility that the tuple 248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# is recursive. Or lots of times a PUT is generated that's never accessed 258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# by a later GET. 268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters""" 298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters"A pickle" is a program for a virtual pickle machine (PM, but more accurately 308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscalled an unpickling machine). It's a sequence of opcodes, interpreted by the 318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersPM, building an arbitrarily complex Python object. 328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersFor the most part, the PM is very simple: there are no looping, testing, or 348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersconditional instructions, no arithmetic and no function calls. Opcodes are 358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersexecuted once each, from first to last, until a STOP opcode is reached. 368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe PM has two data areas, "the stack" and "the memo". 388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersMany opcodes push Python objects onto the stack; e.g., INT pushes a Python 408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersinteger object on the stack, whose value is gotten from a decimal string 418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersliteral immediately following the INT opcode in the pickle bytestream. Other 428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersopcodes take Python objects off the stack. The result of unpickling is 438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterswhatever object is left on the stack when the final STOP opcode is executed. 448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe memo is simply an array of objects, or it can be implemented as a dict 468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersmapping little integers to objects. The memo serves as the PM's "long term 478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersmemory", and the little integers indexing the memo are akin to variable 488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersnames. Some opcodes pop a stack object into the memo at a given index, 498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersand others push a memo object at a given index onto the stack again. 508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersAt heart, that's all the PM has. Subtleties arise for these reasons: 528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Object identity. Objects can be arbitrarily complex, and subobjects 548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters may be shared (for example, the list [a, a] refers to the same object a 558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters twice). It can be vital that unpickling recreate an isomorphic object 568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters graph, faithfully reproducing sharing. 578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Recursive objects. For example, after "L = []; L.append(L)", L is a 598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters list, and L[0] is the same list. This is related to the object identity 608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters point, and some sequences of pickle opcodes are subtle in order to 618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters get the right result in all cases. 628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Things pickle doesn't know everything about. Examples of things pickle 648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters does know everything about are Python's builtin scalar and container 658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters types, like ints and tuples. They generally have opcodes dedicated to 668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters them. For things like module references and instances of user-defined 678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters classes, pickle's knowledge is limited. Historically, many enhancements 688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters have been made to the pickle protocol in order to do a better (faster, 698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and/or more compact) job on those. 708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Backward compatibility and micro-optimization. As explained below, 728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pickle opcodes never go away, not even when better ways to do a thing 738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters get invented. The repertoire of the PM just keeps growing over time. 74fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters For example, protocol 0 had two opcodes for building Python integers (INT 75fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters and LONG), protocol 1 added three more for more-efficient pickling of short 76fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters integers, and protocol 2 added two more for more-efficient pickling of 77fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters long integers (before protocol 2, the only ways to pickle a Python long 78fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters took time quadratic in the number of digits, for both pickling and 79fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters unpickling). "Opcode bloat" isn't so much a subtlety as a source of 808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters wearying complication. 818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersPickle protocols: 848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersFor compatibility, the meaning of a pickle opcode never changes. Instead new 868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspickle opcodes get added, and each version's unpickler can handle all the 878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspickle opcodes in all protocol versions to date. So old pickles continue to 888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersbe readable forever. The pickler can generally be told to restrict itself to 898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersthe subset of opcodes available under previous protocol versions too, so that 908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersusers can create pickles under the current version readable by older 918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersversions. However, a pickle does not contain its version number embedded 928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterswithin it. If an older unpickler tries to read a pickle using a later 938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersprotocol, the result is most likely an exception due to seeing an unknown (in 948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersthe older unpickler) opcode. 958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe original pickle used what's now called "protocol 0", and what was called 978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters"text mode" before Python 2.3. The entire pickle bytestream is made up of 988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersprintable 7-bit ASCII characters, plus the newline character, in protocol 0. 99fdc03462b3e0796ae6474da6f0f9844773d1da8fTim PetersThat's why it was called text mode. Protocol 0 is small and elegant, but 100fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peterssometimes painfully inefficient. 1018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe second major set of additions is now called "protocol 1", and was called 1038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters"binary mode" before Python 2.3. This added many opcodes with arguments 1048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersconsisting of arbitrary bytes, including NUL bytes and unprintable "high bit" 1058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersbytes. Binary mode pickles can be substantially smaller than equivalent 1068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterstext mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte 1078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersint as 4 bytes following the opcode, which is cheaper to unpickle than the 108fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters(perhaps) 11-character decimal string attached to INT. Protocol 1 also added 109fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Petersa number of opcodes that operate on many stack elements at once (like APPENDS 11081098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Petersand SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE). 1118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe third major set of additions came in Python 2.3, and is called "protocol 113fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters2". This added: 114fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 115fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- A better way to pickle instances of new-style classes (NEWOBJ). 116fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 117fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- A way for a pickle to identify its protocol (PROTO). 118fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 119fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- Time- and space- efficient pickling of long ints (LONG{1,4}). 120fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 121fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- Shortcuts for small tuples (TUPLE{1,2,3}}. 122fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 123fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- Dedicated opcodes for bools (NEWTRUE, NEWFALSE). 124fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 125fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- The "extension registry", a vector of popular objects that can be pushed 126fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but 127fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters the registry contents are predefined (there's nothing akin to the memo's 128fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters PUT). 129ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum 1305445594e20baf1ac0c0c1aef30d55d789c4b9694Skip MontanaroAnother independent change with Python 2.3 is the abandonment of any 1315445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanaropretense that it might be safe to load pickles received from untrusted 132ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumparties -- no sufficient security analysis has been done to guarantee 1335445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanarothis and there isn't a use case that warrants the expense of such an 134ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumanalysis. 135ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum 136ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van RossumTo this end, all tests for __safe_for_unpickling__ or for 137ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumcopy_reg.safe_constructors are removed from the unpickling code. 138ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van RossumReferences to these variables in the descriptions below are to be seen 139ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumas describing unpickling in Python 2.2 and before. 1408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters""" 1418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Meta-rule: Descriptions are stored in instances of descriptor objects, 1438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# with plain constructors. No meta-language is defined from which 1448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# descriptors could be constructed. If you want, e.g., XML, write a little 1458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# program to generate XML from the objects. 1468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 1488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Some pickle opcodes have an argument, following the opcode in the 1498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# bytestream. An argument is of a specific type, described by an instance 1508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# of ArgumentDescriptor. These are not to be confused with arguments taken 1518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# off the stack -- ArgumentDescriptor applies only to arguments embedded in 1528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# the opcode stream, immediately following an opcode. 1538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Represents the number of bytes consumed by an argument delimited by the 1558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# next newline character. 1568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersUP_TO_NEWLINE = -1 1578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Represents the number of bytes consumed by a two-argument opcode where 1598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# the first argument gives the number of bytes in the second argument. 160fdb8cfab085d0f412411b090796e9f856ee0cab5Tim PetersTAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int 161fdb8cfab085d0f412411b090796e9f856ee0cab5Tim PetersTAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int 1628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersclass ArgumentDescriptor(object): 1648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __slots__ = ( 1658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # name of descriptor record, also a module global name; a string 1668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'name', 1678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # length of argument, in bytes; an int; UP_TO_NEWLINE and 169fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length 170fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters # cases 1718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'n', 1728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # a function taking a file-like object, reading this kind of argument 1748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # from the object at the current position, advancing the current 1758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # position by n bytes, and returning the value of the argument 1768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'reader', 1778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # human-readable docs for this arg descriptor; a string 1798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'doc', 1808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ) 1818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters def __init__(self, name, n, reader, doc): 1838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(name, str) 1848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.name = name 1858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(n, int) and (n >= 0 or 187fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n in (UP_TO_NEWLINE, 188fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters TAKEN_FROM_ARGUMENT1, 189fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters TAKEN_FROM_ARGUMENT4)) 1908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.n = n 1918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.reader = reader 1938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(doc, str) 1958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.doc = doc 1968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfrom struct import unpack as _unpack 1988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_uint1(f): 20055762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 20255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_uint1(StringIO.StringIO('\xff')) 2038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 255 2048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(1) 2078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if data: 2088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return ord(data) 2098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read uint1") 2108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersuint1 = ArgumentDescriptor( 2128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='uint1', 2138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=1, 2148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_uint1, 2158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="One-byte unsigned integer.") 2168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_uint2(f): 21955762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 22155762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_uint2(StringIO.StringIO('\xff\x00')) 2228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 255 22355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_uint2(StringIO.StringIO('\xff\xff')) 2248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 65535 2258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(2) 2288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == 2: 2298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return _unpack("<H", data)[0] 2308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read uint2") 2318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersuint2 = ArgumentDescriptor( 2338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='uint2', 2348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=2, 2358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_uint2, 2368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Two-byte unsigned integer, little-endian.") 2378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_int4(f): 24055762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 24255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00')) 2438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 255 24455762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31) 2458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters True 2468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(4) 2498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == 4: 2508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return _unpack("<i", data)[0] 2518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read int4") 2528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersint4 = ArgumentDescriptor( 2548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='int4', 2558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=4, 2568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_int4, 2578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Four-byte signed integer, little-endian, 2's complement.") 2588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_stringnl(f, decode=True, stripquotes=True): 26155762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 26355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n")) 2648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'abcd' 2658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 26655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("\n")) 2678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 2688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 2698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: no string quotes around '' 2708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 27155762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False) 2728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 2738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 27455762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("''\n")) 2758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 2768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> read_stringnl(StringIO.StringIO('"abcd"')) 2788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 2798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 2808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: no newline found when trying to read stringnl 2818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Embedded escapes are undone in the result. 28355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'")) 28455762f5f804c4848bbce323b085101d450f89ff6Tim Peters 'a\n\\b\x00c\td' 2858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.readline() 2888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not data.endswith('\n'): 2898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("no newline found when trying to read stringnl") 2908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data[:-1] # lose the newline 2918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if stripquotes: 2938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for q in "'\"": 2948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if data.startswith(q): 2958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not data.endswith(q): 2968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("strinq quote %r not found at both " 2978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "ends of %r" % (q, data)) 2988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data[1:-1] 2998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters break 3008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 3018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("no string quotes around %r" % data) 3028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # I'm not sure when 'string_escape' was added to the std codecs; it's 3048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # crazy not to use it if it's there. 3058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if decode: 3068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data.decode('string_escape') 3078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return data 3088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstringnl = ArgumentDescriptor( 3108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='stringnl', 3118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 3128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_stringnl, 3138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated string. 3148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a repr-style string, with embedded escapes, and 3168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bracketing quotes. 3178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_stringnl_noescape(f): 3208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return read_stringnl(f, decode=False, stripquotes=False) 3218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstringnl_noescape = ArgumentDescriptor( 3238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='stringnl_noescape', 3248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 3258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_stringnl_noescape, 3268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated string. 3278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a str-style string, without embedded escapes, 3298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters or bracketing quotes. It should consist solely of 3308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters printable ASCII characters. 3318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_stringnl_noescape_pair(f): 33455762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 3358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 33655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk")) 337d916cf4ec7014e9f6b25b8d63728bda01a17d3f9Tim Peters 'Queue Empty' 3388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 3398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 340d916cf4ec7014e9f6b25b8d63728bda01a17d3f9Tim Peters return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) 3418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstringnl_noescape_pair = ArgumentDescriptor( 3438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='stringnl_noescape_pair', 3448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 3458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_stringnl_noescape_pair, 3468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A pair of newline-terminated strings. 3478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters These are str-style strings, without embedded 3498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters escapes, or bracketing quotes. They should 3508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters consist solely of printable ASCII characters. 3518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The pair is returned as a single string, with 352d916cf4ec7014e9f6b25b8d63728bda01a17d3f9Tim Peters a single blank separating the two strings. 3538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_string4(f): 35655762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 3578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 35855762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc")) 3598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 36055762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef")) 3618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'abc' 36255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef")) 3638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 3648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 3658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: expected 50331648 bytes in a string4, but only 6 remain 3668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 3678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n = read_int4(f) 3698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if n < 0: 3708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("string4 byte count < 0: %d" % n) 3718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(n) 3728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == n: 3738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return data 3748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("expected %d bytes in a string4, but only %d remain" % 3758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (n, len(data))) 3768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstring4 = ArgumentDescriptor( 3788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="string4", 379fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT4, 3808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_string4, 3818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A counted string. 3828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The first argument is a 4-byte little-endian signed int giving 3848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the number of bytes in the string, and the second argument is 3858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters that many bytes. 3868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_string1(f): 39055762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 3918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 39255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string1(StringIO.StringIO("\x00")) 3938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 39455762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string1(StringIO.StringIO("\x03abcdef")) 3958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'abc' 3968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 3978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n = read_uint1(f) 3998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert n >= 0 4008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(n) 4018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == n: 4028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return data 4038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("expected %d bytes in a string1, but only %d remain" % 4048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (n, len(data))) 4058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstring1 = ArgumentDescriptor( 4078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="string1", 408fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT1, 4098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_string1, 4108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A counted string. 4118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The first argument is a 1-byte unsigned int giving the number 4138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of bytes in the string, and the second argument is that many 4148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bytes. 4158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 4168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_unicodestringnl(f): 41955762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 4208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 42155762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk")) 42255762f5f804c4848bbce323b085101d450f89ff6Tim Peters u'abc\uabcd' 4238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 4248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.readline() 4268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not data.endswith('\n'): 4278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("no newline found when trying to read " 4288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "unicodestringnl") 4298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data[:-1] # lose the newline 4308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return unicode(data, 'raw-unicode-escape') 4318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersunicodestringnl = ArgumentDescriptor( 4338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='unicodestringnl', 4348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 4358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_unicodestringnl, 4368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated Unicode string. 4378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is raw-unicode-escape encoded, so consists of 4398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters printable ASCII characters, and may contain embedded 4408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters escape sequences. 4418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 4428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_unicodestring4(f): 44455762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 4458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 44655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> s = u'abcd\uabcd' 4478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> enc = s.encode('utf-8') 4488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> enc 44955762f5f804c4848bbce323b085101d450f89ff6Tim Peters 'abcd\xea\xaf\x8d' 4508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> n = chr(len(enc)) + chr(0) * 3 # little-endian 4-byte length 4518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk')) 4528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> s == t 4538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters True 4548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1])) 4568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 4578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 4588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: expected 7 bytes in a unicodestring4, but only 6 remain 4598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 4608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n = read_int4(f) 4628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if n < 0: 4638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("unicodestring4 byte count < 0: %d" % n) 4648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(n) 4658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == n: 4668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return unicode(data, 'utf-8') 4678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("expected %d bytes in a unicodestring4, but only %d " 4688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "remain" % (n, len(data))) 4698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersunicodestring4 = ArgumentDescriptor( 4718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="unicodestring4", 472fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT4, 4738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_unicodestring4, 4748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A counted Unicode string. 4758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The first argument is a 4-byte little-endian signed int 4778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters giving the number of bytes in the string, and the second 4788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argument-- the UTF-8 encoding of the Unicode string -- 4798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters contains that many bytes. 4808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 4818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_decimalnl_short(f): 48455762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 4858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 48655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_short(StringIO.StringIO("1234\n56")) 4878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1234 4888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 48955762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_short(StringIO.StringIO("1234L\n56")) 4908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 4918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 4928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: trailing 'L' not allowed in '1234L' 4938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 4948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters s = read_stringnl(f, decode=False, stripquotes=False) 4968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if s.endswith("L"): 4978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("trailing 'L' not allowed in %r" % s) 4988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # It's not necessarily true that the result fits in a Python short int: 5008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the pickle may have been written on a 64-bit box. There's also a hack 5018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # for True and False here. 5028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if s == "00": 5038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return False 5048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters elif s == "01": 5058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return True 5068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters try: 5088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return int(s) 5098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters except OverflowError: 5108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return long(s) 5118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_decimalnl_long(f): 51355762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 5148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 5158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 51655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_long(StringIO.StringIO("1234\n56")) 5178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 5188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 5198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: trailing 'L' required in '1234' 5208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Someday the trailing 'L' will probably go away from this output. 5228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 52355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_long(StringIO.StringIO("1234L\n56")) 5248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1234L 5258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 52655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\n6")) 5278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 123456789012345678901234L 5288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 5298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters s = read_stringnl(f, decode=False, stripquotes=False) 5318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not s.endswith("L"): 5328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("trailing 'L' required in %r" % s) 5338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return long(s) 5348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdecimalnl_short = ArgumentDescriptor( 5378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='decimalnl_short', 5388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 5398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_decimalnl_short, 5408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated decimal integer literal. 5418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This never has a trailing 'L', and the integer fit 5438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters in a short Python int on the box where the pickle 5448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters was written -- but there's no guarantee it will fit 5458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters in a short Python int on the box where the pickle 5468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is read. 5478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 5488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdecimalnl_long = ArgumentDescriptor( 5508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='decimalnl_long', 5518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 5528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_decimalnl_long, 5538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated decimal integer literal. 5548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This has a trailing 'L', and can represent integers 5568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of any size. 5578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 5588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_floatnl(f): 56155762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 5628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 56355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_floatnl(StringIO.StringIO("-1.25\n6")) 5648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters -1.25 5658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 5668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters s = read_stringnl(f, decode=False, stripquotes=False) 5678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return float(s) 5688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfloatnl = ArgumentDescriptor( 5708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='floatnl', 5718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 5728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_floatnl, 5738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated decimal floating literal. 5748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In general this requires 17 significant digits for roundtrip 5768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters identity, and pickling then unpickling infinities, NaNs, and 5778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters minus zero doesn't work across boxes, or on some boxes even 5788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters on itself (e.g., Windows can't read the strings it produces 5798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for infinities or NaNs). 5808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 5818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_float8(f): 58355762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 5848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO, struct 5858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> raw = struct.pack(">d", -1.25) 5868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> raw 58755762f5f804c4848bbce323b085101d450f89ff6Tim Peters '\xbf\xf4\x00\x00\x00\x00\x00\x00' 58855762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_float8(StringIO.StringIO(raw + "\n")) 5898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters -1.25 5908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 5918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(8) 5938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == 8: 5948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return _unpack(">d", data)[0] 5958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read float8") 5968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfloat8 = ArgumentDescriptor( 5998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='float8', 6008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=8, 6018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_float8, 6028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""An 8-byte binary representation of a float, big-endian. 6038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The format is unique to Python, and shared with the struct 6058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters module (format string '>d') "in theory" (the struct and cPickle 6068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters implementations don't share the code -- they should). It's 6078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters strongly related to the IEEE-754 double format, and, in normal 6088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters cases, is in fact identical to the big-endian 754 double format. 6098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters On other boxes the dynamic range is limited to that of a 754 6108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters double, and "add a half and chop" rounding is used to reduce 6118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the precision to 53 bits. However, even on a 754 box, 6128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters infinities, NaNs, and minus zero may not be handled correctly 6138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (may not survive roundtrip pickling intact). 6148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 6158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6165a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum# Protocol 2 formats 6175a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 618c0c12b57070a5b494662bebc418e3958bf5bdbeeTim Petersfrom pickle import decode_long 6195a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6205a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumdef read_long1(f): 6215a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum r""" 6225a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> import StringIO 6235a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\xff\x00")) 6245a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 255L 6255a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\xff\x7f")) 6265a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 32767L 6275a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\x00\xff")) 6285a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -256L 6295a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\x00\x80")) 6305a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -32768L 6315eed340267868c54775b49613b81dff7eb2461bbTim Peters >>> 6325a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """ 6335a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6345a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum n = read_uint1(f) 6355a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum data = f.read(n) 6365a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum if len(data) != n: 6375a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum raise ValueError("not enough data in stream to read long1") 6385a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum return decode_long(data) 6395a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6405a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumlong1 = ArgumentDescriptor( 6415a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum name="long1", 642fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT1, 6435a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum reader=read_long1, 6445a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum doc="""A binary long, little-endian, using 1-byte size. 6455a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6465a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum This first reads one byte as an unsigned size, then reads that 647bdbe74183ceb7aedd71dd1afdf8755ee93e8c3caTim Peters many bytes and interprets them as a little-endian 2's-complement long. 6485a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """) 6495a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6505a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumdef read_long4(f): 6515a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum r""" 6525a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> import StringIO 6535a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x00")) 6545a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 255L 6555a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x7f")) 6565a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 32767L 6575a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\xff")) 6585a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -256L 6595a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\x80")) 6605a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -32768L 6615eed340267868c54775b49613b81dff7eb2461bbTim Peters >>> 6625a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """ 6635a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6645a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum n = read_int4(f) 6655a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum if n < 0: 666784a3f53a8f78995d4c8ca22f612a68828bc4838Neal Norwitz raise ValueError("long4 byte count < 0: %d" % n) 6675a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum data = f.read(n) 6685a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum if len(data) != n: 669784a3f53a8f78995d4c8ca22f612a68828bc4838Neal Norwitz raise ValueError("not enough data in stream to read long4") 6705a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum return decode_long(data) 6715a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6725a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumlong4 = ArgumentDescriptor( 6735a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum name="long4", 674fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT4, 6755a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum reader=read_long4, 6765a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum doc="""A binary representation of a long, little-endian. 6775a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6785a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum This first reads four bytes as a signed size (but requires the 6795a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum size to be >= 0), then reads that many bytes and interprets them 680bdbe74183ceb7aedd71dd1afdf8755ee93e8c3caTim Peters as a little-endian 2's-complement long. 6815a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """) 6825a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6835a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 6858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Object descriptors. The stack used by the pickle machine holds objects, 6868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# and in the stack_before and stack_after attributes of OpcodeInfo 6878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# descriptors we need names to describe the various types of objects that can 6888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# appear on the stack. 6898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersclass StackObject(object): 6918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __slots__ = ( 6928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # name of descriptor record, for info only 6938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'name', 6948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # type of object, or tuple of type objects (meaning the object can 6968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # be of any type in the tuple) 6978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'obtype', 6988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # human-readable docs for this kind of stack object; a string 7008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'doc', 7018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ) 7028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters def __init__(self, name, obtype, doc): 7048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(name, str) 7058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.name = name 7068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(obtype, type) or isinstance(obtype, tuple) 7088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if isinstance(obtype, tuple): 7098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for contained in obtype: 7108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(contained, type) 7118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.obtype = obtype 7128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(doc, str) 7148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.doc = doc 7158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 716c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters def __repr__(self): 717c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters return self.name 718c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 7198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyint = StackObject( 7218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='int', 7228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=int, 7238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A short (as opposed to long) Python integer object.") 7248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspylong = StackObject( 7268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='long', 7278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=long, 7288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A long (as opposed to short) Python integer object.") 7298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyinteger_or_bool = StackObject( 7318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='int_or_bool', 7328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=(int, long, bool), 7338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python integer object (short or long), or " 7348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "a Python bool.") 7358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7365a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumpybool = StackObject( 7375a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum name='bool', 7385a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum obtype=(bool,), 7395a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum doc="A Python bool object.") 7405a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 7418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyfloat = StackObject( 7428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='float', 7438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=float, 7448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python float object.") 7458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspystring = StackObject( 7478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='str', 7488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=str, 7498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python string object.") 7508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyunicode = StackObject( 7528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='unicode', 7538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=unicode, 7548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python Unicode string object.") 7558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspynone = StackObject( 7578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="None", 7588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=type(None), 7598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="The Python None object.") 7608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspytuple = StackObject( 7628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="tuple", 7638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=tuple, 7648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python tuple object.") 7658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspylist = StackObject( 7678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="list", 7688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=list, 7698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python list object.") 7708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspydict = StackObject( 7728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="dict", 7738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=dict, 7748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python dict object.") 7758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersanyobject = StackObject( 7778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='any', 7788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=object, 7798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Any kind of object whatsoever.") 7808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersmarkobject = StackObject( 7828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="mark", 7838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=StackObject, 7848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""'The mark' is a unique object. 7858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Opcodes that operate on a variable number of objects 7878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters generally don't embed the count of objects in the opcode, 7888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters or pull it off the stack. Instead the MARK opcode is used 7898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to push a special marker object on the stack, and then 7908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters some other opcodes grab all the objects from the top of 7918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the stack down to (but not including) the topmost marker 7928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object. 7938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 7948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstackslice = StackObject( 7968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="stackslice", 7978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=StackObject, 7988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""An object representing a contiguous slice of the stack. 7998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is used in conjuction with markobject, to represent all 8018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of the stack following the topmost markobject. For example, 8028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the POP_MARK opcode changes the stack from 8038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters [..., markobject, stackslice] 8058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to 8068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters [...] 8078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters No matter how many object are on the stack after the topmost 8098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markobject, POP_MARK gets rid of all of them (including the 8108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject too). 8118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 8128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 8148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Descriptors for pickle opcodes. 8158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersclass OpcodeInfo(object): 8178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __slots__ = ( 8198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # symbolic name of opcode; a string 8208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'name', 8218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the code used in a bytestream to represent the opcode; a 8238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # one-character string 8248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'code', 8258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # If the opcode has an argument embedded in the byte string, an 8278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # instance of ArgumentDescriptor specifying its type. Note that 8288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # arg.reader(s) can be used to read and decode the argument from 8298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the bytestream s, and arg.doc documents the format of the raw 8308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # argument bytes. If the opcode doesn't have an argument embedded 8318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # in the bytestream, arg should be None. 8328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'arg', 8338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # what the stack looks like before this opcode runs; a list 8358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'stack_before', 8368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # what the stack looks like after this opcode runs; a list 8388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'stack_after', 8398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the protocol number in which this opcode was introduced; an int 8418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'proto', 8428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # human-readable docs for this opcode; a string 8448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'doc', 8458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ) 8468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters def __init__(self, name, code, arg, 8488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before, stack_after, proto, doc): 8498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(name, str) 8508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.name = name 8518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(code, str) 8538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert len(code) == 1 8548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.code = code 8558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert arg is None or isinstance(arg, ArgumentDescriptor) 8578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.arg = arg 8588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(stack_before, list) 8608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for x in stack_before: 8618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(x, StackObject) 8628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.stack_before = stack_before 8638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(stack_after, list) 8658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for x in stack_after: 8668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(x, StackObject) 8678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.stack_after = stack_after 8688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(proto, int) and 0 <= proto <= 2 8708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.proto = proto 8718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(doc, str) 8738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.doc = doc 8748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersI = OpcodeInfo 8768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersopcodes = [ 8778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell integers. 8798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='INT', 8818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='I', 8828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=decimalnl_short, 8838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 8848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyinteger_or_bool], 8858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 8868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an integer or bool. 8878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is a newline-terminated decimal literal string. 8898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The intent may have been that this always fit in a short Python int, 8918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters but INT can be generated in pickles written on a 64-bit box that 8928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters require a Python long on a 32-bit box. The difference between this 8938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and LONG then is that INT skips a trailing 'L', and produces a short 8948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters int whenever possible. 8958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Another difference is due to that, when bool was introduced as a 8978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters distinct type in 2.3, builtin names True and False were also added to 8988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, 8998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". 9008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Leading zeroes are never produced for a genuine integer. The 2.3 9018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (and later) unpicklers special-case these and return bool instead; 9028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters earlier unpicklers ignore the leading "0" and return the int. 9038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BININT', 9068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='J', 9078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=int4, 9088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyint], 9108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 9118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a four-byte signed integer. 9128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This handles the full range of Python (short) integers on a 32-bit 9148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters box, directly as binary bytes (1 for the opcode and 4 for the integer). 9158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the integer is non-negative and fits in 1 or 2 bytes, pickling via 9168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters BININT1 or BININT2 saves space. 9178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BININT1', 9208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='K', 9218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint1, 9228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyint], 9248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 9258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a one-byte unsigned integer. 9268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a space optimization for pickling very small non-negative ints, 9288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters in range(256). 9298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BININT2', 9328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='M', 9338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint2, 9348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyint], 9368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 9378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a two-byte unsigned integer. 9388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a space optimization for pickling small positive ints, in 9408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters range(256, 2**16). Integers in range(256) can also be pickled via 9418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters BININT2, but BININT1 instead saves a byte. 9428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 944fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='LONG', 945fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='L', 946fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=decimalnl_long, 947fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 948fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pylong], 949fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=0, 950fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Push a long integer. 951fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 952fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters The same as INT, except that the literal ends with 'L', and always 953fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters unpickles to a Python long. There doesn't seem a real purpose to the 954fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters trailing 'L'. 955fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 956fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters Note that LONG takes time quadratic in the number of digits when 957fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters unpickling (this is simply due to the nature of decimal->binary 958fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters conversion). Proto 2 added linear-time (in C; still quadratic-time 959fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters in Python) LONG1 and LONG4 opcodes. 960fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 961fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 962fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name="LONG1", 963fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x8a', 964fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=long1, 965fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 966fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pylong], 967fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 968fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Long integer using one-byte length. 969fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 970fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters A more efficient encoding of a Python long; the long1 encoding 971fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters says it all."""), 972fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 973fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name="LONG4", 974fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x8b', 975fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=long4, 976fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 977fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pylong], 978fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 979fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Long integer using found-byte length. 980fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 981fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters A more efficient encoding of a Python long; the long4 encoding 982fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters says it all."""), 983fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 9848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell strings (8-bit, not Unicode). 9858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='STRING', 9878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='S', 9888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl, 9898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pystring], 9918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 9928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python string object. 9938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is a repr-style string, with bracketing quote characters, 9958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and perhaps embedded escapes. The argument extends until the next 9968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters newline character. 9978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINSTRING', 10008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='T', 10018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=string4, 10028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pystring], 10048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 10058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python string object. 10068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are two arguments: the first is a 4-byte little-endian signed int 10088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters giving the number of bytes in the string, and the second is that many 10098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bytes, which are taken literally as the string content. 10108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='SHORT_BINSTRING', 10138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='U', 10148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=string1, 10158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pystring], 10178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 10188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python string object. 10198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are two arguments: the first is a 1-byte unsigned int giving 10218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the number of bytes in the string, and the second is that many bytes, 10228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters which are taken literally as the string content. 10238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell None. 10268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='NONE', 10288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='N', 10298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 10308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pynone], 10328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 10338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push None on the stack."), 10348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1035fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # Ways to spell bools, starting with proto 2. See INT for how this was 1036fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # done before proto 2. 1037fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1038fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='NEWTRUE', 1039fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x88', 1040fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1041fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1042fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pybool], 1043fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1044fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""True. 1045fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1046fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters Push True onto the stack."""), 1047fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1048fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='NEWFALSE', 1049fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x89', 1050fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1051fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1052fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pybool], 1053fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1054fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""True. 1055fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1056fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters Push False onto the stack."""), 1057fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 10588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell Unicode strings. 10598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='UNICODE', 10618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='V', 10628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=unicodestringnl, 10638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyunicode], 10658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, # this may be pure-text, but it's a later addition 10668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python Unicode string object. 10678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is a raw-unicode-escape encoding of a Unicode string, 10698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and so may contain embedded escape sequences. The argument extends 10708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters until the next newline character. 10718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINUNICODE', 10748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='X', 10758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=unicodestring4, 10768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyunicode], 10788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 10798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python Unicode string object. 10808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are two arguments: the first is a 4-byte little-endian signed int 10828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters giving the number of bytes in the string. The second is that many 10838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bytes, and is the UTF-8 encoding of the Unicode string. 10848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell floats. 10878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='FLOAT', 10898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='F', 10908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=floatnl, 10918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyfloat], 10938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 10948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Newline-terminated decimal float literal. 10958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is repr(a_float), and in general requires 17 significant 10978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters digits for roundtrip conversion to be an identity (this is so for 10988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters IEEE-754 double precision values, which is what Python float maps to 10998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters on most boxes). 11008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In general, FLOAT cannot be used to transport infinities, NaNs, or 11028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters minus zero across boxes (or even on a single box, if the platform C 11038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters library can't read the strings it produces for such things -- Windows 11048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is like that), but may do less damage than BINFLOAT on boxes with 11058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters greater precision or dynamic range than IEEE-754 double. 11068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINFLOAT', 11098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='G', 11108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=float8, 11118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 11128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyfloat], 11138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Float stored in binary form, with 8 bytes of data. 11158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This generally requires less than half the space of FLOAT encoding. 11178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In general, BINFLOAT cannot be used to transport infinities, NaNs, or 11188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters minus zero, raises an exception if the exponent exceeds the range of 11198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters an IEEE-754 double, and retains no more than 53 bits of precision (if 11208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters there are more than that, "add a half and chop" rounding is used to 11218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters cut it back to 53 significant bits). 11228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build lists. 11258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='EMPTY_LIST', 11278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code=']', 11288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 11308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push an empty list."), 11338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='APPEND', 11358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='a', 11368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pylist, anyobject], 11388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 11408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Append an object to a list. 11418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pylist anyobject 11438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pylist+[anyobject] 114481098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters 114581098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters although pylist is really extended in-place. 11468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='APPENDS', 11498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='e', 11508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pylist, markobject, stackslice], 11528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Extend a list by a slice of stack objects. 11558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pylist markobject stackslice 11578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pylist+stackslice 115881098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters 115981098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters although pylist is really extended in-place. 11608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='LIST', 11638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='l', 11648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 11668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 11688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a list out of the topmost stack slice, after markobject. 11698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters All the stack entries following the topmost markobject are placed into 11718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a single Python list, which single list object replaces all of the 11728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack from the topmost markobject onward. For example, 11738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject 1 2 3 'abc' 11758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... [1, 2, 3, 'abc'] 11768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build tuples. 11798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='EMPTY_TUPLE', 11818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code=')', 11828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 11848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pytuple], 11858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push an empty tuple."), 11878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='TUPLE', 11898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='t', 11908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 11928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pytuple], 11938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 11948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a tuple out of the topmost stack slice, after markobject. 11958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters All the stack entries following the topmost markobject are placed into 11978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a single Python tuple, which single tuple object replaces all of the 11988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack from the topmost markobject onward. For example, 11998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject 1 2 3 'abc' 12018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... (1, 2, 3, 'abc') 12028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 12038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1204fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='TUPLE1', 1205fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x85', 1206fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1207fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject], 1208fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pytuple], 1209fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1210fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""One-tuple. 1211fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1212fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code pops one value off the stack and pushes a tuple of 1213fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters length 1 whose one item is that value back onto it. IOW: 1214fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1215fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack[-1] = tuple(stack[-1:]) 1216fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1217fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1218fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='TUPLE2', 1219fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x86', 1220fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1221fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject, anyobject], 1222fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pytuple], 1223fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1224fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""One-tuple. 1225fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1226fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code pops two values off the stack and pushes a tuple 1227fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters of length 2 whose items are those values back onto it. IOW: 1228fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1229fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack[-2:] = [tuple(stack[-2:])] 1230fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1231fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1232fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='TUPLE3', 1233fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x87', 1234fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1235fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject, anyobject, anyobject], 1236fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pytuple], 1237fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1238fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""One-tuple. 1239fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1240fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code pops three values off the stack and pushes a tuple 1241fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters of length 3 whose items are those values back onto it. IOW: 1242fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1243fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack[-3:] = [tuple(stack[-3:])] 1244fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1245fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 12468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build dicts. 12478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='EMPTY_DICT', 12498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='}', 12508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 12528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 12548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push an empty dict."), 12558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='DICT', 12578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='d', 12588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 12608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 12628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a dict out of the topmost stack slice, after markobject. 12638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters All the stack entries following the topmost markobject are placed into 12658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a single Python dict, which single dict object replaces all of the 12668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack from the topmost markobject onward. The stack slice alternates 12678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters key, value, key, value, .... For example, 12688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject 1 2 3 'abc' 12708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... {1: 2, 3: 'abc'} 12718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 12728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='SETITEM', 12748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='s', 12758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pydict, anyobject, anyobject], 12778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 12798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Add a key+value pair to an existing dict. 12808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pydict key value 12828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pydict 12838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters where pydict has been modified via pydict[key] = value. 12858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 12868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='SETITEMS', 12888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='u', 12898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pydict, markobject, stackslice], 12918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 12938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Add an arbitrary number of key+value pairs to an existing dict. 12948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The slice of the stack following the topmost markobject is taken as 12968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters an alternating sequence of keys and values, added to the dict 12978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters immediately under the topmost markobject. Everything at and after the 12988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject is popped, leaving the mutated dict at the top 12998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of the stack. 13008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pydict markobject key_1 value_1 ... key_n value_n 13028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pydict 13038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters where pydict has been modified via pydict[key_i] = value_i for i in 13058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1, 2, ..., n, and in that order. 13068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Stack manipulation. 13098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='POP', 13118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='0', 13128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 13148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 13158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Discard the top stack item, shrinking the stack by one item."), 13178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='DUP', 13198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='2', 13208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 13228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject, anyobject], 13238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push the top stack item onto the stack again, duplicating it."), 13258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='MARK', 13278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='(', 13288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[markobject], 13318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push markobject onto the stack. 13338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markobject is a unique object, used by other opcodes to identify a 13358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters region of the stack containing a variable number of objects for them 13368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to work on. See markobject.doc for more detail. 13378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='POP_MARK', 13408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='1', 13418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 13438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 13448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Pop all the stack objects at and above the topmost markobject. 13468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters When an opcode using a variable number of stack objects is done, 13488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters POP_MARK is used to remove those objects, and to remove the markobject 13498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters that delimited their starting position on the stack. 13508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Memo manipulation. There are really only two operations (get and put), 13538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # each in all-text, "short binary", and "long binary" flavors. 13548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='GET', 13568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='g', 13578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=decimalnl_short, 13588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 13608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Read an object from the memo and push it on the stack. 13628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo object to push is given by the newline-teriminated 13648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters decimal string following. BINGET and LONG_BINGET are space-optimized 13658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters versions. 13668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINGET', 13698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='h', 13708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint1, 13718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 13738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 13748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Read an object from the memo and push it on the stack. 13758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo object to push is given by the 1-byte unsigned 13778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters integer following. 13788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='LONG_BINGET', 13818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='j', 13828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=int4, 13838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 13858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 13868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Read an object from the memo and push it on the stack. 13878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo object to push is given by the 4-byte signed 13898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters little-endian integer following. 13908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='PUT', 13938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='p', 13948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=decimalnl_short, 13958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 13978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Store the stack top into the memo. The stack is not popped. 13998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo location to write into is given by the newline- 14018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters terminated decimal string following. BINPUT and LONG_BINPUT are 14028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters space-optimized versions. 14038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINPUT', 14068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='q', 14078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint1, 14088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 14098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 14108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 14118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Store the stack top into the memo. The stack is not popped. 14128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo location to write into is given by the 1-byte 14148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters unsigned integer following. 14158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='LONG_BINPUT', 14188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='r', 14198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=int4, 14208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 14218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 14228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 14238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Store the stack top into the memo. The stack is not popped. 14248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo location to write into is given by the 4-byte 14268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters signed little-endian integer following. 14278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1429fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # Access the extension registry (predefined objects). Akin to the GET 1430fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # family. 1431fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1432fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='EXT1', 1433fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x82', 1434fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=uint1, 1435fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1436fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1437fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1438fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Extension code. 1439fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1440fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code and the similar EXT2 and EXT4 allow using a registry 1441fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters of popular objects that are pickled by name, typically classes. 1442fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters It is envisioned that through a global negotiation and 1443fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters registration process, third parties can set up a mapping between 1444fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters ints and object names. 1445fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1446fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters In order to guarantee pickle interchangeability, the extension 1447fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code registry ought to be global, although a range of codes may 1448fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters be reserved for private use. 1449fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1450fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters EXT1 has a 1-byte integer argument. This is used to index into the 1451fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters extension registry, and the object at that index is pushed on the stack. 1452fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1453fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1454fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='EXT2', 1455fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x83', 1456fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=uint2, 1457fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1458fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1459fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1460fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Extension code. 1461fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1462fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters See EXT1. EXT2 has a two-byte integer argument. 1463fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1464fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1465fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='EXT4', 1466fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x84', 1467fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=int4, 1468fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1469fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1470fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1471fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Extension code. 1472fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1473fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters See EXT1. EXT4 has a four-byte integer argument. 1474fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1475fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 14768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Push a class object, or module function, on the stack, via its module 14778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # and name. 14788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='GLOBAL', 14808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='c', 14818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl_noescape_pair, 14828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 14838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 14848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 14858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a global object (module.attr) on the stack. 14868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Two newline-terminated strings follow the GLOBAL opcode. The first is 14888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters taken as a module name, and the second as a class name. The class 14898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object module.class is pushed on the stack. More accurately, the 14908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object returned by self.find_class(module, class) is pushed on the 14918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack, so unpickling subclasses can override this form of lookup. 14928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build objects of classes pickle doesn't know about directly 14958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # (user-defined classes). I despair of documenting this accurately 14968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # and comprehensibly -- you really have to read the pickle code to 14978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # find all the special cases. 14988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='REDUCE', 15008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='R', 15018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 15028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject, anyobject], 15038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 15048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 15058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an object built from a callable and an argument tuple. 15068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The opcode is named to remind of the __reduce__() method. 15088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... callable pytuple 15108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... callable(*pytuple) 15118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The callable and the argument tuple are the first two items returned 15138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters by a __reduce__ method. Applying the callable to the argtuple is 15148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters supposed to reproduce the original object, or at least get it started. 15158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the __reduce__ method returns a 3-tuple, the last component is an 15168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argument to be passed to the object's __setstate__, and then the REDUCE 15178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode is followed by code to create setstate's argument, and then a 15188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters BUILD opcode to apply __setstate__ to that argument. 15198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are lots of special cases here. The argtuple can be None, in 15218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters which case callable.__basicnew__() is called instead to produce the 15228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object to be pushed on the stack. This appears to be a trick unique 15238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to ExtensionClasses, and is deprecated regardless. 15248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If type(callable) is not ClassType, REDUCE complains unless the 15268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters callable has been registered with the copy_reg module's 15278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters safe_constructors dict, or the callable has a magic 15288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '__safe_for_unpickling__' attribute with a true value. I'm not sure 15298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters why it does this, but I've sure seen this complaint often enough when 15308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I didn't want to <wink>. 15318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 15328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BUILD', 15348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='b', 15358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 15368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject, anyobject], 15378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 15388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 15398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Finish building an object, via __setstate__ or dict update. 15408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... anyobject argument 15428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... anyobject 15438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters where anyobject may have been mutated, as follows: 15458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the object has a __setstate__ method, 15478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters anyobject.__setstate__(argument) 15498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is called. 15518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Else the argument must be a dict, the object must have a __dict__, and 15538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the object is updated via 15548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters anyobject.__dict__.update(argument) 15568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This may raise RuntimeError in restricted execution mode (which 15588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters disallows access to __dict__ directly); in that case, the object 15598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is updated instead via 15608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for k, v in argument.items(): 15628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters anyobject[k] = v 15638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 15648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='INST', 15668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='i', 15678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl_noescape_pair, 15688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 15698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 15708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 15718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a class instance. 15728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is the protocol 0 version of protocol 1's OBJ opcode. 15748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters INST is followed by two newline-terminated strings, giving a 15758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters module and class name, just as for the GLOBAL opcode (and see 15768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters GLOBAL for more details about that). self.find_class(module, name) 15778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is used to get a class object. 15788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In addition, all the objects on the stack following the topmost 15808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markobject are gathered into a tuple and popped (along with the 15818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject), just as for the TUPLE opcode. 15828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Now it gets complicated. If all of these are true: 15848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters + The argtuple is empty (markobject was at the top of the stack 15868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters at the start). 15878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters + It's an old-style class object (the type of the class object is 15898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ClassType). 15908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters + The class object does not have a __getinitargs__ attribute. 15928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters then we want to create an old-style class instance without invoking 15948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters its __init__() method (pickle has waffled on this over the years; not 15958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters calling __init__() is current wisdom). In this case, an instance of 15968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters an old-style dummy class is created, and then we try to rebind its 15978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __class__ attribute to the desired class object. If this succeeds, 15988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the new instance object is pushed on the stack, and we're done. In 15998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters restricted execution mode it can fail (assignment to __class__ is 16008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters disallowed), and I'm not really sure what happens then -- it looks 16018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters like the code ends up calling the class object's __init__ anyway, 16028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters via falling into the next case. 16038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Else (the argtuple is not empty, it's not an old-style class object, 16058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters or the class object does have a __getinitargs__ attribute), the code 16068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters first insists that the class object have a __safe_for_unpickling__ 16078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, 16088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters it doesn't matter whether this attribute has a true or false value, it 1609ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum only matters whether it exists (XXX this is a bug; cPickle 1610ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum requires the attribute to be true). If __safe_for_unpickling__ 1611ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum doesn't exist, UnpicklingError is raised. 16128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Else (the class object does have a __safe_for_unpickling__ attr), 16148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the class object obtained from INST's arguments is applied to the 16158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argtuple obtained from the stack, and the resulting instance object 16168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is pushed on the stack. 16178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 16188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='OBJ', 16208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='o', 16218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 16228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, anyobject, stackslice], 16238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 16248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 16258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a class instance. 16268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is the protocol 1 version of protocol 0's INST opcode, and is 16288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters very much like it. The major difference is that the class object 16298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is taken off the stack, allowing it to be retrieved from the memo 16308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters repeatedly if several instances of the same class are created. This 16318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters can be much more efficient (in both time and space) than repeatedly 16328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters embedding the module and class names in INST opcodes. 16338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Unlike INST, OBJ takes no arguments from the opcode stream. Instead 16358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the class object is taken off the stack, immediately above the 16368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject: 16378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject classobject stackslice 16398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... new_instance_object 16408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters As for INST, the remainder of the stack above the markobject is 16428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters gathered into an argument tuple, and then the logic seems identical, 1643ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum except that no __safe_for_unpickling__ check is done (XXX this is 1644ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum a bug; cPickle does test __safe_for_unpickling__). See INST for 1645ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum the gory details. 16468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 16478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1648fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='NEWOBJ', 1649fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x81', 1650fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1651fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject, anyobject], 1652fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1653fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1654fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Build an object instance. 1655fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1656fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters The stack before should be thought of as containing a class 1657fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters object followed by an argument tuple (the tuple being the stack 1658fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters top). Call these cls and args. They are popped off the stack, 1659fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters and the value returned by cls.__new__(cls, *args) is pushed back 1660fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters onto the stack. 1661fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1662fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 16638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Machine control. 16648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1665fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='PROTO', 1666fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x80', 1667fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=uint1, 1668fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1669fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[], 1670fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1671fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Protocol version indicator. 1672fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1673fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters For protocol 2 and above, a pickle must start with this opcode. 1674fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters The argument is the protocol version, an int in range(2, 256). 1675fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1676fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 16778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='STOP', 16788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='.', 16798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 16808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 16818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 16828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 16838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Stop the unpickling machine. 16848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Every pickle ends with this opcode. The object at the top of the stack 16868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is popped, and that's the result of unpickling. The stack should be 16878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters empty then. 16888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 16898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to deal with persistent IDs. 16918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='PERSID', 16938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='P', 16948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl_noescape, 16958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 16968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 16978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 16988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an object identified by a persistent ID. 16998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The pickle module doesn't define what a persistent ID means. PERSID's 17018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argument is a newline-terminated str-style (no embedded escapes, no 17028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bracketing quote characters) string, which *is* "the persistent ID". 17038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The unpickler passes this string to self.persistent_load(). Whatever 17048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object that returns is pushed on the stack. There is no implementation 17058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of persistent_load() in Python's unpickler: it must be supplied by an 17068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters unpickler subclass. 17078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 17088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINPERSID', 17108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='Q', 17118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 17128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 17138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 17148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 17158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an object identified by a persistent ID. 17168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Like PERSID, except the persistent ID is popped off the stack (instead 17188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of being a string embedded in the opcode bytestream). The persistent 17198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ID is passed to self.persistent_load(), and whatever object that 17208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters returns is pushed on the stack. See PERSID for more detail. 17218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 17228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters] 17238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdel I 17248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Verify uniqueness of .name and .code members. 17268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersname2i = {} 17278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscode2i = {} 17288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfor i, d in enumerate(opcodes): 17308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if d.name in name2i: 17318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("repeated name %r at indices %d and %d" % 17328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (d.name, name2i[d.name], i)) 17338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if d.code in code2i: 17348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("repeated code %r at indices %d and %d" % 17358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (d.code, code2i[d.code], i)) 17368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name2i[d.name] = i 17388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code2i[d.code] = i 17398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdel name2i, code2i, i, d 17418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 17438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Build a code2op dict, mapping opcode characters to OpcodeInfo records. 17448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Also ensure we've got the same stuff as pickle.py, although the 17458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# introspection here is dicey. 17468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscode2op = {} 17488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfor d in opcodes: 17498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code2op[d.code] = d 17508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdel d 17518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef assure_pickle_consistency(verbose=False): 17538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters import pickle, re 17548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters copy = code2op.copy() 17568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for name in pickle.__all__: 17578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not re.match("[A-Z][A-Z0-9_]+$", name): 17588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if verbose: 17598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print "skipping %r: it doesn't look like an opcode name" % name 17608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters continue 17618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters picklecode = getattr(pickle, name) 17628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not isinstance(picklecode, str) or len(picklecode) != 1: 17638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if verbose: 17648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print ("skipping %r: value %r doesn't look like a pickle " 17658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "code" % (name, picklecode)) 17668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters continue 17678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if picklecode in copy: 17688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if verbose: 17698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print "checking name %r w/ code %r for consistency" % ( 17708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name, picklecode) 17718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters d = copy[picklecode] 17728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if d.name != name: 17738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("for pickle code %r, pickle.py uses name %r " 17748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "but we're using name %r" % (picklecode, 17758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name, 17768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters d.name)) 17778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Forget this one. Any left over in copy at the end are a problem 17788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # of a different kind. 17798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters del copy[picklecode] 17808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 17818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("pickle.py appears to have a pickle opcode with " 17828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "name %r and code %r, but we don't" % 17838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (name, picklecode)) 17848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if copy: 17858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"] 17868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for code, d in copy.items(): 17878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters msg.append(" name %r with code %r" % (d.name, code)) 17888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("\n".join(msg)) 17898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersassure_pickle_consistency() 1791c0c12b57070a5b494662bebc418e3958bf5bdbeeTim Petersdel assure_pickle_consistency 17928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 17948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# A pickle opcode generator. 17958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef genops(pickle): 1797a72ded9bc81bd278bf119e92feaeaea2aa16f02dGuido van Rossum """Generate all the opcodes in a pickle. 17988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'pickle' is a file-like object, or string, containing the pickle. 18008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Each opcode in the pickle is generated, from the current pickle position, 18028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stopping after a STOP opcode is delivered. A triple is generated for 18038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters each opcode: 18048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode, arg, pos 18068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode is an OpcodeInfo record, describing the current opcode. 18088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the opcode has an argument embedded in the pickle, arg is its decoded 18108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters value, as a Python object. If the opcode doesn't have an argument, arg 18118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is None. 18128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the pickle has a tell() method, pos was the value of pickle.tell() 18148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters before reading the current opcode. If the pickle is a string object, 18158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters it's wrapped in a StringIO object, and the latter's tell() result is 18168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters used. Else (the pickle doesn't have a tell(), and it's not obvious how 18178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to query its current position) pos is None. 18188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 18198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters import cStringIO as StringIO 18218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if isinstance(pickle, str): 18238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pickle = StringIO.StringIO(pickle) 18248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if hasattr(pickle, "tell"): 18268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters getpos = pickle.tell 18278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 18288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters getpos = lambda: None 18298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters while True: 18318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pos = getpos() 18328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code = pickle.read(1) 18338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode = code2op.get(code) 18348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if opcode is None: 18358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if code == "": 18368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("pickle exhausted before seeing STOP") 18378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 18388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("at position %s, opcode %r unknown" % ( 18398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pos is None and "<unknown>" or pos, 18408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code)) 18418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if opcode.arg is None: 18428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg = None 18438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 18448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg = opcode.arg.reader(pickle) 18458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters yield opcode, arg, pos 18468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if code == '.': 18478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert opcode.name == 'STOP' 18488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters break 18498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 18518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# A symbolic pickle disassembler. 18528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef dis(pickle, out=None, indentlevel=4): 18548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """Produce a symbolic disassembly of a pickle. 18558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'pickle' is a file-like object, or string, containing a (at least one) 18578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pickle. The pickle is disassembled from the current position, through 18588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the first STOP opcode encountered. 18598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Optional arg 'out' is a file-like object to which the disassembly is 18618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters printed. It defaults to sys.stdout. 18628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Optional arg indentlevel is the number of blanks by which to indent 18648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a new MARK level. It defaults to 4. 1865c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1866c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters In addition to printing the disassembly, some sanity checks are made: 1867c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1868c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + All embedded opcode arguments "make sense". 1869c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1870c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + Explicit and implicit pop operations have enough items on the stack. 1871c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1872c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + When an opcode implicitly refers to a markobject, a markobject is 1873c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters actually on the stack. 1874c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1875c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + A memo entry isn't referenced before it's defined. 1876c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1877c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + The markobject isn't stored in the memo. 1878c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1879c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + A memo entry isn't redefined. 18808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 18818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1882c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Most of the hair here is for sanity checks, but most of it is needed 1883c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # anyway to detect when a protocol 0 POP takes a MARK off the stack 1884c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # (which in turn is needed to indent MARK blocks correctly). 1885c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1886c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack = [] # crude emulation of unpickler stack 1887c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters memo = {} # crude emulation of unpicker memo 1888c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters maxproto = -1 # max protocol number seen 1889c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markstack = [] # bytecode positions of MARK opcodes 18908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters indentchunk = ' ' * indentlevel 1891c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = None 18928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for opcode, arg, pos in genops(pickle): 18938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if pos is not None: 18948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print >> out, "%5d:" % pos, 18958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1896d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters line = "%-4s %s%s" % (repr(opcode.code)[1:-1], 1897d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters indentchunk * len(markstack), 1898d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters opcode.name) 18998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1900c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters maxproto = max(maxproto, opcode.proto) 1901c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters before = opcode.stack_before # don't mutate 1902c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters after = opcode.stack_after # don't mutate 190343277d64eb5e008e59009bd03887d66c134445efTim Peters numtopop = len(before) 190443277d64eb5e008e59009bd03887d66c134445efTim Peters 190543277d64eb5e008e59009bd03887d66c134445efTim Peters # See whether a MARK should be popped. 19068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markmsg = None 1907c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markobject in before or (opcode.name == "POP" and 1908c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack and 1909c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack[-1] is markobject): 1910c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters assert markobject not in after 191143277d64eb5e008e59009bd03887d66c134445efTim Peters if __debug__: 191243277d64eb5e008e59009bd03887d66c134445efTim Peters if markobject in before: 191343277d64eb5e008e59009bd03887d66c134445efTim Peters assert before[-1] is stackslice 1914c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markstack: 1915c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markpos = markstack.pop() 1916c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markpos is None: 1917c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markmsg = "(MARK at unknown opcode offset)" 1918c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1919c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markmsg = "(MARK at %d)" % markpos 1920c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Pop everything at and after the topmost markobject. 1921c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters while stack[-1] is not markobject: 1922c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack.pop() 1923c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack.pop() 192443277d64eb5e008e59009bd03887d66c134445efTim Peters # Stop later code from popping too much. 1925c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters try: 192643277d64eb5e008e59009bd03887d66c134445efTim Peters numtopop = before.index(markobject) 1927c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters except ValueError: 1928c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters assert opcode.name == "POP" 192943277d64eb5e008e59009bd03887d66c134445efTim Peters numtopop = 0 1930c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1931c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = markmsg = "no MARK exists on stack" 1932c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1933c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Check for correct memo usage. 1934c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"): 193543277d64eb5e008e59009bd03887d66c134445efTim Peters assert arg is not None 1936c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if arg in memo: 1937c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "memo key %r already defined" % arg 1938c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters elif not stack: 1939c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "stack is empty -- can't store into memo" 1940c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters elif stack[-1] is markobject: 1941c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "can't store markobject in the memo" 1942c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1943c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters memo[arg] = stack[-1] 1944c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1945c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters elif opcode.name in ("GET", "BINGET", "LONG_BINGET"): 1946c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if arg in memo: 1947c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters assert len(after) == 1 1948c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters after = [memo[arg]] # for better stack emulation 1949c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1950c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "memo key %r has never been stored into" % arg 19518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 19528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if arg is not None or markmsg: 19538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # make a mild effort to align arguments 19548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters line += ' ' * (10 - len(opcode.name)) 19558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if arg is not None: 19568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters line += ' ' + repr(arg) 19578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if markmsg: 19588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters line += ' ' + markmsg 19598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print >> out, line 19608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1961c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if errormsg: 1962c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Note that we delayed complaining until the offending opcode 1963c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # was printed. 1964c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters raise ValueError(errormsg) 1965c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1966c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Emulate the stack effects. 196743277d64eb5e008e59009bd03887d66c134445efTim Peters if len(stack) < numtopop: 196843277d64eb5e008e59009bd03887d66c134445efTim Peters raise ValueError("tries to pop %d items from stack with " 196943277d64eb5e008e59009bd03887d66c134445efTim Peters "only %d items" % (numtopop, len(stack))) 197043277d64eb5e008e59009bd03887d66c134445efTim Peters if numtopop: 197143277d64eb5e008e59009bd03887d66c134445efTim Peters del stack[-numtopop:] 1972c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markobject in after: 197343277d64eb5e008e59009bd03887d66c134445efTim Peters assert markobject not in before 19748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markstack.append(pos) 19758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1976c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack.extend(after) 1977c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1978c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters print >> out, "highest protocol among opcodes =", maxproto 1979c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if stack: 1980c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters raise ValueError("stack not empty after STOP: %r" % stack) 19818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 198203e35327f26d353db126a8f840a8890b3593f18aGuido van Rossum_dis_test = r""" 19838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> import pickle 19848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> x = [1, 2, (3, 4), {'abc': u"def"}] 1985570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> pkl = pickle.dumps(x, 0) 1986570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> dis(pkl) 1987d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 1988d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: l LIST (MARK at 0) 1989d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: p PUT 0 1990d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: I INT 1 1991d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: a APPEND 1992d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: I INT 2 1993d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 12: a APPEND 1994d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 13: ( MARK 1995d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: I INT 3 1996d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 17: I INT 4 1997d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 20: t TUPLE (MARK at 13) 1998d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 21: p PUT 1 1999d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 24: a APPEND 2000d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 25: ( MARK 2001d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 26: d DICT (MARK at 25) 2002d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 27: p PUT 2 2003d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 30: S STRING 'abc' 2004d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 37: p PUT 3 2005d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 40: V UNICODE u'def' 2006d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 45: p PUT 4 2007d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 48: s SETITEM 2008d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 49: a APPEND 2009d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 50: . STOP 2010c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 20118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTry again with a "binary" pickle. 20138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2014570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> pkl = pickle.dumps(x, 1) 2015570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> dis(pkl) 2016d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ] EMPTY_LIST 2017d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: q BINPUT 0 2018d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: ( MARK 2019d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: K BININT1 1 2020d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: K BININT1 2 2021d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: ( MARK 2022d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: K BININT1 3 2023d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: K BININT1 4 2024d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 13: t TUPLE (MARK at 8) 2025d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: q BINPUT 1 2026d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 16: } EMPTY_DICT 2027d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 17: q BINPUT 2 2028d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 19: U SHORT_BINSTRING 'abc' 2029d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 24: q BINPUT 3 2030d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 26: X BINUNICODE u'def' 2031d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 34: q BINPUT 4 2032d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 36: s SETITEM 2033d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 37: e APPENDS (MARK at 3) 2034d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 38: . STOP 2035c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 20368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersExercise the INST/OBJ/BUILD family. 20388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> import random 2040f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(random.random, 0)) 2041d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: c GLOBAL 'random random' 2042d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 15: p PUT 0 2043d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 18: . STOP 2044c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 20458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> x = [pickle.PicklingError()] * 2 2047f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(x, 0)) 2048d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2049d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: l LIST (MARK at 0) 2050d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: p PUT 0 2051d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: ( MARK 2052d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: i INST 'pickle PicklingError' (MARK at 5) 2053d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 28: p PUT 1 2054d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 31: ( MARK 2055d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 32: d DICT (MARK at 31) 2056d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 33: p PUT 2 2057d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 36: S STRING 'args' 2058d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 44: p PUT 3 2059d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 47: ( MARK 2060d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 48: t TUPLE (MARK at 47) 2061d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 49: s SETITEM 2062d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 50: b BUILD 2063d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 51: a APPEND 2064d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 52: g GET 1 2065d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 55: a APPEND 2066d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 56: . STOP 2067c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 20688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> dis(pickle.dumps(x, 1)) 2070d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ] EMPTY_LIST 2071d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: q BINPUT 0 2072d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: ( MARK 2073d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: ( MARK 2074d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: c GLOBAL 'pickle PicklingError' 2075d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 27: q BINPUT 1 2076d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 29: o OBJ (MARK at 4) 2077d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 30: q BINPUT 2 2078d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 32: } EMPTY_DICT 2079d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 33: q BINPUT 3 2080d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 35: U SHORT_BINSTRING 'args' 2081d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 41: q BINPUT 4 2082d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 43: ) EMPTY_TUPLE 2083d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 44: s SETITEM 2084d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 45: b BUILD 2085d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 46: h BINGET 2 2086d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 48: e APPENDS (MARK at 3) 2087d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 49: . STOP 2088c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 20898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTry "the canonical" recursive-object test. 20918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L = [] 20938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> T = L, 20948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L.append(T) 20958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L[0] is T 20968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 20978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> T[0] is L 20988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 20998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L[0][0] is L 21008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 21018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> T[0][0] is T 21028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 2103f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(L, 0)) 2104d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2105d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: l LIST (MARK at 0) 2106d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: p PUT 0 2107d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: ( MARK 2108d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: g GET 0 2109d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: t TUPLE (MARK at 5) 2110d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: p PUT 1 2111d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 13: a APPEND 2112d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: . STOP 2113c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 2114c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 21158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> dis(pickle.dumps(L, 1)) 2116d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ] EMPTY_LIST 2117d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: q BINPUT 0 2118d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: ( MARK 2119d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: h BINGET 0 2120d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: t TUPLE (MARK at 3) 2121d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: q BINPUT 1 2122d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: a APPEND 2123d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: . STOP 2124c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 21258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2126c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim PetersNote that, in the protocol 0 pickle of the recursive tuple, the disassembler 2127c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershas to emulate the stack in order to realize that the POP opcode at 16 gets 2128c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petersrid of the MARK at 0. 21298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2130f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(T, 0)) 2131d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2132d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: ( MARK 2133d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: l LIST (MARK at 1) 2134d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: p PUT 0 2135d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: ( MARK 2136d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: g GET 0 2137d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: t TUPLE (MARK at 6) 2138d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: p PUT 1 2139d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: a APPEND 2140d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 15: 0 POP 2141c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 16: 0 POP (MARK at 0) 2142c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 17: g GET 1 2143c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 20: . STOP 2144c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 2145c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 21468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> dis(pickle.dumps(T, 1)) 2147d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2148d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: ] EMPTY_LIST 2149d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: q BINPUT 0 2150d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: ( MARK 2151d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: h BINGET 0 2152d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: t TUPLE (MARK at 4) 2153d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: q BINPUT 1 2154d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: a APPEND 2155d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: 1 POP_MARK (MARK at 0) 2156d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 12: h BINGET 1 2157d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: . STOP 2158c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 2159d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2160d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim PetersTry protocol 2. 2161d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2162d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters>>> dis(pickle.dumps(L, 2)) 2163d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: \x80 PROTO 2 2164d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: ] EMPTY_LIST 2165d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: q BINPUT 0 2166d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: h BINGET 0 2167d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: \x85 TUPLE1 2168d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: q BINPUT 1 2169d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: a APPEND 2170d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: . STOP 2171c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 2 2172d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2173d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters>>> dis(pickle.dumps(T, 2)) 2174d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: \x80 PROTO 2 2175d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: ] EMPTY_LIST 2176d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: q BINPUT 0 2177d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: h BINGET 0 2178d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: \x85 TUPLE1 2179d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: q BINPUT 1 2180d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: a APPEND 2181d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: 0 POP 2182d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 12: h BINGET 1 2183d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: . STOP 2184c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 2 21858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters""" 21868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2187570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum__test__ = {'disassembler_test': _dis_test, 21888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters } 21898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 21908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef _test(): 21918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters import doctest 21928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return doctest.testmod() 21938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 21948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersif __name__ == "__main__": 21958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters _test() 2196