pickletools.py revision 4b23f2b44bdd13758eab6808d6a08b951fbfc4dd
15445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanaro'''"Executable documentation" for the pickle module. 28ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 38ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersExtensive comments about the pickle protocols and pickle-machine opcodes 48ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscan be found here. Some functions meant for external use: 58ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 68ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersgenops(pickle) 78ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Generate all the opcodes in a pickle, as (opcode, arg, position) triples. 88ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 98ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdis(pickle, out=None, indentlevel=4) 108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Print a symbolic disassembly of a pickle. 115445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanaro''' 128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Other ideas: 148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# 158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# - A pickle verifier: read a pickle and check it exhaustively for 16c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters# well-formedness. dis() does a lot of this already. 178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# 188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# - A protocol identifier: examine a pickle and return its protocol number 198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# (== the highest .proto attr value among all the opcodes in the pickle). 20c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters# dis() already prints this info at the end. 218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# 228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# - A pickle optimizer: for example, tuple-building code is sometimes more 238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# elaborate than necessary, catering for the possibility that the tuple 248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# is recursive. Or lots of times a PUT is generated that's never accessed 258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# by a later GET. 268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters""" 298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters"A pickle" is a program for a virtual pickle machine (PM, but more accurately 308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscalled an unpickling machine). It's a sequence of opcodes, interpreted by the 318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersPM, building an arbitrarily complex Python object. 328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersFor the most part, the PM is very simple: there are no looping, testing, or 348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersconditional instructions, no arithmetic and no function calls. Opcodes are 358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersexecuted once each, from first to last, until a STOP opcode is reached. 368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe PM has two data areas, "the stack" and "the memo". 388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersMany opcodes push Python objects onto the stack; e.g., INT pushes a Python 408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersinteger object on the stack, whose value is gotten from a decimal string 418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersliteral immediately following the INT opcode in the pickle bytestream. Other 428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersopcodes take Python objects off the stack. The result of unpickling is 438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterswhatever object is left on the stack when the final STOP opcode is executed. 448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe memo is simply an array of objects, or it can be implemented as a dict 468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersmapping little integers to objects. The memo serves as the PM's "long term 478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersmemory", and the little integers indexing the memo are akin to variable 488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersnames. Some opcodes pop a stack object into the memo at a given index, 498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersand others push a memo object at a given index onto the stack again. 508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersAt heart, that's all the PM has. Subtleties arise for these reasons: 528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Object identity. Objects can be arbitrarily complex, and subobjects 548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters may be shared (for example, the list [a, a] refers to the same object a 558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters twice). It can be vital that unpickling recreate an isomorphic object 568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters graph, faithfully reproducing sharing. 578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Recursive objects. For example, after "L = []; L.append(L)", L is a 598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters list, and L[0] is the same list. This is related to the object identity 608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters point, and some sequences of pickle opcodes are subtle in order to 618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters get the right result in all cases. 628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Things pickle doesn't know everything about. Examples of things pickle 648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters does know everything about are Python's builtin scalar and container 658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters types, like ints and tuples. They generally have opcodes dedicated to 668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters them. For things like module references and instances of user-defined 678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters classes, pickle's knowledge is limited. Historically, many enhancements 688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters have been made to the pickle protocol in order to do a better (faster, 698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and/or more compact) job on those. 708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters+ Backward compatibility and micro-optimization. As explained below, 728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pickle opcodes never go away, not even when better ways to do a thing 738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters get invented. The repertoire of the PM just keeps growing over time. 74fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters For example, protocol 0 had two opcodes for building Python integers (INT 75fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters and LONG), protocol 1 added three more for more-efficient pickling of short 76fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters integers, and protocol 2 added two more for more-efficient pickling of 77fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters long integers (before protocol 2, the only ways to pickle a Python long 78fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters took time quadratic in the number of digits, for both pickling and 79fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters unpickling). "Opcode bloat" isn't so much a subtlety as a source of 808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters wearying complication. 818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersPickle protocols: 848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersFor compatibility, the meaning of a pickle opcode never changes. Instead new 868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspickle opcodes get added, and each version's unpickler can handle all the 878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspickle opcodes in all protocol versions to date. So old pickles continue to 888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersbe readable forever. The pickler can generally be told to restrict itself to 898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersthe subset of opcodes available under previous protocol versions too, so that 908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersusers can create pickles under the current version readable by older 918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersversions. However, a pickle does not contain its version number embedded 928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterswithin it. If an older unpickler tries to read a pickle using a later 938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersprotocol, the result is most likely an exception due to seeing an unknown (in 948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersthe older unpickler) opcode. 958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe original pickle used what's now called "protocol 0", and what was called 978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters"text mode" before Python 2.3. The entire pickle bytestream is made up of 988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersprintable 7-bit ASCII characters, plus the newline character, in protocol 0. 99fdc03462b3e0796ae6474da6f0f9844773d1da8fTim PetersThat's why it was called text mode. Protocol 0 is small and elegant, but 100fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peterssometimes painfully inefficient. 1018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe second major set of additions is now called "protocol 1", and was called 1038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters"binary mode" before Python 2.3. This added many opcodes with arguments 1048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersconsisting of arbitrary bytes, including NUL bytes and unprintable "high bit" 1058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersbytes. Binary mode pickles can be substantially smaller than equivalent 1068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterstext mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte 1078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersint as 4 bytes following the opcode, which is cheaper to unpickle than the 108fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters(perhaps) 11-character decimal string attached to INT. Protocol 1 also added 109fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Petersa number of opcodes that operate on many stack elements at once (like APPENDS 11081098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Petersand SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE). 1118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersThe third major set of additions came in Python 2.3, and is called "protocol 113fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters2". This added: 114fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 115fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- A better way to pickle instances of new-style classes (NEWOBJ). 116fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 117fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- A way for a pickle to identify its protocol (PROTO). 118fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 119fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- Time- and space- efficient pickling of long ints (LONG{1,4}). 120fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 121fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- Shortcuts for small tuples (TUPLE{1,2,3}}. 122fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 123fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- Dedicated opcodes for bools (NEWTRUE, NEWFALSE). 124fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 125fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters- The "extension registry", a vector of popular objects that can be pushed 126fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but 127fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters the registry contents are predefined (there's nothing akin to the memo's 128fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters PUT). 129ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum 1305445594e20baf1ac0c0c1aef30d55d789c4b9694Skip MontanaroAnother independent change with Python 2.3 is the abandonment of any 1315445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanaropretense that it might be safe to load pickles received from untrusted 132ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumparties -- no sufficient security analysis has been done to guarantee 1335445594e20baf1ac0c0c1aef30d55d789c4b9694Skip Montanarothis and there isn't a use case that warrants the expense of such an 134ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumanalysis. 135ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum 136ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van RossumTo this end, all tests for __safe_for_unpickling__ or for 137ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumcopy_reg.safe_constructors are removed from the unpickling code. 138ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van RossumReferences to these variables in the descriptions below are to be seen 139ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossumas describing unpickling in Python 2.2 and before. 1408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters""" 1418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Meta-rule: Descriptions are stored in instances of descriptor objects, 1438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# with plain constructors. No meta-language is defined from which 1448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# descriptors could be constructed. If you want, e.g., XML, write a little 1458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# program to generate XML from the objects. 1468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 1488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Some pickle opcodes have an argument, following the opcode in the 1498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# bytestream. An argument is of a specific type, described by an instance 1508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# of ArgumentDescriptor. These are not to be confused with arguments taken 1518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# off the stack -- ArgumentDescriptor applies only to arguments embedded in 1528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# the opcode stream, immediately following an opcode. 1538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Represents the number of bytes consumed by an argument delimited by the 1558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# next newline character. 1568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersUP_TO_NEWLINE = -1 1578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Represents the number of bytes consumed by a two-argument opcode where 1598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# the first argument gives the number of bytes in the second argument. 160fdb8cfab085d0f412411b090796e9f856ee0cab5Tim PetersTAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int 161fdb8cfab085d0f412411b090796e9f856ee0cab5Tim PetersTAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int 1628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersclass ArgumentDescriptor(object): 1648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __slots__ = ( 1658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # name of descriptor record, also a module global name; a string 1668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'name', 1678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # length of argument, in bytes; an int; UP_TO_NEWLINE and 169fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length 170fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters # cases 1718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'n', 1728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # a function taking a file-like object, reading this kind of argument 1748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # from the object at the current position, advancing the current 1758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # position by n bytes, and returning the value of the argument 1768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'reader', 1778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # human-readable docs for this arg descriptor; a string 1798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'doc', 1808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ) 1818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters def __init__(self, name, n, reader, doc): 1838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(name, str) 1848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.name = name 1858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(n, int) and (n >= 0 or 187fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n in (UP_TO_NEWLINE, 188fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters TAKEN_FROM_ARGUMENT1, 189fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters TAKEN_FROM_ARGUMENT4)) 1908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.n = n 1918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.reader = reader 1938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(doc, str) 1958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.doc = doc 1968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfrom struct import unpack as _unpack 1988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_uint1(f): 20055762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 20255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_uint1(StringIO.StringIO('\xff')) 2038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 255 2048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(1) 2078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if data: 2088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return ord(data) 2098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read uint1") 2108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersuint1 = ArgumentDescriptor( 2128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='uint1', 2138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=1, 2148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_uint1, 2158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="One-byte unsigned integer.") 2168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_uint2(f): 21955762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 22155762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_uint2(StringIO.StringIO('\xff\x00')) 2228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 255 22355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_uint2(StringIO.StringIO('\xff\xff')) 2248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 65535 2258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(2) 2288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == 2: 2298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return _unpack("<H", data)[0] 2308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read uint2") 2318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersuint2 = ArgumentDescriptor( 2338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='uint2', 2348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=2, 2358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_uint2, 2368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Two-byte unsigned integer, little-endian.") 2378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_int4(f): 24055762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 24255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00')) 2438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 255 24455762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31) 2458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters True 2468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(4) 2498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == 4: 2508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return _unpack("<i", data)[0] 2518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read int4") 2528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersint4 = ArgumentDescriptor( 2548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='int4', 2558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=4, 2568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_int4, 2578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Four-byte signed integer, little-endian, 2's complement.") 2588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_stringnl(f, decode=True, stripquotes=True): 26155762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 2628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 26355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n")) 2648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'abcd' 2658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 26655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("\n")) 2678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 2688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 2698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: no string quotes around '' 2708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 27155762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False) 2728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 2738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 27455762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO("''\n")) 2758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 2768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> read_stringnl(StringIO.StringIO('"abcd"')) 2788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 2798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 2808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: no newline found when trying to read stringnl 2818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Embedded escapes are undone in the result. 28355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'")) 28455762f5f804c4848bbce323b085101d450f89ff6Tim Peters 'a\n\\b\x00c\td' 2858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 2868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.readline() 2888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not data.endswith('\n'): 2898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("no newline found when trying to read stringnl") 2908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data[:-1] # lose the newline 2918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if stripquotes: 2938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for q in "'\"": 2948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if data.startswith(q): 2958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not data.endswith(q): 2968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("strinq quote %r not found at both " 2978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "ends of %r" % (q, data)) 2988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data[1:-1] 2998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters break 3008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 3018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("no string quotes around %r" % data) 3028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # I'm not sure when 'string_escape' was added to the std codecs; it's 3048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # crazy not to use it if it's there. 3058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if decode: 3068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data.decode('string_escape') 3078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return data 3088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstringnl = ArgumentDescriptor( 3108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='stringnl', 3118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 3128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_stringnl, 3138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated string. 3148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a repr-style string, with embedded escapes, and 3168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bracketing quotes. 3178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_stringnl_noescape(f): 3208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return read_stringnl(f, decode=False, stripquotes=False) 3218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstringnl_noescape = ArgumentDescriptor( 3238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='stringnl_noescape', 3248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 3258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_stringnl_noescape, 3268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated string. 3278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a str-style string, without embedded escapes, 3298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters or bracketing quotes. It should consist solely of 3308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters printable ASCII characters. 3318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_stringnl_noescape_pair(f): 33455762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 3358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 33655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk")) 337d916cf4ec7014e9f6b25b8d63728bda01a17d3f9Tim Peters 'Queue Empty' 3388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 3398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 340d916cf4ec7014e9f6b25b8d63728bda01a17d3f9Tim Peters return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) 3418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstringnl_noescape_pair = ArgumentDescriptor( 3438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='stringnl_noescape_pair', 3448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 3458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_stringnl_noescape_pair, 3468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A pair of newline-terminated strings. 3478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters These are str-style strings, without embedded 3498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters escapes, or bracketing quotes. They should 3508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters consist solely of printable ASCII characters. 3518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The pair is returned as a single string, with 352d916cf4ec7014e9f6b25b8d63728bda01a17d3f9Tim Peters a single blank separating the two strings. 3538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_string4(f): 35655762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 3578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 35855762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc")) 3598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 36055762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef")) 3618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'abc' 36255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef")) 3638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 3648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 3658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: expected 50331648 bytes in a string4, but only 6 remain 3668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 3678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n = read_int4(f) 3698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if n < 0: 3708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("string4 byte count < 0: %d" % n) 3718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(n) 3728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == n: 3738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return data 3748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("expected %d bytes in a string4, but only %d remain" % 3758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (n, len(data))) 3768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstring4 = ArgumentDescriptor( 3788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="string4", 379fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT4, 3808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_string4, 3818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A counted string. 3828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The first argument is a 4-byte little-endian signed int giving 3848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the number of bytes in the string, and the second argument is 3858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters that many bytes. 3868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 3878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_string1(f): 39055762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 3918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 39255762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string1(StringIO.StringIO("\x00")) 3938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '' 39455762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_string1(StringIO.StringIO("\x03abcdef")) 3958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'abc' 3968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 3978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 3988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n = read_uint1(f) 3998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert n >= 0 4008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(n) 4018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == n: 4028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return data 4038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("expected %d bytes in a string1, but only %d remain" % 4048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (n, len(data))) 4058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstring1 = ArgumentDescriptor( 4078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="string1", 408fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT1, 4098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_string1, 4108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A counted string. 4118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The first argument is a 1-byte unsigned int giving the number 4138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of bytes in the string, and the second argument is that many 4148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bytes. 4158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 4168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_unicodestringnl(f): 41955762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 4208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 42155762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk")) 42255762f5f804c4848bbce323b085101d450f89ff6Tim Peters u'abc\uabcd' 4238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 4248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.readline() 4268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not data.endswith('\n'): 4278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("no newline found when trying to read " 4288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "unicodestringnl") 4298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = data[:-1] # lose the newline 4308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return unicode(data, 'raw-unicode-escape') 4318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersunicodestringnl = ArgumentDescriptor( 4338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='unicodestringnl', 4348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 4358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_unicodestringnl, 4368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated Unicode string. 4378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is raw-unicode-escape encoded, so consists of 4398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters printable ASCII characters, and may contain embedded 4408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters escape sequences. 4418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 4428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_unicodestring4(f): 44455762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 4458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 44655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> s = u'abcd\uabcd' 4478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> enc = s.encode('utf-8') 4488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> enc 44955762f5f804c4848bbce323b085101d450f89ff6Tim Peters 'abcd\xea\xaf\x8d' 4508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> n = chr(len(enc)) + chr(0) * 3 # little-endian 4-byte length 4518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk')) 4528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> s == t 4538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters True 4548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1])) 4568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 4578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 4588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: expected 7 bytes in a unicodestring4, but only 6 remain 4598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 4608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n = read_int4(f) 4628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if n < 0: 4638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("unicodestring4 byte count < 0: %d" % n) 4648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(n) 4658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == n: 4668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return unicode(data, 'utf-8') 4678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("expected %d bytes in a unicodestring4, but only %d " 4688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "remain" % (n, len(data))) 4698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersunicodestring4 = ArgumentDescriptor( 4718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="unicodestring4", 472fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT4, 4738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_unicodestring4, 4748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A counted Unicode string. 4758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The first argument is a 4-byte little-endian signed int 4778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters giving the number of bytes in the string, and the second 4788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argument-- the UTF-8 encoding of the Unicode string -- 4798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters contains that many bytes. 4808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 4818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_decimalnl_short(f): 48455762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 4858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 48655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_short(StringIO.StringIO("1234\n56")) 4878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1234 4888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 48955762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_short(StringIO.StringIO("1234L\n56")) 4908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 4918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 4928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: trailing 'L' not allowed in '1234L' 4938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 4948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters s = read_stringnl(f, decode=False, stripquotes=False) 4968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if s.endswith("L"): 4978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("trailing 'L' not allowed in %r" % s) 4988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 4998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # It's not necessarily true that the result fits in a Python short int: 5008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the pickle may have been written on a 64-bit box. There's also a hack 5018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # for True and False here. 5028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if s == "00": 5038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return False 5048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters elif s == "01": 5058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return True 5068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters try: 5088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return int(s) 5098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters except OverflowError: 5108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return long(s) 5118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_decimalnl_long(f): 51355762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 5148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 5158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 51655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_long(StringIO.StringIO("1234\n56")) 5178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Traceback (most recent call last): 5188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ... 5198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ValueError: trailing 'L' required in '1234' 5208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Someday the trailing 'L' will probably go away from this output. 5228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 52355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_long(StringIO.StringIO("1234L\n56")) 5248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1234L 5258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 52655762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\n6")) 5278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 123456789012345678901234L 5288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 5298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters s = read_stringnl(f, decode=False, stripquotes=False) 5318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not s.endswith("L"): 5328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("trailing 'L' required in %r" % s) 5338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return long(s) 5348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdecimalnl_short = ArgumentDescriptor( 5378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='decimalnl_short', 5388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 5398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_decimalnl_short, 5408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated decimal integer literal. 5418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This never has a trailing 'L', and the integer fit 5438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters in a short Python int on the box where the pickle 5448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters was written -- but there's no guarantee it will fit 5458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters in a short Python int on the box where the pickle 5468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is read. 5478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 5488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdecimalnl_long = ArgumentDescriptor( 5508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='decimalnl_long', 5518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 5528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_decimalnl_long, 5538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated decimal integer literal. 5548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This has a trailing 'L', and can represent integers 5568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of any size. 5578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 5588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_floatnl(f): 56155762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 5628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO 56355762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_floatnl(StringIO.StringIO("-1.25\n6")) 5648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters -1.25 5658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 5668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters s = read_stringnl(f, decode=False, stripquotes=False) 5678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return float(s) 5688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfloatnl = ArgumentDescriptor( 5708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='floatnl', 5718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=UP_TO_NEWLINE, 5728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_floatnl, 5738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""A newline-terminated decimal floating literal. 5748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In general this requires 17 significant digits for roundtrip 5768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters identity, and pickling then unpickling infinities, NaNs, and 5778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters minus zero doesn't work across boxes, or on some boxes even 5788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters on itself (e.g., Windows can't read the strings it produces 5798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for infinities or NaNs). 5808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 5818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef read_float8(f): 58355762f5f804c4848bbce323b085101d450f89ff6Tim Peters r""" 5848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> import StringIO, struct 5858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> raw = struct.pack(">d", -1.25) 5868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters >>> raw 58755762f5f804c4848bbce323b085101d450f89ff6Tim Peters '\xbf\xf4\x00\x00\x00\x00\x00\x00' 58855762f5f804c4848bbce323b085101d450f89ff6Tim Peters >>> read_float8(StringIO.StringIO(raw + "\n")) 5898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters -1.25 5908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 5918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters data = f.read(8) 5938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if len(data) == 8: 5948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return _unpack(">d", data)[0] 5958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("not enough data in stream to read float8") 5968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 5988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfloat8 = ArgumentDescriptor( 5998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='float8', 6008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters n=8, 6018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters reader=read_float8, 6028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""An 8-byte binary representation of a float, big-endian. 6038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The format is unique to Python, and shared with the struct 6058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters module (format string '>d') "in theory" (the struct and cPickle 6068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters implementations don't share the code -- they should). It's 6078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters strongly related to the IEEE-754 double format, and, in normal 6088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters cases, is in fact identical to the big-endian 754 double format. 6098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters On other boxes the dynamic range is limited to that of a 754 6108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters double, and "add a half and chop" rounding is used to reduce 6118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the precision to 53 bits. However, even on a 754 box, 6128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters infinities, NaNs, and minus zero may not be handled correctly 6138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (may not survive roundtrip pickling intact). 6148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 6158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6165a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum# Protocol 2 formats 6175a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 618c0c12b57070a5b494662bebc418e3958bf5bdbeeTim Petersfrom pickle import decode_long 6195a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6205a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumdef read_long1(f): 6215a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum r""" 6225a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> import StringIO 6234b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters >>> read_long1(StringIO.StringIO("\x00")) 6244b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters 0L 6255a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\xff\x00")) 6265a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 255L 6275a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\xff\x7f")) 6285a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 32767L 6295a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\x00\xff")) 6305a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -256L 6315a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long1(StringIO.StringIO("\x02\x00\x80")) 6325a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -32768L 6335a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """ 6345a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6355a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum n = read_uint1(f) 6365a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum data = f.read(n) 6375a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum if len(data) != n: 6385a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum raise ValueError("not enough data in stream to read long1") 6395a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum return decode_long(data) 6405a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6415a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumlong1 = ArgumentDescriptor( 6425a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum name="long1", 643fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT1, 6445a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum reader=read_long1, 6455a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum doc="""A binary long, little-endian, using 1-byte size. 6465a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6475a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum This first reads one byte as an unsigned size, then reads that 648bdbe74183ceb7aedd71dd1afdf8755ee93e8c3caTim Peters many bytes and interprets them as a little-endian 2's-complement long. 6494b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters If the size is 0, that's taken as a shortcut for the long 0L. 6505a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """) 6515a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6525a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumdef read_long4(f): 6535a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum r""" 6545a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> import StringIO 6555a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x00")) 6565a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 255L 6575a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x7f")) 6585a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 32767L 6595a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\xff")) 6605a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -256L 6615a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\x80")) 6625a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum -32768L 6634b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters >>> read_long1(StringIO.StringIO("\x00\x00\x00\x00")) 6644b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters 0L 6655a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """ 6665a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6675a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum n = read_int4(f) 6685a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum if n < 0: 669784a3f53a8f78995d4c8ca22f612a68828bc4838Neal Norwitz raise ValueError("long4 byte count < 0: %d" % n) 6705a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum data = f.read(n) 6715a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum if len(data) != n: 672784a3f53a8f78995d4c8ca22f612a68828bc4838Neal Norwitz raise ValueError("not enough data in stream to read long4") 6735a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum return decode_long(data) 6745a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6755a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumlong4 = ArgumentDescriptor( 6765a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum name="long4", 677fdb8cfab085d0f412411b090796e9f856ee0cab5Tim Peters n=TAKEN_FROM_ARGUMENT4, 6785a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum reader=read_long4, 6795a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum doc="""A binary representation of a long, little-endian. 6805a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6815a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum This first reads four bytes as a signed size (but requires the 6825a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum size to be >= 0), then reads that many bytes and interprets them 6834b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters as a little-endian 2's-complement long. If the size is 0, that's taken 6844b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters as a shortcut for the long 0L, although LONG1 should really be used 6854b23f2b44bdd13758eab6808d6a08b951fbfc4ddTim Peters then instead (and in any case where # of bytes < 256). 6865a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum """) 6875a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6885a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 6898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 6908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Object descriptors. The stack used by the pickle machine holds objects, 6918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# and in the stack_before and stack_after attributes of OpcodeInfo 6928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# descriptors we need names to describe the various types of objects that can 6938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# appear on the stack. 6948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 6958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersclass StackObject(object): 6968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __slots__ = ( 6978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # name of descriptor record, for info only 6988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'name', 6998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # type of object, or tuple of type objects (meaning the object can 7018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # be of any type in the tuple) 7028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'obtype', 7038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # human-readable docs for this kind of stack object; a string 7058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'doc', 7068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ) 7078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters def __init__(self, name, obtype, doc): 7098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(name, str) 7108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.name = name 7118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(obtype, type) or isinstance(obtype, tuple) 7138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if isinstance(obtype, tuple): 7148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for contained in obtype: 7158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(contained, type) 7168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.obtype = obtype 7178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(doc, str) 7198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.doc = doc 7208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 721c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters def __repr__(self): 722c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters return self.name 723c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 7248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyint = StackObject( 7268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='int', 7278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=int, 7288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A short (as opposed to long) Python integer object.") 7298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspylong = StackObject( 7318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='long', 7328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=long, 7338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A long (as opposed to short) Python integer object.") 7348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyinteger_or_bool = StackObject( 7368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='int_or_bool', 7378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=(int, long, bool), 7388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python integer object (short or long), or " 7398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "a Python bool.") 7408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7415a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossumpybool = StackObject( 7425a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum name='bool', 7435a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum obtype=(bool,), 7445a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum doc="A Python bool object.") 7455a2d8f5e9af0cbd513f02eb5576ff497e3693ffeGuido van Rossum 7468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyfloat = StackObject( 7478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='float', 7488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=float, 7498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python float object.") 7508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspystring = StackObject( 7528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='str', 7538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=str, 7548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python string object.") 7558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspyunicode = StackObject( 7578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='unicode', 7588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=unicode, 7598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python Unicode string object.") 7608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspynone = StackObject( 7628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="None", 7638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=type(None), 7648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="The Python None object.") 7658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspytuple = StackObject( 7678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="tuple", 7688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=tuple, 7698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python tuple object.") 7708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspylist = StackObject( 7728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="list", 7738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=list, 7748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python list object.") 7758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterspydict = StackObject( 7778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="dict", 7788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=dict, 7798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="A Python dict object.") 7808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersanyobject = StackObject( 7828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name='any', 7838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=object, 7848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Any kind of object whatsoever.") 7858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersmarkobject = StackObject( 7878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="mark", 7888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=StackObject, 7898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""'The mark' is a unique object. 7908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 7918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Opcodes that operate on a variable number of objects 7928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters generally don't embed the count of objects in the opcode, 7938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters or pull it off the stack. Instead the MARK opcode is used 7948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to push a special marker object on the stack, and then 7958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters some other opcodes grab all the objects from the top of 7968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the stack down to (but not including) the topmost marker 7978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object. 7988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 7998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersstackslice = StackObject( 8018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name="stackslice", 8028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters obtype=StackObject, 8038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""An object representing a contiguous slice of the stack. 8048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is used in conjuction with markobject, to represent all 8068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of the stack following the topmost markobject. For example, 8078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the POP_MARK opcode changes the stack from 8088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters [..., markobject, stackslice] 8108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to 8118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters [...] 8128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters No matter how many object are on the stack after the topmost 8148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markobject, POP_MARK gets rid of all of them (including the 8158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject too). 8168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """) 8178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 8198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Descriptors for pickle opcodes. 8208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersclass OpcodeInfo(object): 8228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __slots__ = ( 8248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # symbolic name of opcode; a string 8258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'name', 8268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the code used in a bytestream to represent the opcode; a 8288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # one-character string 8298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'code', 8308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # If the opcode has an argument embedded in the byte string, an 8328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # instance of ArgumentDescriptor specifying its type. Note that 8338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # arg.reader(s) can be used to read and decode the argument from 8348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the bytestream s, and arg.doc documents the format of the raw 8358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # argument bytes. If the opcode doesn't have an argument embedded 8368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # in the bytestream, arg should be None. 8378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'arg', 8388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # what the stack looks like before this opcode runs; a list 8408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'stack_before', 8418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # what the stack looks like after this opcode runs; a list 8438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'stack_after', 8448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # the protocol number in which this opcode was introduced; an int 8468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'proto', 8478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # human-readable docs for this opcode; a string 8498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'doc', 8508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ) 8518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters def __init__(self, name, code, arg, 8538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before, stack_after, proto, doc): 8548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(name, str) 8558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.name = name 8568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(code, str) 8588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert len(code) == 1 8598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.code = code 8608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert arg is None or isinstance(arg, ArgumentDescriptor) 8628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.arg = arg 8638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(stack_before, list) 8658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for x in stack_before: 8668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(x, StackObject) 8678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.stack_before = stack_before 8688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(stack_after, list) 8708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for x in stack_after: 8718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(x, StackObject) 8728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.stack_after = stack_after 8738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(proto, int) and 0 <= proto <= 2 8758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.proto = proto 8768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert isinstance(doc, str) 8788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters self.doc = doc 8798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersI = OpcodeInfo 8818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersopcodes = [ 8828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell integers. 8848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='INT', 8868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='I', 8878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=decimalnl_short, 8888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 8898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyinteger_or_bool], 8908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 8918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an integer or bool. 8928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is a newline-terminated decimal literal string. 8948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 8958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The intent may have been that this always fit in a short Python int, 8968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters but INT can be generated in pickles written on a 64-bit box that 8978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters require a Python long on a 32-bit box. The difference between this 8988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and LONG then is that INT skips a trailing 'L', and produces a short 8998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters int whenever possible. 9008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Another difference is due to that, when bool was introduced as a 9028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters distinct type in 2.3, builtin names True and False were also added to 9038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, 9048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". 9058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Leading zeroes are never produced for a genuine integer. The 2.3 9068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (and later) unpicklers special-case these and return bool instead; 9078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters earlier unpicklers ignore the leading "0" and return the int. 9088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BININT', 9118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='J', 9128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=int4, 9138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyint], 9158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 9168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a four-byte signed integer. 9178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This handles the full range of Python (short) integers on a 32-bit 9198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters box, directly as binary bytes (1 for the opcode and 4 for the integer). 9208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the integer is non-negative and fits in 1 or 2 bytes, pickling via 9218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters BININT1 or BININT2 saves space. 9228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BININT1', 9258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='K', 9268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint1, 9278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyint], 9298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 9308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a one-byte unsigned integer. 9318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a space optimization for pickling very small non-negative ints, 9338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters in range(256). 9348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BININT2', 9378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='M', 9388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint2, 9398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyint], 9418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 9428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a two-byte unsigned integer. 9438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is a space optimization for pickling small positive ints, in 9458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters range(256, 2**16). Integers in range(256) can also be pickled via 9468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters BININT2, but BININT1 instead saves a byte. 9478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 9488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 949fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='LONG', 950fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='L', 951fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=decimalnl_long, 952fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 953fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pylong], 954fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=0, 955fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Push a long integer. 956fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 957fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters The same as INT, except that the literal ends with 'L', and always 958fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters unpickles to a Python long. There doesn't seem a real purpose to the 959fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters trailing 'L'. 960fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 961fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters Note that LONG takes time quadratic in the number of digits when 962fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters unpickling (this is simply due to the nature of decimal->binary 963fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters conversion). Proto 2 added linear-time (in C; still quadratic-time 964fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters in Python) LONG1 and LONG4 opcodes. 965fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 966fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 967fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name="LONG1", 968fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x8a', 969fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=long1, 970fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 971fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pylong], 972fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 973fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Long integer using one-byte length. 974fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 975fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters A more efficient encoding of a Python long; the long1 encoding 976fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters says it all."""), 977fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 978fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name="LONG4", 979fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x8b', 980fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=long4, 981fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 982fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pylong], 983fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 984fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Long integer using found-byte length. 985fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 986fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters A more efficient encoding of a Python long; the long4 encoding 987fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters says it all."""), 988fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 9898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell strings (8-bit, not Unicode). 9908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='STRING', 9928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='S', 9938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl, 9948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 9958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pystring], 9968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 9978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python string object. 9988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 9998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is a repr-style string, with bracketing quote characters, 10008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and perhaps embedded escapes. The argument extends until the next 10018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters newline character. 10028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINSTRING', 10058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='T', 10068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=string4, 10078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pystring], 10098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 10108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python string object. 10118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are two arguments: the first is a 4-byte little-endian signed int 10138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters giving the number of bytes in the string, and the second is that many 10148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bytes, which are taken literally as the string content. 10158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='SHORT_BINSTRING', 10188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='U', 10198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=string1, 10208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pystring], 10228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 10238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python string object. 10248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are two arguments: the first is a 1-byte unsigned int giving 10268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the number of bytes in the string, and the second is that many bytes, 10278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters which are taken literally as the string content. 10288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell None. 10318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='NONE', 10338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='N', 10348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 10358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pynone], 10378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 10388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push None on the stack."), 10398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1040fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # Ways to spell bools, starting with proto 2. See INT for how this was 1041fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # done before proto 2. 1042fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1043fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='NEWTRUE', 1044fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x88', 1045fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1046fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1047fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pybool], 1048fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1049fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""True. 1050fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1051fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters Push True onto the stack."""), 1052fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1053fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='NEWFALSE', 1054fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x89', 1055fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1056fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1057fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pybool], 1058fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1059fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""True. 1060fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1061fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters Push False onto the stack."""), 1062fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 10638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell Unicode strings. 10648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='UNICODE', 10668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='V', 10678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=unicodestringnl, 10688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyunicode], 10708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, # this may be pure-text, but it's a later addition 10718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python Unicode string object. 10728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is a raw-unicode-escape encoding of a Unicode string, 10748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters and so may contain embedded escape sequences. The argument extends 10758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters until the next newline character. 10768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINUNICODE', 10798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='X', 10808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=unicodestring4, 10818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyunicode], 10838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 10848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a Python Unicode string object. 10858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are two arguments: the first is a 4-byte little-endian signed int 10878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters giving the number of bytes in the string. The second is that many 10888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bytes, and is the UTF-8 encoding of the Unicode string. 10898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 10908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to spell floats. 10928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 10938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='FLOAT', 10948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='F', 10958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=floatnl, 10968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 10978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyfloat], 10988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 10998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Newline-terminated decimal float literal. 11008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The argument is repr(a_float), and in general requires 17 significant 11028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters digits for roundtrip conversion to be an identity (this is so for 11038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters IEEE-754 double precision values, which is what Python float maps to 11048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters on most boxes). 11058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In general, FLOAT cannot be used to transport infinities, NaNs, or 11078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters minus zero across boxes (or even on a single box, if the platform C 11088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters library can't read the strings it produces for such things -- Windows 11098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is like that), but may do less damage than BINFLOAT on boxes with 11108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters greater precision or dynamic range than IEEE-754 double. 11118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINFLOAT', 11148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='G', 11158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=float8, 11168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 11178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pyfloat], 11188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Float stored in binary form, with 8 bytes of data. 11208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This generally requires less than half the space of FLOAT encoding. 11228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In general, BINFLOAT cannot be used to transport infinities, NaNs, or 11238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters minus zero, raises an exception if the exponent exceeds the range of 11248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters an IEEE-754 double, and retains no more than 53 bits of precision (if 11258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters there are more than that, "add a half and chop" rounding is used to 11268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters cut it back to 53 significant bits). 11278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build lists. 11308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='EMPTY_LIST', 11328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code=']', 11338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 11358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push an empty list."), 11388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='APPEND', 11408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='a', 11418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pylist, anyobject], 11438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 11458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Append an object to a list. 11468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pylist anyobject 11488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pylist+[anyobject] 114981098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters 115081098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters although pylist is really extended in-place. 11518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='APPENDS', 11548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='e', 11558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pylist, markobject, stackslice], 11578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Extend a list by a slice of stack objects. 11608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pylist markobject stackslice 11628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pylist+stackslice 116381098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters 116481098ac1c8c88f9f1dcf7511555d2ad1f7124575Tim Peters although pylist is really extended in-place. 11658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='LIST', 11688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='l', 11698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 11718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pylist], 11728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 11738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a list out of the topmost stack slice, after markobject. 11748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters All the stack entries following the topmost markobject are placed into 11768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a single Python list, which single list object replaces all of the 11778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack from the topmost markobject onward. For example, 11788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject 1 2 3 'abc' 11808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... [1, 2, 3, 'abc'] 11818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 11828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build tuples. 11848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='EMPTY_TUPLE', 11868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code=')', 11878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 11898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pytuple], 11908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 11918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push an empty tuple."), 11928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 11938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='TUPLE', 11948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='t', 11958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 11968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 11978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pytuple], 11988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 11998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a tuple out of the topmost stack slice, after markobject. 12008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters All the stack entries following the topmost markobject are placed into 12028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a single Python tuple, which single tuple object replaces all of the 12038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack from the topmost markobject onward. For example, 12048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject 1 2 3 'abc' 12068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... (1, 2, 3, 'abc') 12078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 12088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1209fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='TUPLE1', 1210fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x85', 1211fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1212fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject], 1213fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pytuple], 1214fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1215fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""One-tuple. 1216fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1217fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code pops one value off the stack and pushes a tuple of 1218fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters length 1 whose one item is that value back onto it. IOW: 1219fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1220fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack[-1] = tuple(stack[-1:]) 1221fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1222fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1223fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='TUPLE2', 1224fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x86', 1225fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1226fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject, anyobject], 1227fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pytuple], 1228fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1229fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""One-tuple. 1230fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1231fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code pops two values off the stack and pushes a tuple 1232fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters of length 2 whose items are those values back onto it. IOW: 1233fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1234fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack[-2:] = [tuple(stack[-2:])] 1235fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1236fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1237fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='TUPLE3', 1238fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x87', 1239fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1240fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject, anyobject, anyobject], 1241fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[pytuple], 1242fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1243fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""One-tuple. 1244fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1245fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code pops three values off the stack and pushes a tuple 1246fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters of length 3 whose items are those values back onto it. IOW: 1247fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1248fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack[-3:] = [tuple(stack[-3:])] 1249fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1250fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 12518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build dicts. 12528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='EMPTY_DICT', 12548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='}', 12558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 12578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 12598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push an empty dict."), 12608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='DICT', 12628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='d', 12638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 12658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 12678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a dict out of the topmost stack slice, after markobject. 12688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters All the stack entries following the topmost markobject are placed into 12708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a single Python dict, which single dict object replaces all of the 12718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack from the topmost markobject onward. The stack slice alternates 12728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters key, value, key, value, .... For example, 12738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject 1 2 3 'abc' 12758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... {1: 2, 3: 'abc'} 12768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 12778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='SETITEM', 12798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='s', 12808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pydict, anyobject, anyobject], 12828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 12848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Add a key+value pair to an existing dict. 12858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pydict key value 12878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pydict 12888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters where pydict has been modified via pydict[key] = value. 12908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 12918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 12928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='SETITEMS', 12938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='u', 12948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 12958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[pydict, markobject, stackslice], 12968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[pydict], 12978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 12988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Add an arbitrary number of key+value pairs to an existing dict. 12998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The slice of the stack following the topmost markobject is taken as 13018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters an alternating sequence of keys and values, added to the dict 13028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters immediately under the topmost markobject. Everything at and after the 13038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject is popped, leaving the mutated dict at the top 13048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of the stack. 13058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... pydict markobject key_1 value_1 ... key_n value_n 13078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... pydict 13088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters where pydict has been modified via pydict[key_i] = value_i for i in 13108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1, 2, ..., n, and in that order. 13118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Stack manipulation. 13148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='POP', 13168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='0', 13178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 13198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 13208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Discard the top stack item, shrinking the stack by one item."), 13228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='DUP', 13248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='2', 13258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 13278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject, anyobject], 13288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="Push the top stack item onto the stack again, duplicating it."), 13308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='MARK', 13328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='(', 13338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[markobject], 13368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push markobject onto the stack. 13388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markobject is a unique object, used by other opcodes to identify a 13408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters region of the stack containing a variable number of objects for them 13418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to work on. See markobject.doc for more detail. 13428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='POP_MARK', 13458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='1', 13468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 13478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 13488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 13498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Pop all the stack objects at and above the topmost markobject. 13518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters When an opcode using a variable number of stack objects is done, 13538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters POP_MARK is used to remove those objects, and to remove the markobject 13548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters that delimited their starting position on the stack. 13558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Memo manipulation. There are really only two operations (get and put), 13588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # each in all-text, "short binary", and "long binary" flavors. 13598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='GET', 13618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='g', 13628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=decimalnl_short, 13638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 13658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 13668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Read an object from the memo and push it on the stack. 13678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo object to push is given by the newline-teriminated 13698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters decimal string following. BINGET and LONG_BINGET are space-optimized 13708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters versions. 13718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINGET', 13748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='h', 13758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint1, 13768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 13788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 13798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Read an object from the memo and push it on the stack. 13808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo object to push is given by the 1-byte unsigned 13828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters integer following. 13838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='LONG_BINGET', 13868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='j', 13878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=int4, 13888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 13898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 13908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 13918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Read an object from the memo and push it on the stack. 13928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo object to push is given by the 4-byte signed 13948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters little-endian integer following. 13958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 13968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 13978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='PUT', 13988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='p', 13998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=decimalnl_short, 14008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 14018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 14028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 14038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Store the stack top into the memo. The stack is not popped. 14048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo location to write into is given by the newline- 14068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters terminated decimal string following. BINPUT and LONG_BINPUT are 14078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters space-optimized versions. 14088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINPUT', 14118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='q', 14128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=uint1, 14138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 14148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 14158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 14168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Store the stack top into the memo. The stack is not popped. 14178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo location to write into is given by the 1-byte 14198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters unsigned integer following. 14208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='LONG_BINPUT', 14238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='r', 14248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=int4, 14258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 14268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 14278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 14288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Store the stack top into the memo. The stack is not popped. 14298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The index of the memo location to write into is given by the 4-byte 14318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters signed little-endian integer following. 14328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1434fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # Access the extension registry (predefined objects). Akin to the GET 1435fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters # family. 1436fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1437fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='EXT1', 1438fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x82', 1439fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=uint1, 1440fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1441fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1442fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1443fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Extension code. 1444fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1445fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters This code and the similar EXT2 and EXT4 allow using a registry 1446fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters of popular objects that are pickled by name, typically classes. 1447fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters It is envisioned that through a global negotiation and 1448fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters registration process, third parties can set up a mapping between 1449fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters ints and object names. 1450fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1451fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters In order to guarantee pickle interchangeability, the extension 1452fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code registry ought to be global, although a range of codes may 1453fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters be reserved for private use. 1454fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1455fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters EXT1 has a 1-byte integer argument. This is used to index into the 1456fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters extension registry, and the object at that index is pushed on the stack. 1457fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1458fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1459fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='EXT2', 1460fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x83', 1461fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=uint2, 1462fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1463fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1464fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1465fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Extension code. 1466fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1467fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters See EXT1. EXT2 has a two-byte integer argument. 1468fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1469fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1470fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='EXT4', 1471fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x84', 1472fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=int4, 1473fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1474fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1475fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1476fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Extension code. 1477fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1478fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters See EXT1. EXT4 has a four-byte integer argument. 1479fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1480fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 14818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Push a class object, or module function, on the stack, via its module 14828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # and name. 14838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='GLOBAL', 14858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='c', 14868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl_noescape_pair, 14878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 14888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 14898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 14908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push a global object (module.attr) on the stack. 14918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Two newline-terminated strings follow the GLOBAL opcode. The first is 14938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters taken as a module name, and the second as a class name. The class 14948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object module.class is pushed on the stack. More accurately, the 14958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object returned by self.find_class(module, class) is pushed on the 14968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack, so unpickling subclasses can override this form of lookup. 14978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 14988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 14998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to build objects of classes pickle doesn't know about directly 15008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # (user-defined classes). I despair of documenting this accurately 15018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # and comprehensibly -- you really have to read the pickle code to 15028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # find all the special cases. 15038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='REDUCE', 15058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='R', 15068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 15078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject, anyobject], 15088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 15098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 15108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an object built from a callable and an argument tuple. 15118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The opcode is named to remind of the __reduce__() method. 15138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... callable pytuple 15158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... callable(*pytuple) 15168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The callable and the argument tuple are the first two items returned 15188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters by a __reduce__ method. Applying the callable to the argtuple is 15198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters supposed to reproduce the original object, or at least get it started. 15208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the __reduce__ method returns a 3-tuple, the last component is an 15218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argument to be passed to the object's __setstate__, and then the REDUCE 15228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode is followed by code to create setstate's argument, and then a 15238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters BUILD opcode to apply __setstate__ to that argument. 15248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters There are lots of special cases here. The argtuple can be None, in 15268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters which case callable.__basicnew__() is called instead to produce the 15278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object to be pushed on the stack. This appears to be a trick unique 15288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to ExtensionClasses, and is deprecated regardless. 15298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If type(callable) is not ClassType, REDUCE complains unless the 15318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters callable has been registered with the copy_reg module's 15328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters safe_constructors dict, or the callable has a magic 15338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters '__safe_for_unpickling__' attribute with a true value. I'm not sure 15348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters why it does this, but I've sure seen this complaint often enough when 15358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I didn't want to <wink>. 15368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 15378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BUILD', 15398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='b', 15408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 15418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject, anyobject], 15428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 15438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 15448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Finish building an object, via __setstate__ or dict update. 15458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... anyobject argument 15478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... anyobject 15488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters where anyobject may have been mutated, as follows: 15508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the object has a __setstate__ method, 15528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters anyobject.__setstate__(argument) 15548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is called. 15568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Else the argument must be a dict, the object must have a __dict__, and 15588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the object is updated via 15598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters anyobject.__dict__.update(argument) 15618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This may raise RuntimeError in restricted execution mode (which 15638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters disallows access to __dict__ directly); in that case, the object 15648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is updated instead via 15658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for k, v in argument.items(): 15678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters anyobject[k] = v 15688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 15698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='INST', 15718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='i', 15728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl_noescape_pair, 15738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, stackslice], 15748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 15758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 15768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a class instance. 15778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is the protocol 0 version of protocol 1's OBJ opcode. 15798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters INST is followed by two newline-terminated strings, giving a 15808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters module and class name, just as for the GLOBAL opcode (and see 15818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters GLOBAL for more details about that). self.find_class(module, name) 15828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is used to get a class object. 15838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters In addition, all the objects on the stack following the topmost 15858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markobject are gathered into a tuple and popped (along with the 15868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject), just as for the TUPLE opcode. 15878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Now it gets complicated. If all of these are true: 15898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters + The argtuple is empty (markobject was at the top of the stack 15918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters at the start). 15928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters + It's an old-style class object (the type of the class object is 15948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ClassType). 15958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters + The class object does not have a __getinitargs__ attribute. 15978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 15988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters then we want to create an old-style class instance without invoking 15998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters its __init__() method (pickle has waffled on this over the years; not 16008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters calling __init__() is current wisdom). In this case, an instance of 16018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters an old-style dummy class is created, and then we try to rebind its 16028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters __class__ attribute to the desired class object. If this succeeds, 16038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the new instance object is pushed on the stack, and we're done. In 16048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters restricted execution mode it can fail (assignment to __class__ is 16058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters disallowed), and I'm not really sure what happens then -- it looks 16068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters like the code ends up calling the class object's __init__ anyway, 16078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters via falling into the next case. 16088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Else (the argtuple is not empty, it's not an old-style class object, 16108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters or the class object does have a __getinitargs__ attribute), the code 16118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters first insists that the class object have a __safe_for_unpickling__ 16128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, 16138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters it doesn't matter whether this attribute has a true or false value, it 1614ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum only matters whether it exists (XXX this is a bug; cPickle 1615ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum requires the attribute to be true). If __safe_for_unpickling__ 1616ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum doesn't exist, UnpicklingError is raised. 16178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Else (the class object does have a __safe_for_unpickling__ attr), 16198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the class object obtained from INST's arguments is applied to the 16208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argtuple obtained from the stack, and the resulting instance object 16218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is pushed on the stack. 16222b93c4c70820d0314e8e297c2817d9e03d73df62Tim Peters 16232b93c4c70820d0314e8e297c2817d9e03d73df62Tim Peters NOTE: checks for __safe_for_unpickling__ went away in Python 2.3. 16248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 16258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='OBJ', 16278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='o', 16288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 16298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[markobject, anyobject, stackslice], 16308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 16318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 16328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Build a class instance. 16338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters This is the protocol 1 version of protocol 0's INST opcode, and is 16358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters very much like it. The major difference is that the class object 16368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is taken off the stack, allowing it to be retrieved from the memo 16378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters repeatedly if several instances of the same class are created. This 16388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters can be much more efficient (in both time and space) than repeatedly 16398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters embedding the module and class names in INST opcodes. 16408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Unlike INST, OBJ takes no arguments from the opcode stream. Instead 16428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the class object is taken off the stack, immediately above the 16438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters topmost markobject: 16448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack before: ... markobject classobject stackslice 16468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Stack after: ... new_instance_object 16478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters As for INST, the remainder of the stack above the markobject is 16498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters gathered into an argument tuple, and then the logic seems identical, 1650ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum except that no __safe_for_unpickling__ check is done (XXX this is 1651ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum a bug; cPickle does test __safe_for_unpickling__). See INST for 1652ecb1104342af0e2dca191f7666c60d5ca65069a8Guido van Rossum the gory details. 16532b93c4c70820d0314e8e297c2817d9e03d73df62Tim Peters 16542b93c4c70820d0314e8e297c2817d9e03d73df62Tim Peters NOTE: In Python 2.3, INST and OBJ are identical except for how they 16552b93c4c70820d0314e8e297c2817d9e03d73df62Tim Peters get the class object. That was always the intent; the implementations 16562b93c4c70820d0314e8e297c2817d9e03d73df62Tim Peters had diverged for accidental reasons. 16578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 16588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1659fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='NEWOBJ', 1660fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x81', 1661fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=None, 1662fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[anyobject, anyobject], 1663fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[anyobject], 1664fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1665fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Build an object instance. 1666fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1667fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters The stack before should be thought of as containing a class 1668fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters object followed by an argument tuple (the tuple being the stack 1669fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters top). Call these cls and args. They are popped off the stack, 1670fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters and the value returned by cls.__new__(cls, *args) is pushed back 1671fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters onto the stack. 1672fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1673fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 16748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Machine control. 16758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1676fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters I(name='PROTO', 1677fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters code='\x80', 1678fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters arg=uint1, 1679fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_before=[], 1680fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters stack_after=[], 1681fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters proto=2, 1682fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters doc="""Protocol version indicator. 1683fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 1684fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters For protocol 2 and above, a pickle must start with this opcode. 1685fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters The argument is the protocol version, an int in range(2, 256). 1686fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters """), 1687fdc03462b3e0796ae6474da6f0f9844773d1da8fTim Peters 16888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='STOP', 16898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='.', 16908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 16918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 16928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[], 16938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 16948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Stop the unpickling machine. 16958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 16968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Every pickle ends with this opcode. The object at the top of the stack 16978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is popped, and that's the result of unpickling. The stack should be 16988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters empty then. 16998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 17008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Ways to deal with persistent IDs. 17028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='PERSID', 17048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='P', 17058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=stringnl_noescape, 17068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[], 17078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 17088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=0, 17098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an object identified by a persistent ID. 17108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The pickle module doesn't define what a persistent ID means. PERSID's 17128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters argument is a newline-terminated str-style (no embedded escapes, no 17138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters bracketing quote characters) string, which *is* "the persistent ID". 17148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters The unpickler passes this string to self.persistent_load(). Whatever 17158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters object that returns is pushed on the stack. There is no implementation 17168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of persistent_load() in Python's unpickler: it must be supplied by an 17178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters unpickler subclass. 17188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 17198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters I(name='BINPERSID', 17218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code='Q', 17228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg=None, 17238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_before=[anyobject], 17248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stack_after=[anyobject], 17258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters proto=1, 17268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters doc="""Push an object identified by a persistent ID. 17278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Like PERSID, except the persistent ID is popped off the stack (instead 17298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters of being a string embedded in the opcode bytestream). The persistent 17308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters ID is passed to self.persistent_load(), and whatever object that 17318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters returns is pushed on the stack. See PERSID for more detail. 17328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """), 17338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters] 17348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdel I 17358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Verify uniqueness of .name and .code members. 17378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersname2i = {} 17388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscode2i = {} 17398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfor i, d in enumerate(opcodes): 17418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if d.name in name2i: 17428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("repeated name %r at indices %d and %d" % 17438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (d.name, name2i[d.name], i)) 17448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if d.code in code2i: 17458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("repeated code %r at indices %d and %d" % 17468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (d.code, code2i[d.code], i)) 17478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name2i[d.name] = i 17498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code2i[d.code] = i 17508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdel name2i, code2i, i, d 17528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 17548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Build a code2op dict, mapping opcode characters to OpcodeInfo records. 17558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# Also ensure we've got the same stuff as pickle.py, although the 17568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# introspection here is dicey. 17578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peterscode2op = {} 17598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersfor d in opcodes: 17608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code2op[d.code] = d 17618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdel d 17628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef assure_pickle_consistency(verbose=False): 17648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters import pickle, re 17658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 17668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters copy = code2op.copy() 17678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for name in pickle.__all__: 17688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not re.match("[A-Z][A-Z0-9_]+$", name): 17698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if verbose: 17708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print "skipping %r: it doesn't look like an opcode name" % name 17718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters continue 17728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters picklecode = getattr(pickle, name) 17738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if not isinstance(picklecode, str) or len(picklecode) != 1: 17748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if verbose: 17758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print ("skipping %r: value %r doesn't look like a pickle " 17768ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "code" % (name, picklecode)) 17778ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters continue 17788ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if picklecode in copy: 17798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if verbose: 17808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print "checking name %r w/ code %r for consistency" % ( 17818ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name, picklecode) 17828ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters d = copy[picklecode] 17838ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if d.name != name: 17848ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("for pickle code %r, pickle.py uses name %r " 17858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "but we're using name %r" % (picklecode, 17868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters name, 17878ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters d.name)) 17888ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # Forget this one. Any left over in copy at the end are a problem 17898ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # of a different kind. 17908ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters del copy[picklecode] 17918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 17928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("pickle.py appears to have a pickle opcode with " 17938ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters "name %r and code %r, but we don't" % 17948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters (name, picklecode)) 17958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if copy: 17968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"] 17978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for code, d in copy.items(): 17988ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters msg.append(" name %r with code %r" % (d.name, code)) 17998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("\n".join(msg)) 18008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersassure_pickle_consistency() 1802c0c12b57070a5b494662bebc418e3958bf5bdbeeTim Petersdel assure_pickle_consistency 18038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 18058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# A pickle opcode generator. 18068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef genops(pickle): 1808a72ded9bc81bd278bf119e92feaeaea2aa16f02dGuido van Rossum """Generate all the opcodes in a pickle. 18098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'pickle' is a file-like object, or string, containing the pickle. 18118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Each opcode in the pickle is generated, from the current pickle position, 18138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters stopping after a STOP opcode is delivered. A triple is generated for 18148ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters each opcode: 18158ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18168ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode, arg, pos 18178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18188ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode is an OpcodeInfo record, describing the current opcode. 18198ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18208ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the opcode has an argument embedded in the pickle, arg is its decoded 18218ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters value, as a Python object. If the opcode doesn't have an argument, arg 18228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters is None. 18238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters If the pickle has a tell() method, pos was the value of pickle.tell() 18258ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters before reading the current opcode. If the pickle is a string object, 18268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters it's wrapped in a StringIO object, and the latter's tell() result is 18278ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters used. Else (the pickle doesn't have a tell(), and it's not obvious how 18288ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters to query its current position) pos is None. 18298ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 18308ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18318ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters import cStringIO as StringIO 18328ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18338ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if isinstance(pickle, str): 18348ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pickle = StringIO.StringIO(pickle) 18358ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if hasattr(pickle, "tell"): 18378ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters getpos = pickle.tell 18388ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 18398ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters getpos = lambda: None 18408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18418ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters while True: 18428ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pos = getpos() 18438ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code = pickle.read(1) 18448ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters opcode = code2op.get(code) 18458ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if opcode is None: 18468ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if code == "": 18478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("pickle exhausted before seeing STOP") 18488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 18498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters raise ValueError("at position %s, opcode %r unknown" % ( 18508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pos is None and "<unknown>" or pos, 18518ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters code)) 18528ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if opcode.arg is None: 18538ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg = None 18548ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters else: 18558ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters arg = opcode.arg.reader(pickle) 18568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters yield opcode, arg, pos 18578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if code == '.': 18588ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters assert opcode.name == 'STOP' 18598ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters break 18608ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18618ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters############################################################################## 18628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters# A symbolic pickle disassembler. 18638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef dis(pickle, out=None, indentlevel=4): 18658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """Produce a symbolic disassembly of a pickle. 18668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 'pickle' is a file-like object, or string, containing a (at least one) 18688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters pickle. The pickle is disassembled from the current position, through 18698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters the first STOP opcode encountered. 18708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Optional arg 'out' is a file-like object to which the disassembly is 18728ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters printed. It defaults to sys.stdout. 18738ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 18748ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters Optional arg indentlevel is the number of blanks by which to indent 18758ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters a new MARK level. It defaults to 4. 1876c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1877c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters In addition to printing the disassembly, some sanity checks are made: 1878c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1879c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + All embedded opcode arguments "make sense". 1880c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1881c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + Explicit and implicit pop operations have enough items on the stack. 1882c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1883c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + When an opcode implicitly refers to a markobject, a markobject is 1884c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters actually on the stack. 1885c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1886c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + A memo entry isn't referenced before it's defined. 1887c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1888c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + The markobject isn't stored in the memo. 1889c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1890c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters + A memo entry isn't redefined. 18918ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters """ 18928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1893c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Most of the hair here is for sanity checks, but most of it is needed 1894c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # anyway to detect when a protocol 0 POP takes a MARK off the stack 1895c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # (which in turn is needed to indent MARK blocks correctly). 1896c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1897c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack = [] # crude emulation of unpickler stack 1898c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters memo = {} # crude emulation of unpicker memo 1899c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters maxproto = -1 # max protocol number seen 1900c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markstack = [] # bytecode positions of MARK opcodes 19018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters indentchunk = ' ' * indentlevel 1902c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = None 19038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters for opcode, arg, pos in genops(pickle): 19048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if pos is not None: 19058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print >> out, "%5d:" % pos, 19068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1907d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters line = "%-4s %s%s" % (repr(opcode.code)[1:-1], 1908d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters indentchunk * len(markstack), 1909d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters opcode.name) 19108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1911c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters maxproto = max(maxproto, opcode.proto) 1912c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters before = opcode.stack_before # don't mutate 1913c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters after = opcode.stack_after # don't mutate 191443277d64eb5e008e59009bd03887d66c134445efTim Peters numtopop = len(before) 191543277d64eb5e008e59009bd03887d66c134445efTim Peters 191643277d64eb5e008e59009bd03887d66c134445efTim Peters # See whether a MARK should be popped. 19178ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markmsg = None 1918c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markobject in before or (opcode.name == "POP" and 1919c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack and 1920c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack[-1] is markobject): 1921c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters assert markobject not in after 192243277d64eb5e008e59009bd03887d66c134445efTim Peters if __debug__: 192343277d64eb5e008e59009bd03887d66c134445efTim Peters if markobject in before: 192443277d64eb5e008e59009bd03887d66c134445efTim Peters assert before[-1] is stackslice 1925c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markstack: 1926c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markpos = markstack.pop() 1927c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markpos is None: 1928c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markmsg = "(MARK at unknown opcode offset)" 1929c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1930c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters markmsg = "(MARK at %d)" % markpos 1931c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Pop everything at and after the topmost markobject. 1932c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters while stack[-1] is not markobject: 1933c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack.pop() 1934c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack.pop() 193543277d64eb5e008e59009bd03887d66c134445efTim Peters # Stop later code from popping too much. 1936c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters try: 193743277d64eb5e008e59009bd03887d66c134445efTim Peters numtopop = before.index(markobject) 1938c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters except ValueError: 1939c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters assert opcode.name == "POP" 194043277d64eb5e008e59009bd03887d66c134445efTim Peters numtopop = 0 1941c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1942c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = markmsg = "no MARK exists on stack" 1943c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1944c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Check for correct memo usage. 1945c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"): 194643277d64eb5e008e59009bd03887d66c134445efTim Peters assert arg is not None 1947c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if arg in memo: 1948c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "memo key %r already defined" % arg 1949c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters elif not stack: 1950c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "stack is empty -- can't store into memo" 1951c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters elif stack[-1] is markobject: 1952c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "can't store markobject in the memo" 1953c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1954c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters memo[arg] = stack[-1] 1955c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1956c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters elif opcode.name in ("GET", "BINGET", "LONG_BINGET"): 1957c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if arg in memo: 1958c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters assert len(after) == 1 1959c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters after = [memo[arg]] # for better stack emulation 1960c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters else: 1961c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters errormsg = "memo key %r has never been stored into" % arg 19628ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 19638ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if arg is not None or markmsg: 19648ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters # make a mild effort to align arguments 19658ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters line += ' ' * (10 - len(opcode.name)) 19668ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if arg is not None: 19678ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters line += ' ' + repr(arg) 19688ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters if markmsg: 19698ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters line += ' ' + markmsg 19708ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters print >> out, line 19718ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1972c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if errormsg: 1973c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Note that we delayed complaining until the offending opcode 1974c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # was printed. 1975c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters raise ValueError(errormsg) 1976c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1977c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters # Emulate the stack effects. 197843277d64eb5e008e59009bd03887d66c134445efTim Peters if len(stack) < numtopop: 197943277d64eb5e008e59009bd03887d66c134445efTim Peters raise ValueError("tries to pop %d items from stack with " 198043277d64eb5e008e59009bd03887d66c134445efTim Peters "only %d items" % (numtopop, len(stack))) 198143277d64eb5e008e59009bd03887d66c134445efTim Peters if numtopop: 198243277d64eb5e008e59009bd03887d66c134445efTim Peters del stack[-numtopop:] 1983c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if markobject in after: 198443277d64eb5e008e59009bd03887d66c134445efTim Peters assert markobject not in before 19858ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters markstack.append(pos) 19868ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 1987c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters stack.extend(after) 1988c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 1989c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters print >> out, "highest protocol among opcodes =", maxproto 1990c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters if stack: 1991c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters raise ValueError("stack not empty after STOP: %r" % stack) 19928ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 199303e35327f26d353db126a8f840a8890b3593f18aGuido van Rossum_dis_test = r""" 19948ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> import pickle 19958ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> x = [1, 2, (3, 4), {'abc': u"def"}] 1996570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> pkl = pickle.dumps(x, 0) 1997570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> dis(pkl) 1998d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 1999d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: l LIST (MARK at 0) 2000d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: p PUT 0 2001d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: I INT 1 2002d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: a APPEND 2003d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: I INT 2 2004d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 12: a APPEND 2005d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 13: ( MARK 2006d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: I INT 3 2007d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 17: I INT 4 2008d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 20: t TUPLE (MARK at 13) 2009d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 21: p PUT 1 2010d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 24: a APPEND 2011d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 25: ( MARK 2012d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 26: d DICT (MARK at 25) 2013d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 27: p PUT 2 2014d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 30: S STRING 'abc' 2015d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 37: p PUT 3 2016d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 40: V UNICODE u'def' 2017d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 45: p PUT 4 2018d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 48: s SETITEM 2019d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 49: a APPEND 2020d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 50: . STOP 2021c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 20228ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20238ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTry again with a "binary" pickle. 20248ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2025570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> pkl = pickle.dumps(x, 1) 2026570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum>>> dis(pkl) 2027d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ] EMPTY_LIST 2028d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: q BINPUT 0 2029d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: ( MARK 2030d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: K BININT1 1 2031d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: K BININT1 2 2032d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: ( MARK 2033d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: K BININT1 3 2034d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: K BININT1 4 2035d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 13: t TUPLE (MARK at 8) 2036d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: q BINPUT 1 2037d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 16: } EMPTY_DICT 2038d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 17: q BINPUT 2 2039d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 19: U SHORT_BINSTRING 'abc' 2040d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 24: q BINPUT 3 2041d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 26: X BINUNICODE u'def' 2042d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 34: q BINPUT 4 2043d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 36: s SETITEM 2044d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 37: e APPENDS (MARK at 3) 2045d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 38: . STOP 2046c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 20478ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20488ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersExercise the INST/OBJ/BUILD family. 20498ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20508ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> import random 2051f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(random.random, 0)) 2052d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: c GLOBAL 'random random' 2053d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 15: p PUT 0 2054d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 18: . STOP 2055c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 20568ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> x = [pickle.PicklingError()] * 2 2058f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(x, 0)) 2059d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2060d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: l LIST (MARK at 0) 2061d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: p PUT 0 2062d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: ( MARK 2063d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: i INST 'pickle PicklingError' (MARK at 5) 2064d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 28: p PUT 1 2065d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 31: ( MARK 2066d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 32: d DICT (MARK at 31) 2067d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 33: p PUT 2 2068d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 36: S STRING 'args' 2069d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 44: p PUT 3 2070d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 47: ( MARK 2071d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 48: t TUPLE (MARK at 47) 2072d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 49: s SETITEM 2073d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 50: b BUILD 2074d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 51: a APPEND 2075d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 52: g GET 1 2076d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 55: a APPEND 2077d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 56: . STOP 2078c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 20798ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 20808ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> dis(pickle.dumps(x, 1)) 2081d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ] EMPTY_LIST 2082d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: q BINPUT 0 2083d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: ( MARK 2084d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: ( MARK 2085d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: c GLOBAL 'pickle PicklingError' 2086d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 27: q BINPUT 1 2087d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 29: o OBJ (MARK at 4) 2088d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 30: q BINPUT 2 2089d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 32: } EMPTY_DICT 2090d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 33: q BINPUT 3 2091d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 35: U SHORT_BINSTRING 'args' 2092d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 41: q BINPUT 4 2093d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 43: ) EMPTY_TUPLE 2094d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 44: s SETITEM 2095d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 45: b BUILD 2096d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 46: h BINGET 2 2097d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 48: e APPENDS (MARK at 3) 2098d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 49: . STOP 2099c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 21008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 21018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTry "the canonical" recursive-object test. 21028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 21038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L = [] 21048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> T = L, 21058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L.append(T) 21068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L[0] is T 21078ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 21088ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> T[0] is L 21098ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 21108ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> L[0][0] is L 21118ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 21128ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> T[0][0] is T 21138ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim PetersTrue 2114f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(L, 0)) 2115d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2116d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: l LIST (MARK at 0) 2117d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: p PUT 0 2118d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: ( MARK 2119d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: g GET 0 2120d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: t TUPLE (MARK at 5) 2121d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: p PUT 1 2122d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 13: a APPEND 2123d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: . STOP 2124c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 2125c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 21268ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> dis(pickle.dumps(L, 1)) 2127d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ] EMPTY_LIST 2128d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: q BINPUT 0 2129d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: ( MARK 2130d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: h BINGET 0 2131d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: t TUPLE (MARK at 3) 2132d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: q BINPUT 1 2133d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 9: a APPEND 2134d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: . STOP 2135c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 21368ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2137c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim PetersNote that, in the protocol 0 pickle of the recursive tuple, the disassembler 2138c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershas to emulate the stack in order to realize that the POP opcode at 16 gets 2139c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petersrid of the MARK at 0. 21408ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2141f29d3d6011e41b40282994375454f2020a429d79Guido van Rossum>>> dis(pickle.dumps(T, 0)) 2142d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2143d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: ( MARK 2144d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: l LIST (MARK at 1) 2145d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: p PUT 0 2146d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 6: ( MARK 2147d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: g GET 0 2148d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: t TUPLE (MARK at 6) 2149d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: p PUT 1 2150d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: a APPEND 2151d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 15: 0 POP 2152c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 16: 0 POP (MARK at 0) 2153c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 17: g GET 1 2154c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 20: . STOP 2155c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 0 2156c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Peters 21578ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters>>> dis(pickle.dumps(T, 1)) 2158d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: ( MARK 2159d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 1: ] EMPTY_LIST 2160d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: q BINPUT 0 2161d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 4: ( MARK 2162d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: h BINGET 0 2163d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: t TUPLE (MARK at 4) 2164d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: q BINPUT 1 2165d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: a APPEND 2166d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: 1 POP_MARK (MARK at 0) 2167d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 12: h BINGET 1 2168d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: . STOP 2169c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 1 2170d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2171d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim PetersTry protocol 2. 2172d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2173d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters>>> dis(pickle.dumps(L, 2)) 2174d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: \x80 PROTO 2 2175d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: ] EMPTY_LIST 2176d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: q BINPUT 0 2177d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: h BINGET 0 2178d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: \x85 TUPLE1 2179d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: q BINPUT 1 2180d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: a APPEND 2181d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: . STOP 2182c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 2 2183d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2184d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters>>> dis(pickle.dumps(T, 2)) 2185d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 0: \x80 PROTO 2 2186d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 2: ] EMPTY_LIST 2187d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 3: q BINPUT 0 2188d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 5: h BINGET 0 2189d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 7: \x85 TUPLE1 2190d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 8: q BINPUT 1 2191d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 10: a APPEND 2192d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 11: 0 POP 2193d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 12: h BINGET 1 2194d0f7c86a20e9cef8cbf2c8fb676fcc8d8a7000b4Tim Peters 14: . STOP 2195c1c2b3e0e2af6eb05346b4908f4c4d56618fb7b6Tim Petershighest protocol among opcodes = 2 21968ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters""" 21978ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 2198570283584af6a9aff47d2341d6154055572aaff5Guido van Rossum__test__ = {'disassembler_test': _dis_test, 21998ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters } 22008ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 22018ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersdef _test(): 22028ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters import doctest 22038ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters return doctest.testmod() 22048ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters 22058ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Petersif __name__ == "__main__": 22068ecfc8ef9d6ffe0d9c732a438cb36e1e11480a19Tim Peters _test() 2207