15821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# Protocol Buffers - Google's data interchange format
25821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# Copyright 2008 Google Inc.  All rights reserved.
35821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# http://code.google.com/p/protobuf/
45821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#
55821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# Redistribution and use in source and binary forms, with or without
65821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# modification, are permitted provided that the following conditions are
75821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# met:
85821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#
95821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#     * Redistributions of source code must retain the above copyright
105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# notice, this list of conditions and the following disclaimer.
115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#     * Redistributions in binary form must reproduce the above
125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# copyright notice, this list of conditions and the following disclaimer
135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# in the documentation and/or other materials provided with the
145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# distribution.
155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#     * Neither the name of Google Inc. nor the names of its
165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# contributors may be used to endorse or promote products derived from
175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# this software without specific prior written permission.
185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#
195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)"""Code for decoding protocol buffer primitives.
325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)This code is very similar to encoder.py -- read the docs for that module first.
345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)A "decoder" is a function with the signature:
365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  Decode(buffer, pos, end, message, field_dict)
375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)The arguments are:
385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  buffer:     The string containing the encoded message.
395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  pos:        The current position in the string.
405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  end:        The position in the string where the current message ends.  May be
415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)              less than len(buffer) if we're reading a sub-message.
425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  message:    The message object into which we're parsing.
435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  field_dict: message._fields (avoids a hashtable lookup).
445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)The decoder reads the field and stores it into field_dict, returning the new
455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)buffer position.  A decoder for a repeated field may proactively decode all of
465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)the elements of that field, if they appear consecutively.
475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Note that decoders may throw any of the following:
495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  IndexError:  Indicates a truncated message.
505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  struct.error:  Unpacking of a fixed-width field failed.
515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  message.DecodeError:  Other errors.
525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Decoders are expected to raise an exception if they are called with pos > end.
545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)This allows callers to be lax about bounds checking:  it's fineto read past
555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)"end" as long as you are sure that someone else will notice and throw an
565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)exception later on.
575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Something up the call stack is expected to catch IndexError and struct.error
595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)and convert them to message.DecodeError.
605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Decoders are constructed using decoder constructors with the signature:
625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  MakeDecoder(field_number, is_repeated, is_packed, key, new_default)
635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)The arguments are:
645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  field_number:  The field number of the field we want to decode.
655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  is_repeated:   Is the field a repeated field? (bool)
665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  is_packed:     Is the field a packed field? (bool)
675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  key:           The key to use when looking up the field within field_dict.
685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                 (This is actually the FieldDescriptor but nothing in this
695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                 file should depend on that.)
705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  new_default:   A function which takes a message object as a parameter and
715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                 returns a new instance of the default value for this field.
725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                 (This is called for repeated fields and sub-messages, when an
735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                 instance does not already exist.)
745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)As with encoders, we define a decoder constructor for every type of field.
765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Then, for every field of every message class we construct an actual decoder.
775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)That decoder goes into a dict indexed by tag, so when we decode a message
785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)we repeatedly read a tag, look up the corresponding decoder, and invoke it.
795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)"""
805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)__author__ = 'kenton@google.com (Kenton Varda)'
825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)import struct
845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)from google.protobuf.internal import encoder
855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)from google.protobuf.internal import wire_format
865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)from google.protobuf import message
875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# This will overflow and thus become IEEE-754 "infinity".  We would use
905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# "float('inf')" but it doesn't work on Windows pre-Python-2.6.
915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_POS_INF = 1e10000
925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_NEG_INF = -_POS_INF
935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_NAN = _POS_INF * 0
945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# This is not for optimization, but rather to avoid conflicts with local
975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# variables named "message".
985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_DecodeError = message.DecodeError
995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _VarintDecoder(mask):
1025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Return an encoder for a basic varint value (does not include tag).
1035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  Decoded values will be bitwise-anded with the given mask before being
1055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  returned, e.g. to limit them to 32 bits.  The returned decoder does not
1065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  take the usual "end" parameter -- the caller is expected to do bounds checking
1075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  after the fact (often the caller can defer such checking until later).  The
1085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  decoder returns a (value, new_pos) pair.
1095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
1105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_ord = ord
1125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def DecodeVarint(buffer, pos):
1135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    result = 0
1145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    shift = 0
1155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    while 1:
1165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      b = local_ord(buffer[pos])
1175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      result |= ((b & 0x7f) << shift)
1185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      pos += 1
1195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if not (b & 0x80):
1205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        result &= mask
1215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        return (result, pos)
1225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      shift += 7
1235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if shift >= 64:
1245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Too many bytes when decoding varint.')
1255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return DecodeVarint
1265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _SignedVarintDecoder(mask):
1295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Like _VarintDecoder() but decodes signed values."""
1305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_ord = ord
1325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def DecodeVarint(buffer, pos):
1335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    result = 0
1345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    shift = 0
1355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    while 1:
1365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      b = local_ord(buffer[pos])
1375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      result |= ((b & 0x7f) << shift)
1385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      pos += 1
1395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if not (b & 0x80):
1405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if result > 0x7fffffffffffffff:
1415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          result -= (1 << 64)
1425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          result |= ~mask
1435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        else:
1445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          result &= mask
1455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        return (result, pos)
1465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      shift += 7
1475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if shift >= 64:
1485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Too many bytes when decoding varint.')
1495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return DecodeVarint
1505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_DecodeVarint = _VarintDecoder((1 << 64) - 1)
1535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_DecodeSignedVarint = _SignedVarintDecoder((1 << 64) - 1)
1545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# Use these versions for values which must be limited to 32 bits.
1565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_DecodeVarint32 = _VarintDecoder((1 << 32) - 1)
1575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)_DecodeSignedVarint32 = _SignedVarintDecoder((1 << 32) - 1)
1585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def ReadTag(buffer, pos):
1615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Read a tag from the buffer, and return a (tag_bytes, new_pos) tuple.
1625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  We return the raw bytes of the tag rather than decoding them.  The raw
1645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  bytes can then be used to look up the proper decoder.  This effectively allows
1655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  us to trade some work that would be done in pure-python (decoding a varint)
1665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  for work that is done in C (searching for a byte string in a hash table).
1675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  In a low-level language it would be much cheaper to decode the varint and
1685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  use that, but not in Python.
1695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
1705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  start = pos
1725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  while ord(buffer[pos]) & 0x80:
1735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    pos += 1
1745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  pos += 1
1755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return (buffer[start:pos], pos)
1765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# --------------------------------------------------------------------
1795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _SimpleDecoder(wire_type, decode_value):
1825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Return a constructor for a decoder for fields of a particular type.
1835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  Args:
1855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      wire_type:  The field's wire type.
1865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      decode_value:  A function which decodes an individual value, e.g.
1875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        _DecodeVarint()
1885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
1895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
1905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def SpecificDecoder(field_number, is_repeated, is_packed, key, new_default):
1915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if is_packed:
1925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      local_DecodeVarint = _DecodeVarint
1935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      def DecodePackedField(buffer, pos, end, message, field_dict):
1945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.get(key)
1955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if value is None:
1965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          value = field_dict.setdefault(key, new_default(message))
1975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        (endpoint, pos) = local_DecodeVarint(buffer, pos)
1985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        endpoint += pos
1995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if endpoint > end:
2005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Truncated message.')
2015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        while pos < endpoint:
2025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          (element, pos) = decode_value(buffer, pos)
2035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          value.append(element)
2045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if pos > endpoint:
2055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          del value[-1]   # Discard corrupt value.
2065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Packed element was truncated.')
2075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        return pos
2085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return DecodePackedField
2095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    elif is_repeated:
2105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      tag_bytes = encoder.TagBytes(field_number, wire_type)
2115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      tag_len = len(tag_bytes)
2125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      def DecodeRepeatedField(buffer, pos, end, message, field_dict):
2135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.get(key)
2145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if value is None:
2155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          value = field_dict.setdefault(key, new_default(message))
2165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        while 1:
2175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          (element, new_pos) = decode_value(buffer, pos)
2185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          value.append(element)
2195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # Predict that the next tag is another copy of the same repeated
2205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # field.
2215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          pos = new_pos + tag_len
2225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          if buffer[new_pos:pos] != tag_bytes or new_pos >= end:
2235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)            # Prediction failed.  Return.
2245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)            if new_pos > end:
2255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)              raise _DecodeError('Truncated message.')
2265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)            return new_pos
2275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return DecodeRepeatedField
2285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    else:
2295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      def DecodeField(buffer, pos, end, message, field_dict):
2305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        (field_dict[key], pos) = decode_value(buffer, pos)
2315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if pos > end:
2325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          del field_dict[key]  # Discard corrupt value.
2335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Truncated message.')
2345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        return pos
2355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return DecodeField
2365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return SpecificDecoder
2385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _ModifiedDecoder(wire_type, decode_value, modify_value):
2415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Like SimpleDecoder but additionally invokes modify_value on every value
2425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  before storing it.  Usually modify_value is ZigZagDecode.
2435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
2445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  # Reusing _SimpleDecoder is slightly slower than copying a bunch of code, but
2465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  # not enough to make a significant difference.
2475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def InnerDecode(buffer, pos):
2495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    (result, new_pos) = decode_value(buffer, pos)
2505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return (modify_value(result), new_pos)
2515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return _SimpleDecoder(wire_type, InnerDecode)
2525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _StructPackDecoder(wire_type, format):
2555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Return a constructor for a decoder for a fixed-width field.
2565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  Args:
2585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      wire_type:  The field's wire type.
2595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      format:  The format string to pass to struct.unpack().
2605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
2615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  value_size = struct.calcsize(format)
2635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_unpack = struct.unpack
2645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  # Reusing _SimpleDecoder is slightly slower than copying a bunch of code, but
2665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  # not enough to make a significant difference.
2675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  # Note that we expect someone up-stack to catch struct.error and convert
2695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  # it to _DecodeError -- this way we don't have to set up exception-
2705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  # handling blocks every time we parse one value.
2715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def InnerDecode(buffer, pos):
2735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    new_pos = pos + value_size
2745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    result = local_unpack(format, buffer[pos:new_pos])[0]
2755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return (result, new_pos)
2765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return _SimpleDecoder(wire_type, InnerDecode)
2775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _FloatDecoder():
2805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Returns a decoder for a float field.
2815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  This code works around a bug in struct.unpack for non-finite 32-bit
2835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  floating-point values.
2845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
2855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_unpack = struct.unpack
2875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def InnerDecode(buffer, pos):
2895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # We expect a 32-bit value in little-endian byte order.  Bit 1 is the sign
2905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # bit, bits 2-9 represent the exponent, and bits 10-32 are the significand.
2915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    new_pos = pos + 4
2925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    float_bytes = buffer[pos:new_pos]
2935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
2945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # If this value has all its exponent bits set, then it's non-finite.
2955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # In Python 2.4, struct.unpack will convert it to a finite 64-bit value.
2965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # To avoid that, we parse it specially.
2975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if ((float_bytes[3] in '\x7F\xFF')
2985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        and (float_bytes[2] >= '\x80')):
2995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      # If at least one significand bit is set...
3005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if float_bytes[0:3] != '\x00\x00\x80':
3015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        return (_NAN, new_pos)
3025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      # If sign bit is set...
3035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if float_bytes[3] == '\xFF':
3045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        return (_NEG_INF, new_pos)
3055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return (_POS_INF, new_pos)
3065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # Note that we expect someone up-stack to catch struct.error and convert
3085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # it to _DecodeError -- this way we don't have to set up exception-
3095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # handling blocks every time we parse one value.
3105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    result = local_unpack('<f', float_bytes)[0]
3115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return (result, new_pos)
3125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return _SimpleDecoder(wire_format.WIRETYPE_FIXED32, InnerDecode)
3135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _DoubleDecoder():
3165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Returns a decoder for a double field.
3175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  This code works around a bug in struct.unpack for not-a-number.
3195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
3205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_unpack = struct.unpack
3225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def InnerDecode(buffer, pos):
3245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # We expect a 64-bit value in little-endian byte order.  Bit 1 is the sign
3255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # bit, bits 2-12 represent the exponent, and bits 13-64 are the significand.
3265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    new_pos = pos + 8
3275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    double_bytes = buffer[pos:new_pos]
3285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # If this value has all its exponent bits set and at least one significand
3305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # bit set, it's not a number.  In Python 2.4, struct.unpack will treat it
3315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # as inf or -inf.  To avoid that, we treat it specially.
3325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if ((double_bytes[7] in '\x7F\xFF')
3335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        and (double_bytes[6] >= '\xF0')
3345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        and (double_bytes[0:7] != '\x00\x00\x00\x00\x00\x00\xF0')):
3355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return (_NAN, new_pos)
3365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # Note that we expect someone up-stack to catch struct.error and convert
3385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # it to _DecodeError -- this way we don't have to set up exception-
3395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # handling blocks every time we parse one value.
3405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    result = local_unpack('<d', double_bytes)[0]
3415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return (result, new_pos)
3425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return _SimpleDecoder(wire_format.WIRETYPE_FIXED64, InnerDecode)
3435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# --------------------------------------------------------------------
3465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Int32Decoder = EnumDecoder = _SimpleDecoder(
3495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    wire_format.WIRETYPE_VARINT, _DecodeSignedVarint32)
3505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Int64Decoder = _SimpleDecoder(
3525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    wire_format.WIRETYPE_VARINT, _DecodeSignedVarint)
3535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)UInt32Decoder = _SimpleDecoder(wire_format.WIRETYPE_VARINT, _DecodeVarint32)
3555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)UInt64Decoder = _SimpleDecoder(wire_format.WIRETYPE_VARINT, _DecodeVarint)
3565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)SInt32Decoder = _ModifiedDecoder(
3585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    wire_format.WIRETYPE_VARINT, _DecodeVarint32, wire_format.ZigZagDecode)
3595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)SInt64Decoder = _ModifiedDecoder(
3605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    wire_format.WIRETYPE_VARINT, _DecodeVarint, wire_format.ZigZagDecode)
3615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# Note that Python conveniently guarantees that when using the '<' prefix on
3635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# formats, they will also have the same size across all platforms (as opposed
3645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# to without the prefix, where their sizes depend on the C compiler's basic
3655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# type sizes).
3665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Fixed32Decoder  = _StructPackDecoder(wire_format.WIRETYPE_FIXED32, '<I')
3675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Fixed64Decoder  = _StructPackDecoder(wire_format.WIRETYPE_FIXED64, '<Q')
3685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)SFixed32Decoder = _StructPackDecoder(wire_format.WIRETYPE_FIXED32, '<i')
3695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)SFixed64Decoder = _StructPackDecoder(wire_format.WIRETYPE_FIXED64, '<q')
3705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)FloatDecoder = _FloatDecoder()
3715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)DoubleDecoder = _DoubleDecoder()
3725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)BoolDecoder = _ModifiedDecoder(
3745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    wire_format.WIRETYPE_VARINT, _DecodeVarint, bool)
3755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def StringDecoder(field_number, is_repeated, is_packed, key, new_default):
3785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Returns a decoder for a string field."""
3795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_DecodeVarint = _DecodeVarint
3815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_unicode = unicode
3825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
3835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  assert not is_packed
3845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if is_repeated:
3855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_bytes = encoder.TagBytes(field_number,
3865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                                 wire_format.WIRETYPE_LENGTH_DELIMITED)
3875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_len = len(tag_bytes)
3885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeRepeatedField(buffer, pos, end, message, field_dict):
3895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      value = field_dict.get(key)
3905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value is None:
3915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.setdefault(key, new_default(message))
3925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      while 1:
3935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        (size, pos) = local_DecodeVarint(buffer, pos)
3945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        new_pos = pos + size
3955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if new_pos > end:
3965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Truncated string.')
3975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value.append(local_unicode(buffer[pos:new_pos], 'utf-8'))
3985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Predict that the next tag is another copy of the same repeated field.
3995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        pos = new_pos + tag_len
4005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if buffer[new_pos:pos] != tag_bytes or new_pos == end:
4015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # Prediction failed.  Return.
4025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          return new_pos
4035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeRepeatedField
4045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  else:
4055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeField(buffer, pos, end, message, field_dict):
4065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      (size, pos) = local_DecodeVarint(buffer, pos)
4075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      new_pos = pos + size
4085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if new_pos > end:
4095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Truncated string.')
4105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      field_dict[key] = local_unicode(buffer[pos:new_pos], 'utf-8')
4115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return new_pos
4125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeField
4135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def BytesDecoder(field_number, is_repeated, is_packed, key, new_default):
4165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Returns a decoder for a bytes field."""
4175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_DecodeVarint = _DecodeVarint
4195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  assert not is_packed
4215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if is_repeated:
4225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_bytes = encoder.TagBytes(field_number,
4235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                                 wire_format.WIRETYPE_LENGTH_DELIMITED)
4245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_len = len(tag_bytes)
4255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeRepeatedField(buffer, pos, end, message, field_dict):
4265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      value = field_dict.get(key)
4275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value is None:
4285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.setdefault(key, new_default(message))
4295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      while 1:
4305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        (size, pos) = local_DecodeVarint(buffer, pos)
4315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        new_pos = pos + size
4325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if new_pos > end:
4335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Truncated string.')
4345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value.append(buffer[pos:new_pos])
4355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Predict that the next tag is another copy of the same repeated field.
4365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        pos = new_pos + tag_len
4375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if buffer[new_pos:pos] != tag_bytes or new_pos == end:
4385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # Prediction failed.  Return.
4395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          return new_pos
4405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeRepeatedField
4415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  else:
4425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeField(buffer, pos, end, message, field_dict):
4435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      (size, pos) = local_DecodeVarint(buffer, pos)
4445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      new_pos = pos + size
4455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if new_pos > end:
4465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Truncated string.')
4475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      field_dict[key] = buffer[pos:new_pos]
4485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return new_pos
4495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeField
4505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def GroupDecoder(field_number, is_repeated, is_packed, key, new_default):
4535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Returns a decoder for a group field."""
4545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  end_tag_bytes = encoder.TagBytes(field_number,
4565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                                   wire_format.WIRETYPE_END_GROUP)
4575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  end_tag_len = len(end_tag_bytes)
4585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  assert not is_packed
4605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if is_repeated:
4615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_bytes = encoder.TagBytes(field_number,
4625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                                 wire_format.WIRETYPE_START_GROUP)
4635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_len = len(tag_bytes)
4645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeRepeatedField(buffer, pos, end, message, field_dict):
4655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      value = field_dict.get(key)
4665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value is None:
4675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.setdefault(key, new_default(message))
4685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      while 1:
4695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.get(key)
4705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if value is None:
4715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          value = field_dict.setdefault(key, new_default(message))
4725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Read sub-message.
4735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        pos = value.add()._InternalParse(buffer, pos, end)
4745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Read end tag.
4755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        new_pos = pos+end_tag_len
4765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if buffer[pos:new_pos] != end_tag_bytes or new_pos > end:
4775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Missing group end tag.')
4785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Predict that the next tag is another copy of the same repeated field.
4795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        pos = new_pos + tag_len
4805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if buffer[new_pos:pos] != tag_bytes or new_pos == end:
4815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # Prediction failed.  Return.
4825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          return new_pos
4835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeRepeatedField
4845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  else:
4855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeField(buffer, pos, end, message, field_dict):
4865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      value = field_dict.get(key)
4875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value is None:
4885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.setdefault(key, new_default(message))
4895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      # Read sub-message.
4905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      pos = value._InternalParse(buffer, pos, end)
4915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      # Read end tag.
4925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      new_pos = pos+end_tag_len
4935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if buffer[pos:new_pos] != end_tag_bytes or new_pos > end:
4945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Missing group end tag.')
4955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return new_pos
4965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeField
4975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
4995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def MessageDecoder(field_number, is_repeated, is_packed, key, new_default):
5005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Returns a decoder for a message field."""
5015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_DecodeVarint = _DecodeVarint
5035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  assert not is_packed
5055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if is_repeated:
5065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_bytes = encoder.TagBytes(field_number,
5075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)                                 wire_format.WIRETYPE_LENGTH_DELIMITED)
5085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    tag_len = len(tag_bytes)
5095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeRepeatedField(buffer, pos, end, message, field_dict):
5105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      value = field_dict.get(key)
5115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value is None:
5125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.setdefault(key, new_default(message))
5135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      while 1:
5145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.get(key)
5155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if value is None:
5165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          value = field_dict.setdefault(key, new_default(message))
5175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Read length.
5185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        (size, pos) = local_DecodeVarint(buffer, pos)
5195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        new_pos = pos + size
5205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if new_pos > end:
5215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Truncated message.')
5225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Read sub-message.
5235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if value.add()._InternalParse(buffer, pos, new_pos) != new_pos:
5245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # The only reason _InternalParse would return early is if it
5255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # encountered an end-group tag.
5265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Unexpected end-group tag.')
5275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # Predict that the next tag is another copy of the same repeated field.
5285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        pos = new_pos + tag_len
5295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if buffer[new_pos:pos] != tag_bytes or new_pos == end:
5305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          # Prediction failed.  Return.
5315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          return new_pos
5325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeRepeatedField
5335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  else:
5345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    def DecodeField(buffer, pos, end, message, field_dict):
5355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      value = field_dict.get(key)
5365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value is None:
5375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.setdefault(key, new_default(message))
5385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      # Read length.
5395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      (size, pos) = local_DecodeVarint(buffer, pos)
5405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      new_pos = pos + size
5415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if new_pos > end:
5425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Truncated message.')
5435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      # Read sub-message.
5445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value._InternalParse(buffer, pos, new_pos) != new_pos:
5455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # The only reason _InternalParse would return early is if it encountered
5465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # an end-group tag.
5475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Unexpected end-group tag.')
5485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return new_pos
5495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return DecodeField
5505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# --------------------------------------------------------------------
5535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)MESSAGE_SET_ITEM_TAG = encoder.TagBytes(1, wire_format.WIRETYPE_START_GROUP)
5555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def MessageSetItemDecoder(extensions_by_number):
5575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Returns a decoder for a MessageSet item.
5585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  The parameter is the _extensions_by_number map for the message class.
5605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  The message set message looks like this:
5625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    message MessageSet {
5635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      repeated group Item = 1 {
5645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        required int32 type_id = 2;
5655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        required string message = 3;
5665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      }
5675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    }
5685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """
5695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  type_id_tag_bytes = encoder.TagBytes(2, wire_format.WIRETYPE_VARINT)
5715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  message_tag_bytes = encoder.TagBytes(3, wire_format.WIRETYPE_LENGTH_DELIMITED)
5725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  item_end_tag_bytes = encoder.TagBytes(1, wire_format.WIRETYPE_END_GROUP)
5735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_ReadTag = ReadTag
5755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_DecodeVarint = _DecodeVarint
5765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_SkipField = SkipField
5775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def DecodeItem(buffer, pos, end, message, field_dict):
579ba5b9a6411cb1792fd21f0a078d7a25cd1ceec16Ben Murdoch    message_set_item_start = pos
5805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    type_id = -1
5815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    message_start = -1
5825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    message_end = -1
5835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
5845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # Technically, type_id and message can appear in any order, so we need
5855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # a little loop here.
5865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    while 1:
5875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      (tag_bytes, pos) = local_ReadTag(buffer, pos)
5885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if tag_bytes == type_id_tag_bytes:
5895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        (type_id, pos) = local_DecodeVarint(buffer, pos)
5905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      elif tag_bytes == message_tag_bytes:
5915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        (size, message_start) = local_DecodeVarint(buffer, pos)
5925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        pos = message_end = message_start + size
5935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      elif tag_bytes == item_end_tag_bytes:
5945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        break
5955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      else:
5965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        pos = SkipField(buffer, pos, end, tag_bytes)
5975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        if pos == -1:
5985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)          raise _DecodeError('Missing group end tag.')
5995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if pos > end:
6015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      raise _DecodeError('Truncated message.')
6025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if type_id == -1:
6045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      raise _DecodeError('MessageSet item missing type_id.')
6055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if message_start == -1:
6065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      raise _DecodeError('MessageSet item missing message.')
6075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    extension = extensions_by_number.get(type_id)
6095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if extension is not None:
6105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      value = field_dict.get(extension)
6115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value is None:
6125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        value = field_dict.setdefault(
6135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)            extension, extension.message_type._concrete_class())
6145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      if value._InternalParse(buffer, message_start,message_end) != message_end:
6155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # The only reason _InternalParse would return early is if it encountered
6165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        # an end-group tag.
6175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        raise _DecodeError('Unexpected end-group tag.')
618ba5b9a6411cb1792fd21f0a078d7a25cd1ceec16Ben Murdoch    else:
619ba5b9a6411cb1792fd21f0a078d7a25cd1ceec16Ben Murdoch      if not message._unknown_fields:
620ba5b9a6411cb1792fd21f0a078d7a25cd1ceec16Ben Murdoch        message._unknown_fields = []
621ba5b9a6411cb1792fd21f0a078d7a25cd1ceec16Ben Murdoch      message._unknown_fields.append((MESSAGE_SET_ITEM_TAG,
622ba5b9a6411cb1792fd21f0a078d7a25cd1ceec16Ben Murdoch                                      buffer[message_set_item_start:pos]))
6235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return pos
6255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return DecodeItem
6275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# --------------------------------------------------------------------
6295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# Optimization is not as heavy here because calls to SkipField() are rare,
6305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# except for handling end-group tags.
6315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _SkipVarint(buffer, pos, end):
6335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Skip a varint value.  Returns the new position."""
6345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  while ord(buffer[pos]) & 0x80:
6365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    pos += 1
6375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  pos += 1
6385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if pos > end:
6395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    raise _DecodeError('Truncated message.')
6405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return pos
6415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _SkipFixed64(buffer, pos, end):
6435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Skip a fixed64 value.  Returns the new position."""
6445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  pos += 8
6465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if pos > end:
6475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    raise _DecodeError('Truncated message.')
6485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return pos
6495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _SkipLengthDelimited(buffer, pos, end):
6515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Skip a length-delimited value.  Returns the new position."""
6525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  (size, pos) = _DecodeVarint(buffer, pos)
6545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  pos += size
6555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if pos > end:
6565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    raise _DecodeError('Truncated message.')
6575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return pos
6585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _SkipGroup(buffer, pos, end):
6605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Skip sub-group.  Returns the new position."""
6615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  while 1:
6635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    (tag_bytes, pos) = ReadTag(buffer, pos)
6645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    new_pos = SkipField(buffer, pos, end, tag_bytes)
6655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    if new_pos == -1:
6665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      return pos
6675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    pos = new_pos
6685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _EndGroup(buffer, pos, end):
6705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Skipping an END_GROUP tag returns -1 to tell the parent loop to break."""
6715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return -1
6735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _SkipFixed32(buffer, pos, end):
6755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Skip a fixed32 value.  Returns the new position."""
6765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  pos += 4
6785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  if pos > end:
6795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    raise _DecodeError('Truncated message.')
6805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return pos
6815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _RaiseInvalidWireType(buffer, pos, end):
6835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Skip function for unknown wire types.  Raises an exception."""
6845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  raise _DecodeError('Tag had invalid wire type.')
6865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)def _FieldSkipper():
6885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  """Constructs the SkipField function."""
6895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
6905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  WIRETYPE_TO_SKIPPER = [
6915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _SkipVarint,
6925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _SkipFixed64,
6935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _SkipLengthDelimited,
6945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _SkipGroup,
6955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _EndGroup,
6965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _SkipFixed32,
6975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _RaiseInvalidWireType,
6985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      _RaiseInvalidWireType,
6995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)      ]
7005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
7015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  wiretype_mask = wire_format.TAG_TYPE_MASK
7025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  local_ord = ord
7035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
7045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  def SkipField(buffer, pos, end, tag_bytes):
7055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    """Skips a field with the specified tag.
7065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
7075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    |pos| should point to the byte immediately after the tag.
7085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
7095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    Returns:
7105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        The new position (after the tag value), or -1 if the tag is an end-group
7115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)        tag (in which case the calling loop should break).
7125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    """
7135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
7145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    # The wire type is always in the first byte since varints are little-endian.
7155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    wire_type = local_ord(tag_bytes[0]) & wiretype_mask
7165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)    return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end)
7175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
7185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)  return SkipField
7195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)
7205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)SkipField = _FieldSkipper()
721