gencodec.py revision dc36472472361d8a360d97adbc7f1a3a8fce4493
134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum""" Unicode Mapping Parser and Codec Generator. 234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossumThis script parses Unicode mapping files as available from the Unicode 4a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburgsite (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec 5a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburgmodules from them. The codecs use the standard character mapping codec 6a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburgto actually apply the mapping. 734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossumSynopsis: gencodec.py dir codec_prefix 934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 1034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossumAll files in dir are scanned and those producing non-empty mappings 1134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumwill be written to <codec_prefix><mapname>.py with <mapname> being the 1234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumfirst part of the map's filename ('a' in a.b.c.txt) converted to 1334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumlowercase with hyphens replaced by underscores. 1434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 15bae57a88a6e34fa25ba0a5812171a407dc4fc4b1Fred DrakeThe tool also writes marshalled versions of the mapping tables to the 1634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumsame location (with .mapping extension). 1734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 18bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André LemburgWritten by Marc-Andre Lemburg (mal@lemburg.com). 1934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 2034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum(c) Copyright CNRI, All Rights Reserved. NO WARRANTY. 21a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg(c) Copyright Guido van Rossum, 2000. 22bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg 23bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André LemburgTable generation: 24c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg(c) Copyright Marc-Andre Lemburg, 2005. 25bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg Licensed to PSF under a Contributor Agreement. 2634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 2734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum"""#" 2834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 29c5f05e45cffa16f45f1332cec531c045893f928fChristian Heimesimport re, os, marshal, codecs 3034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 31c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg# Maximum allowed size of charmap tables 32c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André LemburgMAX_TABLE_SIZE = 8192 33c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 34c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg# Standard undefined Unicode code point 35c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André LemburgUNI_UNDEFINED = unichr(0xFFFE) 3634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 3734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossummapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)' 3834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '\s+' 3934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '((?:(?:0x[0-9a-fA-Z]+|<[A-Za-z]+>)\+?)*)' 4034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '\s*' 4134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '(#.+)?') 4234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 43dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xiclunadef parsecodes(codes, len=len, range=range): 4434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 4534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ Converts code combinations to either a single code integer 4634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum or a tuple of integers. 4734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 4834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum meta-codes (in angular brackets, e.g. <LR> and <RL>) are 4934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum ignored. 5034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 5134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum Empty codes or illegal ones are returned as None. 5234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 5334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ 5434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not codes: 5534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return None 56aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald l = codes.split('+') 5734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if len(l) == 1: 58aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald return int(l[0],16) 5934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for i in range(len(l)): 6034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 61aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald l[i] = int(l[i],16) 6234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum except ValueError: 6334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum l[i] = None 64dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xicluna l = [x for x in l if x is not None] 6534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if len(l) == 1: 6634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return l[0] 6734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 6834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return tuple(l) 6934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 70aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwalddef readmap(filename): 7134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 7234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f = open(filename,'r') 7334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum lines = f.readlines() 7434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.close() 7534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc2uni = {} 76a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg identity = [] 77a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg unmapped = range(256) 78c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 79c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # UTC mapping tables per convention don't include the identity 80c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # mappings for code points 0x00 - 0x1F and 0x7F, unless these are 81c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # explicitly mapped to different characters or undefined 82c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for i in range(32) + [127]: 83c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg identity.append(i) 84c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg unmapped.remove(i) 85c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg enc2uni[i] = (i, 'CONTROL CHARACTER') 86c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 8734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for line in lines: 88aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald line = line.strip() 8934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not line or line[0] == '#': 9034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum continue 9134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum m = mapRE.match(line) 9234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not m: 9334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum #print '* not matched: %s' % repr(line) 9434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum continue 9534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc,uni,comment = m.groups() 9634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc = parsecodes(enc) 9734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum uni = parsecodes(uni) 98c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if comment is None: 9934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum comment = '' 10034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 101c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg comment = comment[1:].strip() 102a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg if enc < 256: 103c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if enc in unmapped: 104c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg unmapped.remove(enc) 105a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg if enc == uni: 106a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg identity.append(enc) 107c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg enc2uni[enc] = (uni,comment) 108a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg else: 10934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc2uni[enc] = (uni,comment) 110c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 111a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg # If there are more identity-mapped entries than unmapped entries, 112771bc377f81f877d9840402aab765c7bd957c79dWalter Dörwald # it pays to generate an identity dictionary first, and add explicit 113a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg # mappings to None for the rest 114c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if len(identity) >= len(unmapped): 115a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg for enc in unmapped: 116a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg enc2uni[enc] = (None, "") 117a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg enc2uni['IDENTITY'] = 256 118a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg 11934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return enc2uni 12034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 121bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburgdef hexrepr(t, precision=4): 12234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 12334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if t is None: 12434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return 'None' 12534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 12634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum len(t) 12734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum except: 128bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg return '0x%0*X' % (precision, t) 129c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg try: 130bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg return '(' + ', '.join(['0x%0*X' % (precision, item) 131bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg for item in t]) + ')' 132c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg except TypeError, why: 133c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg print '* failed to convert %r: %s' % (t, why) 134c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg raise 13534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 136bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburgdef python_mapdef_code(varname, map, comments=1, precisions=(2, 4)): 13734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 138c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l = [] 139c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append = l.append 140dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xicluna if "IDENTITY" in map: 141c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append("%s = codecs.make_identity_dict(range(%d))" % 142c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (varname, map["IDENTITY"])) 143c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append("%s.update({" % varname) 144c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg splits = 1 145c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg del map["IDENTITY"] 146c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg identity = 1 14734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 148c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append("%s = {" % varname) 149c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg splits = 0 150c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg identity = 0 15134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 152dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xicluna mappings = sorted(map.items()) 153c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg i = 0 154bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg key_precision, value_precision = precisions 155c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for mapkey, mapvalue in mappings: 156c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment = '' 157c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapkey, tuple): 158c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapkey, mapcomment) = mapkey 159c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapvalue, tuple): 160c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapvalue, mapcomment) = mapvalue 161c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapkey is None: 162c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg continue 163c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if (identity and 164c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapkey == mapvalue and 165c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapkey < 256): 166c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # No need to include identity mappings, since these 167c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # are already set for the first 256 code points. 168c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg continue 169bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg key = hexrepr(mapkey, key_precision) 170bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg value = hexrepr(mapvalue, value_precision) 171c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapcomment and comments: 172c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %s: %s,\t# %s' % (key, value, mapcomment)) 173c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 174c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %s: %s,' % (key, value)) 175c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg i += 1 176c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if i == 4096: 177c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Split the definition into parts to that the Python 178c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # parser doesn't dump core 179c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if splits == 0: 180c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('}') 181c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 182c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('})') 183c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('%s.update({' % varname) 184c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg i = 0 185c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg splits = splits + 1 186c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if splits == 0: 187c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('}') 18834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 189c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('})') 190c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 191c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return l 192c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 193bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburgdef python_tabledef_code(varname, map, comments=1, key_precision=2): 194c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 195c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l = [] 196c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append = l.append 197c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('%s = (' % varname) 198c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 199c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Analyze map and create table dict 200dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xicluna mappings = sorted(map.items()) 201c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg table = {} 202c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg maxkey = 0 203dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xicluna if 'IDENTITY' in map: 204c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for key in range(256): 205c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg table[key] = (key, '') 206c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg maxkey = 255 207c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg del map['IDENTITY'] 208c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for mapkey, mapvalue in mappings: 209c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment = '' 210c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapkey, tuple): 211c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapkey, mapcomment) = mapkey 212c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapvalue, tuple): 213c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapvalue, mapcomment) = mapvalue 214c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapkey is None: 215c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg continue 216c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg table[mapkey] = (mapvalue, mapcomment) 217c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapkey > maxkey: 218c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg maxkey = mapkey 219c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if maxkey > MAX_TABLE_SIZE: 220c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Table too large 221c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return None 222c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 223c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Create table code 224c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for key in range(maxkey + 1): 225c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if key not in table: 226c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapvalue = None 227c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment = 'UNDEFINED' 228c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 229c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapvalue, mapcomment = table[key] 230c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapvalue is None: 231c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapchar = UNI_UNDEFINED 232c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 233c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapvalue, tuple): 234c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # 1-n mappings not supported 235c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return None 23634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 237c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapchar = unichr(mapvalue) 238c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapcomment and comments: 239c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %r\t# %s -> %s' % (mapchar, 240bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg hexrepr(key, key_precision), 241c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment)) 242c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 243c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %r' % mapchar) 24434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 245c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(')') 246c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return l 247c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 248abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwalddef codegen(name, map, encodingname, comments=1): 24934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 25034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ Returns Python source for the given map. 25134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 25234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum Comments are included in the source, if comments is true (default). 25334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 25434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ 255c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Generate code 256c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg decoding_map_code = python_mapdef_code( 257c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 'decoding_map', 258c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg map, 259c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg comments=comments) 260c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg decoding_table_code = python_tabledef_code( 261c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 'decoding_table', 262c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg map, 263c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg comments=comments) 264c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg encoding_map_code = python_mapdef_code( 265c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 'encoding_map', 266c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg codecs.make_encoding_map(map), 267bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg comments=comments, 268bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg precisions=(4, 2)) 269c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 2703f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis if decoding_table_code: 2713f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis suffix = 'table' 2723f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis else: 2733f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis suffix = 'map' 2743f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis 27534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum l = [ 27634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '''\ 277abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald""" Python Character Mapping Codec %s generated from '%s' with gencodec.py. 27834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 27934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum"""#" 28034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumimport codecs 28234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### Codec APIs 28434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumclass Codec(codecs.Codec): 28634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum def encode(self,input,errors='strict'): 2883f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis return codecs.charmap_encode(input,errors,encoding_%s) 289536cf99536bce562cfcb44a856fac1c84b9de4c3Tim Peters 2903f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis def decode(self,input,errors='strict'): 2913f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis return codecs.charmap_decode(input,errors,decoding_%s) 2923f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis''' % (encodingname, name, suffix, suffix)] 2933f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis l.append('''\ 294abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwaldclass IncrementalEncoder(codecs.IncrementalEncoder): 295abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald def encode(self, input, final=False): 2963f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis return codecs.charmap_encode(input,self.errors,encoding_%s)[0] 297abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald 298abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwaldclass IncrementalDecoder(codecs.IncrementalDecoder): 2993f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis def decode(self, input, final=False): 3003f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis return codecs.charmap_decode(input,self.errors,decoding_%s)[0]''' % 3013f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis (suffix, suffix)) 302abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald 303abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald l.append(''' 30434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumclass StreamWriter(Codec,codecs.StreamWriter): 30534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum pass 30670c4378dbcfdcbeef6fb3aa348f32ed862fe8eb7Tim Peters 30734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumclass StreamReader(Codec,codecs.StreamReader): 30834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum pass 30934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 31034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### encodings module API 31134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 31234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef getregentry(): 313df676c5ffdbc1c7cfa11d759936bfddebb53067eJack Diederich return codecs.CodecInfo( 314df676c5ffdbc1c7cfa11d759936bfddebb53067eJack Diederich name=%r, 3153f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis encode=Codec().encode, 3163f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis decode=Codec().decode, 317abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald incrementalencoder=IncrementalEncoder, 318abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald incrementaldecoder=IncrementalDecoder, 3193f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis streamreader=StreamReader, 3203f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis streamwriter=StreamWriter, 321df676c5ffdbc1c7cfa11d759936bfddebb53067eJack Diederich ) 322abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald''' % encodingname.replace('_', '-')) 32334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 324bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg # Add decoding table or map (with preference to the table) 325bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg if not decoding_table_code: 326bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg l.append(''' 32734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### Decoding Map 328c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg''') 329bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg l.extend(decoding_map_code) 330bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg else: 331c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.append(''' 332c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg### Decoding Table 333c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg''') 334c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.extend(decoding_table_code) 33570c4378dbcfdcbeef6fb3aa348f32ed862fe8eb7Tim Peters 336bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg # Add encoding map 3373f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis if decoding_table_code: 3383f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis l.append(''' 3393f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis### Encoding table 3403f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwisencoding_table=codecs.charmap_build(decoding_table) 3413f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis''') 3423f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis else: 3433f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis l.append(''' 34434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### Encoding Map 34534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum''') 3463f767795f6784ca6bf797b055be67fce5bf2fa06Martin v. Löwis l.extend(encoding_map_code) 347bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg 348bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg # Final new-line 3495d23f9a8a33324d00d1e73f40f7c07e914b295f1Walter Dörwald l.append('') 350536cf99536bce562cfcb44a856fac1c84b9de4c3Tim Peters 351abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald return '\n'.join(l).expandtabs() 35234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 353abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwalddef pymap(name,map,pyfile,encodingname,comments=1): 35434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 355abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald code = codegen(name,map,encodingname,comments) 35634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f = open(pyfile,'w') 35734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.write(code) 35834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.close() 35934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 36034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef marshalmap(name,map,marshalfile): 36134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 36234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum d = {} 36334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for e,(u,c) in map.items(): 36434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum d[e] = (u,c) 36534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f = open(marshalfile,'wb') 36634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum marshal.dump(d,f) 36734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.close() 36834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 369abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwalddef convertdir(dir, dirprefix='', nameprefix='', comments=1): 37034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 37134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum mapnames = os.listdir(dir) 37234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for mapname in mapnames: 373c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mappathname = os.path.join(dir, mapname) 374bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg if not os.path.isfile(mappathname): 375bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg continue 37634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum name = os.path.split(mapname)[1] 377aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald name = name.replace('-','_') 378aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald name = name.split('.')[0] 379aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald name = name.lower() 380abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald name = nameprefix + name 38134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum codefile = name + '.py' 38234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum marshalfile = name + '.mapping' 38334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print 'converting %s to %s and %s' % (mapname, 384abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald dirprefix + codefile, 385abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald dirprefix + marshalfile) 38634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 38734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum map = readmap(os.path.join(dir,mapname)) 38834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not map: 38934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print '* map is empty; skipping' 39034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 391abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald pymap(mappathname, map, dirprefix + codefile,name,comments) 392abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald marshalmap(mappathname, map, dirprefix + marshalfile) 393c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg except ValueError, why: 394c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg print '* conversion failed: %s' % why 395c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg raise 39634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 397abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwalddef rewritepythondir(dir, dirprefix='', comments=1): 39870c4378dbcfdcbeef6fb3aa348f32ed862fe8eb7Tim Peters 39934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum mapnames = os.listdir(dir) 40034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for mapname in mapnames: 401a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg if not mapname.endswith('.mapping'): 40234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum continue 403abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald name = mapname[:-len('.mapping')] 404abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald codefile = name + '.py' 40534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print 'converting %s to %s' % (mapname, 406abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald dirprefix + codefile) 40734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 40834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum map = marshal.load(open(os.path.join(dir,mapname), 40934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 'rb')) 41034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not map: 41134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print '* map is empty; skipping' 41234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 413abb02e59946f9ea3076e96e3b03b51d1cebd46b4Walter Dörwald pymap(mapname, map, dirprefix + codefile,name,comments) 41434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum except ValueError, why: 41534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print '* conversion failed: %s' % why 41634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 41734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumif __name__ == '__main__': 41834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 41934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum import sys 42034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if 1: 421dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xicluna convertdir(*sys.argv[1:]) 42234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 423dc36472472361d8a360d97adbc7f1a3a8fce4493Florent Xicluna rewritepythondir(*sys.argv[1:]) 424