gencodec.py revision bd20ea55bc7a044a773e6824f7fcef4f5669d44c
134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum""" Unicode Mapping Parser and Codec Generator. 234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossumThis script parses Unicode mapping files as available from the Unicode 4a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburgsite (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec 5a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburgmodules from them. The codecs use the standard character mapping codec 6a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburgto actually apply the mapping. 734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossumSynopsis: gencodec.py dir codec_prefix 934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 1034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossumAll files in dir are scanned and those producing non-empty mappings 1134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumwill be written to <codec_prefix><mapname>.py with <mapname> being the 1234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumfirst part of the map's filename ('a' in a.b.c.txt) converted to 1334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumlowercase with hyphens replaced by underscores. 1434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 15bae57a88a6e34fa25ba0a5812171a407dc4fc4b1Fred DrakeThe tool also writes marshalled versions of the mapping tables to the 1634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumsame location (with .mapping extension). 1734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 18bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André LemburgWritten by Marc-Andre Lemburg (mal@lemburg.com). 1934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 2034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum(c) Copyright CNRI, All Rights Reserved. NO WARRANTY. 21a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg(c) Copyright Guido van Rossum, 2000. 22bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg 23bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André LemburgTable generation: 24c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg(c) Copyright Marc-Andre Lemburg, 2005. 25bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg Licensed to PSF under a Contributor Agreement. 2634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 2734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum"""#" 2834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 29c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburgimport re, os, time, marshal, codecs 3034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 31c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg# Maximum allowed size of charmap tables 32c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André LemburgMAX_TABLE_SIZE = 8192 33c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 34c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg# Standard undefined Unicode code point 35c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André LemburgUNI_UNDEFINED = unichr(0xFFFE) 3634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 3734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van RossummapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)' 3834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '\s+' 3934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '((?:(?:0x[0-9a-fA-Z]+|<[A-Za-z]+>)\+?)*)' 4034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '\s*' 4134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '(#.+)?') 4234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 4334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef parsecodes(codes, 44aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald len=len, filter=filter,range=range): 4534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 4634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ Converts code combinations to either a single code integer 4734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum or a tuple of integers. 4834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 4934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum meta-codes (in angular brackets, e.g. <LR> and <RL>) are 5034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum ignored. 5134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 5234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum Empty codes or illegal ones are returned as None. 5334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 5434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ 5534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not codes: 5634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return None 57aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald l = codes.split('+') 5834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if len(l) == 1: 59aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald return int(l[0],16) 6034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for i in range(len(l)): 6134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 62aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald l[i] = int(l[i],16) 6334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum except ValueError: 6434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum l[i] = None 6534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum l = filter(lambda x: x is not None, l) 6634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if len(l) == 1: 6734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return l[0] 6834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 6934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return tuple(l) 7034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 71aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwalddef readmap(filename): 7234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 7334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f = open(filename,'r') 7434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum lines = f.readlines() 7534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.close() 7634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc2uni = {} 77a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg identity = [] 78a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg unmapped = range(256) 79c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 80c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # UTC mapping tables per convention don't include the identity 81c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # mappings for code points 0x00 - 0x1F and 0x7F, unless these are 82c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # explicitly mapped to different characters or undefined 83c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for i in range(32) + [127]: 84c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg identity.append(i) 85c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg unmapped.remove(i) 86c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg enc2uni[i] = (i, 'CONTROL CHARACTER') 87c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 8834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for line in lines: 89aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald line = line.strip() 9034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not line or line[0] == '#': 9134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum continue 9234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum m = mapRE.match(line) 9334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not m: 9434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum #print '* not matched: %s' % repr(line) 9534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum continue 9634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc,uni,comment = m.groups() 9734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc = parsecodes(enc) 9834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum uni = parsecodes(uni) 99c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if comment is None: 10034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum comment = '' 10134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 102c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg comment = comment[1:].strip() 103a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg if enc < 256: 104c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if enc in unmapped: 105c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg unmapped.remove(enc) 106a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg if enc == uni: 107a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg identity.append(enc) 108c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg enc2uni[enc] = (uni,comment) 109a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg else: 11034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum enc2uni[enc] = (uni,comment) 111c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 112a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg # If there are more identity-mapped entries than unmapped entries, 113771bc377f81f877d9840402aab765c7bd957c79dWalter Dörwald # it pays to generate an identity dictionary first, and add explicit 114a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg # mappings to None for the rest 115c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if len(identity) >= len(unmapped): 116a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg for enc in unmapped: 117a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg enc2uni[enc] = (None, "") 118a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg enc2uni['IDENTITY'] = 256 119a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg 12034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return enc2uni 12134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 122bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburgdef hexrepr(t, precision=4): 12334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 12434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if t is None: 12534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return 'None' 12634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 12734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum len(t) 12834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum except: 129bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg return '0x%0*X' % (precision, t) 130c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg try: 131bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg return '(' + ', '.join(['0x%0*X' % (precision, item) 132bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg for item in t]) + ')' 133c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg except TypeError, why: 134c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg print '* failed to convert %r: %s' % (t, why) 135c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg raise 13634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 137bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburgdef python_mapdef_code(varname, map, comments=1, precisions=(2, 4)): 13834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 139c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l = [] 140c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append = l.append 141c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if map.has_key("IDENTITY"): 142c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append("%s = codecs.make_identity_dict(range(%d))" % 143c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (varname, map["IDENTITY"])) 144c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append("%s.update({" % varname) 145c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg splits = 1 146c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg del map["IDENTITY"] 147c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg identity = 1 14834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 149c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append("%s = {" % varname) 150c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg splits = 0 151c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg identity = 0 15234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 153c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mappings = map.items() 154c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mappings.sort() 155c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg i = 0 156bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg key_precision, value_precision = precisions 157c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for mapkey, mapvalue in mappings: 158c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment = '' 159c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapkey, tuple): 160c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapkey, mapcomment) = mapkey 161c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapvalue, tuple): 162c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapvalue, mapcomment) = mapvalue 163c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapkey is None: 164c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg continue 165c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if (identity and 166c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapkey == mapvalue and 167c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapkey < 256): 168c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # No need to include identity mappings, since these 169c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # are already set for the first 256 code points. 170c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg continue 171bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg key = hexrepr(mapkey, key_precision) 172bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg value = hexrepr(mapvalue, value_precision) 173c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapcomment and comments: 174c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %s: %s,\t# %s' % (key, value, mapcomment)) 175c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 176c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %s: %s,' % (key, value)) 177c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg i += 1 178c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if i == 4096: 179c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Split the definition into parts to that the Python 180c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # parser doesn't dump core 181c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if splits == 0: 182c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('}') 183c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 184c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('})') 185c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('%s.update({' % varname) 186c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg i = 0 187c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg splits = splits + 1 188c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if splits == 0: 189c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('}') 19034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 191c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('})') 192c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 193c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return l 194c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 195bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburgdef python_tabledef_code(varname, map, comments=1, key_precision=2): 196c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 197c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l = [] 198c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append = l.append 199c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append('%s = (' % varname) 200c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 201c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Analyze map and create table dict 202c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mappings = map.items() 203c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mappings.sort() 204c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg table = {} 205c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg maxkey = 0 206c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if map.has_key('IDENTITY'): 207c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for key in range(256): 208c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg table[key] = (key, '') 209c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg maxkey = 255 210c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg del map['IDENTITY'] 211c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for mapkey, mapvalue in mappings: 212c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment = '' 213c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapkey, tuple): 214c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapkey, mapcomment) = mapkey 215c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapvalue, tuple): 216c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg (mapvalue, mapcomment) = mapvalue 217c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapkey is None: 218c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg continue 219c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg table[mapkey] = (mapvalue, mapcomment) 220c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapkey > maxkey: 221c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg maxkey = mapkey 222c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if maxkey > MAX_TABLE_SIZE: 223c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Table too large 224c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return None 225c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 226c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Create table code 227c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg for key in range(maxkey + 1): 228c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if key not in table: 229c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapvalue = None 230c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment = 'UNDEFINED' 231c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 232c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapvalue, mapcomment = table[key] 233c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapvalue is None: 234c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapchar = UNI_UNDEFINED 235c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 236c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if isinstance(mapvalue, tuple): 237c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # 1-n mappings not supported 238c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return None 23934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 240c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapchar = unichr(mapvalue) 241c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if mapcomment and comments: 242c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %r\t# %s -> %s' % (mapchar, 243bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg hexrepr(key, key_precision), 244c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mapcomment)) 245c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 246c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(' %r' % mapchar) 24734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 248c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg append(')') 249c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return l 250c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 251c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburgdef codegen(name, map, comments=1): 25234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 25334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ Returns Python source for the given map. 25434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 25534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum Comments are included in the source, if comments is true (default). 25634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 25734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum """ 258c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg # Generate code 259c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg decoding_map_code = python_mapdef_code( 260c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 'decoding_map', 261c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg map, 262c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg comments=comments) 263c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg decoding_table_code = python_tabledef_code( 264c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 'decoding_table', 265c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg map, 266c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg comments=comments) 267c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg encoding_map_code = python_mapdef_code( 268c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 'encoding_map', 269c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg codecs.make_encoding_map(map), 270bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg comments=comments, 271bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg precisions=(4, 2)) 272c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 27334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum l = [ 27434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum '''\ 275a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg""" Python Character Mapping Codec generated from '%s' with gencodec.py. 27634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 27734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum"""#" 27834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 27934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumimport codecs 28034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### Codec APIs 28234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumclass Codec(codecs.Codec): 28434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum def encode(self,input,errors='strict'): 28634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 28734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return codecs.charmap_encode(input,errors,encoding_map) 28870c4378dbcfdcbeef6fb3aa348f32ed862fe8eb7Tim Peters 28934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum def decode(self,input,errors='strict'): 290c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg''' % name 291c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg ] 292c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg if decoding_table_code: 293c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.append('''\ 294c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return codecs.charmap_decode(input,errors,decoding_table)''') 295c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg else: 296c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.append('''\ 297c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg return codecs.charmap_decode(input,errors,decoding_map)''') 298c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 299c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.append(''' 30034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumclass StreamWriter(Codec,codecs.StreamWriter): 30134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum pass 30270c4378dbcfdcbeef6fb3aa348f32ed862fe8eb7Tim Peters 30334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumclass StreamReader(Codec,codecs.StreamReader): 30434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum pass 30534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 30634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### encodings module API 30734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 30834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef getregentry(): 30934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 31034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum return (Codec().encode,Codec().decode,StreamReader,StreamWriter) 311bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg''') 31234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 313bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg # Add decoding table or map (with preference to the table) 314bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg if not decoding_table_code: 315bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg l.append(''' 31634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### Decoding Map 317c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg''') 318bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg l.extend(decoding_map_code) 319bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg else: 320c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.append(''' 321c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg### Decoding Table 322c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg''') 323c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.extend(decoding_table_code) 32470c4378dbcfdcbeef6fb3aa348f32ed862fe8eb7Tim Peters 325bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg # Add encoding map 326c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.append(''' 32734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum### Encoding Map 32834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum''') 329c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg l.extend(encoding_map_code) 330bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg 331bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg # Final new-line 332bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg l.append('\n') 333c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg 334aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald return '\n'.join(l) 33534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 33634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef pymap(name,map,pyfile,comments=1): 33734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 33834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum code = codegen(name,map,comments) 33934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f = open(pyfile,'w') 34034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.write(code) 34134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.close() 34234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 34334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef marshalmap(name,map,marshalfile): 34434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 34534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum d = {} 34634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for e,(u,c) in map.items(): 34734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum d[e] = (u,c) 34834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f = open(marshalfile,'wb') 34934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum marshal.dump(d,f) 35034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum f.close() 35134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 35234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef convertdir(dir,prefix='',comments=1): 35334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 35434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum mapnames = os.listdir(dir) 35534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for mapname in mapnames: 356c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg mappathname = os.path.join(dir, mapname) 357bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg if not os.path.isfile(mappathname): 358bd20ea55bc7a044a773e6824f7fcef4f5669d44cMarc-André Lemburg continue 35934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum name = os.path.split(mapname)[1] 360aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald name = name.replace('-','_') 361aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald name = name.split('.')[0] 362aaab30e00cc3e8d90c71b8657c284feeb4ac1413Walter Dörwald name = name.lower() 36334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum codefile = name + '.py' 36434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum marshalfile = name + '.mapping' 36534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print 'converting %s to %s and %s' % (mapname, 36634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum prefix + codefile, 36734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum prefix + marshalfile) 36834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 36934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum map = readmap(os.path.join(dir,mapname)) 37034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not map: 37134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print '* map is empty; skipping' 37234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 373c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg pymap(mappathname, map, prefix + codefile,comments) 374c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg marshalmap(mappathname, map, prefix + marshalfile) 375c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg except ValueError, why: 376c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg print '* conversion failed: %s' % why 377c5694c8bf4bf2008b42e0107fb245415df4147fdMarc-André Lemburg raise 37834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 37934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumdef rewritepythondir(dir,prefix='',comments=1): 38070c4378dbcfdcbeef6fb3aa348f32ed862fe8eb7Tim Peters 38134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum mapnames = os.listdir(dir) 38234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum for mapname in mapnames: 383a866df806dd0ffd439bbba873ab9f3da7080e0a0Marc-André Lemburg if not mapname.endswith('.mapping'): 38434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum continue 38534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum codefile = mapname[:-len('.mapping')] + '.py' 38634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print 'converting %s to %s' % (mapname, 38734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum prefix + codefile) 38834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum try: 38934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum map = marshal.load(open(os.path.join(dir,mapname), 39034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 'rb')) 39134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if not map: 39234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print '* map is empty; skipping' 39334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 39434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum pymap(mapname, map, prefix + codefile,comments) 39534a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum except ValueError, why: 39634a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum print '* conversion failed: %s' % why 39734a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 39834a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossumif __name__ == '__main__': 39934a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum 40034a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum import sys 40134a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum if 1: 40234a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum apply(convertdir,tuple(sys.argv[1:])) 40334a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum else: 40434a79115c5d5be53581f49ced5a5a17171cabb7dGuido van Rossum apply(rewritepythondir,tuple(sys.argv[1:])) 405