1* Copyright (C) 2004-2013, International Business Machines 2* Corporation and others. All Rights Reserved. 3* 4* file name: changes.txt 5* encoding: US-ASCII 6* tab size: 8 (not used) 7* indentation:4 8* 9* created on: 2004may06 10* created by: Markus W. Scherer 11* 12* change log for Unicode updates 13 14---------------------------------------------------------------------------- *** 15 16Unicode 6.3 update 17 18http://www.unicode.org/review/pri249/ -- beta review 19http://www.unicode.org/reports/uax-proposed-updates.html 20http://www.unicode.org/versions/beta-6.3.0.html#notable_issues 21http://www.unicode.org/reports/tr44/tr44-11.html 22 23*** ICU Trac 24 25- ticket 10128: update ICU to Unicode 6.3 beta 26- ticket 10168: update ICU to Unicode 6.3 final 27- C++ branches/markus/uni63 at r33552 from trunk at r33551 28- Java branches/markus/uni63 at r33550 from trunk at r33553 29 30- ticket 10142: implement Unicode 6.3 bidi algorithm additions 31 32*** Unicode version numbers 33- makedata.mak 34- uchar.h 35 (configure.in & configure: have been modified to extract the version from uchar.h) 36- com.ibm.icu.util.VersionInfo 37- com.ibm.icu.dev.test.lang.UCharacterTest.VERSION_ 38 39- Run ICU4C "configure" _after_ updating the Unicode version number in uchar.h 40 so that the makefiles see the new version number. 41 42*** data files & enums & parser code 43 44* file preparation 45 46- download UCD, UCA & IDNA files 47- make sure that the Unicode data folder passed into preparseucd.py 48 includes a copy of the latest IdnaMappingTable.txt (can be in some subfolder) 49- modify preparseucd.py: 50 parse new file BidiBrackets.txt 51 with new properties bpb=Bidi_Paired_Bracket and bpt=Bidi_Paired_Bracket_Type 52- ~/svn.icutools/trunk/src/unicode$ py/preparseucd.py ~/unidata/uni63/20130425 ~/svn.icu/uni63/src ~/svn.icutools/trunk/src 53- This writes files (especially ppucd.txt) to the ICU4C unidata and testdata subfolders. 54- Check test file diffs for previously commented-out, known-failing data lines; 55 probably need to keep those commented out. 56 57* PropertyAliases.txt changes 58- 1 new Enumerated Property 59 bpt ; Bidi_Paired_Bracket_Type 60 -> uchar.h & UProperty.java & UCharacter.BidiPairedBracketType 61 -> ubidi_props.h & .c & UBiDiProps.java 62 -> remember to write the max value at UBIDI_MAX_VALUES_INDEX 63 -> uprops.cpp 64 -> change ubidi.icu format version from 2.0 to 2.1 65- 1 new Miscellaneous Property 66 bpb ; Bidi_Paired_Bracket 67 -> uchar.h & UProperty.java 68 -> ppucd.h & .cpp 69 70* PropertyValueAliases.txt changes 71- 3 Bidi_Paired_Bracket_Type (bpt) values: 72 bpt; c ; Close 73 bpt; n ; None 74 bpt; o ; Open 75 -> uchar.h & UCharacter.BidiPairedBracketType 76 -> ubidi_props.h & .c & UBiDiProps.java 77 -> change ubidi.icu format version from 2.0 to 2.1 78- 4 new Bidi_Class (bc) values: 79 bc ; FSI ; First_Strong_Isolate 80 bc ; LRI ; Left_To_Right_Isolate 81 bc ; RLI ; Right_To_Left_Isolate 82 bc ; PDI ; Pop_Directional_Isolate 83 -> uchar.h & UCharacterEnums.ECharacterDirection 84 -> until the bidi code gets updated, 85 Roozbeh suggests mapping the new bc values to ON (Other_Neutral) 86- 3 new Word_Break (WB) values: 87 WB ; HL ; Hebrew_Letter 88 WB ; SQ ; Single_Quote 89 WB ; DQ ; Double_Quote 90 -> uchar.h & UCharacter.WordBreak 91 -> first time Word_Break numeric constants exceed 4 bits (now 17 values) 92- 2 new script codes from ISO 15924 http://www.unicode.org/iso15924/codechanges.html 93 (added 2012-10-16) 94 Aghb 239 Caucasian Albanian 95 Mahj 314 Mahajani 96 -> uscript.h 97 -> com.ibm.icu.lang.UScript 98 find USCRIPT_([^ ]+) *= ([0-9]+),(.+) 99 replace public static final int \1 = \2;\3 100 -> preparseucd.py _scripts_only_in_iso15924 101 -> add to expectedLong and expectedShort names in cintltst/cucdapi.c/TestUScriptCodeAPI() 102 and in com.ibm.icu.dev.test.lang.TestUScript.java 103 -> update Script metadata: SCRIPT_PROPS[] in uscript_props.cpp & UScript.ScriptMetadata 104 (not strictly necessary for NOT_ENCODED scripts) 105 106* generate normalization data files 107- ~/svn.icu/uni63/dbg$ export LD_LIBRARY_PATH=~/svn.icu/uni63/dbg/lib 108- ~/svn.icu/uni63/dbg$ SRC_DATA_IN=~/svn.icu/uni63/src/source/data/in 109- ~/svn.icu/uni63/dbg$ UNIDATA=~/svn.icu/uni63/src/source/data/unidata 110- ~/svn.icu/uni63/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfc.nrm -s $UNIDATA/norm2 nfc.txt 111- ~/svn.icu/uni63/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt 112- ~/svn.icu/uni63/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc_cf.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt nfkc_cf.txt 113- ~/svn.icu/uni63/dbg$ bin/gennorm2 -o $SRC_DATA_IN/uts46.nrm -s $UNIDATA/norm2 nfc.txt uts46.txt 114 115* build ICU (make install) 116 so that the tools build can pick up the new definitions from the installed header files. 117 118~/svn.icu/uni63/dbg$ echo;echo;make -j5 install > out.txt 2>&1 ; tail -n 20 out.txt 119 120* build Unicode tools using CMake+make 121 122~/svn.icutools/trunk/src/unicode/c/icudefs.txt: 123 124# Location (--prefix) of where ICU was installed. 125set(ICU_INST_DIR /home/mscherer/svn.icu/uni63/inst) 126# Location of the ICU source tree. 127set(ICU_SRC_DIR /home/mscherer/svn.icu/uni63/src) 128 129~/svn.icutools/trunk/dbg/unicode/c$ cmake ../../../src/unicode/c 130~/svn.icutools/trunk/dbg/unicode/c$ make 131 132* generate core properties data files 133- ~/svn.icutools/trunk/dbg/unicode/c$ genprops/genprops ~/svn.icu/uni63/src 134- ~/svn.icutools/trunk/dbg/unicode/c$ genuca/genuca -i ~/svn.icu/uni63/dbg/data/out/build/icudt52l ~/svn.icu/uni63/src 135- rebuild ICU (make install) & tools 136- run genuca again (see step above) so that it picks up the new case mappings and nfc.nrm 137- rebuild ICU (make install) & tools 138 139* update uts46test.cpp and UTS46Test.java if there are new characters that are equivalent to 140 sequences with non-LDH ASCII (that is, their decompositions contain '=' or similar) 141- grep IdnaMappingTable.txt or uts46.txt for "disallowed_STD3_valid" on non-ASCII characters 142- Unicode 6.0..6.3: U+2260, U+226E, U+226F 143- nothing new in 6.3, no test file to update 144 145* update Java data files 146- refresh just the UCD-related files, just to be safe 147- see (ICU4C)/source/data/icu4j-readme.txt 148- mkdir /tmp/icu4j 149- ~/svn.icu/uni63/dbg$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 150 output: 151 ... 152 Unicode .icu files built to ./out/build/icudt52l 153 mkdir -p ./out/icu4j/com/ibm/icu/impl/data/icudt52b 154 mkdir -p ./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt52b 155 echo pnames.icu ubidi.icu ucase.icu uprops.icu > ./out/icu4j/add.txt 156 LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/icupkg ./out/tmp/icudt52l.dat ./out/icu4j/icudt52b.dat -a ./out/icu4j/add.txt -s ./out/build/icudt52l -x '*' -tb -d ./out/icu4j/com/ibm/icu/impl/data/icudt52b 157 mv ./out/icu4j/"com/ibm/icu/impl/data/icudt52b/zoneinfo64.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt52b/metaZones.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt52b/timezoneTypes.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt52b/windowsZones.res" "./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt52b" 158 jar cf ./out/icu4j/icudata.jar -C ./out/icu4j com/ibm/icu/impl/data/icudt52b/ 159 mkdir -p /tmp/icu4j/main/shared/data 160 cp ./out/icu4j/icudata.jar /tmp/icu4j/main/shared/data 161 jar cf ./out/icu4j/icutzdata.jar -C ./out/icu4j/tzdata com/ibm/icu/impl/data/icudt52b/ 162 mkdir -p /tmp/icu4j/main/shared/data 163 cp ./out/icu4j/icutzdata.jar /tmp/icu4j/main/shared/data 164 make[1]: Leaving directory `/home/mscherer/svn.icu/uni63/dbg/data' 165- copy the big-endian Unicode data files to another location, 166 separate from the other data files 167 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt52b/coll 168 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt52b/brkitr 169 ~/svn.icu/uni63/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt52b/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt52b 170 ~/svn.icu/uni63/dbg/data/out/icu4j$ rm /tmp/icu4j/com/ibm/icu/impl/data/icudt52b/cnvalias.icu 171 ~/svn.icu/uni63/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt52b/*.nrm /tmp/icu4j/com/ibm/icu/impl/data/icudt52b 172 ~/svn.icu/uni63/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt52b/coll/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt52b/coll 173 ~/svn.icu/uni63/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt52b/brkitr/* /tmp/icu4j/com/ibm/icu/impl/data/icudt52b/brkitr 174- refresh ICU4J 175 ~/svn.icu/uni63/dbg/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt52b 176 177* refresh Java test .txt files 178- copy new .txt files into ICU4J's main/tests/core/src/com/ibm/icu/dev/data/unicode 179 180* UCA -- mostly skipped for ICU 52 / Unicode 6.3, except update coll/* files 181 182- get output from Mark's tools; look in http://www.unicode.org/Public/UCA/<beta version>/ 183- CLDR root files for ICU are in CollationAuxiliary.zip; unpack that 184- update source/data/unidata/FractionalUCA.txt with FractionalUCA_SHORT.txt 185- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt 186 (note removing the underscore before "Rules") 187- update (ICU4C)/source/test/testdata/CollationTest_*.txt 188 and (ICU4J)/main/tests/collate/src/com/ibm/icu/dev/data/CollationTest_*.txt 189 with output from Mark's Unicode tools (..._CLDR_..._SHORT.txt) 190- check test file diffs for previously commented-out, known-failing data lines; 191 probably need to keep those commented out 192- check FractionalUCA.txt for manual changes of lead bytes from IMPLICIT to Hani 193- run genuca, see command line above 194- rebuild ICU4C 195- refresh ICU4J collation data: 196 (subset of instructions above for properties data refresh, except copies all coll/*) 197 ~/svn.icu/uni63/dbg$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 198 ~/svn.icu/uni63/dbg$ mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt52b/coll 199 ~/svn.icu/uni63/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt52b/coll/* /tmp/icu4j/com/ibm/icu/impl/data/icudt52b/coll 200 ~/svn.icu/uni63/dbg/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt52b 201- run all tests with the *_SHORT.txt or the full files (the full ones have comments, useful for debugging) 202- note on intltest: if collate/UCAConformanceTest fails, then 203 utility/MultithreadTest/TestCollators will fail as well; 204 fix the conformance test before looking into the multi-thread test 205 206* test ICU, fix test code where necessary 207 208* When refreshing all of ICU4J data from ICU4C 209- ~/svn.icu/uni63/dbg$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 210- cp /tmp/icu4j/main/shared/data/icudata.jar ~/svn.icu4j/trunk/src/main/shared/data 211or 212- ~/svn.icu/uni63/dbg$ make ICU4J_ROOT=~/svn.icu4j/trunk/src icu4j-data-install 213 214*** LayoutEngine script information 215- skipped for Unicode 6.3: no new scripts 216 217*** merge the Unicode update branches back onto the trunk 218- do not merge the icudata.jar and testdata.jar, 219 instead rebuild them from merged & tested ICU4C 220 221---------------------------------------------------------------------------- *** 222 223Unicode 6.2 update 224 225http://www.unicode.org/review/pri230/ 226http://www.unicode.org/versions/beta-6.2.0.html 227http://www.unicode.org/reports/tr44/tr44-9.html#Unicode_6.2.0 228http://www.unicode.org/review/pri227/ Changes to Script Extensions Property Values 229http://www.unicode.org/review/pri228/ Changing some common characters from Punctuation to Symbol 230http://www.unicode.org/review/pri229/ Linebreaking Changes for Pictographic Symbols 231http://www.unicode.org/reports/tr46/tr46-8.html IDNA 232http://unicode.org/Public/idna/6.2.0/ 233 234*** ICU Trac 235 236- ticket 9515: Unicode 6.2: final ICU update 237 238- ticket 9514: UCA 6.2: fix UCARules.txt 239 240- ticket 9437: update ICU to Unicode 6.2 241- C++ branches/markus/uni62 at r32050 from trunk at r32041 242- Java branches/markus/uni62 at r32068 from trunk at r32066 243 244*** Unicode version numbers 245- makedata.mak 246- uchar.h 247 (configure.in & configure: have been modified to extract the version from uchar.h) 248- com.ibm.icu.util.VersionInfo 249- com.ibm.icu.dev.test.lang.UCharacterTest.VERSION_ 250 251*** data files & enums & parser code 252 253* file preparation 254 255- download UCD, UCA & IDNA files 256- make sure that the Unicode data folder passed into preparseucd.py 257 includes a copy of the latest IdnaMappingTable.txt (can be in some subfolder) 258- modify preparseucd.py: NamesList.txt is now in UTF-8 259- ~/svn.icu/tools/trunk/src/unicode$ py/preparseucd.py ~/uni62/20120816 ~/svn.icu/uni62/src ~/svn.icu/tools/trunk/src 260- This writes files (especially ppucd.txt) to the ICU4C unidata and testdata subfolders. 261- Check test file diffs for previously commented-out, known-failing data lines; 262 probably need to keep those commented out. 263 264* PropertyValueAliases.txt changes 265- 1 new Line_Break (lb) value: 266 lb ; RI ; Regional_Indicator 267 -> uchar.h & UCharacter.LineBreak 268- 1 new Word_Break (WB) value: 269 WB ; RI ; Regional_Indicator 270 -> uchar.h & UCharacter.WordBreak 271- 1 new Grapheme_Cluster_Break (GCB) value: 272 GCB; RI ; Regional_Indicator 273 -> uchar.h & UCharacter.GraphemeClusterBreak 274 275* 3 new numeric values 276 The new value -1, which was really supposed to be NaN but that would have required 277 new UnicodeData.txt syntax, can already be represented as a "fraction" of -1/1, 278 but encodeNumericValue() in corepropsbuilder.cpp had to be fixed. 279 cp;12456;na=CUNEIFORM NUMERIC SIGN NIGIDAMIN;nv=-1 280 cp;12457;na=CUNEIFORM NUMERIC SIGN NIGIDAESH;nv=-1 281 The two new values 216000 and 432000 require an addition to the encoding of numeric values. 282 cp;12432;na=CUNEIFORM NUMERIC SIGN SHAR2 TIMES GAL PLUS DISH;nv=216000 283 cp;12433;na=CUNEIFORM NUMERIC SIGN SHAR2 TIMES GAL PLUS MIN;nv=432000 284 -> uprops.h, uchar.c & UCharacterProperty.java 285 -> cucdtst.c & UCharacterTest.java 286 287* generate normalization data files 288- ~/svn.icu/uni62/dbg$ export LD_LIBRARY_PATH=~/svn.icu/uni62/dbg/lib 289- ~/svn.icu/uni62/dbg$ SRC_DATA_IN=~/svn.icu/uni62/src/source/data/in 290- ~/svn.icu/uni62/dbg$ UNIDATA=~/svn.icu/uni62/src/source/data/unidata 291- ~/svn.icu/uni62/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfc.nrm -s $UNIDATA/norm2 nfc.txt 292- ~/svn.icu/uni62/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt 293- ~/svn.icu/uni62/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc_cf.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt nfkc_cf.txt 294- ~/svn.icu/uni62/dbg$ bin/gennorm2 -o $SRC_DATA_IN/uts46.nrm -s $UNIDATA/norm2 nfc.txt uts46.txt 295 296* build ICU (make install) 297 so that the tools build can pick up the new definitions from the installed header files. 298* build Unicode tools using CMake+make 299 300* generate core properties data files 301- ~/svn.icu/tools/trunk/dbg/unicode$ c/genprops/genprops ~/svn.icu/uni62/src 302- in initial bootstrapping, change the UCA version 303 in source/data/unidata/FractionalUCA.txt to match the new Unicode version 304- ~/svn.icu/tools/trunk/dbg/unicode$ c/genuca/genuca -i ~/svn.icu/uni62/dbg/data/out/build/icudt50l ~/svn.icu/uni62/src 305- rebuild ICU (make install) & tools 306 + if genrb fails to build coll/root.res with an U_INVALID_FORMAT_ERROR, 307 check if the UCA version in FractionalUCA.txt matches the new Unicode version 308 (see step above) 309- run genuca again (see step above) so that it picks up the new case mappings and nfc.nrm 310- rebuild ICU (make install) & tools 311 312* update uts46test.cpp and UTS46Test.java if there are new characters that are equivalent to 313 sequences with non-LDH ASCII (that is, their decompositions contain '=' or similar) 314- grep IdnaMappingTable.txt or uts46.txt for "disallowed_STD3_valid" on non-ASCII characters 315- Unicode 6.0..6.2: U+2260, U+226E, U+226F 316- nothing new in 6.2, no test file to update 317 318* update Java data files 319- refresh just the UCD-related files, just to be safe 320- see (ICU4C)/source/data/icu4j-readme.txt 321- mkdir /tmp/icu4j 322- ~/svn.icu/uni62/dbg$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 323 output: 324 ... 325 Unicode .icu files built to ./out/build/icudt50l 326 mkdir -p ./out/icu4j/com/ibm/icu/impl/data/icudt50b 327 mkdir -p ./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt50b 328 echo pnames.icu ubidi.icu ucase.icu uprops.icu > ./out/icu4j/add.txt 329 LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/icupkg ./out/tmp/icudt50l.dat ./out/icu4j/icudt50b.dat -a ./out/icu4j/add.txt -s ./out/build/icudt50l -x '*' -tb -d ./out/icu4j/com/ibm/icu/impl/data/icudt50b 330 mv ./out/icu4j/"com/ibm/icu/impl/data/icudt50b/zoneinfo64.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt50b/metaZones.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt50b/timezoneTypes.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt50b/windowsZones.res" "./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt50b" 331 jar cf ./out/icu4j/icudata.jar -C ./out/icu4j com/ibm/icu/impl/data/icudt50b/ 332 mkdir -p /tmp/icu4j/main/shared/data 333 cp ./out/icu4j/icudata.jar /tmp/icu4j/main/shared/data 334 jar cf ./out/icu4j/icutzdata.jar -C ./out/icu4j/tzdata com/ibm/icu/impl/data/icudt50b/ 335 mkdir -p /tmp/icu4j/main/shared/data 336 cp ./out/icu4j/icutzdata.jar /tmp/icu4j/main/shared/data 337 make[1]: Leaving directory `/home/mscherer/svn.icu/uni62/dbg/data' 338- copy the big-endian Unicode data files to another location, 339 separate from the other data files 340 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt50b/coll 341 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt50b/brkitr 342 ~/svn.icu/uni62/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt50b/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt50b 343 ~/svn.icu/uni62/dbg/data/out/icu4j$ rm /tmp/icu4j/com/ibm/icu/impl/data/icudt50b/cnvalias.icu 344 ~/svn.icu/uni62/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt50b/*.nrm /tmp/icu4j/com/ibm/icu/impl/data/icudt50b 345 ~/svn.icu/uni62/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt50b/coll/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt50b/coll 346 ~/svn.icu/uni62/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt50b/brkitr/* /tmp/icu4j/com/ibm/icu/impl/data/icudt50b/brkitr 347- refresh ICU4J 348 ~/svn.icu/uni62/dbg/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt50b 349 350* refresh Java test .txt files 351- copy new .txt files into ICU4J's main/tests/core/src/com/ibm/icu/dev/data/unicode 352 353* UCA 354 355- get output from Mark's tools; look in http://www.unicode.org/Public/UCA/<beta version>/ 356- CLDR root files for ICU are in CollationAuxiliary.zip; unpack that 357- update source/data/unidata/FractionalUCA.txt with FractionalUCA_SHORT.txt 358- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt 359 (note removing the underscore before "Rules") 360- update (ICU4C)/source/test/testdata/CollationTest_*.txt 361 and (ICU4J)/main/tests/collate/src/com/ibm/icu/dev/data/CollationTest_*.txt 362 with output from Mark's Unicode tools (..._CLDR_..._SHORT.txt) 363- check test file diffs for previously commented-out, known-failing data lines; 364 probably need to keep those commented out 365- check FractionalUCA.txt for manual changes of lead bytes from IMPLICIT to Hani 366- run genuca, see command line above 367- rebuild ICU4C 368- refresh ICU4J collation data: 369 (subset of instructions above for properties data refresh, except copies all coll/*) 370 ~/svn.icu/uni62/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 371 ~/svn.icu/uni62/bld$ mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt50b/coll 372 ~/svn.icu/uni62/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt50b/coll/* /tmp/icu4j/com/ibm/icu/impl/data/icudt50b/coll 373 ~/svn.icu/uni62/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt50b 374- run all tests with the *_SHORT.txt or the full files (the full ones have comments, useful for debugging) 375- note on intltest: if collate/UCAConformanceTest fails, then 376 utility/MultithreadTest/TestCollators will fail as well; 377 fix the conformance test before looking into the multi-thread test 378 379* test ICU, fix test code where necessary 380 381* When refreshing all of ICU4J data from ICU4C 382- ~/svn.icu/uni62/dbg$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 383- cp /tmp/icu4j/main/shared/data/icudata.jar ~/svn.icu4j/trunk/src/main/shared/data 384or 385- ~/svn.icu/uni62/dbg$ make ICU4J_ROOT=~/svn.icu4j/trunk/src icu4j-data-install 386 387*** LayoutEngine script information 388- skipped for Unicode 6.2: no new scripts 389 390*** merge the Unicode update branches back onto the trunk 391- do not merge the icudata.jar and testdata.jar, 392 instead rebuild them from merged & tested ICU4C 393 394---------------------------------------------------------------------------- *** 395 396Future Unicode update 397 398Tools simplified since the Unicode 6.1 update. See 399- http://site.icu-project.org/design/props/ppucd 400- http://bugs.icu-project.org/trac/wiki/Markus/ReviewTicket8972 401 402* Unicode version numbers 403- icutools/unicode/makedefs.sh was deleted, so one fewer place for version & path updates 404 405* file preparation 406- ucdcopy.py, idna2nrm.py and genpname/preparse.pl replaced by preparseucd.py: 407- ~/svn.icu/tools/trunk/src/unicode$ py/preparseucd.py ~/uni61/20120118 ~/svn.icu/trunk/src ~/svn.icu/tools/trunk/src 408- This writes files (especially ppucd.txt) to the ICU4C unidata and testdata subfolders. 409- Check test file diffs for previously commented-out, known-failing data lines; 410 probably need to keep those commented out. 411 412* PropertyValueAliases.txt changes 413- Script codes that are in ISO 15924 but not in Unicode are now listed in 414 preparseucd.py, in the _scripts_only_in_iso15924 variable. 415 If there are new ISO codes, then add them. 416 If Unicode adds some of them, then remove them from the .py variable. 417 418* UnicodeData.txt changes 419- No more manual changes for CJK ranges for algorithmic names; 420 those are now written to ppucd.txt and genprops reads them from there. 421 422* generate core properties data files (makeprops.sh was deleted) 423- ~/svn.icu/tools/trunk/dbg/unicode$ c/genprops/genprops ~/svn.icu/trunk/src 424 425* no more manual updates of source/data/unidata/norm2/nfkc_cf.txt 426- it is now generated by preparseucd.py 427 428* no more separate idna2nrm.py run and manual copying to generate source/data/unidata/norm2/uts46.txt 429- it is now generated by preparseucd.py 430- make sure that the Unicode data folder passed into preparseucd.py 431 includes a copy of http://www.unicode.org/Public/idna/6.1.0/IdnaMappingTable.txt 432 (can be in some subfolder) 433 434* generate normalization data files 435- ~/svn.icu/trunk/dbg$ export LD_LIBRARY_PATH=~/svn.icu/trunk/dbg/lib 436- ~/svn.icu/trunk/dbg$ SRC_DATA_IN=~/svn.icu/trunk/src/source/data/in 437- ~/svn.icu/trunk/dbg$ UNIDATA=~/svn.icu/trunk/src/source/data/unidata 438- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfc.nrm -s $UNIDATA/norm2 nfc.txt 439- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt 440- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc_cf.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt nfkc_cf.txt 441- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/uts46.nrm -s $UNIDATA/norm2 nfc.txt uts46.txt 442 443* build ICU (make install) 444* build Unicode tools using CMake+make 445 446* new way to call genuca (makeuca.sh was deleted) 447- ~/svn.icu/tools/trunk/dbg/unicode$ c/genuca/genuca -i ~/svn.icu/trunk/dbg/data/out/build/icudt49l ~/svn.icu/trunk/src 448 449---------------------------------------------------------------------------- *** 450 451Unicode 6.1 update 452 453*** ICU Trac 454 455- ticket 8995 final update to Unicode 6.1 456- ticket 8994 regenerate source/layout/CanonData.cpp 457 458- ticket 8961 support Unicode "Age" value *names* 459- ticket 8963 support multiple character name aliases & types 460 461- ticket 8827 "update ICU to Unicode 6.1" 462- C++ branches/markus/uni61 at r30864 from trunk at r30843 463- Java branches/markus/uni61 at r30865 from trunk at r30863 464 465*** Unicode version numbers 466- makedata.mak 467- uchar.h 468 (configure.in & configure: have been modified to extract the version from uchar.h) 469- com.ibm.icu.util.VersionInfo 470- icutools/unicode/makedefs.sh 471 + also review & update other definitions in that file, 472 e.g. the ICU version in this path: BLD_DATA_FILES=$ICU_BLD/data/out/build/icudt49l 473 474*** data files & enums & parser code 475 476* file preparation 477 478~/svn.icu/tools/trunk/src/unicode/c/genprops/misc$ ./ucdcopy.py ~/uni61/20111205/ucd ~/uni61/processed 479- This prepares both unidata and testdata files in respective output subfolders. 480- Check test file diffs for previously commented-out, known-failing data lines; 481 probably need to keep those commented out. 482 483* PropertyValueAliases.txt changes 484- 11 new block names: 485 Arabic_Extended_A 486 Arabic_Mathematical_Alphabetic_Symbols 487 Chakma 488 Meetei_Mayek_Extensions 489 Meroitic_Cursive 490 Meroitic_Hieroglyphs 491 Miao 492 Sharada 493 Sora_Sompeng 494 Sundanese_Supplement 495 Takri 496 -> add to uchar.h 497 -> add to UCharacter.UnicodeBlock IDs 498 Eclipse find UBLOCK_([^ ]+) = ([0-9]+), (/.+) 499 replace public static final int \1_ID = \2; \3 500 -> add to UCharacter.UnicodeBlock objects 501 Eclipse find UBLOCK_([^ ]+) = [0-9]+, (/.+) 502 replace public static final UnicodeBlock \1 = new UnicodeBlock("\1", \1_ID); \2 503- 1 new Joining_Group (jg) value: 504 Rohingya_Yeh 505 -> uchar.h & UCharacter.JoiningGroup 506- 2 new Line_Break (lb) values: 507 CJ=Conditional_Japanese_Starter 508 HL=Hebrew_Letter 509 -> uchar.h & UCharacter.LineBreak 510- 7 new scripts: 511 sc ; Cakm ; Chakma 512 sc ; Merc ; Meroitic_Cursive 513 sc ; Mero ; Meroitic_Hieroglyphs 514 sc ; Plrd ; Miao 515 sc ; Shrd ; Sharada 516 sc ; Sora ; Sora_Sompeng 517 sc ; Takr ; Takri 518 -> remove these from SyntheticPropertyValueAliases.txt 519 -> fix expectedLong names in cucdapi.c/TestUScriptCodeAPI() 520 and in com.ibm.icu.dev.test.lang.TestUScript.java 521- 2 new script codes from ISO 15924 http://www.unicode.org/iso15924/codechanges.html 522 (added 2011-06-21) 523 Khoj 322 Khojki 524 Tirh 326 Tirhuta 525 and another one added 2011-12-09 526 Hluw 080 Anatolian Hieroglyphs (Luwian Hieroglyphs, Hittite Hieroglyphs) 527 -> uscript.h 528 -> com.ibm.icu.lang.UScript 529 find USCRIPT_([^ ]+) *= ([0-9]+),(.+) 530 replace public static final int \1 = \2;\3 531 -> SyntheticPropertyValueAliases.txt 532 -> add to expectedLong and expectedShort names in cintltst/cucdapi.c/TestUScriptCodeAPI() 533 and in com.ibm.icu.dev.test.lang.TestUScript.java 534 535* UnicodeData.txt changes 536- the last Unihan code point changes from U+9FCB to U+9FCC 537 search for both 9FCB (end) and 9FCC (limit) (regex 9FC[BC], case-insensitive) 538 + do change gennames.c 539 + do change swapCJK() in ucol.cpp & ImplicitCEGenerator.java 540 541* DerivedBidiClass.txt changes 542- 2 new default-AL blocks: 543# Arabic Extended-A: U+08A0 - U+08FF (was default-R) 544# Arabic Mathematical Alphabetic Symbols: 545# U+1EE00 - U+1EEFF (was default-R) 546- 2 new default-R blocks: 547# Meroitic Hieroglyphs: 548# U+10980 - U+1099F 549# Meroitic Cursive: U+109A0 - U+109FF 550 -> should be picked up by the explicit data in the file 551 552* NameAliases.txt changes 553- from 554 # Each line has two fields 555 # First field: Code point 556 # Second field: Alias 557- to 558 # Each line has three fields, as described here: 559 # 560 # First field: Code point 561 # Second field: Alias 562 # Third field: Type 563- Also, the file previously allowed multiple aliases but only now does it 564 actually provide multiple, even multiple of the same type. For example, 565 FEFF;BYTE ORDER MARK;alternate 566 FEFF;BOM;abbreviation 567 FEFF;ZWNBSP;abbreviation 568- This breaks our gennames parser, unames.icu data structure, and API. 569 Fix gennames to only pick up "correction" aliases. 570 New ticket #8963 for further changes. 571 572* run genpname/preparse.pl (on Linux) 573 + cd ~/svn.icu/tools/trunk/src/unicode/c/genpname 574 + make sure that data.h is writable 575 + perl preparse.pl ~/svn.icu/trunk/src > out.txt 576 + preparse.pl shows no errors, out.txt Info and Warning lines look ok 577 578* build ICU (make install) 579 so that the tools build can pick up the new definitions from the installed header files. 580* build Unicode tools (at least genpname) using CMake+make 581 582* run genpname 583 (builds both pnames.icu and propname_data.h) 584- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/data/in 585- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/common --csource 586 587* build ICU (make install) 588* build Unicode tools using CMake+make 589 590* update source/data/unidata/norm2/nfkc_cf.txt 591- follow the instructions in nfkc_cf.txt for updating it from DerivedNormalizationProps.txt 592 593* update source/data/unidata/norm2/uts46.txt 594- download http://www.unicode.org/Public/idna/6.1.0/IdnaMappingTable.txt 595 to ~/svn.icu/tools/trunk/src/unicode/py 596- adjust idna2nrm.py to remove "; NV8": For UTS #46, we do not care about "not valid in IDNA2008". 597- ~/svn.icu/tools/trunk/src/unicode/py$ ./idna2nrm.py 598- ~/svn.icu/tools/trunk/src/unicode/py$ cp uts46.txt ~/svn.icu/trunk/src/source/data/unidata/norm2 599 600* update uts46test.cpp and UTS46Test.java if there are new characters that are equivalent to 601 sequences with non-LDH ASCII (that is, their decompositions contain '=' or similar) 602- grep IdnaMappingTable.txt or uts46.txt for "disallowed_STD3_valid" on non-ASCII characters 603- Unicode 6.0..6.1: U+2260, U+226E, U+226F 604- nothing new in 6.1, no test file to update 605 606* generate core properties data files 607- in initial bootstrapping, change the UCA version 608 in source/data/unidata/FractionalUCA.txt to match the new Unicode version 609- ~/svn.icu/tools/trunk/src/unicode$ ./makeprops.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld 610- rebuild ICU & tools 611 + if genrb fails to build coll/root.res with an U_INVALID_FORMAT_ERROR, 612 check if the UCA version in FractionalUCA.txt matches the new Unicode version 613 (see step above) 614- run makeuca.sh so that genuca picks up the new case mappings and nfc.nrm: 615 ~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld 616- rebuild ICU & tools 617 618* update Java data files 619- refresh just the UCD-related files, just to be safe 620- see (ICU4C)/source/data/icu4j-readme.txt 621- mkdir /tmp/icu4j 622- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 623 output: 624 ... 625 Unicode .icu files built to ./out/build/icudt49l 626 mkdir -p ./out/icu4j/com/ibm/icu/impl/data/icudt49b 627 mkdir -p ./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt49b 628 echo pnames.icu ubidi.icu ucase.icu uprops.icu > ./out/icu4j/add.txt 629 LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/icupkg ./out/tmp/icudt49l.dat ./out/icu4j/icudt49b.dat -a ./out/icu4j/add.txt -s ./out/build/icudt49l -x '*' -tb -d ./out/icu4j/com/ibm/icu/impl/data/icudt49b 630 mv ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/zoneinfo64.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/metaZones.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/timezoneTypes.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/windowsZones.res" "./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt49b" 631 jar cf ./out/icu4j/icudata.jar -C ./out/icu4j com/ibm/icu/impl/data/icudt49b/ 632 mkdir -p /tmp/icu4j/main/shared/data 633 cp ./out/icu4j/icudata.jar /tmp/icu4j/main/shared/data 634 jar cf ./out/icu4j/icutzdata.jar -C ./out/icu4j/tzdata com/ibm/icu/impl/data/icudt49b/ 635 mkdir -p /tmp/icu4j/main/shared/data 636 cp ./out/icu4j/icutzdata.jar /tmp/icu4j/main/shared/data 637 make[1]: Leaving directory `/home/mscherer/svn.icu/trunk/bld/data' 638- copy the big-endian Unicode data files to another location, 639 separate from the other data files 640 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll 641 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/brkitr 642 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt49b 643 ~/svn.icu/trunk/bld/data/out/icu4j$ rm /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/cnvalias.icu 644 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/*.nrm /tmp/icu4j/com/ibm/icu/impl/data/icudt49b 645 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/coll/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll 646 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/brkitr/* /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/brkitr 647- refresh ICU4J 648 ~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt49b 649 650* refresh Java test .txt files 651- copy new .txt files into ICU4J's main/tests/core/src/com/ibm/icu/dev/data/unicode 652 653* test ICU so far, fix test code where necessary 654- temporarily ignore collation issues that look like UCA/UCD mismatches, 655 until UCA data is updated 656 657* UCA 658 659- get output from Mark's tools; look in 660 http://www.unicode.org/Public/UCA/6.1.0/CollationAuxiliary-<dev. version>.txt 661- update source/data/unidata/FractionalUCA.txt with FractionalUCA_SHORT.txt 662- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt 663 (note removing the underscore before "Rules") 664- update (ICU)/source/test/testdata/CollationTest_*.txt 665 and (ICU4J)/main/tests/collate/src/com/ibm/icu/dev/data/CollationTest_*.txt 666 with output from Mark's Unicode tools (..._CLDR_..._SHORT.txt) 667- check test file diffs for previously commented-out, known-failing data lines; 668 probably need to keep those commented out 669- check FractionalUCA.txt for manual changes of lead bytes from IMPLICIT to Hani 670- run makeuca.sh: 671 ~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld 672- rebuild ICU4C 673- refresh ICU4J collation data: 674 (subset of instructions above for properties data refresh, except copies all coll/*) 675 ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 676 ~/svn.icu/trunk/bld$ mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll 677 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/coll/* /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll 678 ~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt49b 679- run all tests with the *_SHORT.txt or the full files (the full ones have comments, useful for debugging) 680- note on intltest: if collate/UCAConformanceTest fails, then 681 utility/MultithreadTest/TestCollators will fail as well; 682 fix the conformance test before looking into the multi-thread test 683 684* When refreshing all of ICU4J data from ICU4C 685- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 686- cp /tmp/icu4j/main/shared/data/icudata.jar ~/svn.icu4j/trunk/src/main/shared/data 687or 688- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=~/svn.icu4j/trunk/src icu4j-data-install 689 690*** LayoutEngine script information 691 692(For details see the Unicode 5.2 change log below.) 693 694* Run icu4j-tools: com.ibm.icu.dev.tool.layout.ScriptNameBuilder. 695 This generates LEScripts.h, LELanguages.h, ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp 696 in the working directory. 697 (It also generates ScriptRunData.cpp, which is no longer needed.) 698 699 The generated files have a current copyright date and "@draft" statement. 700 701- diff current <icu>/source/layout files vs. generated ones 702 ~/svn.icu4j/trunk/src$ kdiff3 ~/svn.icu/trunk/src/source/layout tools/misc/src/com/ibm/icu/dev/tool/layout 703 review and manually merge desired changes; 704 fix gratuitous changes, incorrect @draft and missing aliases; 705 Unicode-derived script codes should be "born stable" like constants in uchar.h, uscript.h etc. 706- if you just copy the above files, then 707 fix mixed line endings, review the diffs as above and restore changes to API tags etc.; 708 manually re-add the "Indic script xyz v.2" tags in ScriptAndLanguageTags.h 709 710*** merge the Unicode update branches back onto the trunk 711- do not merge the icudata.jar and testdata.jar, 712 instead rebuild them from merged & tested ICU4C 713 714---------------------------------------------------------------------------- *** 715 716ICU 4.8 (no Unicode update, just new script codes) 717 718* 9 new script codes from ISO 15924 http://www.unicode.org/iso15924/codechanges.html 719 (added 2010-12-21) 720 Afak 439 Afaka 721 Jurc 510 Jurchen 722 Mroo 199 Mro, Mru 723 Nshu 499 Nüshu 724 Shrd 319 Sharada, Śāradā 725 Sora 398 Sora Sompeng 726 Takr 321 Takri, Ṭākrī, Ṭāṅkrī 727 Tang 520 Tangut 728 Wole 480 Woleai 729 -> uscript.h 730 -> com.ibm.icu.lang.UScript 731 find USCRIPT_([^ ]+) *= ([0-9]+),(.+) 732 replace public static final int \1 = \2;\3 733 -> genpname/SyntheticPropertyValueAliases.txt 734 -> add to expectedLong and expectedShort names in cintltst/cucdapi.c/TestUScriptCodeAPI() 735 and in com.ibm.icu.dev.test.lang.TestUScript.java 736 737* run genpname/preparse.pl (on Linux) 738 + cd ~/svn.icu/tools/trunk/src/unicode/c/genpname 739 + make sure that data.h is writable 740 + perl preparse.pl ~/svn.icu/trunk/src > out.txt 741 + preparse.pl shows no errors, out.txt Info and Warning lines look ok 742 743* rebuild Unicode tools (at least genpname) using make 744- You might first need to "make install" ICU so that the tools build can pick 745 up the new definitions from the installed header files. 746 747* run genpname 748 (builds both pnames.icu and propname_data.h) 749- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/data/in 750- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/common --csource 751- rebuild ICU & tools 752 753* run genprops 754- ~/svn.icu/tools/trunk/bld/unicode/c$ genprops/genprops -d ~/svn.icu/trunk/src/source/data/in -s ~/svn.icu/trunk/src/source/data/unidata -i ~/svn.icu/trunk/dbg/data/out/build/icudt48l -u 6.0 755- ~/svn.icu/tools/trunk/bld/unicode/c$ genprops/genprops -d ~/svn.icu/trunk/src/source/common --csource -s ~/svn.icu/trunk/src/source/data/unidata -i ~/svn.icu/trunk/dbg/data/out/build/icudt48l -u 6.0 756- rebuild ICU & tools 757 758* update Java data files 759- refresh just the UCD-related files, just to be safe 760- see (ICU4C)/source/data/icu4j-readme.txt 761- mkdir /tmp/icu4j 762- ~/svn.icu/trunk/dbg$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 763- copy the big-endian Unicode data files to another location, 764 separate from the other data files 765 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt48b 766 ~/svn.icu/trunk/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt48b/pnames.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt48b 767 ~/svn.icu/trunk/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt48b/uprops.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt48b 768- refresh ICU4J 769 ~/svn.icu/trunk/dbg/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt48b 770 771* should have updated the layout engine script codes but forgot 772 773---------------------------------------------------------------------------- *** 774 775Unicode 6.0 update 776 777*** related ICU Trac tickets 778 7797264 Unicode 6.0 Update 780 781*** Unicode version numbers 782- makedata.mak 783- uchar.h 784 (configure.in & configure: have been modified to extract the version from uchar.h) 785- com.ibm.icu.util.VersionInfo 786 787*** data files & enums & parser code 788 789* file preparation 790 791~/svn.icu/tools/trunk/src/unicode/c/genprops/misc$ ./ucdcopy.py ~/uni60/20100720/ucd ~/uni60/processed 792- This now prepares both unidata and testdata files in respective output subfolders. 793 794* PropertyAliases.txt changes 795- new Script_Extensions property defined in the new ScriptExtensions.txt file 796 but not listed in PropertyAliases.txt; reported to unicode.org; 797 -> added to tools/trunk/src/unicode/c/genpname/SyntheticPropertyAliases.txt 798 scx; Script_Extensions 799 -> uchar.h with new UProperty section 800 -> com.ibm.icu.lang.UProperty, parallel with uchar.h 801 802* PropertyValueAliases.txt changes 803- 12 new block names: 804 Alchemical_Symbols 805 Bamum_Supplement 806 Batak 807 Brahmi 808 CJK_Unified_Ideographs_Extension_D 809 Emoticons 810 Ethiopic_Extended_A 811 Kana_Supplement 812 Mandaic 813 Miscellaneous_Symbols_And_Pictographs 814 Playing_Cards 815 Transport_And_Map_Symbols 816 -> add to uchar.h 817 -> add to UCharacter.UnicodeBlock 818 Eclipse find UBLOCK_([^ ]+) = [0-9]+, (/.+) 819 replace public static final UnicodeBlock \1 = new UnicodeBlock("\1", \1_ID); \2 820- Joining_Group (jg) values: 821 Teh_Marbuta_Goal becomes the new canonical value for the old Hamza_On_Heh_Goal which becomes an alias 822 -> uchar.h & UCharacter.JoiningGroup 823- 3 new scripts: 824 sc ; Batk ; Batak 825 sc ; Brah ; Brahmi 826 sc ; Mand ; Mandaic 827 -> remove these from SyntheticPropertyValueAliases.txt 828 -> add alias USCRIPT_MANDAIC to USCRIPT_MANDAEAN 829 -> fix expectedLong names in cucdapi.c/TestUScriptCodeAPI() 830 and in com.ibm.icu.dev.test.lang.TestUScript.java 831- 13 new script codes from ISO 15924 http://www.unicode.org/iso15924/codechanges.html 832 (added 2009-11-11..2010-07-18) 833 Bass 259 Bassa Vah 834 Dupl 755 Duployan shortand 835 Elba 226 Elbasan 836 Gran 343 Grantha 837 Kpel 436 Kpelle 838 Loma 437 Loma 839 Mend 438 Mende 840 Merc 101 Meroitic Cursive 841 Narb 106 Old North Arabian 842 Nbat 159 Nabataean 843 Palm 126 Palmyrene 844 Sind 318 Sindhi 845 Wara 262 Warang Citi 846 -> uscript.h 847 -> com.ibm.icu.lang.UScript 848 find USCRIPT_([^ ]+) *= ([0-9]+),(.+) 849 replace public static final int \1 = \2;\3 850 -> SyntheticPropertyValueAliases.txt 851 -> add to expectedLong and expectedShort names in cintltst/cucdapi.c/TestUScriptCodeAPI() 852 and in com.ibm.icu.dev.test.lang.TestUScript.java 853- ISO 15924 name change 854 Mero 100 Meroitic Hieroglyphs (was Meroitic) 855 -> add new alias USCRIPT_MEROITIC_HIEROGLYPHS to USCRIPT_MEROITIC 856- property value alias added for Cham, was already moved out of SyntheticPropertyValueAliases.txt 857 858* UnicodeData.txt changes 859- new CJK block: 860 2B740;<CJK Ideograph Extension D, First>;Lo;0;L;;;;;N;;;;; 861 2B81D;<CJK Ideograph Extension D, Last>;Lo;0;L;;;;;N;;;;; 862 -> add to tools/trunk/src/unicode/c/gennames/gennames.c, with new ucdVersion 863 864* build Unicode tools using CMake+make 865 866* run genpname/preparse.pl (on Linux) 867 + cd ~/svn.icu/tools/trunk/src/unicode/c/genpname 868 + make sure that data.h is writable 869 + perl preparse.pl ~/svn.icu/trunk/src > out.txt 870 + preparse.pl shows no errors, out.txt Info and Warning lines look ok 871 872* rebuild Unicode tools (at least genpname) using make 873- You might first need to "make install" ICU so that the tools build can pick 874 up the new definitions from the installed header files. 875 876* run genpname 877- ~/svn.icu/tools/trunk/bld/unicode$ c/genpname/genpname -v -d ~/svn.icu/trunk/src/source/data/in 878- rebuild ICU & tools 879 880* update source/data/unidata/norm2/nfkc_cf.txt 881- follow the instructions in nfkc_cf.txt for updating it from DerivedNormalizationProps.txt 882 883* update source/data/unidata/norm2/uts46.txt 884- download http://www.unicode.org/Public/idna/6.0.0/IdnaMappingTable.txt 885 to ~/svn.icu/tools/trunk/src/unicode/py 886- adjust idna2nrm.py to handle new disallowed_STD3_valid and disallowed_STD3_mapped values 887- ~/svn.icu/tools/trunk/src/unicode/py$ ./idna2nrm.py 888- ~/svn.icu/tools/trunk/src/unicode/py$ cp uts46.txt ~/svn.icu/trunk/src/source/data/unidata/norm2 889 890* update uts46test.cpp and UTS46Test.java if there are new characters that are equivalent to 891 sequences with non-LDH ASCII (that is, their decompositions contain '=' or similar) 892- grep IdnaMappingTable.txt or uts46.txt for "disallowed_STD3_valid" on non-ASCII characters 893- Unicode 6.0: U+2260, U+226E, U+226F 894 895* generate core properties data files 896- ~/svn.icu/tools/trunk/src/unicode$ ./makeprops.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld 897- rebuild ICU & tools 898- run makeuca.sh so that genuca picks up the new nfc.nrm: 899 ~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld 900- rebuild ICU & tools 901 902* implement new Script_Extensions property (provisional) 903- parser & generator: genprops & uprops.icu 904- uscript.h, uprops.h, uchar.c, uniset_props.cpp and others, plus cintltst/cucdapi.c & intltest/usettest.cpp 905- UScript.java, UCharacterProperty.java, UnicodeSet.java, TestUScript.java, UnicodeSetTest.java 906 907* switch ubidi.icu, ucase.icu and uprops.icu from UTrie to UTrie2 908- (one-time change) 909- genbidi/gencase/genprops tools changes 910- re-run makeprops.sh (see above) 911- UCharacterProperty.java, UCharacterTypeIterator.java, 912 UBiDiProps.java, UCaseProps.java, and several others with minor changes; 913 UCharacterPropertyReader.java deleted and its code folded into UCharacterProperty.java 914 915* update Java data files 916- refresh just the UCD-related files, just to be safe 917- see (ICU4C)/source/data/icu4j-readme.txt 918- mkdir /tmp/icu4j 919- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 920 output: 921 ... 922 Unicode .icu files built to ./out/build/icudt45l 923 mkdir -p ./out/icu4j/com/ibm/icu/impl/data/icudt45b 924 echo ubidi.icu ucase.icu uprops.icu > ./out/icu4j/add.txt 925 LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/icupkg ./out/tmp/icudt45l.dat ./out/icu4j/icudt45b.dat -a ./out/icu4j/add.txt -s ./out/build/icudt45l -x '*' -tb -d ./out/icu4j/com/ibm/icu/impl/data/icudt45b 926 jar cf ./out/icu4j/icudata.jar -C ./out/icu4j com/ibm/icu/impl/data/icudt45b 927 mkdir -p /tmp/icu4j/main/shared/data 928 cp ./out/icu4j/icudata.jar /tmp/icu4j/main/shared/data 929- copy the big-endian Unicode data files to another location, 930 separate from the other data files 931 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll 932 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/brkitr 933 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt45b 934 ~/svn.icu/trunk/bld/data/out/icu4j$ rm /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/cnvalias.icu 935 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/*.nrm /tmp/icu4j/com/ibm/icu/impl/data/icudt45b 936 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/coll/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll 937 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/brkitr/* /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/brkitr 938- refresh ICU4J 939 ~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt45b 940 941* refresh Java test .txt files 942- copy new .txt files into ICU4J's main/tests/core/src/com/ibm/icu/dev/data/unicode 943 944* un-hardcode normalization skippable (NF*_Inert) test data 945- removes one manual step from the Unicode upgrade, and removes dependency on one of Mark's tools 946 947* copy updated break iterator test files 948- now handled by early ucdcopy.py and 949 copying the uni60/processed/testdata files to ~/svn.icu/trunk/src/source/test/testdata 950 (old instructions: 951 copy from (Unicode 6.0)/ucd/auxiliary/*BreakTest-6....txt 952 to ~/svn.icu/trunk/src/source/test/testdata) 953- they are not used in ICU4J 954 955* UCA 956 957- get output from Mark's tools; look in 958 http://www.unicode.org/~book/incoming/mark/uca6.0.0/ 959 http://www.macchiato.com/unicode/utc/additional-uca-files 960 http://www.unicode.org/Public/UCA/6.0.0/ 961 http://www.unicode.org/~mdavis/uca/ 962- update source/data/unidata/FractionalUCA.txt with FractionalUCA_SHORT.txt 963- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt 964- update Han-implicit ranges for new CJK extensions: 965 swapCJK() in ucol.cpp & ImplicitCEGenerator.java 966- genuca: allow bytes 02 for U+FFFE, new merge-sort character; 967 do not add it into invuca so that tailoring primary-after an ignorable works 968- genuca: permit space between [variable top] bytes 969- ucol.cpp: treat noncharacters like unassigned rather than ignorable 970- run makeuca.sh: 971 ~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld 972- rebuild ICU4C 973- refresh ICU4J collation data: 974 (subset of instructions above for properties data refresh, except copies all coll/*) 975 ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 976 mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll 977 ~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/coll/* /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll 978 ~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt45b 979- update (ICU)/source/test/testdata/CollationTest_*.txt 980 and (ICU4J)/main/tests/collate/src/com/ibm/icu/dev/data/CollationTest_*.txt 981 with output from Mark's Unicode tools 982- run all tests with the *_SHORT.txt or the full files (the full ones have comments) 983- note on intltest: if collate/UCAConformanceTest fails, then 984 utility/MultithreadTest/TestCollators will fail as well; 985 fix the conformance test before looking into the multi-thread test 986 987* When refreshing all of ICU4J data from ICU4C 988- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install 989- cp /tmp/icu4j/main/shared/data/icudata.jar ~/svn.icu4j/trunk/src/main/shared/data 990or 991- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=~/svn.icu4j/trunk/src icu4j-data-install 992 993*** LayoutEngine script information 994 995(For details see the Unicode 5.2 change log below.) 996 997* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguages.h, 998ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (It also generates 999ScriptRunData.cpp, which is no longer needed.) 1000 1001The generated files have a current copyright date and "@draft" statement. 1002 1003* copy the above files into <icu>/source/layout, replacing the old files. 1004* fix mixed line endings 1005* review the diffs and fix incorrect @draft and missing aliases; 1006 Unicode-derived script codes should be "born stable" like constants in uchar.h, uscript.h etc. 1007* manually re-add the "Indic script xyz v.2" tags in ScriptAndLanguageTags.h 1008 1009---------------------------------------------------------------------------- *** 1010 1011Unicode 5.2 update 1012 1013*** related ICU Trac tickets 1014 10157084 Unicode 5.2 1016 10177167 verify collation bytes 10187235 Java test NAME_ALIAS 10197236 Java DerivedCoreProperties.txt test 10207237 Java BidiTest.txt 10217238 UTrie2 in core unidata 10227239 test for tailoring gaps 10237240 Java fix CollationMiscTest 10247243 update layout engine for Unicode 5.2 1025 1026*** Unicode version numbers 1027- makedata.mak 1028- uchar.h 1029- configure.in & configure 1030- update ucdVersion in gennames.c if an algorithmic range changes 1031 1032*** data files & enums & parser code 1033 1034* file preparation 1035 1036python source\tools\genprops\misc\ucdcopy.py "C:\Documents and Settings\mscherer\My Documents\unicode\ucd\5.2.0" C:\svn\icuproj\icu\trunk\source\data\unidata 1037- includes finding files regardless of version numbers, 1038 copying them, and performing the equivalent processing of the 1039 ucdstrip and ucdmerge tools on the desired set of files 1040 1041* notes on changes 1042- PropertyAliases.txt 1043 moved from numeric to enumerated: 1044 ccc ; Canonical_Combining_Class 1045 new string properties: 1046 NFKC_CF ; NFKC_Casefold 1047 Name_Alias; Name_Alias 1048 new binary properties: 1049 Cased ; Cased 1050 CI ; Case_Ignorable 1051 CWCF ; Changes_When_Casefolded 1052 CWCM ; Changes_When_Casemapped 1053 CWKCF ; Changes_When_NFKC_Casefolded 1054 CWL ; Changes_When_Lowercased 1055 CWT ; Changes_When_Titlecased 1056 CWU ; Changes_When_Uppercased 1057 new CJK Unihan properties (not supported by ICU) 1058- PropertyValueAliases.txt 1059 new block names 1060 new scripts 1061 one script code change: 1062 sc ; Qaai ; Inherited 1063 -> 1064 sc ; Zinh ; Inherited ; Qaai 1065 new Line_Break (lb) value: 1066 lb ; CP ; Close_Parenthesis 1067 new Joining_Group (jg) values: Farsi_Yeh, Nya 1068 other new values: 1069 ccc; 214; ATA ; Attached_Above 1070- DerivedBidiClass.txt 1071 new default-R range: U+1E800 - U+1EFFF 1072- UnicodeData.txt 1073 all of the ISO comments are gone 1074 new CJK block end: 1075 9FC3;<CJK Ideograph, Last> -> 9FCB;<CJK Ideograph, Last> 1076 new CJK block: 1077 2A700;<CJK Ideograph Extension C, First>;Lo;0;L;;;;;N;;;;; 1078 2B734;<CJK Ideograph Extension C, Last>;Lo;0;L;;;;;N;;;;; 1079 1080* genpname 1081- run preparse.pl 1082 + cd \svn\icuproj\icu\trunk\source\tools\genpname 1083 + make sure that data.h is writable 1084 + perl preparse.pl \svn\icuproj\icu\trunk > out.txt 1085 + preparse.pl complains with errors like the following: 1086 Error: sc:Egyp already set to Egyptian_Hieroglyphs, cannot set to Egyp at preparse.pl line 1322, <GEN6> line 34. 1087 This is because ICU 4.0 had scripts from ISO 15924 which are now 1088 added to Unicode 5.2, and the Perl script shows a conflict between SyntheticPropertyValueAliases.txt 1089 and PropertyValueAliases.txt. 1090 -> Removed duplicate script entries from SyntheticPropertyValueAliases.txt: 1091 Egyp, Java, Lana, Mtei, Orkh, Armi, Avst, Kthi, Phli, Prti, Samr, Tavt 1092 + preparse.pl complains with errors about block names missing from uchar.h; add them 1093 1094* uchar.h & uscript.h & uprops.h & uprops.c & genprops 1095- new block & script values 1096 + 26 new blocks 1097 copy new blocks from Blocks.txt 1098 MS VC++ 2008 regular expression: 1099 find "^{[0-9A-F]+}\.\.{[0-9A-F]+}; {[A-Z].+}$" 1100 replace with " UBLOCK_\3 = 172, /*[\1]*/" 1101 + several new script values already added in ICU 4.0 for ISO 15924 coverage 1102 (removed from SyntheticPropertyValueAliases.txt, see genpname notes above) 1103 + 3 new script values added for ISO 15924 and Unicode 5.2 coverage 1104 + 1 new script value added for ISO 15924 coverage (not in Unicode 5.2) 1105 (added to SyntheticPropertyValueAliases.txt) 1106- new Joining Group (JG) values: Farsi_Yeh, Nya 1107- new Line_Break (lb) value: 1108 lb ; CP ; Close_Parenthesis 1109 1110* hardcoded Unihan range end/limit 1111- Unihan range end moves from 9FC3 to 9FCB 1112 search for both 9FC3 (end) and 9FC4 (limit) (regex 9FC[34], case-insensitive) 1113 + do change gennames.c 1114 1115* Compare definitions of new binary properties with what we used to use 1116 in algorithms, to see if the definitions changed. 1117- Verified that definitions for Cased and Case_Ignorable are unchanged. 1118 The gencase tool now parses the newly public Case_Ignorable values 1119 in case the definition changes in the future. 1120 1121* uchar.c & uprops.h & uprops.c & genprops 1122- new numeric values that didn't exist in Unicode data before: 1123 1/7, 1/9, 1/10, 3/10, 1/16, 3/16 1124 the ones with denominators >9 cannot be supported by uprops.icu formatVersion 5, 1125 therefore redesign the encoding of numeric types and values for formatVersion 6; 1126 design for simple numbers up to at least 144 ("one gross"), 1127 large values up to at least 10^20, 1128 and fractions with numerators -1..17 and denominators 1..16 1129 to cover current and expected future values 1130 (e.g., more Han numeric values, Meroitic twelfths) 1131 1132* reimplement Hangul_Syllable_Type for new Jamo characters 1133- the old code assumed that all Jamo characters are in the 11xx block 1134- Unicode 5.2 fills holes there and adds new Jamo characters in 1135 A960..A97F; Hangul Jamo Extended-A 1136 and in 1137 D7B0..D7FF; Hangul Jamo Extended-B 1138- Hangul_Syllable_Type can be trivially derived from a subset of 1139 Grapheme_Cluster_Break values 1140 1141* build Unicode data source code for hardcoding core data 1142C:\svn\icuproj\icu\trunk\source\data>NMAKE /f makedata.mak ICUMAKE=\svn\icuproj\icu\trunk\source\data\ CFG=x86\release uni-core-data 1143 1144ICU data make path is \svn\icuproj\icu\trunk\source\data\ 1145ICU root path is \svn\icuproj\icu\trunk 1146Information: cannot find "ucmlocal.mk". Not building user-additional converter files. 1147Information: cannot find "brklocal.mk". Not building user-additional break iterator files. 1148Information: cannot find "reslocal.mk". Not building user-additional resource bundle files. 1149Information: cannot find "collocal.mk". Not building user-additional resource bundle files. 1150Information: cannot find "rbnflocal.mk". Not building user-additional resource bundle files. 1151Information: cannot find "trnslocal.mk". Not building user-additional transliterator files. 1152Information: cannot find "misclocal.mk". Not building user-additional miscellaenous files. 1153Information: cannot find "spreplocal.mk". Not building user-additional stringprep files. 1154Creating data file for Unicode Property Names 1155Creating data file for Unicode Character Properties 1156Creating data file for Unicode Case Mapping Properties 1157Creating data file for Unicode BiDi/Shaping Properties 1158Creating data file for Unicode Normalization 1159Unicode .icu files built to "\svn\icuproj\icu\trunk\source\data\out\build\icudt43l" 1160Unicode .c source files built to "\svn\icuproj\icu\trunk\source\data\out\tmp" 1161 1162- copy the .c source files to C:\svn\icuproj\icu\trunk\source\common 1163 and rebuild the common library 1164 1165*** UCA 1166 1167- update FractionalUCA.txt with new canonical closure (output from Mark's Unicode tools) 1168- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt from Mark's Unicode tools 1169- update source/test/testdata/CollationTest_*.txt with output from Mark's Unicode tools 1170[ Begin obsolete instructions: 1171 Starting with UCA 5.2, we use the CollationTest_*_SHORT.txt files not the *_STUB.txt files. 1172 - generate the source/test/testdata/CollationTest_*_STUB.txt files via source/tools/genuca/genteststub.py 1173 on Windows: 1174 python C:\svn\icuproj\icu\trunk\source\tools\genuca\genteststub.py CollationTest_NON_IGNORABLE_SHORT.txt CollationTest_NON_IGNORABLE_STUB.txt 1175 python C:\svn\icuproj\icu\trunk\source\tools\genuca\genteststub.py CollationTest_SHIFTED_SHORT.txt CollationTest_SHIFTED_STUB.txt 1176 End obsolete instructions] 1177- run all tests with the *_SHORT.txt or the full files (the full ones have comments) 1178 not just the *_STUB.txt files 1179- note on intltest: if collate/UCAConformanceTest fails, then 1180 utility/MultithreadTest/TestCollators will fail as well; 1181 fix the conformance test before looking into the multi-thread test 1182 1183*** Implement Cased & Case_Ignorable properties 1184- via UProperty; call ucase.h functions ucase_getType() and ucase_getTypeOrIgnorable() 1185- Problem: These properties should be disjoint, but aren't 1186- UTC 2009nov decision: skip all Case_Ignorable regardless of whether they are Cased or not 1187- change ucase.icu to be able to store any combination of Cased and Case_Ignorable 1188 1189*** Implement Changes_When_Xyz properties 1190- without stored data 1191 1192*** Implement Name_Alias property 1193- add it as another name field in unames.icu 1194- make it available via u_charName() and UCharNameChoice and 1195- consider it in u_charFromName() 1196 1197*** Break iterators 1198 1199* Update break iterator rules to new UAX versions and new property values 1200* Update source/test/testdata/<boundary>Test.txt files from <unicode.org ucd>/ucd/auxiliary 1201 1202*** new BidiTest file 1203- review format and data 1204- copy BidiTest.txt to source/test/testdata 1205- write test code using this data 1206- fix ICU code where it fails the conformance test 1207 1208*** Java 1209- generally, find and update code corresponding to C/C++ 1210- UCharacter.UnicodeBlock constants: 1211 a) add an _ID integer per new block, update COUNT 1212 b) add a class instance per new block 1213 Visual Studio regex: 1214 find UBLOCK_{[^ ]+} = [0-9]+, {/.+} 1215 replace with public static final UnicodeBlock \1 = new UnicodeBlock("\1", \1_ID); \2 1216- CHAR_NAME_ALIAS -> UCharacter.getNameAlias() and getCharFromNameAlias() 1217 1218- port test changes to Java 1219 1220*** LayoutEngine script information 1221 1222(For comparison, see the Unicode 5.1 update: http://bugs.icu-project.org/trac/changeset/23833) 1223 1224* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguages.h, 1225ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (It also generates 1226ScriptRunData.cpp, which is no longer needed.) 1227 1228The generated files have a current copyright date and "@draft" statement. 1229 1230-> Eric Mader wrote in email on 20090930: 1231 "I think the tool has been modified to update @draft to @stable for 1232 older scripts and to add @draft for new scripts. 1233 (I worked with an intern on this last year.) 1234 You should check the output after you run it." 1235 1236* copy the above files into <icu>/source/layout, replacing the old files. 1237* fix mixed line endings 1238* review the diffs and fix incorrect @draft and missing aliases 1239* manually re-add the "Indic script xyz v.2" tags in ScriptAndLanguageTags.h 1240 1241Add new default entries to the indicClassTables array in <icu>/source/layout/IndicClassTables.cpp 1242and the complexTable array in <icu>/source/layoutex/ParagraphLayout.cpp. (This step should be automated...) 1243 1244-> Eric Mader wrote in email on 20090930: 1245 "This is just a matter of making sure that all the per-script tables have 1246 entries for any new scripts that were added. 1247 If any new Indic characters were added, then the class tables in 1248 IndicClassTables.cpp should be updated to reflect this. 1249 John Emmons should know how to do this if it's required." 1250 1251* rebuild the layout and layoutex libraries. 1252 1253*** Documentation 1254- Update User Guide 1255 + Jamo_Short_Name, sfc->scf, binary property value aliases 1256 1257---------------------------------------------------------------------------- *** 1258 1259Unicode 5.1 update 1260 1261*** related ICU Trac tickets 1262 12635696 Update to Unicode 5.1 1264 1265*** Unicode version numbers 1266- makedata.mak 1267- uchar.h 1268- configure.in & configure 1269- update ucdVersion in gennames.c if an algorithmic range changes 1270 1271*** data files & enums & parser code 1272 1273* file preparation 1274- ucdstrip: 1275 DerivedCoreProperties.txt 1276 DerivedNormalizationProps.txt 1277 NormalizationTest.txt 1278 PropList.txt 1279 Scripts.txt 1280 GraphemeBreakProperty.txt 1281 SentenceBreakProperty.txt 1282 WordBreakProperty.txt 1283- ucdstrip and ucdmerge: 1284 EastAsianWidth.txt 1285 LineBreak.txt 1286 1287* my ucd2unidata.bat (needs to be updated each time with UCD and file version numbers) 1288copy 5.1.0\ucd\BidiMirroring.txt ..\unidata\ 1289copy 5.1.0\ucd\Blocks.txt ..\unidata\ 1290copy 5.1.0\ucd\CaseFolding.txt ..\unidata\ 1291copy 5.1.0\ucd\DerivedAge.txt ..\unidata\ 1292copy 5.1.0\ucd\extracted\DerivedBidiClass.txt ..\unidata\ 1293copy 5.1.0\ucd\extracted\DerivedJoiningGroup.txt ..\unidata\ 1294copy 5.1.0\ucd\extracted\DerivedJoiningType.txt ..\unidata\ 1295copy 5.1.0\ucd\extracted\DerivedNumericValues.txt ..\unidata\ 1296copy 5.1.0\ucd\NormalizationCorrections.txt ..\unidata\ 1297copy 5.1.0\ucd\PropertyAliases.txt ..\unidata\ 1298copy 5.1.0\ucd\PropertyValueAliases.txt ..\unidata\ 1299copy 5.1.0\ucd\SpecialCasing.txt ..\unidata\ 1300copy 5.1.0\ucd\UnicodeData.txt ..\unidata\ 1301 1302ucdstrip < 5.1.0\ucd\DerivedCoreProperties.txt > ..\unidata\DerivedCoreProperties.txt 1303ucdstrip < 5.1.0\ucd\DerivedNormalizationProps.txt > ..\unidata\DerivedNormalizationProps.txt 1304ucdstrip < 5.1.0\ucd\NormalizationTest.txt > ..\unidata\NormalizationTest.txt 1305ucdstrip < 5.1.0\ucd\PropList.txt > ..\unidata\PropList.txt 1306ucdstrip < 5.1.0\ucd\Scripts.txt > ..\unidata\Scripts.txt 1307ucdstrip < 5.1.0\ucd\auxiliary\GraphemeBreakProperty.txt > ..\unidata\GraphemeBreakProperty.txt 1308ucdstrip < 5.1.0\ucd\auxiliary\SentenceBreakProperty.txt > ..\unidata\SentenceBreakProperty.txt 1309ucdstrip < 5.1.0\ucd\auxiliary\WordBreakProperty.txt > ..\unidata\WordBreakProperty.txt 1310ucdstrip < 5.1.0\ucd\EastAsianWidth.txt | ucdmerge > ..\unidata\EastAsianWidth.txt 1311ucdstrip < 5.1.0\ucd\LineBreak.txt | ucdmerge > ..\unidata\LineBreak.txt 1312 1313* genpname 1314- run preparse.pl 1315 + cd \svn\icuproj\icu\uni51\source\tools\genpname 1316 + make sure that data.h is writable 1317 + perl preparse.pl \svn\icuproj\icu\uni51 > out.txt 1318 + preparse.pl complains with errors like the following: 1319 Error: sc:Cari already set to Carian, cannot set to Cari at preparse.pl line 1308, <GEN6> line 30. 1320 This is because ICU 3.8 had scripts from ISO 15924 which are now 1321 added to Unicode 5.1, and the script shows a conflict between SyntheticPropertyValueAliases.txt 1322 and PropertyValueAliases.txt. 1323 -> Removed duplicate script entries from SyntheticPropertyValueAliases.txt: 1324 Cari, Cham, Kali, Lepc, Lyci, Lydi, Olck, Rjng, Saur, Sund, Vaii 1325 + PropertyValueAliases.txt now explicitly contains values for boolean properties: 1326 N/Y, No/Yes, F/T, False/True 1327 -> Added N/No and Y/Yes to preparse.pl function read_PropertyValueAliases. 1328 It will use further values from the file if present. 1329 1330* uchar.h & uscript.h & uprops.h & uprops.c & genprops 1331- new block & script values 1332 + 17 new blocks 1333 + 11 new script values already added in ICU 3.8 for ISO 15924 coverage 1334 (removed from SyntheticPropertyValueAliases.txt) 1335 + 14 new script values added for ISO 15924 coverage (not in Unicode 5.1) 1336 (added to SyntheticPropertyValueAliases.txt) 1337- uprops.icu (uprops.h) only provides 7 bits for script codes. 1338 In ICU 4.0 there are USCRIPT_CODE_LIMIT=130 script codes now. 1339 There is none above 127 yet which is the script code for an 1340 assigned Unicode character, so ICU 4.0 uprops.icu does not store any 1341 script code values greater than 127. 1342 However, it does need to store the maximum script value=USCRIPT_CODE_LIMIT-1=129 1343 in a parallel bit field, and that overflows now. 1344 Also, future values >=128 would be incompatible anyway. 1345 uprops.h is modified to move around several of the bit fields 1346 in the properties vector words, and now uses 8 bits for the script code. 1347 Two other bit fields also grow to accommodate future growth: 1348 Block (current count: 172) grows from 8 to 9 bits, 1349 and Word_Break grows from 4 to 5 bits. 1350- renamed property Simple_Case_Folding (sfc->scf) 1351 + nothing to be done: handled as normal alias 1352- new property JSN Jamo_Short_Name 1353 + no new API: only contributes to the Name property 1354- new Grapheme_Cluster_Break (GCB) value: SM=SpacingMark 1355- new Joining Group (JG) value: Burushashki_Yeh_Barree 1356- new Sentence_Break (SB) values: 1357 SB ; CR ; CR 1358 SB ; EX ; Extend 1359 SB ; LF ; LF 1360 SB ; SC ; SContinue 1361- new Word_Break (WB) values: 1362 WB ; CR ; CR 1363 WB ; Extend ; Extend 1364 WB ; LF ; LF 1365 WB ; MB ; MidNumLet 1366 1367* Further changes in the 2008-02-29 update: 1368- Default_Ignorable_Code_Point: The new file removes Cc, Cs, noncharacters from DICP 1369 because they should not normally be invisible. 1370- new Joining Group (JG) value Burushashki_Yeh_Barree was renamed to Burushaski_Yeh_Barree (one 'h' removed) 1371- new Grapheme_Cluster_Break (GCB) value: PP=Prepend 1372- new Word_Break (WB) value: NL=Newline 1373 1374* hardcoded Unihan range end/limit (see Unicode 4.1 update for comparison) 1375- Unihan range end moves from 9FBB to 9FC3 1376 search for both 9FBB (end) and 9FBC (limit) (regex 9FB[BC], case-insensitive) 1377 + do change gennames.c 1378 1379* build Unicode data source code for hardcoding core data 1380C:\svn\icuproj\icu\uni51\source\data>NMAKE /f makedata.mak ICUMAKE=\svn\icuproj\icu\uni51\source\data\ CFG=debug uni-core-data 1381 1382ICU data make path is \svn\icuproj\icu\uni51\source\data\ 1383ICU root path is \svn\icuproj\icu\uni51 1384Information: cannot find "ucmlocal.mk". Not building user-additional converter files. 1385Information: cannot find "brklocal.mk". Not building user-additional break iterator files. 1386Information: cannot find "reslocal.mk". Not building user-additional resource bundle files. 1387Information: cannot find "collocal.mk". Not building user-additional resource bundle files. 1388Information: cannot find "rbnflocal.mk". Not building user-additional resource bundle files. 1389Information: cannot find "trnslocal.mk". Not building user-additional transliterator files. 1390Information: cannot find "misclocal.mk". Not building user-additional miscellaenous files. 1391Creating data file for Unicode Character Properties 1392Creating data file for Unicode Case Mapping Properties 1393Creating data file for Unicode BiDi/Shaping Properties 1394Creating data file for Unicode Normalization 1395Unicode .icu files built to "\svn\icuproj\icu\uni51\source\data\out\build\icudt39l" 1396Unicode .c source files built to "\svn\icuproj\icu\uni51\source\data\out\tmp" 1397 1398- copy the .c source files to C:\svn\icuproj\icu\uni51\source\common 1399 and rebuild the common library 1400 1401*** Break iterators 1402 1403* Update break iterator rules to new UAX versions and new property values 1404 1405*** UCA 1406 1407* update FractionalUCA.txt and UCARules.txt with new canonical closure 1408 1409*** Test suites 1410- Test that APIs using Unicode property value aliases (like UnicodeSet) 1411 support all of the boolean values N/Y, No/Yes, F/T, False/True 1412 -> TestBinaryValues() tests in both cintltst and intltest 1413 1414*** LayoutEngine script information 1415* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguage.h, 1416ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (it also generates 1417ScriptRunData.cpp, which is no longer needed.) 1418 1419The generated files have a current copyright date and "@draft" statement. 1420 1421* copy the above files into <icu>/source/layout, replacing the old files. 1422 1423Add new default entries to the indicClassTables array in <icu>/source/layout/IndicClassTables.cpp 1424and the complexTable array in <icu>/source/layoutex/ParagraphLayout.cpp. (This step should be automated...) 1425 1426* rebuild the layout and layoutex libraries. 1427 1428*** Documentation 1429- Update User Guide 1430 + Jamo_Short_Name, sfc->scf, binary property value aliases 1431 1432---------------------------------------------------------------------------- *** 1433 1434Unicode 5.0 update 1435 1436*** related Jitterbugs 1437 14385084 RFE: Update to Unicode 5.0 1439 1440*** data files & enums & parser code 1441 1442* file preparation 1443- ucdstrip: 1444 DerivedCoreProperties.txt 1445 DerivedNormalizationProps.txt 1446 NormalizationTest.txt 1447 PropList.txt 1448 Scripts.txt 1449 GraphemeBreakProperty.txt 1450 SentenceBreakProperty.txt 1451 WordBreakProperty.txt 1452- ucdstrip and ucdmerge: 1453 EastAsianWidth.txt 1454 LineBreak.txt 1455 1456* my ucd2unidata.bat (needs to be updated each time with UCD and file version numbers) 1457copy 5.0.0\ucd\BidiMirroring.txt ..\unidata\ 1458copy 5.0.0\ucd\Blocks.txt ..\unidata\ 1459copy 5.0.0\ucd\CaseFolding.txt ..\unidata\ 1460copy 5.0.0\ucd\DerivedAge.txt ..\unidata\ 1461copy 5.0.0\ucd\extracted\DerivedBidiClass.txt ..\unidata\ 1462copy 5.0.0\ucd\extracted\DerivedJoiningGroup.txt ..\unidata\ 1463copy 5.0.0\ucd\extracted\DerivedJoiningType.txt ..\unidata\ 1464copy 5.0.0\ucd\extracted\DerivedNumericValues.txt ..\unidata\ 1465copy 5.0.0\ucd\NormalizationCorrections.txt ..\unidata\ 1466copy 5.0.0\ucd\PropertyAliases.txt ..\unidata\ 1467copy 5.0.0\ucd\PropertyValueAliases.txt ..\unidata\ 1468copy 5.0.0\ucd\SpecialCasing.txt ..\unidata\ 1469copy 5.0.0\ucd\UnicodeData.txt ..\unidata\ 1470 1471ucdstrip < 5.0.0\ucd\DerivedCoreProperties.txt > ..\unidata\DerivedCoreProperties.txt 1472ucdstrip < 5.0.0\ucd\DerivedNormalizationProps.txt > ..\unidata\DerivedNormalizationProps.txt 1473ucdstrip < 5.0.0\ucd\NormalizationTest.txt > ..\unidata\NormalizationTest.txt 1474ucdstrip < 5.0.0\ucd\PropList.txt > ..\unidata\PropList.txt 1475ucdstrip < 5.0.0\ucd\Scripts.txt > ..\unidata\Scripts.txt 1476ucdstrip < 5.0.0\ucd\auxiliary\GraphemeBreakProperty.txt > ..\unidata\GraphemeBreakProperty.txt 1477ucdstrip < 5.0.0\ucd\auxiliary\SentenceBreakProperty.txt > ..\unidata\SentenceBreakProperty.txt 1478ucdstrip < 5.0.0\ucd\auxiliary\WordBreakProperty.txt > ..\unidata\WordBreakProperty.txt 1479ucdstrip < 5.0.0\ucd\EastAsianWidth.txt | ucdmerge > ..\unidata\EastAsianWidth.txt 1480ucdstrip < 5.0.0\ucd\LineBreak.txt | ucdmerge > ..\unidata\LineBreak.txt 1481 1482* update FractionalUCA.txt and UCARules.txt with new canonical closure 1483 1484* genpname 1485- run preparse.pl 1486 + make sure that data.h is writable 1487 + perl preparse.pl \cvs\oss\icu > out.txt 1488 1489* uchar.h & uscript.h & uprops.h & uprops.c & genprops 1490- new block & script values 1491 + script values already added in ICU 3.6 because all of ISO 15924 is now covered 1492 1493* build Unicode data source code for hardcoding core data 1494C:\cvs\oss\icu\source\data>NMAKE /f makedata.mak ICUMAKE=\cvs\oss\icu\source\data\ CFG=debug uni-core-data 1495 1496ICU data make path is \cvs\oss\icu\source\data\ 1497ICU root path is \cvs\oss\icu 1498Information: cannot find "ucmlocal.mk". Not building user-additional converter files. 1499[etc.] 1500Creating data file for Unicode Character Properties 1501Creating data file for Unicode Case Mapping Properties 1502Creating data file for Unicode BiDi/Shaping Properties 1503Creating data file for Unicode Normalization 1504Unicode .icu files built to "\cvs\oss\icu\source\data\out\build\icudt35l" 1505Unicode .c source files built to "\cvs\oss\icu\source\data\out\tmp" 1506 1507- copy the .c source files to C:\cvs\oss\icu\source\common 1508 and rebuild the common library 1509 1510*** Unicode version numbers 1511- makedata.mak 1512- uchar.h 1513- configure.in 1514 1515*** LayoutEngine script information 1516* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguage.h, 1517ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (it also generates 1518ScriptRunData.cpp, which is no longer needed.) 1519 1520The generated files have a current copyright date and "@draft" statement. 1521 1522* copy the above files into <icu>/source/layout, replacing the old files. 1523 1524Add new default entries to the indicClassTables array in <icu>/source/layout/IndicClassTables.cpp 1525and the complexTable array in <icu>/source/layoutex/ParagraphLayout.cpp. (This step should be automated...) 1526 1527* rebuild the layout and layoutex libraries. 1528 1529---------------------------------------------------------------------------- *** 1530 1531Unicode 4.1 update 1532 1533*** related Jitterbugs 1534 15354332 RFE: Update to Unicode 4.1 15364157 RBBI, TR29 4.1 updates 1537 1538*** data files & enums & parser code 1539 1540* file preparation 1541- ucdstrip: 1542 DerivedCoreProperties.txt 1543 DerivedNormalizationProps.txt 1544 NormalizationTest.txt 1545 GraphemeBreakProperty.txt 1546 SentenceBreakProperty.txt 1547 WordBreakProperty.txt 1548- ucdstrip and ucdmerge: 1549 EastAsianWidth.txt 1550 LineBreak.txt 1551 1552* add new files to the repository 1553 GraphemeBreakProperty.txt 1554 SentenceBreakProperty.txt 1555 WordBreakProperty.txt 1556 1557* update FractionalUCA.txt and UCARules.txt with new canonical closure 1558 1559* genpname 1560- handle new enumerated properties in sub read_uchar 1561- run preparse.pl 1562 1563* uchar.h & uscript.h & uprops.h & uprops.c & genprops 1564- new binary properties 1565 + Pattern_Syntax 1566 + Pattern_White_Space 1567- new enumerated properties 1568 + Grapheme_Cluster_Break 1569 + Sentence_Break 1570 + Word_Break 1571- new block & script & line break values 1572 1573* gencase 1574- case-ignorable changes 1575 see http://www.unicode.org/versions/Unicode4.1.0/#CaseMods 1576 now: (D47a) Word_Break=MidLetter or Mn, Me, Cf, Lm, Sk 1577 1578*** Unicode version numbers 1579- makedata.mak 1580- uchar.h 1581- configure.in 1582 1583*** tests 1584- verify that u_charMirror() round-trips 1585- test all new properties and some new values of old properties 1586 1587*** other code 1588 1589* hardcoded Unihan range end/limit 1590- Unihan range end moves from 9FA5 to 9FBB 1591 search for both 9FA5 (end) and 9FA6 (limit) (regex 9FA[56], case-insensitive) 1592 + do not modify BOCU/BOCSU code because that would change the encoding 1593 and break binary compatibility! 1594 + similarly, do not change the GB 18030 range data (ucnvmbcs.c), 1595 NamePrepProfile.txt 1596 + ignore trietest.c: test data is arbitrary 1597 + ignore tstnorm.cpp: test optimization, not important 1598 + ignore collation: 9FA[56] only appears in comments; swapCJK() uses the whole block up to 9FFF 1599 + do change line_th.txt and word_th.txt 1600 by replacing hardcoded ranges with the new property values 1601 + do change gennames.c 1602 1603source\data\brkitr\line_th.txt(229): \u33E0-\u33FE \u3400-\u4DB5 \u4E00-\u9FA5 \uA000-\uA48C \uA490-\uA4C6 1604source\data\brkitr\word_th.txt(23): \u33E0-\u33FE \u3400-\u4DB5 \u4E00-\u9FA5 \uA000-\uA48C \uA490-\uA4C6 1605source\tools\gennames\gennames.c(971): 0x4e00, 0x9fa5, 1606 1607* case mappings 1608- compare new special casing context conditions with previous ones 1609 see http://www.unicode.org/versions/Unicode4.1.0/#CaseMods 1610 1611* genpname 1612- consider storing only the short name if it is the same as the long name 1613 1614*** other reviews 1615- UAX #29 changes (grapheme/word/sentence breaks) 1616- UAX #14 changes (line breaks) 1617- Pattern_Syntax & Pattern_White_Space 1618 1619---------------------------------------------------------------------------- *** 1620 1621Unicode 4.0.1 update 1622 1623*** related Jitterbugs 1624 16253170 RFE: Update to Unicode 4.0.1 16263171 Add new Unicode 4.0.1 properties 16273520 use Unicode 4.0.1 updates for break iteration 1628 1629*** data files & enums & parser code 1630 1631* file preparation 1632- ucdstrip: DerivedNormalizationProps.txt, NormalizationTest.txt, DerivedCoreProperties.txt 1633- ucdstrip and ucdmerge: EastAsianWidth.txt, LineBreak.txt 1634 1635* file fixes 1636- fix UnicodeData.txt general categories of Ethiopic digits Nd->No 1637 according to PRI #26 1638 http://www.unicode.org/review/resolved-pri.html#pri26 1639- undone again because no corrigendum in sight; 1640 instead modified tests to not check consistency on this for Unicode 4.0.1 1641 1642* ucdterms.txt 1643- update from http://www.unicode.org/copyright.html 1644 formatted for plain text 1645 1646* uchar.h & uprops.h & uprops.c & genprops 1647- add UBLOCK_CYRILLIC_SUPPLEMENT because the block is renamed 1648- add U_LB_INSEPARABLE due to a spelling fix 1649 + put short name comment only on line with new constant 1650 for genpname perl script parser 1651- new binary properties 1652 + STerm 1653 + Variation_Selector 1654 1655* genpname 1656- fix genpname perl script so that it doesn't choke on more than 2 names per property value 1657- perl script: correctly calculate the maximum number of fields per row 1658 1659* uscript.h 1660- new script code Hrkt=Katakana_Or_Hiragana 1661 1662* gennorm.c track changes in DerivedNormalizationProps.txt 1663- "FNC" -> "FC_NFKC" 1664- single field "NFD_NO" -> two fields "NFD_QC; N" etc. 1665 1666* genprops/props2.c track changes in DerivedNumericValues.txt 1667- changed from 3 columns to 2, dropping the numeric type 1668 + assume that the type is always numeric for Han characters, 1669 and that only those are added in addition to what UnicodeData.txt lists 1670 1671*** Unicode version numbers 1672- makedata.mak 1673- uchar.h 1674- configure.in 1675 1676*** tests 1677- update test of default bidi classes according to PRI #28 1678 /tsutil/cucdtst/TestUnicodeData 1679 http://www.unicode.org/review/resolved-pri.html#pri28 1680- bidi tests: change exemplar character for ES depending on Unicode version 1681- change hardcoded expected property values where they change 1682 1683*** other code 1684 1685* name matching 1686- read UCD.html 1687 1688* scripts 1689- use new Hrkt=Katakana_Or_Hiragana 1690 1691* ZWJ & ZWNJ 1692- are now part of combining character sequences 1693- break iteration used to assume that LB classes did not overlap; now they do for ZWJ & ZWNJ 1694