encoding.html revision 6dbcaf8686cec01223029997a2d22054a2507e55
1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd"> 2<html> 3<head> 4<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> 5<style type="text/css"><!-- 6TD {font-size: 14pt; font-family: Verdana,Arial,Helvetica} 7BODY {font-size: 14pt; font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em} 8H1 {font-size: 20pt; font-family: Verdana,Arial,Helvetica} 9H2 {font-size: 18pt; font-family: Verdana,Arial,Helvetica} 10H3 {font-size: 16pt; font-family: Verdana,Arial,Helvetica} 11A:link, A:visited, A:active { text-decoration: underline } 12--></style> 13<title>Encodings support</title> 14</head> 15<body bgcolor="#8b7765" text="#000000" link="#000000" vlink="#000000"> 16<table border="0" width="100%" cellpadding="5" cellspacing="0" align="center"><tr> 17<td width="180"> 18<a href="http://www.gnome.org/"><img src="smallfootonly.gif" alt="Gnome Logo"></a><a href="http://www.w3.org/Status"><img src="w3c.png" alt="W3C Logo"></a><a href="http://www.redhat.com/"><img src="redhat.gif" alt="Red Hat Logo"></a> 19</td> 20<td><table border="0" width="90%" cellpadding="2" cellspacing="0" align="center" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3" bgcolor="#fffacd"><tr><td align="center"> 21<h1>The XML C library for Gnome</h1> 22<h2>Encodings support</h2> 23</td></tr></table></td></tr></table></td> 24</tr></table> 25<table border="0" cellpadding="4" cellspacing="0" width="100%" align="center"><tr><td bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="2" width="100%"><tr> 26<td valign="top" width="200" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td> 27<table width="100%" border="0" cellspacing="1" cellpadding="3"> 28<tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Main Menu</b></center></td></tr> 29<tr><td bgcolor="#fffacd"><ul> 30<li><a href="index.html">Home</a></li> 31<li><a href="intro.html">Introduction</a></li> 32<li><a href="FAQ.html">FAQ</a></li> 33<li><a href="docs.html">Documentation</a></li> 34<li><a href="bugs.html">Reporting bugs and getting help</a></li> 35<li><a href="help.html">How to help</a></li> 36<li><a href="downloads.html">Downloads</a></li> 37<li><a href="news.html">News</a></li> 38<li><a href="XMLinfo.html">XML</a></li> 39<li><a href="XSLT.html">XSLT</a></li> 40<li><a href="python.html">Python and bindings</a></li> 41<li><a href="architecture.html">libxml architecture</a></li> 42<li><a href="tree.html">The tree output</a></li> 43<li><a href="interface.html">The SAX interface</a></li> 44<li><a href="xmldtd.html">Validation & DTDs</a></li> 45<li><a href="xmlmem.html">Memory Management</a></li> 46<li><a href="encoding.html">Encodings support</a></li> 47<li><a href="xmlio.html">I/O Interfaces</a></li> 48<li><a href="catalog.html">Catalog support</a></li> 49<li><a href="library.html">The parser interfaces</a></li> 50<li><a href="entities.html">Entities or no entities</a></li> 51<li><a href="namespaces.html">Namespaces</a></li> 52<li><a href="upgrade.html">Upgrading 1.x code</a></li> 53<li><a href="threads.html">Thread safety</a></li> 54<li><a href="DOM.html">DOM Principles</a></li> 55<li><a href="example.html">A real example</a></li> 56<li><a href="contribs.html">Contributions</a></li> 57<li> 58<a href="xml.html">flat page</a>, <a href="site.xsl">stylesheet</a> 59</li> 60</ul></td></tr> 61</table> 62<table width="100%" border="0" cellspacing="1" cellpadding="3"> 63<tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>API Indexes</b></center></td></tr> 64<tr><td bgcolor="#fffacd"><ul> 65<li><a href="APIchunk0.html">Alphabetic</a></li> 66<li><a href="APIconstructors.html">Constructors</a></li> 67<li><a href="APIfunctions.html">Functions/Types</a></li> 68<li><a href="APIfiles.html">Modules</a></li> 69<li><a href="APIsymbols.html">Symbols</a></li> 70</ul></td></tr> 71</table> 72<table width="100%" border="0" cellspacing="1" cellpadding="3"> 73<tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Related links</b></center></td></tr> 74<tr><td bgcolor="#fffacd"><ul> 75<li><a href="http://mail.gnome.org/archives/xml/">Mail archive</a></li> 76<li><a href="http://xmlsoft.org/XSLT/">XSLT libxslt</a></li> 77<li><a href="http://phd.cs.unibo.it/gdome2/">DOM gdome2</a></li> 78<li><a href="ftp://xmlsoft.org/">FTP</a></li> 79<li><a href="http://www.fh-frankfurt.de/~igor/projects/libxml/">Windows binaries</a></li> 80<li><a href="http://garypennington.net/libxml2/">Solaris binaries</a></li> 81<li><a href="http://bugzilla.gnome.org/buglist.cgi?product=libxml">Bug Tracker</a></li> 82</ul></td></tr> 83</table> 84</td></tr></table></td> 85<td valign="top" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%"><tr><td><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table border="0" cellpadding="3" cellspacing="1" width="100%"><tr><td bgcolor="#fffacd"> 86<p>Table of Content:</p> 87<ol> 88<li><a href="encoding.html#What">What does internationalization support 89 mean ?</a></li> 90<li><a href="encoding.html#internal">The internal encoding, how and 91 why</a></li> 92<li><a href="encoding.html#implemente">How is it implemented ?</a></li> 93<li><a href="encoding.html#Default">Default supported encodings</a></li> 94<li><a href="encoding.html#extend">How to extend the existing 95 support</a></li> 96</ol> 97<h3><a name="What">What does internationalization support mean ?</a></h3> 98<p>XML was designed from the start to allow the support of any character set 99by using Unicode. Any conformant XML parser has to support the UTF-8 and 100UTF-16 default encodings which can both express the full unicode ranges. UTF8 101is a variable length encoding whose greatest point are to resuse the same 102emcoding for ASCII and to save space for Western encodings, but it is a bit 103more complex to handle in practice. UTF-16 use 2 bytes per characters (and 104sometimes combines two pairs), it makes implementation easier, but looks a 105bit overkill for Western languages encoding. Moreover the XML specification 106allows document to be encoded in other encodings at the condition that they 107are clearly labelled as such. For example the following is a wellformed XML 108document encoded in ISO-8859 1 and using accentuated letter that we French 109likes for both markup and content:</p> 110<pre><?xml version="1.0" encoding="ISO-8859-1"?> 111<tr�s>l�</tr�s></pre> 112<p>Having internationalization support in libxml means the foolowing:</p> 113<ul> 114<li>the document is properly parsed</li> 115<li>informations about it's encoding are saved</li> 116<li>it can be modified</li> 117<li>it can be saved in its original encoding</li> 118<li>it can also be saved in another encoding supported by libxml (for 119 example straight UTF8 or even an ASCII form)</li> 120</ul> 121<p>Another very important point is that the whole libxml API, with the 122exception of a few routines to read with a specific encoding or save to a 123specific encoding, is completely agnostic about the original encoding of the 124document.</p> 125<p>It should be noted too that the HTML parser embedded in libxml now obbey 126the same rules too, the following document will be (as of 2.2.2) handled in 127an internationalized fashion by libxml too:</p> 128<pre><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" 129 "http://www.w3.org/TR/REC-html40/loose.dtd">; 130<html lang="fr"> 131<head> 132 <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1"> 133</head> 134<body> 135<p>W3C cr�e des standards pour le Web.</body> 136</html></pre> 137<h3><a name="internal">The internal encoding, how and why</a></h3> 138<p>One of the core decision was to force all documents to be converted to a 139default internal encoding, and that encoding to be UTF-8, here are the 140rationale for those choices:</p> 141<ul> 142<li>keeping the native encoding in the internal form would force the libxml 143 users (or the code associated) to be fully aware of the encoding of the 144 original document, for examples when adding a text node to a document, 145 the content would have to be provided in the document encoding, i.e. the 146 client code would have to check it before hand, make sure it's conformant 147 to the encoding, etc ... Very hard in practice, though in some specific 148 cases this may make sense.</li> 149<li>the second decision was which encoding. From the XML spec only UTF8 and 150 UTF16 really makes sense as being the two only encodings for which there 151 is amndatory support. UCS-4 (32 bits fixed size encoding) could be 152 considered an intelligent choice too since it's a direct Unicode mapping 153 support. I selected UTF-8 on the basis of efficiency and compatibility 154 with surrounding software: 155 <ul> 156<li>UTF-8 while a bit more complex to convert from/to (i.e. slightly 157 more costly to import and export CPU wise) is also far more compact 158 than UTF-16 (and UCS-4) for a majority of the documents I see it used 159 for right now (RPM RDF catalogs, advogato data, various configuration 160 file formats, etc.) and the key point for today's computer 161 architecture is efficient uses of caches. If one nearly double the 162 memory requirement to store the same amount of data, this will trash 163 caches (main memory/external caches/internal caches) and my take is 164 that this harms the system far more than the CPU requirements needed 165 for the conversion to UTF-8</li> 166<li>Most of libxml version 1 users were using it with straight ASCII 167 most of the time, doing the conversion with an internal encoding 168 requiring all their code to be rewritten was a serious show-stopper 169 for using UTF-16 or UCS-4.</li> 170<li>UTF-8 is being used as the de-facto internal encoding standard for 171 related code like the <a href="http://www.pango.org/">pango</a> 172 upcoming Gnome text widget, and a lot of Unix code (yep another place 173 where Unix programmer base takes a different approach from Microsoft 174 - they are using UTF-16)</li> 175</ul> 176</li> 177</ul> 178<p>What does this mean in practice for the libxml user:</p> 179<ul> 180<li>xmlChar, the libxml data type is a byte, those bytes must be assembled 181 as UTF-8 valid strings. The proper way to terminate an xmlChar * string 182 is simply to append 0 byte, as usual.</li> 183<li>One just need to make sure that when using chars outside the ASCII set, 184 the values has been properly converted to UTF-8</li> 185</ul> 186<h3><a name="implemente">How is it implemented ?</a></h3> 187<p>Let's describe how all this works within libxml, basically the I18N 188(internationalization) support get triggered only during I/O operation, i.e. 189when reading a document or saving one. Let's look first at the reading 190sequence:</p> 191<ol> 192<li>when a document is processed, we usually don't know the encoding, a 193 simple heuristic allows to detect UTF-18 and UCS-4 from whose where the 194 ASCII range (0-0x7F) maps with ASCII</li> 195<li>the xml declaration if available is parsed, including the encoding 196 declaration. At that point, if the autodetected encoding is different 197 from the one declared a call to xmlSwitchEncoding() is issued.</li> 198<li>If there is no encoding declaration, then the input has to be in either 199 UTF-8 or UTF-16, if it is not then at some point when processing the 200 input, the converter/checker of UTF-8 form will raise an encoding error. 201 You may end-up with a garbled document, or no document at all ! Example: 202 <pre>~/XML -> /xmllint err.xml 203err.xml:1: error: Input is not proper UTF-8, indicate encoding ! 204<tr�s>l�</tr�s> 205 ^ 206err.xml:1: error: Bytes: 0xE8 0x73 0x3E 0x6C 207<tr�s>l�</tr�s> 208 ^</pre> 209</li> 210<li>xmlSwitchEncoding() does an encoding name lookup, canonalize it, and 211 then search the default registered encoding converters for that encoding. 212 If it's not within the default set and iconv() support has been compiled 213 it, it will ask iconv for such an encoder. If this fails then the parser 214 will report an error and stops processing: 215 <pre>~/XML -> /xmllint err2.xml 216err2.xml:1: error: Unsupported encoding UnsupportedEnc 217<?xml version="1.0" encoding="UnsupportedEnc"?> 218 ^</pre> 219</li> 220<li>From that point the encoder process progressingly the input (it is 221 plugged as a front-end to the I/O module) for that entity. It captures 222 and convert on-the-fly the document to be parsed to UTF-8. The parser 223 itself just does UTF-8 checking of this input and process it 224 transparently. The only difference is that the encoding information has 225 been added to the parsing context (more precisely to the input 226 corresponding to this entity).</li> 227<li>The result (when using DOM) is an internal form completely in UTF-8 228 with just an encoding information on the document node.</li> 229</ol> 230<p>Ok then what's happen when saving the document (assuming you 231colllected/built an xmlDoc DOM like structure) ? It depends on the function 232called, xmlSaveFile() will just try to save in the original encoding, while 233xmlSaveFileTo() and xmlSaveFileEnc() can optionally save to a given 234encoding:</p> 235<ol> 236<li>if no encoding is given, libxml will look for an encoding value 237 associated to the document and if it exists will try to save to that 238 encoding, 239 <p>otherwise everything is written in the internal form, i.e. UTF-8</p> 240</li> 241<li>so if an encoding was specified, either at the API level or on the 242 document, libxml will again canonalize the encoding name, lookup for a 243 converter in the registered set or through iconv. If not found the 244 function will return an error code</li> 245<li>the converter is placed before the I/O buffer layer, as another kind of 246 buffer, then libxml will simply push the UTF-8 serialization to through 247 that buffer, which will then progressively be converted and pushed onto 248 the I/O layer.</li> 249<li>It is possible that the converter code fails on some input, for example 250 trying to push an UTF-8 encoded chinese character through the UTF-8 to 251 ISO-8859-1 converter won't work. Since the encoders are progressive they 252 will just report the error and the number of bytes converted, at that 253 point libxml will decode the offending character, remove it from the 254 buffer and replace it with the associated charRef encoding &#123; and 255 resume the convertion. This guarante that any document will be saved 256 without losses (except for markup names where this is not legal, this is 257 a problem in the current version, in pactice avoid using non-ascci 258 characters for tags or attributes names @@). A special "ascii" encoding 259 name is used to save documents to a pure ascii form can be used when 260 portability is really crucial</li> 261</ol> 262<p>Here is a few examples based on the same test document:</p> 263<pre>~/XML -> /xmllint isolat1 264<?xml version="1.0" encoding="ISO-8859-1"?> 265<tr�s>l�</tr�s> 266~/XML -> /xmllint --encode UTF-8 isolat1 267<?xml version="1.0" encoding="UTF-8"?> 268<très>l� �</très> 269~/XML -> </pre> 270<p>The same processing is applied (and reuse most of the code) for HTML I18N 271processing. Looking up and modifying the content encoding is a bit more 272difficult since it is located in a <meta> tag under the <head>, 273so a couple of functions htmlGetMetaEncoding() and htmlSetMetaEncoding() have 274been provided. The parser also attempts to switch encoding on the fly when 275detecting such a tag on input. Except for that the processing is the same 276(and again reuses the same code).</p> 277<h3><a name="Default">Default supported encodings</a></h3> 278<p>libxml has a set of default converters for the following encodings 279(located in encoding.c):</p> 280<ol> 281<li>UTF-8 is supported by default (null handlers)</li> 282<li>UTF-16, both little and big endian</li> 283<li>ISO-Latin-1 (ISO-8859-1) covering most western languages</li> 284<li>ASCII, useful mostly for saving</li> 285<li>HTML, a specific handler for the conversion of UTF-8 to ASCII with HTML 286 predefined entities like &copy; for the Copyright sign.</li> 287</ol> 288<p>More over when compiled on an Unix platfor with iconv support the full set 289of encodings supported by iconv can be instantly be used by libxml. On a 290linux machine with glibc-2.1 the list of supported encodings and aliases fill 2913 full pages, and include UCS-4, the full set of ISO-Latin encodings, and the 292various Japanese ones.</p> 293<h4>Encoding aliases</h4> 294<p>From 2.2.3, libxml has support to register encoding names aliases. The 295goal is to be able to parse document whose encoding is supported but where 296the name differs (for example from the default set of names accepted by 297iconv). The following functions allow to register and handle new aliases for 298existing encodings. Once registered libxml will automatically lookup the 299aliases when handling a document:</p> 300<ul> 301<li>int xmlAddEncodingAlias(const char *name, const char *alias);</li> 302<li>int xmlDelEncodingAlias(const char *alias);</li> 303<li>const char * xmlGetEncodingAlias(const char *alias);</li> 304<li>void xmlCleanupEncodingAliases(void);</li> 305</ul> 306<h3><a name="extend">How to extend the existing support</a></h3> 307<p>Well adding support for new encoding, or overriding one of the encoders 308(assuming it is buggy) should not be hard, just write an input and output 309conversion routines to/from UTF-8, and register them using 310xmlNewCharEncodingHandler(name, xxxToUTF8, UTF8Toxxx), and they will be 311called automatically if the parser(s) encounter such an encoding name 312(register it uppercase, this will help). The description of the encoders, 313their arguments and expected return values are described in the encoding.h 314header.</p> 315<p>A quick note on the topic of subverting the parser to use a different 316internal encoding than UTF-8, in some case people will absolutely want to 317keep the internal encoding different, I think it's still possible (but the 318encoding must be compliant with ASCII on the same subrange) though I didn't 319tried it. The key is to override the default conversion routines (by 320registering null encoders/decoders for your charsets), and bypass the UTF-8 321checking of the parser by setting the parser context charset 322(ctxt->charset) to something different than XML_CHAR_ENCODING_UTF8, but 323there is no guarantee taht this will work. You may also have some troubles 324saving back.</p> 325<p>Basically proper I18N support is important, this requires at least 326libxml-2.0.0, but a lot of features and corrections are really available only 327starting 2.2.</p> 328<p><a href="bugs.html">Daniel Veillard</a></p> 329</td></tr></table></td></tr></table></td></tr></table></td> 330</tr></table></td></tr></table> 331</body> 332</html> 333
