1
21. FTS3 Tokenizers
3
4  When creating a new full-text table, FTS3 allows the user to select
5  the text tokenizer implementation to be used when indexing text
6  by specifying a "tokenize" clause as part of the CREATE VIRTUAL TABLE
7  statement:
8
9    CREATE VIRTUAL TABLE <table-name> USING fts3(
10      <columns ...> [, tokenize <tokenizer-name> [<tokenizer-args>]]
11    );
12
13  The built-in tokenizers (valid values to pass as <tokenizer name>) are
14  "simple" and "porter".
15
16  <tokenizer-args> should consist of zero or more white-space separated
17  arguments to pass to the selected tokenizer implementation. The 
18  interpretation of the arguments, if any, depends on the individual 
19  tokenizer.
20
212. Custom Tokenizers
22
23  FTS3 allows users to provide custom tokenizer implementations. The 
24  interface used to create a new tokenizer is defined and described in 
25  the fts3_tokenizer.h source file.
26
27  Registering a new FTS3 tokenizer is similar to registering a new 
28  virtual table module with SQLite. The user passes a pointer to a
29  structure containing pointers to various callback functions that
30  make up the implementation of the new tokenizer type. For tokenizers,
31  the structure (defined in fts3_tokenizer.h) is called
32  "sqlite3_tokenizer_module".
33
34  FTS3 does not expose a C-function that users call to register new
35  tokenizer types with a database handle. Instead, the pointer must
36  be encoded as an SQL blob value and passed to FTS3 through the SQL
37  engine by evaluating a special scalar function, "fts3_tokenizer()".
38  The fts3_tokenizer() function may be called with one or two arguments,
39  as follows:
40
41    SELECT fts3_tokenizer(<tokenizer-name>);
42    SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>);
43  
44  Where <tokenizer-name> is a string identifying the tokenizer and
45  <sqlite3_tokenizer_module ptr> is a pointer to an sqlite3_tokenizer_module
46  structure encoded as an SQL blob. If the second argument is present,
47  it is registered as tokenizer <tokenizer-name> and a copy of it
48  returned. If only one argument is passed, a pointer to the tokenizer
49  implementation currently registered as <tokenizer-name> is returned,
50  encoded as a blob. Or, if no such tokenizer exists, an SQL exception
51  (error) is raised.
52
53  SECURITY: If the fts3 extension is used in an environment where potentially
54    malicious users may execute arbitrary SQL (i.e. gears), they should be
55    prevented from invoking the fts3_tokenizer() function, possibly using the
56    authorisation callback.
57
58  See "Sample code" below for an example of calling the fts3_tokenizer()
59  function from C code.
60
613. ICU Library Tokenizers
62
63  If this extension is compiled with the SQLITE_ENABLE_ICU pre-processor 
64  symbol defined, then there exists a built-in tokenizer named "icu" 
65  implemented using the ICU library. The first argument passed to the
66  xCreate() method (see fts3_tokenizer.h) of this tokenizer may be
67  an ICU locale identifier. For example "tr_TR" for Turkish as used
68  in Turkey, or "en_AU" for English as used in Australia. For example:
69
70    "CREATE VIRTUAL TABLE thai_text USING fts3(text, tokenizer icu th_TH)"
71
72  The ICU tokenizer implementation is very simple. It splits the input
73  text according to the ICU rules for finding word boundaries and discards
74  any tokens that consist entirely of white-space. This may be suitable
75  for some applications in some locales, but not all. If more complex
76  processing is required, for example to implement stemming or 
77  discard punctuation, this can be done by creating a tokenizer 
78  implementation that uses the ICU tokenizer as part of its implementation.
79
80  When using the ICU tokenizer this way, it is safe to overwrite the
81  contents of the strings returned by the xNext() method (see
82  fts3_tokenizer.h).
83
844. Sample code.
85
86  The following two code samples illustrate the way C code should invoke
87  the fts3_tokenizer() scalar function:
88
89      int registerTokenizer(
90        sqlite3 *db, 
91        char *zName, 
92        const sqlite3_tokenizer_module *p
93      ){
94        int rc;
95        sqlite3_stmt *pStmt;
96        const char zSql[] = "SELECT fts3_tokenizer(?, ?)";
97      
98        rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
99        if( rc!=SQLITE_OK ){
100          return rc;
101        }
102      
103        sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
104        sqlite3_bind_blob(pStmt, 2, &p, sizeof(p), SQLITE_STATIC);
105        sqlite3_step(pStmt);
106      
107        return sqlite3_finalize(pStmt);
108      }
109      
110      int queryTokenizer(
111        sqlite3 *db, 
112        char *zName,  
113        const sqlite3_tokenizer_module **pp
114      ){
115        int rc;
116        sqlite3_stmt *pStmt;
117        const char zSql[] = "SELECT fts3_tokenizer(?)";
118      
119        *pp = 0;
120        rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
121        if( rc!=SQLITE_OK ){
122          return rc;
123        }
124      
125        sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
126        if( SQLITE_ROW==sqlite3_step(pStmt) ){
127          if( sqlite3_column_type(pStmt, 0)==SQLITE_BLOB ){
128            memcpy(pp, sqlite3_column_blob(pStmt, 0), sizeof(*pp));
129          }
130        }
131      
132        return sqlite3_finalize(pStmt);
133      }
134