15821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 25821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2006 Oct 10 35821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 45821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The author disclaims copyright to this source code. In place of 55821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a legal notice, here is a blessing: 65821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 75821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** May you do good and not evil. 85821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** May you find forgiveness for yourself and forgive others. 95821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** May you share freely, never taking more than you give. 105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)****************************************************************************** 125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This is an SQLite module implementing full-text search. 145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The code in this file is only compiled if: 185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** * The FTS3 module is being built as an extension 205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (in which case SQLITE_CORE is not defined), or 215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** * The FTS3 module is being built into the core of 235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). 245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* The full-text index is stored in a series of b+tree (-like) 275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** structures called segments which map terms to doclists. The 285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** structures are like b+trees in layout, but are constructed from the 295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** bottom up in optimal fashion and are not updatable. Since trees 305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** are built from the bottom up, things will be described from the 315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** bottom up. 325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)**** Varints **** 355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The basic unit of encoding is a variable-length integer called a 365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint. We encode variable-length integers in little-endian order 375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** using seven bits * per byte as follows: 385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** KEY: 405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A = 0xxxxxxx 7 bits of data and one flag bit 415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** B = 1xxxxxxx 7 bits of data and one flag bit 425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7 bits - A 445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 14 bits - BA 455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 21 bits - BBA 465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** and so on. 475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This is similar in concept to how sqlite encodes "varints" but 495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the encoding is not the same. SQLite varints are big-endian 505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** are are limited to 9 bytes in length whereas FTS3 varints are 515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** little-endian and can be up to 10 bytes in length (in theory). 525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Example encodings: 545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1: 0x01 565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 127: 0x7f 575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 128: 0x81 0x00 585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)**** Document lists **** 615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A doclist (document list) holds a docid-sorted list of hits for a 625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** given term. Doclists hold docids and associated token positions. 635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A docid is the unique integer identifier for a single document. 645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A position is the index of a word within the document. The first 655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** word of the document has a position of 0. 665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** FTS3 used to optionally store character offsets using a compile-time 685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** option. But that functionality is no longer supported. 695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A doclist is stored like this: 715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** array { 735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint docid; 745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** array { (position list for column 0) 755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint position; (2 more than the delta from previous position) 765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** } 775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** array { 785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint POS_COLUMN; (marks start of position list for new column) 795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint column; (index of new column) 805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** array { 815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint position; (2 more than the delta from previous position) 825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** } 835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** } 845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint POS_END; (marks end of positions for this document. 855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** } 865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Here, array { X } means zero or more occurrences of X, adjacent in 885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** memory. A "position" is an index of a token in the token stream 895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** generated by the tokenizer. Note that POS_END and POS_COLUMN occur 905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in the same logical place as the position element, and act as sentinals 915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** ending a position list array. POS_END is 0. POS_COLUMN is 1. 925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The positions numbers are not stored literally but rather as two more 935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** than the difference from the prior position, or the just the position plus 945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2 for the first position. Example: 955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** label: A B C D E F G H I J K 975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** value: 123 5 9 1 1 14 35 0 234 72 0 985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The 123 value is the first docid. For column zero in this document 1005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** there are two matches at positions 3 and 10 (5-2 and 9-2+3). The 1 1015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** at D signals the start of a new column; the 1 at E indicates that the 1025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** new column is column number 1. There are two positions at 12 and 45 1035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (14-2 and 35-2+12). The 0 at H indicate the end-of-document. The 1045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 234 at I is the next docid. It has one position 72 (72-2) and then 1055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** terminates with the 0 at K. 1065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A "position-list" is the list of positions for multiple columns for 1085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a single docid. A "column-list" is the set of positions for a single 1095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** column. Hence, a position-list consists of one or more column-lists, 1105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a document record consists of a docid followed by a position-list and 1115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a doclist consists of one or more document records. 1125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A bare doclist omits the position information, becoming an 1145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** array of varint-encoded docids. 1155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)**** Segment leaf nodes **** 1175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Segment leaf nodes store terms and doclists, ordered by term. Leaf 1185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** nodes are written using LeafWriter, and read using LeafReader (to 1195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** iterate through a single leaf node's data) and LeavesReader (to 1205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** iterate through a segment's entire leaf layer). Leaf nodes have 1215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the format: 1225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint iHeight; (height from leaf level, always 0) 1245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nTerm; (length of first term) 1255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** char pTerm[nTerm]; (content of first term) 1265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nDoclist; (length of term's associated doclist) 1275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** char pDoclist[nDoclist]; (content of doclist) 1285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** array { 1295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (further terms are delta-encoded) 1305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nPrefix; (length of prefix shared with previous term) 1315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nSuffix; (length of unshared suffix) 1325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** char pTermSuffix[nSuffix];(unshared suffix of next term) 1335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nDoclist; (length of term's associated doclist) 1345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** char pDoclist[nDoclist]; (content of doclist) 1355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** } 1365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Here, array { X } means zero or more occurrences of X, adjacent in 1385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** memory. 1395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Leaf nodes are broken into blocks which are stored contiguously in 1415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the %_segments table in sorted order. This means that when the end 1425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** of a node is reached, the next term is in the node with the next 1435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** greater node id. 1445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** New data is spilled to a new leaf node when the current node 1465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** exceeds LEAF_MAX bytes (default 2048). New data which itself is 1475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** larger than STANDALONE_MIN (default 1024) is placed in a standalone 1485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** node (a leaf node with a single term and doclist). The goal of 1495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** these settings is to pack together groups of small doclists while 1505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** making it efficient to directly access large doclists. The 1515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** assumption is that large doclists represent terms which are more 1525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** likely to be query targets. 1535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** TODO(shess) It may be useful for blocking decisions to be more 1555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** dynamic. For instance, it may make more sense to have a 2.5k leaf 1565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** node rather than splitting into 2k and .5k nodes. My intuition is 1575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** that this might extend through 2x or 4x the pagesize. 1585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)**** Segment interior nodes **** 1615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Segment interior nodes store blockids for subtree nodes and terms 1625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to describe what data is stored by the each subtree. Interior 1635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** nodes are written using InteriorWriter, and read using 1645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** InteriorReader. InteriorWriters are created as needed when 1655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** SegmentWriter creates new leaf nodes, or when an interior node 1665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** itself grows too big and must be split. The format of interior 1675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** nodes: 1685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint iHeight; (height from leaf level, always >0) 1705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint iBlockid; (block id of node's leftmost subtree) 1715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** optional { 1725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nTerm; (length of first term) 1735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** char pTerm[nTerm]; (content of first term) 1745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** array { 1755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (further terms are delta-encoded) 1765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nPrefix; (length of shared prefix with previous term) 1775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varint nSuffix; (length of unshared suffix) 1785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** char pTermSuffix[nSuffix]; (unshared suffix of next term) 1795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** } 1805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** } 1815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Here, optional { X } means an optional element, while array { X } 1835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** means zero or more occurrences of X, adjacent in memory. 1845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** An interior node encodes n terms separating n+1 subtrees. The 1865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** subtree blocks are contiguous, so only the first subtree's blockid 1875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is encoded. The subtree at iBlockid will contain all terms less 1885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** than the first term encoded (or all terms if no term is encoded). 1895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Otherwise, for terms greater than or equal to pTerm[i] but less 1905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** than pTerm[i+1], the subtree for that term will be rooted at 1915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** iBlockid+i. Interior nodes only store enough term data to 1925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** distinguish adjacent children (if the rightmost term of the left 1935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** child is "something", and the leftmost term of the right child is 1945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** "wicked", only "w" is stored). 1955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** New data is spilled to a new interior node at the same height when 1975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the current node exceeds INTERIOR_MAX bytes (default 2048). 1985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** INTERIOR_MIN_TERMS (default 7) keeps large terms from monopolizing 1995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** interior nodes and making the tree too skinny. The interior nodes 2005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** at a given height are naturally tracked by interior nodes at 2015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** height+1, and so on. 2025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)**** Segment directory **** 2055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The segment directory in table %_segdir stores meta-information for 2065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** merging and deleting segments, and also the root node of the 2075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** segment's tree. 2085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The root node is the top node of the segment's tree after encoding 2105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the entire segment, restricted to ROOT_MAX bytes (default 1024). 2115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This could be either a leaf node or an interior node. If the top 2125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** node requires more than ROOT_MAX bytes, it is flushed to %_segments 2135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** and a new root interior node is generated (which should always fit 2145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** within ROOT_MAX because it only needs space for 2 varints, the 2155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** height and the blockid of the previous root). 2165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The meta-information in the segment directory is: 2185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** level - segment level (see below) 2195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** idx - index within level 2205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** - (level,idx uniquely identify a segment) 2215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** start_block - first leaf node 2225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** leaves_end_block - last leaf node 2235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** end_block - last block (including interior nodes) 2245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** root - contents of root node 2255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If the root node is a leaf node, then start_block, 2275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** leaves_end_block, and end_block are all 0. 2285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)**** Segment merging **** 2315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** To amortize update costs, segments are grouped into levels and 2325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** merged in batches. Each increase in level represents exponentially 2335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** more documents. 2345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** New documents (actually, document updates) are tokenized and 2365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** written individually (using LeafWriter) to a level 0 segment, with 2375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** incrementing idx. When idx reaches MERGE_COUNT (default 16), all 2385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** level 0 segments are merged into a single level 1 segment. Level 1 2395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is populated like level 0, and eventually MERGE_COUNT level 1 2405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** segments are merged to a single level 2 segment (representing 2415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** MERGE_COUNT^2 updates), and so on. 2425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A segment merge traverses all segments at a given level in 2445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** parallel, performing a straightforward sorted merge. Since segment 2455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** leaf nodes are written in to the %_segments table in order, this 2465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** merge traverses the underlying sqlite disk structures efficiently. 2475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** After the merge, all segment blocks from the merged level are 2485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** deleted. 2495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** MERGE_COUNT controls how often we merge segments. 16 seems to be 2515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** somewhat of a sweet spot for insertion performance. 32 and 64 show 2525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** very similar performance numbers to 16 on insertion, though they're 2535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a tiny bit slower (perhaps due to more overhead in merge-time 2545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** sorting). 8 is about 20% slower than 16, 4 about 50% slower than 2555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 16, 2 about 66% slower than 16. 2565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** At query time, high MERGE_COUNT increases the number of segments 2585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** which need to be scanned and merged. For instance, with 100k docs 2595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** inserted: 2605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** MERGE_COUNT segments 2625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 16 25 2635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 8 12 2645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 4 10 2655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2 6 2665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This appears to have only a moderate impact on queries for very 2685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** frequent terms (which are somewhat dominated by segment merge 2695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** costs), and infrequent and non-existent terms still seem to be fast 2705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** even with many segments. 2715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** TODO(shess) That said, it would be nice to have a better query-side 2735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** argument for MERGE_COUNT of 16. Also, it is possible/likely that 2745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** optimizations to things like doclist merging will swing the sweet 2755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** spot around. 2765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)**** Handling of deletions and updates **** 2805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Since we're using a segmented structure, with no docid-oriented 2815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** index into the term index, we clearly cannot simply update the term 2825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** index when a document is deleted or updated. For deletions, we 2835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** write an empty doclist (varint(docid) varint(POS_END)), for updates 2845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** we simply write the new doclist. Segment merges overwrite older 2855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** data for a particular docid with newer data, so deletes or updates 2865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** will eventually overtake the earlier data and knock it out. The 2875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** query logic likewise merges doclists so that newer data knocks out 2885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** older data. 2895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** TODO(shess) Provide a VACUUM type operation to clear out all 2915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** deletions and duplications. This would basically be a forced merge 2925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** into a single segment. 2935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 2945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define CHROMIUM_FTS3_CHANGES 1 2955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) 2975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#if defined(SQLITE_ENABLE_FTS3) && !defined(SQLITE_CORE) 2995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# define SQLITE_CORE 1 3005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 3015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include "fts3Int.h" 3035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include <assert.h> 3055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include <stdlib.h> 3065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include <stddef.h> 3075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include <stdio.h> 3085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include <string.h> 3095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include <stdarg.h> 3105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#include "fts3.h" 3125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#ifndef SQLITE_CORE 3135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)# include "sqlite3ext.h" 3145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) SQLITE_EXTENSION_INIT1 3155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 3165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 3185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Write a 64-bit variable-length integer to memory starting at p[0]. 3195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The length of data written will be between 1 and FTS3_VARINT_MAX bytes. 3205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The number of bytes written is returned. 3215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 3225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3PutVarint(char *p, sqlite_int64 v){ 3235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) unsigned char *q = (unsigned char *) p; 3245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite_uint64 vu = v; 3255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) do{ 3265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *q++ = (unsigned char) ((vu & 0x7f) | 0x80); 3275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) vu >>= 7; 3285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }while( vu!=0 ); 3295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) q[-1] &= 0x7f; /* turn off high bit in final byte */ 3305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( q - (unsigned char *)p <= FTS3_VARINT_MAX ); 3315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return (int) (q - (unsigned char *)p); 3325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 3335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 3355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Read a 64-bit variable-length integer from memory starting at p[0]. 3365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return the number of bytes read, or 0 on error. 3375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The value is stored in *v. 3385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 3395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3GetVarint(const char *p, sqlite_int64 *v){ 3405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const unsigned char *q = (const unsigned char *) p; 3415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite_uint64 x = 0, y = 1; 3425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( (*q&0x80)==0x80 && q-(unsigned char *)p<FTS3_VARINT_MAX ){ 3435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) x += y * (*q++ & 0x7f); 3445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) y <<= 7; 3455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 3465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) x += y * (*q++); 3475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *v = (sqlite_int64) x; 3485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return (int) (q - (unsigned char *)p); 3495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 3505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 3525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Similar to sqlite3Fts3GetVarint(), except that the output is truncated to a 3535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 32-bit integer before it is returned. 3545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 3555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3GetVarint32(const char *p, int *pi){ 3565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite_int64 i; 3575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int ret = sqlite3Fts3GetVarint(p, &i); 3585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pi = (int) i; 3595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return ret; 3605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 3615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 3635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return the number of bytes required to encode v as a varint 3645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 3655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3VarintLen(sqlite3_uint64 v){ 3665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i = 0; 3675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) do{ 3685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) i++; 3695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) v >>= 7; 3705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }while( v!=0 ); 3715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return i; 3725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 3735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 3755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Convert an SQL-style quoted string into a normal string by removing 3765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the quote characters. The conversion is done in-place. If the 3775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** input does not begin with a quote character, then this routine 3785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is a no-op. 3795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 3805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Examples: 3815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 3825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** "abc" becomes abc 3835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 'xyz' becomes xyz 3845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** [pqr] becomes pqr 3855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** `mno` becomes mno 3865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 3875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 3885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)void sqlite3Fts3Dequote(char *z){ 3895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char quote; /* Quote character (if any ) */ 3905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) quote = z[0]; 3925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( quote=='[' || quote=='\'' || quote=='"' || quote=='`' ){ 3935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iIn = 1; /* Index of next byte to read from input */ 3945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iOut = 0; /* Index of next byte to write to output */ 3955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If the first byte was a '[', then the close-quote character is a ']' */ 3975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( quote=='[' ) quote = ']'; 3985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( ALWAYS(z[iIn]) ){ 4005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( z[iIn]==quote ){ 4015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( z[iIn+1]!=quote ) break; 4025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) z[iOut++] = quote; 4035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iIn += 2; 4045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 4055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) z[iOut++] = z[iIn++]; 4065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 4075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 4085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) z[iOut] = '\0'; 4095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 4105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 4115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 4135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Read a single varint from the doclist at *pp and advance *pp to point 4145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to the first byte past the end of the varint. Add the value of the varint 4155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to *pVal. 4165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 4175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3GetDeltaVarint(char **pp, sqlite3_int64 *pVal){ 4185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iVal; 4195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp += sqlite3Fts3GetVarint(*pp, &iVal); 4205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pVal += iVal; 4215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 4225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 4245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** As long as *pp has not reached its end (pEnd), then do the same 4255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** as fts3GetDeltaVarint(): read a single varint and add it to *pVal. 4265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** But if we have reached the end of the varint, just set *pp=0 and 4275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** leave *pVal unchanged. 4285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 4295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3GetDeltaVarint2(char **pp, char *pEnd, sqlite3_int64 *pVal){ 4305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *pp>=pEnd ){ 4315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp = 0; 4325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 4335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(pp, pVal); 4345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 4355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 4365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 4385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The xDisconnect() virtual table method. 4395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 4405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3DisconnectMethod(sqlite3_vtab *pVtab){ 4415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pVtab; 4425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 4435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( p->nPendingData==0 ); 4455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( p->pSegments==0 ); 4465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Free any prepared statements held */ 4485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; i<SizeofArray(p->aStmt); i++){ 4495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_finalize(p->aStmt[i]); 4505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 4515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(p->zSegmentsTbl); 4525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(p->zReadExprlist); 4535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(p->zWriteExprlist); 4545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Invoke the tokenizer destructor to free the tokenizer. */ 4565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->pTokenizer->pModule->xDestroy(p->pTokenizer); 4575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(p); 4595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 4605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 4615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 4635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Construct one or more SQL statements from the format string given 4645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** and then evaluate those statements. The success code is written 4655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** into *pRc. 4665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 4675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If *pRc is initially non-zero then this routine is a no-op. 4685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 4695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3DbExec( 4705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pRc, /* Success code */ 4715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db, /* Database in which to run SQL */ 4725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zFormat, /* Format string for SQL */ 4735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ... /* Arguments to the format string */ 4745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 4755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) va_list ap; 4765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zSql; 4775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *pRc ) return; 4785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) va_start(ap, zFormat); 4795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zSql = sqlite3_vmprintf(zFormat, ap); 4805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) va_end(ap); 4815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( zSql==0 ){ 4825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pRc = SQLITE_NOMEM; 4835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 4845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pRc = sqlite3_exec(db, zSql, 0, 0, 0); 4855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zSql); 4865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 4875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 4885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 4905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The xDestroy() virtual table method. 4915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 4925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3DestroyMethod(sqlite3_vtab *pVtab){ 4935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; /* Return code */ 4945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pVtab; 4955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db = p->db; 4965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 4975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Drop the shadow tables */ 4985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, "DROP TABLE IF EXISTS %Q.'%q_content'", p->zDb, p->zName); 4995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, "DROP TABLE IF EXISTS %Q.'%q_segments'", p->zDb,p->zName); 5005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, "DROP TABLE IF EXISTS %Q.'%q_segdir'", p->zDb, p->zName); 5015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, "DROP TABLE IF EXISTS %Q.'%q_docsize'", p->zDb, p->zName); 5025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, "DROP TABLE IF EXISTS %Q.'%q_stat'", p->zDb, p->zName); 5035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If everything has worked, invoke fts3DisconnectMethod() to free the 5055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** memory associated with the Fts3Table structure and return SQLITE_OK. 5065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** Otherwise, return an SQLite error code. 5075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 5085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return (rc==SQLITE_OK ? fts3DisconnectMethod(pVtab) : rc); 5095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 5105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 5135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Invoke sqlite3_declare_vtab() to declare the schema for the FTS3 table 5145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** passed as the first argument. This is done as part of the xConnect() 5155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** and xCreate() methods. 5165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 5175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If *pRc is non-zero when this function is called, it is a no-op. 5185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Otherwise, if an error occurs, an SQLite error code is stored in *pRc 5195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** before returning. 5205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 5215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3DeclareVtab(int *pRc, Fts3Table *p){ 5225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *pRc==SQLITE_OK ){ 5235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; /* Iterator variable */ 5245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 5255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zSql; /* SQL statement passed to declare_vtab() */ 5265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zCols; /* List of user defined columns */ 5275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Create a list of user columns for the virtual table */ 5295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCols = sqlite3_mprintf("%Q, ", p->azColumn[0]); 5305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=1; zCols && i<p->nColumn; i++){ 5315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCols = sqlite3_mprintf("%z%Q, ", zCols, p->azColumn[i]); 5325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 5335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Create the whole "CREATE TABLE" statement to pass to SQLite */ 5355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zSql = sqlite3_mprintf( 5365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "CREATE TABLE x(%s %Q HIDDEN, docid HIDDEN)", zCols, p->zName 5375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 5385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zCols || !zSql ){ 5395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 5405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 5415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_declare_vtab(p->db, zSql); 5425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 5435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zSql); 5455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zCols); 5465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pRc = rc; 5475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 5485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 5495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 5515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Create the backing store tables (%_content, %_segments and %_segdir) 5525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** required by the FTS3 table passed as the only argument. This is done 5535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** as part of the vtab xCreate() method. 5545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 5555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If the p->bHasDocsize boolean is true (indicating that this is an 5565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** FTS4 table, not an FTS3 table) then also create the %_docsize and 5575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** %_stat tables required by FTS4. 5585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 5595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3CreateTables(Fts3Table *p){ 5605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; /* Return code */ 5615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; /* Iterator variable */ 5625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zContentCols; /* Columns of %_content table */ 5635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db = p->db; /* The database connection */ 5645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Create a list of user columns for the content table */ 5665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zContentCols = sqlite3_mprintf("docid INTEGER PRIMARY KEY"); 5675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; zContentCols && i<p->nColumn; i++){ 5685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *z = p->azColumn[i]; 5695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zContentCols = sqlite3_mprintf("%z, 'c%d%q'", zContentCols, i, z); 5705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 5715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( zContentCols==0 ) rc = SQLITE_NOMEM; 5725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 5735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Create the content table */ 5745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 5755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "CREATE TABLE %Q.'%q_content'(%s)", 5765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName, zContentCols 5775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 5785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zContentCols); 5795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Create other tables */ 5805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 5815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "CREATE TABLE %Q.'%q_segments'(blockid INTEGER PRIMARY KEY, block BLOB);", 5825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName 5835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 5845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 5855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "CREATE TABLE %Q.'%q_segdir'(" 5865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "level INTEGER," 5875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "idx INTEGER," 5885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "start_block INTEGER," 5895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "leaves_end_block INTEGER," 5905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "end_block INTEGER," 5915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "root BLOB," 5925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "PRIMARY KEY(level, idx)" 5935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ");", 5945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName 5955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 5965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p->bHasDocsize ){ 5975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 5985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "CREATE TABLE %Q.'%q_docsize'(docid INTEGER PRIMARY KEY, size BLOB);", 5995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName 6005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 6015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p->bHasStat ){ 6035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 6045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "CREATE TABLE %Q.'%q_stat'(id INTEGER PRIMARY KEY, value BLOB);", 6055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName 6065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 6075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 6095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 6105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 6115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 6125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Store the current database page-size in bytes in p->nPgsz. 6135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 6145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If *pRc is non-zero when this function is called, it is a no-op. 6155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Otherwise, if an error occurs, an SQLite error code is stored in *pRc 6165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** before returning. 6175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 6185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3DatabasePageSize(int *pRc, Fts3Table *p){ 6195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *pRc==SQLITE_OK ){ 6205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 6215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zSql; /* SQL text "PRAGMA %Q.page_size" */ 6225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_stmt *pStmt; /* Compiled "PRAGMA %Q.page_size" statement */ 6235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 6245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zSql = sqlite3_mprintf("PRAGMA %Q.page_size", p->zDb); 6255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zSql ){ 6265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 6275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 6285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_prepare(p->db, zSql, -1, &pStmt, 0); 6295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 6305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_step(pStmt); 6315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->nPgsz = sqlite3_column_int(pStmt, 0); 6325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_finalize(pStmt); 6335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( rc==SQLITE_AUTH ){ 6345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->nPgsz = 1024; 6355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_OK; 6365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( p->nPgsz>0 || rc!=SQLITE_OK ); 6395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zSql); 6405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pRc = rc; 6415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 6435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 6445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 6455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** "Special" FTS4 arguments are column specifications of the following form: 6465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 6475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** <key> = <value> 6485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 6495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** There may not be whitespace surrounding the "=" character. The <value> 6505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** term may be quoted, but the <key> may not. 6515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 6525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3IsSpecialColumn( 6535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *z, 6545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnKey, 6555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pzValue 6565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 6575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zValue; 6585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zCsr = z; 6595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 6605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( *zCsr!='=' ){ 6615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *zCsr=='\0' ) return 0; 6625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr++; 6635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 6655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnKey = (int)(zCsr-z); 6665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zValue = sqlite3_mprintf("%s", &zCsr[1]); 6675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( zValue ){ 6685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3Dequote(zValue); 6695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pzValue = zValue; 6715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 1; 6725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 6735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 6745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 6755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Append the output of a printf() style formatting to an existing string. 6765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 6775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3Appendf( 6785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pRc, /* IN/OUT: Error code */ 6795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pz, /* IN/OUT: Pointer to string buffer */ 6805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zFormat, /* Printf format string to append */ 6815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ... /* Arguments for printf format string */ 6825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 6835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *pRc==SQLITE_OK ){ 6845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) va_list ap; 6855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *z; 6865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) va_start(ap, zFormat); 6875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) z = sqlite3_vmprintf(zFormat, ap); 6885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( z && *pz ){ 6895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *z2 = sqlite3_mprintf("%s%s", *pz, z); 6905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(z); 6915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) z = z2; 6925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( z==0 ) *pRc = SQLITE_NOMEM; 6945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(*pz); 6955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pz = z; 6965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 6975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 6985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 6995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 7005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return a copy of input string zInput enclosed in double-quotes (") and 7015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** with all double quote characters escaped. For example: 7025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** fts3QuoteId("un \"zip\"") -> "un \"\"zip\"\"" 7045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The pointer returned points to memory obtained from sqlite3_malloc(). It 7065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is the callers responsibility to call sqlite3_free() to release this 7075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** memory. 7085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 7095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static char *fts3QuoteId(char const *zInput){ 7105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nRet; 7115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zRet; 7125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nRet = 2 + strlen(zInput)*2 + 1; 7135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zRet = sqlite3_malloc(nRet); 7145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( zRet ){ 7155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 7165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *z = zRet; 7175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *(z++) = '"'; 7185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; zInput[i]; i++){ 7195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( zInput[i]=='"' ) *(z++) = '"'; 7205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *(z++) = zInput[i]; 7215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 7225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *(z++) = '"'; 7235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *(z++) = '\0'; 7245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 7255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return zRet; 7265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 7275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 7285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 7295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return a list of comma separated SQL expressions that could be used 7305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in a SELECT statement such as the following: 7315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** SELECT <list of expressions> FROM %_content AS x ... 7335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to return the docid, followed by each column of text data in order 7355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** from left to write. If parameter zFunc is not NULL, then instead of 7365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** being returned directly each column of text data is passed to an SQL 7375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** function named zFunc first. For example, if zFunc is "unzip" and the 7385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** table has the three user-defined columns "a", "b", and "c", the following 7395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** string is returned: 7405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** "docid, unzip(x.'a'), unzip(x.'b'), unzip(x.'c')" 7425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The pointer returned points to a buffer allocated by sqlite3_malloc(). It 7445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is the responsibility of the caller to eventually free it. 7455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If *pRc is not SQLITE_OK when this function is called, it is a no-op (and 7475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a NULL pointer is returned). Otherwise, if an OOM error is encountered 7485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** by this function, NULL is returned and *pRc is set to SQLITE_NOMEM. If 7495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** no error occurs, *pRc is left unmodified. 7505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 7515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static char *fts3ReadExprList(Fts3Table *p, const char *zFunc, int *pRc){ 7525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zRet = 0; 7535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zFree = 0; 7545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zFunction; 7555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 7565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 7575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zFunc ){ 7585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zFunction = ""; 7595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 7605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zFree = zFunction = fts3QuoteId(zFunc); 7615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 7625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3Appendf(pRc, &zRet, "docid"); 7635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; i<p->nColumn; i++){ 7645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3Appendf(pRc, &zRet, ",%s(x.'c%d%q')", zFunction, i, p->azColumn[i]); 7655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 7665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zFree); 7675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return zRet; 7685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 7695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 7705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 7715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return a list of N comma separated question marks, where N is the number 7725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** of columns in the %_content table (one for the docid plus one for each 7735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** user-defined text column). 7745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If argument zFunc is not NULL, then all but the first question mark 7765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is preceded by zFunc and an open bracket, and followed by a closed 7775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** bracket. For example, if zFunc is "zip" and the FTS3 table has three 7785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** user-defined text columns, the following string is returned: 7795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** "?, zip(?), zip(?), zip(?)" 7815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The pointer returned points to a buffer allocated by sqlite3_malloc(). It 7835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is the responsibility of the caller to eventually free it. 7845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 7855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If *pRc is not SQLITE_OK when this function is called, it is a no-op (and 7865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a NULL pointer is returned). Otherwise, if an OOM error is encountered 7875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** by this function, NULL is returned and *pRc is set to SQLITE_NOMEM. If 7885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** no error occurs, *pRc is left unmodified. 7895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 7905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static char *fts3WriteExprList(Fts3Table *p, const char *zFunc, int *pRc){ 7915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zRet = 0; 7925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zFree = 0; 7935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zFunction; 7945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 7955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 7965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zFunc ){ 7975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zFunction = ""; 7985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 7995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zFree = zFunction = fts3QuoteId(zFunc); 8005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 8015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3Appendf(pRc, &zRet, "?"); 8025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; i<p->nColumn; i++){ 8035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3Appendf(pRc, &zRet, ",%s(?)", zFunction); 8045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 8055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zFree); 8065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return zRet; 8075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 8085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 8105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is the implementation of both the xConnect and xCreate 8115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** methods of the FTS3 virtual table. 8125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 8135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The argv[] array contains the following: 8145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 8155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** argv[0] -> module name ("fts3" or "fts4") 8165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** argv[1] -> database name 8175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** argv[2] -> table name 8185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** argv[...] -> "column name" and other module argument fields. 8195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 8205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3InitVtab( 8215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isCreate, /* True for xCreate, false for xConnect */ 8225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db, /* The SQLite database connection */ 8235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) void *pAux, /* Hash table containing tokenizers */ 8245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int argc, /* Number of elements in argv array */ 8255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char * const *argv, /* xCreate/xConnect argument array */ 8265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab **ppVTab, /* Write the resulting vtab structure here */ 8275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pzErr /* Write any error message here */ 8285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 8295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Hash *pHash = (Fts3Hash *)pAux; 8305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = 0; /* Pointer to allocated vtab */ 8315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; /* Return code */ 8325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; /* Iterator variable */ 8335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nByte; /* Size of allocation used for *p */ 8345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol; /* Column index */ 8355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nString = 0; /* Bytes required to hold all column names */ 8365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nCol = 0; /* Number of columns in the FTS table */ 8375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zCsr; /* Space for holding column names */ 8385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nDb; /* Bytes required to hold database name */ 8395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nName; /* Bytes required to hold table name */ 8405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isFts4 = (argv[0][3]=='4'); /* True for FTS4, false for FTS3 */ 8415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int bNoDocsize = 0; /* True to omit %_docsize table */ 8425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char **aCol; /* Array of column names */ 8435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_tokenizer *pTokenizer = 0; /* Tokenizer for this table */ 8445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zCompress = 0; 8465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zUncompress = 0; 8475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( strlen(argv[0])==4 ); 8495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( (sqlite3_strnicmp(argv[0], "fts4", 4)==0 && isFts4) 8505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || (sqlite3_strnicmp(argv[0], "fts3", 4)==0 && !isFts4) 8515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 8525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDb = (int)strlen(argv[1]) + 1; 8545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nName = (int)strlen(argv[2]) + 1; 8555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aCol = (const char **)sqlite3_malloc(sizeof(const char *) * (argc-2) ); 8575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aCol ) return SQLITE_NOMEM; 8585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memset((void *)aCol, 0, sizeof(const char *) * (argc-2)); 8595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Loop through all of the arguments passed by the user to the FTS3/4 8615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** module (i.e. all the column names and special arguments). This loop 8625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** does the following: 8635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** 8645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** + Figures out the number of columns the FTSX table will have, and 8655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** the number of bytes of space that must be allocated to store copies 8665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** of the column names. 8675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** 8685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** + If there is a tokenizer specification included in the arguments, 8695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** initializes the tokenizer pTokenizer. 8705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 8715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=3; rc==SQLITE_OK && i<argc; i++){ 8725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char const *z = argv[i]; 8735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nKey; 8745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zVal; 8755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Check if this is a tokenizer specification */ 8775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !pTokenizer 8785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && strlen(z)>8 8795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && 0==sqlite3_strnicmp(z, "tokenize", 8) 8805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && 0==sqlite3Fts3IsIdChar(z[8]) 8815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 8825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3InitTokenizer(pHash, &z[9], &pTokenizer, pzErr); 8835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 8845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 8855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Check if it is an FTS4 special argument. */ 8865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) else if( isFts4 && fts3IsSpecialColumn(z, &nKey, &zVal) ){ 8875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zVal ){ 8885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 8895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) goto fts3_init_out; 8905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 8915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( nKey==9 && 0==sqlite3_strnicmp(z, "matchinfo", 9) ){ 8925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( strlen(zVal)==4 && 0==sqlite3_strnicmp(zVal, "fts3", 4) ){ 8935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) bNoDocsize = 1; 8945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 8955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pzErr = sqlite3_mprintf("unrecognized matchinfo: %s", zVal); 8965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_ERROR; 8975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 8985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( nKey==8 && 0==sqlite3_strnicmp(z, "compress", 8) ){ 8995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCompress = zVal; 9005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zVal = 0; 9015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( nKey==10 && 0==sqlite3_strnicmp(z, "uncompress", 10) ){ 9025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zUncompress = zVal; 9035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zVal = 0; 9045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 9055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pzErr = sqlite3_mprintf("unrecognized parameter: %s", z); 9065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_ERROR; 9075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zVal); 9095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Otherwise, the argument is a column name. */ 9125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) else { 9135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nString += (int)(strlen(z) + 1); 9145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aCol[nCol++] = z; 9155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) goto fts3_init_out; 9185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( nCol==0 ){ 9205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nString==0 ); 9215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aCol[0] = "content"; 9225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nString = 8; 9235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nCol = 1; 9245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTokenizer==0 ){ 9275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3InitTokenizer(pHash, "simple", &pTokenizer, pzErr); 9285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) goto fts3_init_out; 9295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pTokenizer ); 9315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Allocate and populate the Fts3Table structure. */ 9345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nByte = sizeof(Fts3Table) + /* Fts3Table */ 9355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nCol * sizeof(char *) + /* azColumn */ 9365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nName + /* zName */ 9375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDb + /* zDb */ 9385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nString; /* Space for azColumn strings */ 9395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p = (Fts3Table*)sqlite3_malloc(nByte); 9405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p==0 ){ 9415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 9425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) goto fts3_init_out; 9435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memset(p, 0, nByte); 9455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->db = db; 9465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->nColumn = nCol; 9475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->nPendingData = 0; 9485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->azColumn = (char **)&p[1]; 9495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->pTokenizer = pTokenizer; 9505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->nNodeSize = 1000; 9515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->nMaxPendingData = FTS3_MAX_PENDING_DATA; 9525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->bHasDocsize = (isFts4 && bNoDocsize==0); 9535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->bHasStat = isFts4; 9545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3HashInit(&p->pendingTerms, FTS3_HASH_STRING, 1); 9555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Fill in the zName and zDb fields of the vtab structure. */ 9575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr = (char *)&p->azColumn[nCol]; 9585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zName = zCsr; 9595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(zCsr, argv[2], nName); 9605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += nName; 9615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb = zCsr; 9625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(zCsr, argv[1], nDb); 9635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += nDb; 9645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Fill in the azColumn array */ 9665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(iCol=0; iCol<nCol; iCol++){ 9675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *z; 9685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n; 9695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) z = (char *)sqlite3Fts3NextToken(aCol[iCol], &n); 9705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(zCsr, z, n); 9715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr[n] = '\0'; 9725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3Dequote(zCsr); 9735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->azColumn[iCol] = zCsr; 9745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += n+1; 9755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( zCsr <= &((char *)p)[nByte] ); 9765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (zCompress==0)!=(zUncompress==0) ){ 9795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char const *zMiss = (zCompress==0 ? "compress" : "uncompress"); 9805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_ERROR; 9815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pzErr = sqlite3_mprintf("missing %s parameter in fts4 constructor", zMiss); 9825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zReadExprlist = fts3ReadExprList(p, zUncompress, &rc); 9845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zWriteExprlist = fts3WriteExprList(p, zCompress, &rc); 9855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) goto fts3_init_out; 9865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If this is an xCreate call, create the underlying tables in the 9885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** database. TODO: For xConnect(), it could verify that said tables exist. 9895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 9905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( isCreate ){ 9915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3CreateTables(p); 9925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 9935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 9945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Figure out the page-size for the database. This is required in order to 9955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** estimate the cost of loading large doclists from the database (see 9965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** function sqlite3Fts3SegReaderCost() for details). 9975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 9985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DatabasePageSize(&rc, p); 9995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Declare the table schema to SQLite. */ 10015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DeclareVtab(&rc, p); 10025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)fts3_init_out: 10045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zCompress); 10055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zUncompress); 10065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free((void *)aCol); 10075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ){ 10085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p ){ 10095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DisconnectMethod((sqlite3_vtab *)p); 10105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( pTokenizer ){ 10115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTokenizer->pModule->xDestroy(pTokenizer); 10125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 10135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 10145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppVTab = &p->base; 10155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 10165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 10175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 10185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 10205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The xConnect() and xCreate() methods for the virtual table. All the 10215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** work is done in function fts3InitVtab(). 10225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 10235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3ConnectMethod( 10245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db, /* Database connection */ 10255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) void *pAux, /* Pointer to tokenizer hash table */ 10265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int argc, /* Number of elements in argv array */ 10275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char * const *argv, /* xCreate/xConnect argument array */ 10285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab **ppVtab, /* OUT: New sqlite3_vtab object */ 10295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pzErr /* OUT: sqlite3_malloc'd error message */ 10305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 10315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return fts3InitVtab(0, db, pAux, argc, argv, ppVtab, pzErr); 10325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 10335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3CreateMethod( 10345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db, /* Database connection */ 10355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) void *pAux, /* Pointer to tokenizer hash table */ 10365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int argc, /* Number of elements in argv array */ 10375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char * const *argv, /* xCreate/xConnect argument array */ 10385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab **ppVtab, /* OUT: New sqlite3_vtab object */ 10395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pzErr /* OUT: sqlite3_malloc'd error message */ 10405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 10415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return fts3InitVtab(1, db, pAux, argc, argv, ppVtab, pzErr); 10425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 10435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 10455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of the xBestIndex method for FTS3 tables. There 10465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** are three possible strategies, in order of preference: 10475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 10485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 1. Direct lookup by rowid or docid. 10495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 2. Full-text search using a MATCH operator on a non-docid column. 10505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 3. Linear scan of %_content table. 10515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 10525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3BestIndexMethod(sqlite3_vtab *pVTab, sqlite3_index_info *pInfo){ 10535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pVTab; 10545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; /* Iterator variable */ 10555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCons = -1; /* Index of constraint to use */ 10565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* By default use a full table scan. This is an expensive option, 10585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** so search through the constraints to see if a more efficient 10595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** strategy is possible. 10605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 10615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->idxNum = FTS3_FULLSCAN_SEARCH; 10625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->estimatedCost = 500000; 10635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; i<pInfo->nConstraint; i++){ 10645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) struct sqlite3_index_constraint *pCons = &pInfo->aConstraint[i]; 10655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCons->usable==0 ) continue; 10665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* A direct lookup on the rowid or docid column. Assign a cost of 1.0. */ 10685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCons->op==SQLITE_INDEX_CONSTRAINT_EQ 10695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && (pCons->iColumn<0 || pCons->iColumn==p->nColumn+1 ) 10705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 10715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->idxNum = FTS3_DOCID_SEARCH; 10725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->estimatedCost = 1.0; 10735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iCons = i; 10745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 10755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* A MATCH constraint. Use a full-text search. 10775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** 10785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** If there is more than one MATCH constraint available, use the first 10795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** one encountered. If there is both a MATCH constraint and a direct 10805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** rowid/docid lookup, prefer the MATCH strategy. This is done even 10815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** though the rowid/docid lookup is faster than a MATCH query, selecting 10825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** it would lead to an "unable to use function MATCH in the requested 10835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** context" error. 10845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 10855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCons->op==SQLITE_INDEX_CONSTRAINT_MATCH 10865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && pCons->iColumn>=0 && pCons->iColumn<=p->nColumn 10875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 10885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->idxNum = FTS3_FULLTEXT_SEARCH + pCons->iColumn; 10895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->estimatedCost = 2.0; 10905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iCons = i; 10915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 10925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 10935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 10945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 10955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iCons>=0 ){ 10965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->aConstraintUsage[iCons].argvIndex = 1; 10975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pInfo->aConstraintUsage[iCons].omit = 1; 10985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 10995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 11005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 11015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 11025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 11035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of xOpen method. 11045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 11055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3OpenMethod(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCsr){ 11065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab_cursor *pCsr; /* Allocated cursor */ 11075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 11085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(pVTab); 11095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 11105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Allocate a buffer large enough for an Fts3Cursor structure. If the 11115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** allocation succeeds, zero it and return SQLITE_OK. Otherwise, 11125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** if the allocation fails, return SQLITE_NOMEM. 11135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 11145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppCsr = pCsr = (sqlite3_vtab_cursor *)sqlite3_malloc(sizeof(Fts3Cursor)); 11155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !pCsr ){ 11165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_NOMEM; 11175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 11185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memset(pCsr, 0, sizeof(Fts3Cursor)); 11195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 11205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 11215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 11225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 11235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Close the cursor. For additional information see the documentation 11245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** on the xClose method of the virtual table interface. 11255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 11265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3CloseMethod(sqlite3_vtab_cursor *pCursor){ 11275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; 11285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 ); 11295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_finalize(pCsr->pStmt); 11305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3ExprFree(pCsr->pExpr); 11315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3FreeDeferredTokens(pCsr); 11325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pCsr->aDoclist); 11335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pCsr->aMatchinfo); 11345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pCsr); 11355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 11365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 11375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 11385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 11395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Position the pCsr->pStmt statement so that it is on the row 11405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** of the %_content table that contains the last match. Return 11415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** SQLITE_OK on success. 11425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 11435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3CursorSeek(sqlite3_context *pContext, Fts3Cursor *pCsr){ 11445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->isRequireSeek ){ 11455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->isRequireSeek = 0; 11465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_bind_int64(pCsr->pStmt, 1, pCsr->iPrevId); 11475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( SQLITE_ROW==sqlite3_step(pCsr->pStmt) ){ 11485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 11495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 11505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = sqlite3_reset(pCsr->pStmt); 11515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 11525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If no row was found and no error has occured, then the %_content 11535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** table is missing a row that is present in the full-text index. 11545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** The data structures are corrupt. 11555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 11565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_CORRUPT; 11575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 11585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->isEof = 1; 11595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pContext ){ 11605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_error_code(pContext, rc); 11615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 11625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 11635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 11645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 11655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 11665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 11675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 11685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 11695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 11705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is used to process a single interior node when searching 11715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a b-tree for a term or term prefix. The node data is passed to this 11725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** function via the zNode/nNode parameters. The term to search for is 11735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** passed in zTerm/nTerm. 11745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 11755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If piFirst is not NULL, then this function sets *piFirst to the blockid 11765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** of the child node that heads the sub-tree that may contain the term. 11775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 11785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If piLast is not NULL, then *piLast is set to the right-most child node 11795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** that heads a sub-tree that may contain a term for which zTerm/nTerm is 11805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a prefix. 11815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 11825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If an OOM error occurs, SQLITE_NOMEM is returned. Otherwise, SQLITE_OK. 11835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 11845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3ScanInteriorNode( 11855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zTerm, /* Term to select leaves for */ 11865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nTerm, /* Size of term zTerm in bytes */ 11875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zNode, /* Buffer containing segment interior node */ 11885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nNode, /* Size of buffer at zNode */ 11895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 *piFirst, /* OUT: Selected child node */ 11905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 *piLast /* OUT: Selected child node */ 11915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 11925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; /* Return code */ 11935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zCsr = zNode; /* Cursor to iterate through node */ 11945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zEnd = &zCsr[nNode];/* End of interior node buffer */ 11955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zBuffer = 0; /* Buffer to load terms into */ 11965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nAlloc = 0; /* Size of allocated buffer */ 11975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isFirstTerm = 1; /* True when processing first term on page */ 11985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iChild; /* Block id of child node to descend to */ 11995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Skip over the 'height' varint that occurs at the start of every 12015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** interior node. Then load the blockid of the left-child of the b-tree 12025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** node into variable iChild. 12035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** 12045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** Even if the data structure on disk is corrupted, this (reading two 12055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** varints from the buffer) does not risk an overread. If zNode is a 12065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** root node, then the buffer comes from a SELECT statement. SQLite does 12075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** not make this guarantee explicitly, but in practice there are always 12085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** either more than 20 bytes of allocated space following the nNode bytes of 12095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** contents, or two zero bytes. Or, if the node is read from the %_segments 12105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** table, then there are always 20 bytes of zeroed padding following the 12115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** nNode bytes of content (see sqlite3Fts3ReadBlock() for details). 12125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 12135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += sqlite3Fts3GetVarint(zCsr, &iChild); 12145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += sqlite3Fts3GetVarint(zCsr, &iChild); 12155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( zCsr>zEnd ){ 12165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_CORRUPT; 12175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 12185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( zCsr<zEnd && (piFirst || piLast) ){ 12205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int cmp; /* memcmp() result */ 12215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nSuffix; /* Size of term suffix */ 12225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nPrefix = 0; /* Size of term prefix */ 12235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nBuffer; /* Total term size */ 12245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Load the next term on the node into zBuffer. Use realloc() to expand 12265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** the size of zBuffer if required. */ 12275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !isFirstTerm ){ 12285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += sqlite3Fts3GetVarint32(zCsr, &nPrefix); 12295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 12305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) isFirstTerm = 0; 12315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += sqlite3Fts3GetVarint32(zCsr, &nSuffix); 12325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* NOTE(shess): Previous code checked for negative nPrefix and 12345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** nSuffix and suffix overrunning zEnd. Additionally corrupt if 12355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** the prefix is longer than the previous term, or if the suffix 12365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** causes overflow. 12375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 12385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( nPrefix<0 || nSuffix<0 /* || nPrefix>nBuffer */ 12395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || &zCsr[nSuffix]<zCsr || &zCsr[nSuffix]>zEnd ){ 12405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_CORRUPT; 12415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) goto finish_scan; 12425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 12435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( nPrefix+nSuffix>nAlloc ){ 12445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zNew; 12455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nAlloc = (nPrefix+nSuffix) * 2; 12465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zNew = (char *)sqlite3_realloc(zBuffer, nAlloc); 12475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zNew ){ 12485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 12495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) goto finish_scan; 12505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 12515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zBuffer = zNew; 12525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 12535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(&zBuffer[nPrefix], zCsr, nSuffix); 12545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nBuffer = nPrefix + nSuffix; 12555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zCsr += nSuffix; 12565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Compare the term we are searching for with the term just loaded from 12585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** the interior node. If the specified term is greater than or equal 12595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** to the term from the interior node, then all terms on the sub-tree 12605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** headed by node iChild are smaller than zTerm. No need to search 12615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** iChild. 12625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** 12635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** If the interior node term is larger than the specified term, then 12645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** the tree headed by iChild may contain the specified term. 12655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 12665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) cmp = memcmp(zTerm, zBuffer, (nBuffer>nTerm ? nTerm : nBuffer)); 12675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( piFirst && (cmp<0 || (cmp==0 && nBuffer>nTerm)) ){ 12685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *piFirst = iChild; 12695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) piFirst = 0; 12705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 12715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( piLast && cmp<0 ){ 12735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *piLast = iChild; 12745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) piLast = 0; 12755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 12765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iChild++; 12785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }; 12795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( piFirst ) *piFirst = iChild; 12815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( piLast ) *piLast = iChild; 12825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) finish_scan: 12845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zBuffer); 12855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 12865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 12875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 12895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 12905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The buffer pointed to by argument zNode (size nNode bytes) contains an 12915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** interior node of a b-tree segment. The zTerm buffer (size nTerm bytes) 12925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** contains a term. This function searches the sub-tree headed by the zNode 12935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** node for the range of leaf nodes that may contain the specified term 12945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** or terms for which the specified term is a prefix. 12955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 12965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If piLeaf is not NULL, then *piLeaf is set to the blockid of the 12975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** left-most leaf node in the tree that may contain the specified term. 12985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If piLeaf2 is not NULL, then *piLeaf2 is set to the blockid of the 12995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** right-most leaf node that may contain a term for which the specified 13005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** term is a prefix. 13015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 13025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** It is possible that the range of returned leaf nodes does not contain 13035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the specified term or any terms for which it is a prefix. However, if the 13045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** segment does contain any such terms, they are stored within the identified 13055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** range. Because this function only inspects interior segment nodes (and 13065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** never loads leaf nodes into memory), it is not possible to be sure. 13075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 13085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If an error occurs, an error code other than SQLITE_OK is returned. 13095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 13105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3SelectLeaf( 13115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p, /* Virtual table handle */ 13125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zTerm, /* Term to select leaves for */ 13135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nTerm, /* Size of term zTerm in bytes */ 13145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zNode, /* Buffer containing segment interior node */ 13155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nNode, /* Size of buffer at zNode */ 13165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 *piLeaf, /* Selected leaf node */ 13175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 *piLeaf2 /* Selected leaf node */ 13185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 13195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 13205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iHeight; /* Height of this node in tree */ 13215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( piLeaf || piLeaf2 ); 13235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3GetVarint32(zNode, &iHeight); 13255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3ScanInteriorNode(zTerm, nTerm, zNode, nNode, piLeaf, piLeaf2); 13265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( !piLeaf2 || !piLeaf || rc!=SQLITE_OK || (*piLeaf<=*piLeaf2) ); 13275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK && iHeight>1 ){ 13295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zBlob = 0; /* Blob read from %_segments table */ 13305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nBlob; /* Size of zBlob in bytes */ 13315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( piLeaf && piLeaf2 && (*piLeaf!=*piLeaf2) ){ 13335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3ReadBlock(p, *piLeaf, &zBlob, &nBlob); 13345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 13355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, 0); 13365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 13375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zBlob); 13385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) piLeaf = 0; 13395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zBlob = 0; 13405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 13415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 13435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3ReadBlock(p, piLeaf ? *piLeaf : *piLeaf2, &zBlob, &nBlob); 13445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 13455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 13465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, piLeaf2); 13475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 13485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zBlob); 13495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 13505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 13525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 13535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 13555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is used to create delta-encoded serialized lists of FTS3 13565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varints. Each call to this function appends a single varint to a list. 13575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 13585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3PutDeltaVarint( 13595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp, /* IN/OUT: Output pointer */ 13605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 *piPrev, /* IN/OUT: Previous value written to list */ 13615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iVal /* Write this value to the list */ 13625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 13635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( iVal-*piPrev > 0 || (*piPrev==0 && iVal==0) ); 13645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp += sqlite3Fts3PutVarint(*pp, iVal-*piPrev); 13655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *piPrev = iVal; 13665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 13675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 13695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** When this function is called, *ppPoslist is assumed to point to the 13705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** start of a position-list. After it returns, *ppPoslist points to the 13715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** first byte after the position-list. 13725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 13735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A position list is list of positions (delta encoded) and columns for 13745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a single document record of a doclist. So, in other words, this 13755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** routine advances *ppPoslist so that it points to the next docid in 13765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the doclist, or to the first byte past the end of the doclist. 13775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 13785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If pp is not NULL, then the contents of the position list are copied 13795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to *pp. *pp is set to point to the first byte past the last byte copied 13805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** before this function returns. 13815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 13825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3PoslistCopy(char **pp, char **ppPoslist){ 13835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pEnd = *ppPoslist; 13845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char c = 0; 13855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 13865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* The end of a position list is marked by a zero encoded as an FTS3 13875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** varint. A single POS_END (0) byte. Except, if the 0 byte is preceded by 13885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** a byte with the 0x80 bit set, then it is not a varint 0, but the tail 13895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** of some other, multi-byte, value. 13905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** 13915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** The following while-loop moves pEnd to point to the first byte that is not 13925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** immediately preceded by a byte with the 0x80 bit set. Then increments 13935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** pEnd once more so that it points to the byte immediately following the 13945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** last byte in the position-list. 13955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 13965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( *pEnd | c ){ 13975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) c = *pEnd++ & 0x80; 13985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) testcase( c!=0 && (*pEnd)==0 ); 13995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 14005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pEnd++; /* Advance past the POS_END terminator byte */ 14015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 14025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pp ){ 14035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n = (int)(pEnd - *ppPoslist); 14045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = *pp; 14055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(p, *ppPoslist, n); 14065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p += n; 14075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp = p; 14085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 14095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppPoslist = pEnd; 14105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 14115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 14125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 14135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** When this function is called, *ppPoslist is assumed to point to the 14145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** start of a column-list. After it returns, *ppPoslist points to the 14155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to the terminator (POS_COLUMN or POS_END) byte of the column-list. 14165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 14175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A column-list is list of delta-encoded positions for a single column 14185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** within a single document within a doclist. 14195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 14205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The column-list is terminated either by a POS_COLUMN varint (1) or 14215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a POS_END varint (0). This routine leaves *ppPoslist pointing to 14225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the POS_COLUMN or POS_END that terminates the column-list. 14235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 14245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If pp is not NULL, then the contents of the column-list are copied 14255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to *pp. *pp is set to point to the first byte past the last byte copied 14265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** before this function returns. The POS_COLUMN or POS_END terminator 14275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is not copied into *pp. 14285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 14295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3ColumnlistCopy(char **pp, char **ppPoslist){ 14305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pEnd = *ppPoslist; 14315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char c = 0; 14325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 14335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* A column-list is terminated by either a 0x01 or 0x00 byte that is 14345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** not part of a multi-byte varint. 14355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 14365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( 0xFE & (*pEnd | c) ){ 14375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) c = *pEnd++ & 0x80; 14385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) testcase( c!=0 && ((*pEnd)&0xfe)==0 ); 14395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 14405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pp ){ 14415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n = (int)(pEnd - *ppPoslist); 14425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = *pp; 14435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(p, *ppPoslist, n); 14445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p += n; 14455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp = p; 14465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 14475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppPoslist = pEnd; 14485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 14495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 14505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 14515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Value used to signify the end of an position-list. This is safe because 14525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** it is not possible to have a document with 2^31 terms. 14535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 14545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define POSITION_LIST_END 0x7fffffff 14555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 14565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 14575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is used to help parse position-lists. When this function is 14585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** called, *pp may point to the start of the next varint in the position-list 14595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** being parsed, or it may point to 1 byte past the end of the position-list 14605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (in which case **pp will be a terminator bytes POS_END (0) or 14615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (1)). 14625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 14635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If *pp points past the end of the current position-list, set *pi to 14645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** POSITION_LIST_END and return. Otherwise, read the next varint from *pp, 14655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** increment the current value of *pi by the value read, and set *pp to 14665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** point to the next value before returning. 14675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 14685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Before calling this routine *pi must be initialized to the value of 14695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the previous position, or zero if we are reading the first position 14705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in the position-list. Because positions are delta-encoded, the value 14715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** of the previous position is needed in order to compute the value of 14725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the next position. 14735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 14745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3ReadNextPos( 14755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp, /* IN/OUT: Pointer into position-list buffer */ 14765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 *pi /* IN/OUT: Value read from position-list */ 14775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 14785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (**pp)&0xFE ){ 14795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(pp, pi); 14805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pi -= 2; 14815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 14825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pi = POSITION_LIST_END; 14835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 14845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 14855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 14865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 14875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If parameter iCol is not 0, write an POS_COLUMN (1) byte followed by 14885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the value of iCol encoded as a varint to *pp. This will start a new 14895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** column list. 14905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 14915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Set *pp to point to the byte just after the last byte written before 14925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** returning (do not modify it if iCol==0). Return the total number of bytes 14935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** written (0 if iCol==0). 14945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 14955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3PutColNumber(char **pp, int iCol){ 14965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n = 0; /* Number of bytes written */ 14975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iCol ){ 14985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = *pp; /* Output pointer */ 14995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) n = 1 + sqlite3Fts3PutVarint(&p[1], iCol); 15005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *p = 0x01; 15015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp = &p[n]; 15025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 15035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return n; 15045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 15055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 15075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Compute the union of two position lists. The output written 15085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** into *pp contains all positions of both *pp1 and *pp2 in sorted 15095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** order and with any duplicates removed. All pointers are 15105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** updated appropriately. The caller is responsible for insuring 15115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** that there is enough space in *pp to hold the complete output. 15125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 15135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3PoslistMerge( 15145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp, /* Output buffer */ 15155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp1, /* Left input list */ 15165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp2 /* Right input list */ 15175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 15185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = *pp; 15195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p1 = *pp1; 15205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p2 = *pp2; 15215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( *p1 || *p2 ){ 15235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol1; /* The current column index in pp1 */ 15245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol2; /* The current column index in pp2 */ 15255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *p1==POS_COLUMN ) sqlite3Fts3GetVarint32(&p1[1], &iCol1); 15275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) else if( *p1==POS_END ) iCol1 = POSITION_LIST_END; 15285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) else iCol1 = 0; 15295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *p2==POS_COLUMN ) sqlite3Fts3GetVarint32(&p2[1], &iCol2); 15315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) else if( *p2==POS_END ) iCol2 = POSITION_LIST_END; 15325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) else iCol2 = 0; 15335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iCol1==iCol2 ){ 15355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 i1 = 0; /* Last position from pp1 */ 15365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 i2 = 0; /* Last position from pp2 */ 15375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iPrev = 0; 15385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n = fts3PutColNumber(&p, iCol1); 15395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1 += n; 15405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2 += n; 15415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* At this point, both p1 and p2 point to the start of column-lists 15435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** for the same column (the column with index iCol1 and iCol2). 15445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** A column-list is a list of non-negative delta-encoded varints, each 15455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** incremented by 2 before being stored. Each list is terminated by a 15465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** POS_END (0) or POS_COLUMN (1). The following block merges the two lists 15475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** and writes the results to buffer p. p is left pointing to the byte 15485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** after the list written. No terminator (POS_END or POS_COLUMN) is 15495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** written to the output. 15505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 15515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&p1, &i1); 15525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&p2, &i2); 15535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) do { 15545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, (i1<i2) ? i1 : i2); 15555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iPrev -= 2; 15565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( i1==i2 ){ 15575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ReadNextPos(&p1, &i1); 15585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ReadNextPos(&p2, &i2); 15595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( i1<i2 ){ 15605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ReadNextPos(&p1, &i1); 15615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 15625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ReadNextPos(&p2, &i2); 15635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 15645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }while( i1!=POSITION_LIST_END || i2!=POSITION_LIST_END ); 15655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( iCol1<iCol2 ){ 15665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1 += fts3PutColNumber(&p, iCol1); 15675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ColumnlistCopy(&p, &p1); 15685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 15695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2 += fts3PutColNumber(&p, iCol2); 15705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ColumnlistCopy(&p, &p2); 15715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 15725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 15735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *p++ = POS_END; 15755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp = p; 15765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp1 = p1 + 1; 15775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp2 = p2 + 1; 15785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 15795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 15805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 15815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** nToken==1 searches for adjacent positions. 15825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 15835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is used to merge two position lists into one. When it is 15845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** called, *pp1 and *pp2 must both point to position lists. A position-list is 15855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the part of a doclist that follows each document id. For example, if a row 15865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** contains: 15875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 15885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 'a b c'|'x y z'|'a b b a' 15895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 15905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Then the position list for this row for token 'b' would consist of: 15915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 15925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 0x02 0x01 0x02 0x03 0x03 0x00 15935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 15945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** When this function returns, both *pp1 and *pp2 are left pointing to the 15955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** byte following the 0x00 terminator of their respective position lists. 15965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 15975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If isSaveLeft is 0, an entry is added to the output position list for 15985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** each position in *pp2 for which there exists one or more positions in 15995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** *pp1 so that (pos(*pp2)>pos(*pp1) && pos(*pp2)-pos(*pp1)<=nToken). i.e. 16005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** when the *pp1 token appears before the *pp2 token, but not more than nToken 16015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** slots before it. 16025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 16035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3PoslistPhraseMerge( 16045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp, /* IN/OUT: Preallocated output buffer */ 16055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nToken, /* Maximum difference in token positions */ 16065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isSaveLeft, /* Save the left position */ 16075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isExact, /* If *pp1 is exactly nTokens before *pp2 */ 16085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp1, /* IN/OUT: Left input list */ 16095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp2 /* IN/OUT: Right input list */ 16105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 16115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = (pp ? *pp : 0); 16125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p1 = *pp1; 16135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p2 = *pp2; 16145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol1 = 0; 16155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol2 = 0; 16165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Never set both isSaveLeft and isExact for the same invocation. */ 16185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( isSaveLeft==0 || isExact==0 ); 16195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( *p1!=0 && *p2!=0 ); 16215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *p1==POS_COLUMN ){ 16225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1++; 16235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1 += sqlite3Fts3GetVarint32(p1, &iCol1); 16245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *p2==POS_COLUMN ){ 16265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2++; 16275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2 += sqlite3Fts3GetVarint32(p2, &iCol2); 16285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( 1 ){ 16315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iCol1==iCol2 ){ 16325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pSave = p; 16335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iPrev = 0; 16345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iPos1 = 0; 16355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iPos2 = 0; 16365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pp && iCol1 ){ 16385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *p++ = POS_COLUMN; 16395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p += sqlite3Fts3PutVarint(p, iCol1); 16405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( *p1!=POS_END && *p1!=POS_COLUMN ); 16435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( *p2!=POS_END && *p2!=POS_COLUMN ); 16445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&p1, &iPos1); iPos1 -= 2; 16455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&p2, &iPos2); iPos2 -= 2; 16465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( 1 ){ 16485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iPos2==iPos1+nToken 16495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || (isExact==0 && iPos2>iPos1 && iPos2<=iPos1+nToken) 16505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 16515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iSave; 16525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !pp ){ 16535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p2); 16545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p1); 16555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp1 = p1; 16565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp2 = p2; 16575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 1; 16585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iSave = isSaveLeft ? iPos1 : iPos2; 16605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, iSave+2); iPrev -= 2; 16615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pSave = 0; 16625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (!isSaveLeft && iPos2<=(iPos1+nToken)) || iPos2<=iPos1 ){ 16645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (*p2&0xFE)==0 ) break; 16655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&p2, &iPos2); iPos2 -= 2; 16665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 16675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (*p1&0xFE)==0 ) break; 16685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&p1, &iPos1); iPos1 -= 2; 16695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pSave ){ 16735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pp && p ); 16745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p = pSave; 16755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ColumnlistCopy(0, &p1); 16785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ColumnlistCopy(0, &p2); 16795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( (*p1&0xFE)==0 && (*p2&0xFE)==0 ); 16805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( 0==*p1 || 0==*p2 ) break; 16815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1++; 16835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1 += sqlite3Fts3GetVarint32(p1, &iCol1); 16845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2++; 16855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2 += sqlite3Fts3GetVarint32(p2, &iCol2); 16865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 16875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 16885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Advance pointer p1 or p2 (whichever corresponds to the smaller of 16895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** iCol1 and iCol2) so that it points to either the 0x00 that marks the 16905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** end of the position list, or the 0x01 that precedes the next 16915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** column-number in the position list. 16925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 16935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) else if( iCol1<iCol2 ){ 16945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ColumnlistCopy(0, &p1); 16955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( 0==*p1 ) break; 16965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1++; 16975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p1 += sqlite3Fts3GetVarint32(p1, &iCol1); 16985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 16995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ColumnlistCopy(0, &p2); 17005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( 0==*p2 ) break; 17015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2++; 17025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p2 += sqlite3Fts3GetVarint32(p2, &iCol2); 17035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 17045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 17055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 17065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p2); 17075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p1); 17085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp1 = p1; 17095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp2 = p2; 17105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !pp || *pp==p ){ 17115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 0; 17125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 17135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *p++ = 0x00; 17145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp = p; 17155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 1; 17165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 17175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 17185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 17195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Merge two position-lists as required by the NEAR operator. 17205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 17215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3PoslistNearMerge( 17225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp, /* Output buffer */ 17235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aTmp, /* Temporary buffer space */ 17245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nRight, /* Maximum difference in token positions */ 17255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nLeft, /* Maximum difference in token positions */ 17265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp1, /* IN/OUT: Left input list */ 17275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pp2 /* IN/OUT: Right input list */ 17285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 17295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p1 = *pp1; 17305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p2 = *pp2; 17315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 17325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !pp ){ 17335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( fts3PoslistPhraseMerge(0, nRight, 0, 0, pp1, pp2) ) return 1; 17345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp1 = p1; 17355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp2 = p2; 17365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return fts3PoslistPhraseMerge(0, nLeft, 0, 0, pp2, pp1); 17375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 17385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pTmp1 = aTmp; 17395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pTmp2; 17405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aTmp2; 17415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int res = 1; 17425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 17435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistPhraseMerge(&pTmp1, nRight, 0, 0, pp1, pp2); 17445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aTmp2 = pTmp2 = pTmp1; 17455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp1 = p1; 17465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pp2 = p2; 17475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistPhraseMerge(&pTmp2, nLeft, 1, 0, pp2, pp1); 17485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTmp1!=aTmp && pTmp2!=aTmp2 ){ 17495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistMerge(pp, &aTmp, &aTmp2); 17505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( pTmp1!=aTmp ){ 17515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(pp, &aTmp); 17525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( pTmp2!=aTmp2 ){ 17535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(pp, &aTmp2); 17545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 17555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) res = 0; 17565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 17575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 17585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return res; 17595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 17605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 17615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 17625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 17635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Values that may be used as the first parameter to fts3DoclistMerge(). 17645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 17655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_NOT 2 /* D + D -> D */ 17665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_AND 3 /* D + D -> D */ 17675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_OR 4 /* D + D -> D */ 17685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_POS_OR 5 /* P + P -> P */ 17695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_PHRASE 6 /* P + P -> D */ 17705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_POS_PHRASE 7 /* P + P -> P */ 17715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_NEAR 8 /* P + P -> D */ 17725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#define MERGE_POS_NEAR 9 /* P + P -> P */ 17735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 17745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 17755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Merge the two doclists passed in buffer a1 (size n1 bytes) and a2 17765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (size n2 bytes). The output is written to pre-allocated buffer aBuffer, 17775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** which is guaranteed to be large enough to hold the results. The number 17785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** of bytes written to aBuffer is stored in *pnBuffer before returning. 17795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 17805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If successful, SQLITE_OK is returned. Otherwise, if a malloc error 17815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** occurs while allocating a temporary buffer as part of the merge operation, 17825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** SQLITE_NOMEM is returned. 17835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 17845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3DoclistMerge( 17855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int mergetype, /* One of the MERGE_XXX constants */ 17865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nParam1, /* Used by MERGE_NEAR and MERGE_POS_NEAR */ 17875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nParam2, /* Used by MERGE_NEAR and MERGE_POS_NEAR */ 17885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aBuffer, /* Pre-allocated output buffer */ 17895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnBuffer, /* OUT: Bytes written to aBuffer */ 17905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *a1, /* Buffer containing first doclist */ 17915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n1, /* Size of buffer a1 */ 17925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *a2, /* Buffer containing second doclist */ 17935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n2, /* Size of buffer a2 */ 17945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnDoc /* OUT: Number of docids in output */ 17955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 17965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 i1 = 0; 17975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 i2 = 0; 17985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iPrev = 0; 17995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = aBuffer; 18015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p1 = a1; 18025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p2 = a2; 18035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pEnd1 = &a1[n1]; 18045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pEnd2 = &a2[n2]; 18055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nDoc = 0; 18065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( mergetype==MERGE_OR || mergetype==MERGE_POS_OR 18085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || mergetype==MERGE_AND || mergetype==MERGE_NOT 18095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || mergetype==MERGE_PHRASE || mergetype==MERGE_POS_PHRASE 18105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || mergetype==MERGE_NEAR || mergetype==MERGE_POS_NEAR 18115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 18125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aBuffer ){ 18145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnBuffer = 0; 18155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_NOMEM; 18165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Read the first docid from each doclist */ 18195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) switch( mergetype ){ 18235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case MERGE_OR: 18245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case MERGE_POS_OR: 18255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p1 || p2 ){ 18265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p2 && p1 && i1==i2 ){ 18275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, i1); 18285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( mergetype==MERGE_POS_OR ) fts3PoslistMerge(&p, &p1, &p2); 18295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( !p2 || (p1 && i1<i2) ){ 18325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, i1); 18335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( mergetype==MERGE_POS_OR ) fts3PoslistCopy(&p, &p1); 18345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 18365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, i2); 18375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( mergetype==MERGE_POS_OR ) fts3PoslistCopy(&p, &p2); 18385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 18425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case MERGE_AND: 18445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p1 && p2 ){ 18455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( i1==i2 ){ 18465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, i1); 18475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDoc++; 18505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( i1<i2 ){ 18515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 18535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 18575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case MERGE_NOT: 18595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p1 ){ 18605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p2 && i1==i2 ){ 18615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( !p2 || i1<i2 ){ 18645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, i1); 18655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 18675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 18715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case MERGE_POS_PHRASE: 18735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case MERGE_PHRASE: { 18745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **ppPos = (mergetype==MERGE_PHRASE ? 0 : &p); 18755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p1 && p2 ){ 18765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( i1==i2 ){ 18775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pSave = p; 18785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iPrevSave = iPrev; 18795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, i1); 18805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( 0==fts3PoslistPhraseMerge(ppPos, nParam1, 0, 1, &p1, &p2) ){ 18815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p = pSave; 18825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iPrev = iPrevSave; 18835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 18845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDoc++; 18855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( i1<i2 ){ 18895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p1); 18905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 18915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 18925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p2); 18935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 18945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 18975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 18985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 18995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) default: assert( mergetype==MERGE_POS_NEAR || mergetype==MERGE_NEAR ); { 19005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aTmp = 0; 19015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **ppPos = 0; 19025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( mergetype==MERGE_POS_NEAR ){ 19045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ppPos = &p; 19055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aTmp = sqlite3_malloc(2*(n1+n2+1)); 19065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aTmp ){ 19075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_NOMEM; 19085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p1 && p2 ){ 19125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( i1==i2 ){ 19135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pSave = p; 19145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iPrevSave = iPrev; 19155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PutDeltaVarint(&p, &iPrev, i1); 19165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !fts3PoslistNearMerge(ppPos, aTmp, nParam1, nParam2, &p1, &p2) ){ 19185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iPrev = iPrevSave; 19195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p = pSave; 19205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 19235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 19245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( i1<i2 ){ 19255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p1); 19265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p1, pEnd1, &i1); 19275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 19285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p2); 19295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint2(&p2, pEnd2, &i2); 19305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aTmp); 19335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 19345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pnDoc ) *pnDoc = nDoc; 19385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnBuffer = (int)(p-aBuffer); 19395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 19405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 19415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 19435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** A pointer to an instance of this structure is used as the context 19445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** argument to sqlite3Fts3SegReaderIterate() 19455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 19465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)typedef struct TermSelect TermSelect; 19475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)struct TermSelect { 19485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isReqPos; 19495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aaOutput[16]; /* Malloc'd output buffer */ 19505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int anOutput[16]; /* Size of output in bytes */ 19515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)}; 19525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 19545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Merge all doclists in the TermSelect.aaOutput[] array into a single 19555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** doclist stored in TermSelect.aaOutput[0]. If successful, delete all 19565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** other doclists (except the aaOutput[0] one) and return SQLITE_OK. 19575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 19585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If an OOM error occurs, return SQLITE_NOMEM. In this case it is 19595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the responsibility of the caller to free any doclists left in the 19605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** TermSelect.aaOutput[] array. 19615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 19625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3TermSelectMerge(TermSelect *pTS){ 19635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int mergetype = (pTS->isReqPos ? MERGE_POS_OR : MERGE_OR); 19645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aOut = 0; 19655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nOut = 0; 19665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 19675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Loop through the doclists in the aaOutput[] array. Merge them all 19695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** into a single doclist. 19705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 19715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; i<SizeofArray(pTS->aaOutput); i++){ 19725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTS->aaOutput[i] ){ 19735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aOut ){ 19745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aOut = pTS->aaOutput[i]; 19755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nOut = pTS->anOutput[i]; 19765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[i] = 0; 19775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 19785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nNew = nOut + pTS->anOutput[i]; 19795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aNew = sqlite3_malloc(nNew); 19805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aNew ){ 19815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aOut); 19825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_NOMEM; 19835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DoclistMerge(mergetype, 0, 0, 19855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aNew, &nNew, pTS->aaOutput[i], pTS->anOutput[i], aOut, nOut, 0 19865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 19875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pTS->aaOutput[i]); 19885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aOut); 19895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[i] = 0; 19905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aOut = aNew; 19915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nOut = nNew; 19925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 19955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 19965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[0] = aOut; 19975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->anOutput[0] = nOut; 19985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 19995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 20005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 20025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is used as the sqlite3Fts3SegReaderIterate() callback when 20035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** querying the full-text index for a doclist associated with a term or 20045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** term-prefix. 20055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 20065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3TermSelectCb( 20075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p, /* Virtual table object */ 20085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) void *pContext, /* Pointer to TermSelect structure */ 20095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zTerm, 20105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nTerm, 20115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aDoclist, 20125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nDoclist 20135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 20145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) TermSelect *pTS = (TermSelect *)pContext; 20155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(p); 20175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(zTerm); 20185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(nTerm); 20195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTS->aaOutput[0]==0 ){ 20215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If this is the first term selected, copy the doclist to the output 20225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** buffer using memcpy(). TODO: Add a way to transfer control of the 20235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** aDoclist buffer from the caller so as to avoid the memcpy(). 20245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 20255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[0] = sqlite3_malloc(nDoclist); 20265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->anOutput[0] = nDoclist; 20275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTS->aaOutput[0] ){ 20285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(pTS->aaOutput[0], aDoclist, nDoclist); 20295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 20305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_NOMEM; 20315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 20325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 20335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int mergetype = (pTS->isReqPos ? MERGE_POS_OR : MERGE_OR); 20345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aMerge = aDoclist; 20355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nMerge = nDoclist; 20365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iOut; 20375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(iOut=0; iOut<SizeofArray(pTS->aaOutput); iOut++){ 20395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aNew; 20405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nNew; 20415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTS->aaOutput[iOut]==0 ){ 20425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( iOut>0 ); 20435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[iOut] = aMerge; 20445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->anOutput[iOut] = nMerge; 20455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 20465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 20475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nNew = nMerge + pTS->anOutput[iOut]; 20495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aNew = sqlite3_malloc(nNew); 20505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aNew ){ 20515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( aMerge!=aDoclist ){ 20525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aMerge); 20535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 20545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_NOMEM; 20555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 20565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DoclistMerge(mergetype, 0, 0, aNew, &nNew, 20575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[iOut], pTS->anOutput[iOut], aMerge, nMerge, 0 20585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 20595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iOut>0 ) sqlite3_free(aMerge); 20615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pTS->aaOutput[iOut]); 20625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[iOut] = 0; 20635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aMerge = aNew; 20655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nMerge = nNew; 20665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (iOut+1)==SizeofArray(pTS->aaOutput) ){ 20675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->aaOutput[iOut] = aMerge; 20685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTS->anOutput[iOut] = nMerge; 20695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 20705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 20715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 20725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 20735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 20745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3DeferredTermSelect( 20765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3DeferredToken *pToken, /* Phrase token */ 20775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isTermPos, /* True to include positions */ 20785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnOut, /* OUT: Size of list */ 20795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **ppOut /* OUT: Body of list */ 20805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 20815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aSource; 20825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nSource; 20835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 20845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aSource = sqlite3Fts3DeferredDoclist(pToken, &nSource); 20855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aSource ){ 20865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnOut = 0; 20875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppOut = 0; 20885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( isTermPos ){ 20895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppOut = sqlite3_malloc(nSource); 20905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !*ppOut ) return SQLITE_NOMEM; 20915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(*ppOut, aSource, nSource); 20925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnOut = nSource; 20935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 20945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 docid; 20955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnOut = sqlite3Fts3GetVarint(aSource, &docid); 20965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppOut = sqlite3_malloc(*pnOut); 20975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !*ppOut ) return SQLITE_NOMEM; 20985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3PutVarint(*ppOut, docid); 20995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 21025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 21035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3SegReaderCursor( 21055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p, /* FTS3 table handle */ 21065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iLevel, /* Level of segments to scan */ 21075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zTerm, /* Term to query for */ 21085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nTerm, /* Size of zTerm in bytes */ 21095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isPrefix, /* True for a prefix search */ 21105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isScan, /* True to scan from zTerm to EOF */ 21115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReaderCursor *pCsr /* Cursor object to populate */ 21125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 21135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; 21145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc2; 21155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iAge = 0; 21165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_stmt *pStmt = 0; 21175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReader *pPending = 0; 21185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( iLevel==FTS3_SEGCURSOR_ALL 21205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || iLevel==FTS3_SEGCURSOR_PENDING 21215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || iLevel>=0 21225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 21235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( FTS3_SEGCURSOR_PENDING<0 ); 21245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( FTS3_SEGCURSOR_ALL<0 ); 21255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( iLevel==FTS3_SEGCURSOR_ALL || (zTerm==0 && isPrefix==1) ); 21265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( isPrefix==0 || isScan==0 ); 21275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memset(pCsr, 0, sizeof(Fts3SegReaderCursor)); 21305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If iLevel is less than 0, include a seg-reader for the pending-terms. */ 21325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( isScan==0 || fts3HashCount(&p->pendingTerms)==0 ); 21335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iLevel<0 && isScan==0 ){ 21345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3SegReaderPending(p, zTerm, nTerm, isPrefix, &pPending); 21355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK && pPending ){ 21365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nByte = (sizeof(Fts3SegReader *) * 16); 21375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->apSegment = (Fts3SegReader **)sqlite3_malloc(nByte); 21385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->apSegment==0 ){ 21395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 21405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 21415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->apSegment[0] = pPending; 21425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->nSegment = 1; 21435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pPending = 0; 21445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iLevel!=FTS3_SEGCURSOR_PENDING ){ 21495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 21505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3AllSegdirs(p, iLevel, &pStmt); 21515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( rc==SQLITE_OK && SQLITE_ROW==(rc = sqlite3_step(pStmt)) ){ 21535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Read the values returned by the SELECT into local variables. */ 21555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iStartBlock = sqlite3_column_int64(pStmt, 1); 21565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iLeavesEndBlock = sqlite3_column_int64(pStmt, 2); 21575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iEndBlock = sqlite3_column_int64(pStmt, 3); 21585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nRoot = sqlite3_column_bytes(pStmt, 4); 21595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char const *zRoot = sqlite3_column_blob(pStmt, 4); 21605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If nSegment is a multiple of 16 the array needs to be extended. */ 21625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (pCsr->nSegment%16)==0 ){ 21635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReader **apNew; 21645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nByte = (pCsr->nSegment + 16)*sizeof(Fts3SegReader*); 21655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) apNew = (Fts3SegReader **)sqlite3_realloc(pCsr->apSegment, nByte); 21665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !apNew ){ 21675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 21685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) goto finished; 21695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->apSegment = apNew; 21715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If zTerm is not NULL, and this segment is not stored entirely on its 21745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** root node, the range of leaves scanned can be reduced. Do this. */ 21755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iStartBlock && zTerm ){ 21765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 *pi = (isPrefix ? &iLeavesEndBlock : 0); 21775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3SelectLeaf(p, zTerm, nTerm, zRoot, nRoot, &iStartBlock, pi); 21785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) goto finished; 21795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( isPrefix==0 && isScan==0 ) iLeavesEndBlock = iStartBlock; 21805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3SegReaderNew(iAge, iStartBlock, iLeavesEndBlock, 21835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iEndBlock, zRoot, nRoot, &pCsr->apSegment[pCsr->nSegment] 21845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 21855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) goto finished; 21865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->nSegment++; 21875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iAge++; 21885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 21905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) finished: 21925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc2 = sqlite3_reset(pStmt); 21935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_DONE ) rc = rc2; 21945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3SegReaderFree(pPending); 21955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 21975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 21985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 21995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3TermSegReaderCursor( 22015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr, /* Virtual table cursor handle */ 22025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zTerm, /* Term to query for */ 22035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nTerm, /* Size of zTerm in bytes */ 22045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isPrefix, /* True for a prefix search */ 22055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReaderCursor **ppSegcsr /* OUT: Allocated seg-reader cursor */ 22065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 22075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReaderCursor *pSegcsr; /* Object to allocate and return */ 22085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_NOMEM; /* Return code */ 22095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pSegcsr = sqlite3_malloc(sizeof(Fts3SegReaderCursor)); 22115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pSegcsr ){ 22125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pCsr->base.pVtab; 22135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 22145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nCost = 0; 22155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3SegReaderCursor( 22165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p, FTS3_SEGCURSOR_ALL, zTerm, nTerm, isPrefix, 0, pSegcsr); 22175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; rc==SQLITE_OK && i<pSegcsr->nSegment; i++){ 22195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3SegReaderCost(pCsr, pSegcsr->apSegment[i], &nCost); 22205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 22215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pSegcsr->nCost = nCost; 22225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 22235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppSegcsr = pSegcsr; 22255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 22265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 22275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3SegReaderCursorFree(Fts3SegReaderCursor *pSegcsr){ 22295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3SegReaderFinish(pSegcsr); 22305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pSegcsr); 22315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 22325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 22345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function retreives the doclist for the specified term (or term 22355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** prefix) from the database. 22365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 22375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The returned doclist may be in one of two formats, depending on the 22385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** value of parameter isReqPos. If isReqPos is zero, then the doclist is 22395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** a sorted list of delta-compressed docids (a bare doclist). If isReqPos 22405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is non-zero, then the returned list is in the same format as is stored 22415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in the database without the found length specifier at the start of on-disk 22425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** doclists. 22435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 22445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3TermSelect( 22455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p, /* Virtual table handle */ 22465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3PhraseToken *pTok, /* Token to query for */ 22475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iColumn, /* Column to query (or -ve for all columns) */ 22485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isReqPos, /* True to include position lists in output */ 22495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnOut, /* OUT: Size of buffer at *ppOut */ 22505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **ppOut /* OUT: Malloced result buffer */ 22515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 22525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 22535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReaderCursor *pSegcsr; /* Seg-reader cursor for this term */ 22545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) TermSelect tsc; /* Context object for fts3TermSelectCb() */ 22555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegFilter filter; /* Segment term filter configuration */ 22565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pSegcsr = pTok->pSegcsr; 22585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memset(&tsc, 0, sizeof(TermSelect)); 22595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) tsc.isReqPos = isReqPos; 22605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) filter.flags = FTS3_SEGMENT_IGNORE_EMPTY 22625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) | (pTok->isPrefix ? FTS3_SEGMENT_PREFIX : 0) 22635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) | (isReqPos ? FTS3_SEGMENT_REQUIRE_POS : 0) 22645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) | (iColumn<p->nColumn ? FTS3_SEGMENT_COLUMN_FILTER : 0); 22655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) filter.iCol = iColumn; 22665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) filter.zTerm = pTok->z; 22675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) filter.nTerm = pTok->n; 22685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3SegReaderStart(p, pSegcsr, &filter); 22705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( SQLITE_OK==rc 22715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && SQLITE_ROW==(rc = sqlite3Fts3SegReaderStep(p, pSegcsr)) 22725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 22735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3TermSelectCb(p, (void *)&tsc, 22745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pSegcsr->zTerm, pSegcsr->nTerm, pSegcsr->aDoclist, pSegcsr->nDoclist 22755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 22765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 22775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 22795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3TermSelectMerge(&tsc); 22805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 22815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 22825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppOut = tsc.aaOutput[0]; 22835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnOut = tsc.anOutput[0]; 22845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 22855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 22865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; i<SizeofArray(tsc.aaOutput); i++){ 22875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(tsc.aaOutput[i]); 22885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 22895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 22905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3SegReaderCursorFree(pSegcsr); 22925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTok->pSegcsr = 0; 22935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 22945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 22955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 22965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 22975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function counts the total number of docids in the doclist stored 22985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in buffer aList[], size nList bytes. 22995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 23005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If the isPoslist argument is true, then it is assumed that the doclist 23015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** contains a position-list following each docid. Otherwise, it is assumed 23025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** that the doclist is simply a list of docids stored as delta encoded 23035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** varints. 23045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 23055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3DoclistCountDocids(int isPoslist, char *aList, int nList){ 23065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nDoc = 0; /* Return value */ 23075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( aList ){ 23085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aEnd = &aList[nList]; /* Pointer to one byte after EOF */ 23095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = aList; /* Cursor */ 23105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !isPoslist ){ 23115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* The number of docids in the list is the same as the number of 23125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** varints. In FTS3 a varint consists of a single byte with the 0x80 23135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** bit cleared and zero or more bytes with the 0x80 bit set. So to 23145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** count the varints in the buffer, just count the number of bytes 23155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** with the 0x80 bit clear. */ 23165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p<aEnd ) nDoc += (((*p++)&0x80)==0); 23175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 23185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p<aEnd ){ 23195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDoc++; 23205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( (*p++)&0x80 ); /* Skip docid varint */ 23215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p); /* Skip over position list */ 23225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 23265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return nDoc; 23275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 23285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 23295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 23305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Call sqlite3Fts3DeferToken() for each token in the expression pExpr. 23315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 23325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3DeferExpression(Fts3Cursor *pCsr, Fts3Expr *pExpr){ 23335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; 23345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr ){ 23355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3DeferExpression(pCsr, pExpr->pLeft); 23365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 23375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3DeferExpression(pCsr, pExpr->pRight); 23385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->eType==FTSQUERY_PHRASE ){ 23405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol = pExpr->pPhrase->iColumn; 23415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; 23425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; rc==SQLITE_OK && i<pExpr->pPhrase->nToken; i++){ 23435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3PhraseToken *pToken = &pExpr->pPhrase->aToken[i]; 23445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pToken->pDeferred==0 ){ 23455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3DeferToken(pCsr, pToken, iCol); 23465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 23515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 23525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 23535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 23545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function removes the position information from a doclist. When 23555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** called, buffer aList (size *pnList bytes) contains a doclist that includes 23565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** position information. This function removes the position information so 23575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** that aList contains only docids, and adjusts *pnList to reflect the new 23585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** (possibly reduced) size of the doclist. 23595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 23605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3DoclistStripPositions( 23615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aList, /* IN/OUT: Buffer containing doclist */ 23625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnList /* IN/OUT: Size of doclist in bytes */ 23635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 23645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( aList ){ 23655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aEnd = &aList[*pnList]; /* Pointer to one byte after EOF */ 23665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *p = aList; /* Input cursor */ 23675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pOut = aList; /* Output cursor */ 23685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 23695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( p<aEnd ){ 23705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 delta; 23715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p += sqlite3Fts3GetVarint(p, &delta); 23725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &p); 23735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pOut += sqlite3Fts3PutVarint(pOut, delta); 23745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 23765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnList = (int)(pOut - aList); 23775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 23785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 23795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 23805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 23815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return a DocList corresponding to the phrase *pPhrase. 23825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 23835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If this function returns SQLITE_OK, but *pnOut is set to a negative value, 23845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** then no tokens in the phrase were looked up in the full-text index. This 23855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is only possible when this function is called from within xFilter(). The 23865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** caller should assume that all documents match the phrase. The actual 23875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** filtering will take place in xNext(). 23885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 23895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3PhraseSelect( 23905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr, /* Virtual table cursor handle */ 23915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Phrase *pPhrase, /* Phrase to return a doclist for */ 23925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isReqPos, /* True if output should contain positions */ 23935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **paOut, /* OUT: Pointer to malloc'd result buffer */ 23945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnOut /* OUT: Size of buffer at *paOut */ 23955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 23965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pOut = 0; 23975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nOut = 0; 23985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; 23995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int ii; 24005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol = pPhrase->iColumn; 24015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isTermPos = (pPhrase->nToken>1 || isReqPos); 24025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pCsr->base.pVtab; 24035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isFirst = 1; 24045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iPrevTok = 0; 24065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nDoc = 0; 24075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If this is an xFilter() evaluation, create a segment-reader for each 24095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** phrase token. Or, if this is an xNext() or snippet/offsets/matchinfo 24105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** evaluation, only create segment-readers if there are no Fts3DeferredToken 24115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** objects attached to the phrase-tokens. 24125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 24135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(ii=0; ii<pPhrase->nToken; ii++){ 24145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3PhraseToken *pTok = &pPhrase->aToken[ii]; 24155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTok->pSegcsr==0 ){ 24165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( (pCsr->eEvalmode==FTS3_EVAL_FILTER) 24175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || (pCsr->eEvalmode==FTS3_EVAL_NEXT && pCsr->pDeferred==0) 24185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || (pCsr->eEvalmode==FTS3_EVAL_MATCHINFO && pTok->bFulltext) 24195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 24205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3TermSegReaderCursor( 24215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr, pTok->z, pTok->n, pTok->isPrefix, &pTok->pSegcsr 24225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 24235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) return rc; 24245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(ii=0; ii<pPhrase->nToken; ii++){ 24295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3PhraseToken *pTok; /* Token to find doclist for */ 24305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iTok = 0; /* The token being queried this iteration */ 24315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pList = 0; /* Pointer to token doclist */ 24325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nList = 0; /* Size of buffer at pList */ 24335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Select a token to process. If this is an xFilter() call, then tokens 24355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** are processed in order from least to most costly. Otherwise, tokens 24365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** are processed in the order in which they occur in the phrase. 24375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 24385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->eEvalmode==FTS3_EVAL_MATCHINFO ){ 24395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( isReqPos ); 24405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iTok = ii; 24415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTok = &pPhrase->aToken[iTok]; 24425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTok->bFulltext==0 ) continue; 24435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( pCsr->eEvalmode==FTS3_EVAL_NEXT || isReqPos ){ 24445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iTok = ii; 24455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTok = &pPhrase->aToken[iTok]; 24465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 24475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nMinCost = 0x7FFFFFFF; 24485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int jj; 24495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Find the remaining token with the lowest cost. */ 24515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(jj=0; jj<pPhrase->nToken; jj++){ 24525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReaderCursor *pSegcsr = pPhrase->aToken[jj].pSegcsr; 24535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pSegcsr && pSegcsr->nCost<nMinCost ){ 24545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iTok = jj; 24555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nMinCost = pSegcsr->nCost; 24565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTok = &pPhrase->aToken[iTok]; 24595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* This branch is taken if it is determined that loading the doclist 24615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** for the next token would require more IO than loading all documents 24625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** currently identified by doclist pOut/nOut. No further doclists will 24635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** be loaded from the full-text index for this phrase. 24645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 24655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( nMinCost>nDoc && ii>0 ){ 24665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3DeferExpression(pCsr, pCsr->pExpr); 24675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 24685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->eEvalmode==FTS3_EVAL_NEXT && pTok->pDeferred ){ 24725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3DeferredTermSelect(pTok->pDeferred, isTermPos, &nList, &pList); 24735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 24745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTok->pSegcsr ){ 24755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3TermSelect(p, pTok, iCol, isTermPos, &nList, &pList); 24765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pTok->bFulltext = 1; 24785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( rc!=SQLITE_OK || pCsr->eEvalmode || pTok->pSegcsr==0 ); 24805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) break; 24815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( isFirst ){ 24835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pOut = pList; 24845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nOut = nList; 24855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->eEvalmode==FTS3_EVAL_FILTER && pPhrase->nToken>1 ){ 24865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDoc = fts3DoclistCountDocids(1, pOut, nOut); 24875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 24885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) isFirst = 0; 24895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iPrevTok = iTok; 24905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 24915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Merge the new term list and the current output. */ 24925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aLeft, *aRight; 24935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nLeft, nRight; 24945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nDist; 24955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int mt; 24965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 24975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If this is the final token of the phrase, and positions were not 24985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** requested by the caller, use MERGE_PHRASE instead of POS_PHRASE. 24995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** This drops the position information from the output list. 25005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 25015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) mt = MERGE_POS_PHRASE; 25025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( ii==pPhrase->nToken-1 && !isReqPos ) mt = MERGE_PHRASE; 25035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( iPrevTok!=iTok ); 25055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iPrevTok<iTok ){ 25065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aLeft = pOut; 25075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nLeft = nOut; 25085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aRight = pList; 25095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nRight = nList; 25105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDist = iTok-iPrevTok; 25115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iPrevTok = iTok; 25125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 25135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aRight = pOut; 25145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nRight = nOut; 25155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aLeft = pList; 25165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nLeft = nList; 25175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDist = iPrevTok-iTok; 25185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 25195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pOut = aRight; 25205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DoclistMerge( 25215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) mt, nDist, 0, pOut, &nOut, aLeft, nLeft, aRight, nRight, &nDoc 25225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 25235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aLeft); 25245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 25255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nOut==0 || pOut!=0 ); 25265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 25275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 25295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( ii!=pPhrase->nToken ){ 25305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pCsr->eEvalmode==FTS3_EVAL_FILTER && isReqPos==0 ); 25315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DoclistStripPositions(pOut, &nOut); 25325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 25335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *paOut = pOut; 25345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnOut = nOut; 25355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 25365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pOut); 25375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 25385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 25395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 25405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 25425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function merges two doclists according to the requirements of a 25435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** NEAR operator. 25445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 25455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Both input doclists must include position information. The output doclist 25465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** includes position information if the first argument to this function 25475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is MERGE_POS_NEAR, or does not if it is MERGE_NEAR. 25485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 25495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3NearMerge( 25505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int mergetype, /* MERGE_POS_NEAR or MERGE_NEAR */ 25515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nNear, /* Parameter to NEAR operator */ 25525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nTokenLeft, /* Number of tokens in LHS phrase arg */ 25535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aLeft, /* Doclist for LHS (incl. positions) */ 25545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nLeft, /* Size of LHS doclist in bytes */ 25555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nTokenRight, /* As nTokenLeft */ 25565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aRight, /* As aLeft */ 25575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nRight, /* As nRight */ 25585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **paOut, /* OUT: Results of merge (malloced) */ 25595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnOut /* OUT: Sized of output buffer */ 25605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 25615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aOut; /* Buffer to write output doclist to */ 25625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 25635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( mergetype==MERGE_POS_NEAR || MERGE_NEAR ); 25655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aOut = sqlite3_malloc(nLeft+nRight+1); 25675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( aOut==0 ){ 25685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 25695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 25705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3DoclistMerge(mergetype, nNear+nTokenRight, nNear+nTokenLeft, 25715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aOut, pnOut, aLeft, nLeft, aRight, nRight, 0 25725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 25735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ){ 25745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aOut); 25755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aOut = 0; 25765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 25775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 25785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *paOut = aOut; 25805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 25815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 25825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 25845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is used as part of the processing for the snippet() and 25855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** offsets() functions. 25865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 25875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Both pLeft and pRight are expression nodes of type FTSQUERY_PHRASE. Both 25885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** have their respective doclists (including position information) loaded 25895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in Fts3Expr.aDoclist/nDoclist. This function removes all entries from 25905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** each doclist that are not within nNear tokens of a corresponding entry 25915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in the other doclist. 25925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 25935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3ExprNearTrim(Fts3Expr *pLeft, Fts3Expr *pRight, int nNear){ 25945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 25955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 25965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pLeft->eType==FTSQUERY_PHRASE ); 25975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pRight->eType==FTSQUERY_PHRASE ); 25985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pLeft->isLoaded && pRight->isLoaded ); 25995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pLeft->aDoclist==0 || pRight->aDoclist==0 ){ 26015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pLeft->aDoclist); 26025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pRight->aDoclist); 26035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pRight->aDoclist = 0; 26045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft->aDoclist = 0; 26055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_OK; 26065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 26075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aOut; /* Buffer in which to assemble new doclist */ 26085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nOut; /* Size of buffer aOut in bytes */ 26095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 26115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist, 26125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist, 26135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) &aOut, &nOut 26145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 26155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) return rc; 26165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pRight->aDoclist); 26175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pRight->aDoclist = aOut; 26185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pRight->nDoclist = nOut; 26195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 26215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist, 26225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist, 26235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) &aOut, &nOut 26245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 26255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pLeft->aDoclist); 26265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft->aDoclist = aOut; 26275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft->nDoclist = nOut; 26285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 26305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 26315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 26345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Allocate an Fts3SegReaderArray for each token in the expression pExpr. 26355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The allocated objects are stored in the Fts3PhraseToken.pArray member 26365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** variables of each token structure. 26375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 26385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3ExprAllocateSegReaders( 26395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr, /* FTS3 table */ 26405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pExpr, /* Expression to create seg-readers for */ 26415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnExpr /* OUT: Number of AND'd expressions */ 26425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 26435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; /* Return code */ 26445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pCsr->eEvalmode==FTS3_EVAL_FILTER ); 26465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pnExpr && pExpr->eType!=FTSQUERY_AND ){ 26475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) (*pnExpr)++; 26485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pnExpr = 0; 26495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->eType==FTSQUERY_PHRASE ){ 26525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Phrase *pPhrase = pExpr->pPhrase; 26535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int ii; 26545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(ii=0; rc==SQLITE_OK && ii<pPhrase->nToken; ii++){ 26565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3PhraseToken *pTok = &pPhrase->aToken[ii]; 26575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pTok->pSegcsr==0 ){ 26585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3TermSegReaderCursor( 26595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr, pTok->z, pTok->n, pTok->isPrefix, &pTok->pSegcsr 26605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 26615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 26645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3ExprAllocateSegReaders(pCsr, pExpr->pLeft, pnExpr); 26655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 26665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3ExprAllocateSegReaders(pCsr, pExpr->pRight, pnExpr); 26675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 26705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 26715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 26735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Free the Fts3SegReaderArray objects associated with each token in the 26745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** expression pExpr. In other words, this function frees the resources 26755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** allocated by fts3ExprAllocateSegReaders(). 26765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 26775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3ExprFreeSegReaders(Fts3Expr *pExpr){ 26785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr ){ 26795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Phrase *pPhrase = pExpr->pPhrase; 26805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pPhrase ){ 26815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int kk; 26825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(kk=0; kk<pPhrase->nToken; kk++){ 26835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3SegReaderCursorFree(pPhrase->aToken[kk].pSegcsr); 26845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pPhrase->aToken[kk].pSegcsr = 0; 26855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ExprFreeSegReaders(pExpr->pLeft); 26885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ExprFreeSegReaders(pExpr->pRight); 26895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 26905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 26915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 26925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 26935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return the sum of the costs of all tokens in the expression pExpr. This 26945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** function must be called after Fts3SegReaderArrays have been allocated 26955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** for all tokens using fts3ExprAllocateSegReaders(). 26965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 26975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3ExprCost(Fts3Expr *pExpr){ 26985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nCost; /* Return value */ 26995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->eType==FTSQUERY_PHRASE ){ 27005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Phrase *pPhrase = pExpr->pPhrase; 27015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int ii; 27025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nCost = 0; 27035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(ii=0; ii<pPhrase->nToken; ii++){ 27045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3SegReaderCursor *pSegcsr = pPhrase->aToken[ii].pSegcsr; 27055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pSegcsr ) nCost += pSegcsr->nCost; 27065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 27075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 27085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nCost = fts3ExprCost(pExpr->pLeft) + fts3ExprCost(pExpr->pRight); 27095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 27105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return nCost; 27115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 27125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 27135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 27145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The following is a helper function (and type) for fts3EvalExpr(). It 27155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** must be called after Fts3SegReaders have been allocated for every token 27165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in the expression. See the context it is called from in fts3EvalExpr() 27175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** for further explanation. 27185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 27195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)typedef struct ExprAndCost ExprAndCost; 27205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)struct ExprAndCost { 27215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pExpr; 27225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nCost; 27235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)}; 27245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3ExprAssignCosts( 27255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pExpr, /* Expression to create seg-readers for */ 27265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ExprAndCost **ppExprCost /* OUT: Write to *ppExprCost */ 27275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 27285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->eType==FTSQUERY_AND ){ 27295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ExprAssignCosts(pExpr->pLeft, ppExprCost); 27305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ExprAssignCosts(pExpr->pRight, ppExprCost); 27315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 27325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) (*ppExprCost)->pExpr = pExpr; 27335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) (*ppExprCost)->nCost = fts3ExprCost(pExpr); 27345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) (*ppExprCost)++; 27355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 27365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 27375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 27385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 27395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Evaluate the full-text expression pExpr against FTS3 table pTab. Store 27405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the resulting doclist in *paOut and *pnOut. This routine mallocs for 27415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the space needed to store the output. The caller is responsible for 27425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** freeing the space when it has finished. 27435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 27445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is called in two distinct contexts: 27455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 27465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** * From within the virtual table xFilter() method. In this case, the 27475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** output doclist contains entries for all rows in the table, based on 27485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** data read from the full-text index. 27495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 27505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** In this case, if the query expression contains one or more tokens that 27515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** are very common, then the returned doclist may contain a superset of 27525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the documents that actually match the expression. 27535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 27545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** * From within the virtual table xNext() method. This call is only made 27555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** if the call from within xFilter() found that there were very common 27565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** tokens in the query expression and did return a superset of the 27575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** matching documents. In this case the returned doclist contains only 27585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** entries that correspond to the current row of the table. Instead of 27595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** reading the data for each token from the full-text index, the data is 27605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** already available in-memory in the Fts3PhraseToken.pDeferred structures. 27615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** See fts3EvalDeferred() for how it gets there. 27625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 27635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** In the first case above, Fts3Cursor.doDeferred==0. In the second (if it is 27645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** required) Fts3Cursor.doDeferred==1. 27655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 27665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If the SQLite invokes the snippet(), offsets() or matchinfo() function 27675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** as part of a SELECT on an FTS3 table, this function is called on each 27685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** individual phrase expression in the query. If there were very common tokens 27695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** found in the xFilter() call, then this function is called once for phrase 27705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** for each row visited, and the returned doclist contains entries for the 27715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** current row only. Otherwise, if there were no very common tokens, then this 27725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** function is called once only for each phrase in the query and the returned 27735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** doclist contains entries for all rows of the table. 27745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 27755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Fts3Cursor.doDeferred==1 when this function is called on phrases as a 27765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** result of a snippet(), offsets() or matchinfo() invocation. 27775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 27785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3EvalExpr( 27795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *p, /* Virtual table cursor handle */ 27805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pExpr, /* Parsed fts3 expression */ 27815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **paOut, /* OUT: Pointer to malloc'd result buffer */ 27825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnOut, /* OUT: Size of buffer at *paOut */ 27835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int isReqPos /* Require positions in output buffer */ 27845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 27855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; /* Return code */ 27865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 27875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Zero the output parameters. */ 27885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *paOut = 0; 27895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnOut = 0; 27905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 27915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr ){ 27925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pExpr->eType==FTSQUERY_NEAR || pExpr->eType==FTSQUERY_OR 27935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || pExpr->eType==FTSQUERY_AND || pExpr->eType==FTSQUERY_NOT 27945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || pExpr->eType==FTSQUERY_PHRASE 27955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 27965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pExpr->eType==FTSQUERY_PHRASE || isReqPos==0 ); 27975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 27985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->eType==FTSQUERY_PHRASE ){ 27995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3PhraseSelect(p, pExpr->pPhrase, 28005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) isReqPos || (pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR), 28015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) paOut, pnOut 28025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 28035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ExprFreeSegReaders(pExpr); 28045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( p->eEvalmode==FTS3_EVAL_FILTER && pExpr->eType==FTSQUERY_AND ){ 28055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ExprAndCost *aExpr = 0; /* Array of AND'd expressions and costs */ 28065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nExpr = 0; /* Size of aExpr[] */ 28075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aRet = 0; /* Doclist to return to caller */ 28085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nRet = 0; /* Length of aRet[] in bytes */ 28095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nDoc = 0x7FFFFFFF; 28105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( !isReqPos ); 28125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3ExprAllocateSegReaders(p, pExpr, &nExpr); 28145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 28155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nExpr>1 ); 28165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aExpr = sqlite3_malloc(sizeof(ExprAndCost) * nExpr); 28175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !aExpr ) rc = SQLITE_NOMEM; 28185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 28205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int ii; /* Used to iterate through expressions */ 28215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ExprAssignCosts(pExpr, &aExpr); 28235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aExpr -= nExpr; 28245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(ii=0; ii<nExpr; ii++){ 28255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aNew; 28265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nNew; 28275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int jj; 28285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ExprAndCost *pBest = 0; 28295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(jj=0; jj<nExpr; jj++){ 28315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ExprAndCost *pCand = &aExpr[jj]; 28325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCand->pExpr && (pBest==0 || pCand->nCost<pBest->nCost) ){ 28335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pBest = pCand; 28345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pBest->nCost>nDoc ){ 28385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3DeferExpression(p, p->pExpr); 28395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 28405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 28415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3EvalExpr(p, pBest->pExpr, &aNew, &nNew, 0); 28425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) break; 28435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pBest->pExpr = 0; 28445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( ii==0 ){ 28455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aRet = aNew; 28465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nRet = nNew; 28475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) nDoc = fts3DoclistCountDocids(0, aRet, nRet); 28485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 28495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DoclistMerge( 28505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) MERGE_AND, 0, 0, aRet, &nRet, aRet, nRet, aNew, nNew, &nDoc 28515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 28525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aNew); 28535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 28595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *paOut = aRet; 28605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pnOut = nRet; 28615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 28625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( *paOut==0 ); 28635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aRet); 28645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aExpr); 28665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ExprFreeSegReaders(pExpr); 28675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 28695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aLeft; 28705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aRight; 28715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nLeft; 28725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nRight; 28735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pExpr->eType==FTSQUERY_NEAR 28755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || pExpr->eType==FTSQUERY_OR 28765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || pExpr->eType==FTSQUERY_NOT 28775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || (pExpr->eType==FTSQUERY_AND && p->eEvalmode==FTS3_EVAL_NEXT) 28785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 28795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( 0==(rc = fts3EvalExpr(p, pExpr->pRight, &aRight, &nRight, isReqPos)) 28815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && 0==(rc = fts3EvalExpr(p, pExpr->pLeft, &aLeft, &nLeft, isReqPos)) 28825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 28835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) switch( pExpr->eType ){ 28845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case FTSQUERY_NEAR: { 28855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pLeft; 28865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pRight; 28875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int mergetype = MERGE_NEAR; 28885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){ 28895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) mergetype = MERGE_POS_NEAR; 28905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft = pExpr->pLeft; 28925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( pLeft->eType==FTSQUERY_NEAR ){ 28935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft=pLeft->pRight; 28945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 28955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pRight = pExpr->pRight; 28965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pRight->eType==FTSQUERY_PHRASE ); 28975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pLeft->eType==FTSQUERY_PHRASE ); 28985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 28995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3NearMerge(mergetype, pExpr->nNear, 29005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pLeft->pPhrase->nToken, aLeft, nLeft, 29015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pRight->pPhrase->nToken, aRight, nRight, 29025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) paOut, pnOut 29035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 29045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aLeft); 29055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 29065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 29085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case FTSQUERY_OR: { 29095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Allocate a buffer for the output. The maximum size is the 29105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** sum of the sizes of the two input buffers. The +1 term is 29115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** so that a buffer of zero bytes is never allocated - this can 29125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** cause fts3DoclistMerge() to incorrectly return SQLITE_NOMEM. 29135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 29145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *aBuffer = sqlite3_malloc(nRight+nLeft+1); 29155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3DoclistMerge(MERGE_OR, 0, 0, aBuffer, pnOut, 29165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aLeft, nLeft, aRight, nRight, 0 29175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 29185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *paOut = aBuffer; 29195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aLeft); 29205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 29215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 29235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) default: { 29245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( FTSQUERY_NOT==MERGE_NOT && FTSQUERY_AND==MERGE_AND ); 29255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DoclistMerge(pExpr->eType, 0, 0, aLeft, pnOut, 29265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) aLeft, nLeft, aRight, nRight, 0 29275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 29285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *paOut = aLeft; 29295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 29305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(aRight); 29345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 29375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( rc==SQLITE_OK || *paOut==0 ); 29385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 29395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 29405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 29415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 29425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is called from within xNext() for each row visited by 29435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** an FTS3 query. If evaluating the FTS3 query expression within xFilter() 29445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** was able to determine the exact set of matching rows, this function sets 29455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** *pbRes to true and returns SQLITE_IO immediately. 29465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 29475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Otherwise, if evaluating the query expression within xFilter() returned a 29485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** superset of the matching documents instead of an exact set (this happens 29495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** when the query includes very common tokens and it is deemed too expensive to 29505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** load their doclists from disk), this function tests if the current row 29515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** really does match the FTS3 query. 29525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 29535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If an error occurs, an SQLite error code is returned. Otherwise, SQLITE_OK 29545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** is returned and *pbRes is set to true if the current row matches the 29555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** FTS3 query (and should be included in the results returned to SQLite), or 29565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** false otherwise. 29575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 29585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3EvalDeferred( 29595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr, /* FTS3 cursor pointing at row to test */ 29605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pbRes /* OUT: Set to true if row is a match */ 29615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 29625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; 29635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->pDeferred==0 ){ 29645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pbRes = 1; 29655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 29665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3CursorSeek(0, pCsr); 29675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 29685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3FreeDeferredDoclists(pCsr); 29695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3CacheDeferredDoclists(pCsr); 29705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 29725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *a = 0; 29735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int n = 0; 29745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3EvalExpr(pCsr, pCsr->pExpr, &a, &n, 0); 29755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( n>=0 ); 29765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pbRes = (n>0); 29775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(a); 29785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 29805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 29815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 29825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 29835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 29845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Advance the cursor to the next row in the %_content table that 29855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** matches the search criteria. For a MATCH search, this will be 29865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the next row that matches. For a full-table scan, this will be 29875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** simply the next row in the %_content table. For a docid lookup, 29885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** this routine simply sets the EOF flag. 29895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 29905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Return SQLITE_OK if nothing goes wrong. SQLITE_OK is returned 29915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** even if we reach end-of-file. The fts3EofMethod() will be called 29925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** subsequently to determine whether or not an EOF was hit. 29935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 29945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3NextMethod(sqlite3_vtab_cursor *pCursor){ 29955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int res; 29965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; /* Return code */ 29975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; 29985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 29995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->eEvalmode = FTS3_EVAL_NEXT; 30005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) do { 30015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->aDoclist==0 ){ 30025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( SQLITE_ROW!=sqlite3_step(pCsr->pStmt) ){ 30035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->isEof = 1; 30045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_reset(pCsr->pStmt); 30055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 30065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 30075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->iPrevId = sqlite3_column_int64(pCsr->pStmt, 0); 30085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 30095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->pNextId>=&pCsr->aDoclist[pCsr->nDoclist] ){ 30105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->isEof = 1; 30115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 30125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 30135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_reset(pCsr->pStmt); 30145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&pCsr->pNextId, &pCsr->iPrevId); 30155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->isRequireSeek = 1; 30165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->isMatchinfoNeeded = 1; 30175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 30185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }while( SQLITE_OK==(rc = fts3EvalDeferred(pCsr, &res)) && res==0 ); 30195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 30215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 30225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 30245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This is the xFilter interface for the virtual table. See 30255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the virtual table xFilter method documentation for additional 30265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** information. 30275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 30285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If idxNum==FTS3_FULLSCAN_SEARCH then do a full table scan against 30295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the %_content table. 30305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 30315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If idxNum==FTS3_DOCID_SEARCH then do a docid lookup for a single entry 30325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** in the %_content table. 30335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 30345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If idxNum>=FTS3_FULLTEXT_SEARCH then use the full text index. The 30355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** column on the left-hand side of the MATCH operator is column 30365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** number idxNum-FTS3_FULLTEXT_SEARCH, 0 indexed. argv[0] is the right-hand 30375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** side of the MATCH operator. 30385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 30395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3FilterMethod( 30405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */ 30415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int idxNum, /* Strategy index */ 30425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *idxStr, /* Unused */ 30435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nVal, /* Number of elements in apVal */ 30445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_value **apVal /* Arguments for the indexing scheme */ 30455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 30465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *azSql[] = { 30475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "SELECT %s FROM %Q.'%q_content' AS x WHERE docid = ?", /* non-full-scan */ 30485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "SELECT %s FROM %Q.'%q_content' AS x ", /* full-scan */ 30495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }; 30505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 30515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zSql; /* SQL statement used to access %_content */ 30525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pCursor->pVtab; 30535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; 30545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(idxStr); 30565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(nVal); 30575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( idxNum>=0 && idxNum<=(FTS3_FULLTEXT_SEARCH+p->nColumn) ); 30595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nVal==0 || nVal==1 ); 30605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( (nVal==0)==(idxNum==FTS3_FULLSCAN_SEARCH) ); 30615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( p->pSegments==0 ); 30625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* In case the cursor has been used before, clear it now. */ 30645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_finalize(pCsr->pStmt); 30655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pCsr->aDoclist); 30665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3ExprFree(pCsr->pExpr); 30675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memset(&pCursor[1], 0, sizeof(Fts3Cursor)-sizeof(sqlite3_vtab_cursor)); 30685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( idxNum!=FTS3_DOCID_SEARCH && idxNum!=FTS3_FULLSCAN_SEARCH ){ 30705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol = idxNum-FTS3_FULLTEXT_SEARCH; 30715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zQuery = (const char *)sqlite3_value_text(apVal[0]); 30725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( zQuery==0 && sqlite3_value_type(apVal[0])!=SQLITE_NULL ){ 30745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_NOMEM; 30755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 30765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3ExprParse(p->pTokenizer, p->azColumn, p->nColumn, 30785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) iCol, zQuery, -1, &pCsr->pExpr 30795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 30805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ){ 30815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_ERROR ){ 30825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->base.zErrMsg = sqlite3_mprintf("malformed MATCH expression: [%s]", 30835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zQuery); 30845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 30855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 30865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 30875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3ReadLock(p); 30895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) return rc; 30905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3EvalExpr(pCsr, pCsr->pExpr, &pCsr->aDoclist, &pCsr->nDoclist, 0); 30925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3SegmentsClose(p); 30935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) return rc; 30945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->pNextId = pCsr->aDoclist; 30955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->iPrevId = 0; 30965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 30975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 30985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Compile a SELECT statement for this cursor. For a full-table-scan, the 30995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** statement loops through all rows of the %_content table. For a 31005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** full-text query or docid lookup, the statement retrieves a single 31015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** row by docid. 31025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 31035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zSql = (char *)azSql[idxNum==FTS3_FULLSCAN_SEARCH]; 31045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zSql = sqlite3_mprintf(zSql, p->zReadExprlist, p->zDb, p->zName); 31055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zSql ){ 31065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 31075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 31085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_prepare_v2(p->db, zSql, -1, &pCsr->pStmt, 0); 31095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zSql); 31105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 31115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK && idxNum==FTS3_DOCID_SEARCH ){ 31125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_bind_value(pCsr->pStmt, 1, apVal[0]); 31135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 31145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->eSearch = (i16)idxNum; 31155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 31165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) return rc; 31175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return fts3NextMethod(pCursor); 31185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 31195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 31205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 31215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This is the xEof method of the virtual table. SQLite calls this 31225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** routine to find out if it has reached the end of a result set. 31235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 31245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3EofMethod(sqlite3_vtab_cursor *pCursor){ 31255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return ((Fts3Cursor *)pCursor)->isEof; 31265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 31275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 31285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 31295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This is the xRowid method. The SQLite core calls this routine to 31305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** retrieve the rowid for the current row of the result set. fts3 31315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** exposes %_content.docid as the rowid for the virtual table. The 31325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** rowid should be written to *pRowid. 31335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 31345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3RowidMethod(sqlite3_vtab_cursor *pCursor, sqlite_int64 *pRowid){ 31355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr = (Fts3Cursor *) pCursor; 31365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr->aDoclist ){ 31375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pRowid = pCsr->iPrevId; 31385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 31395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* This branch runs if the query is implemented using a full-table scan 31405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** (not using the full-text index). In this case grab the rowid from the 31415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** SELECT statement. 31425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 31435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pCsr->isRequireSeek==0 ); 31445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pRowid = sqlite3_column_int64(pCsr->pStmt, 0); 31455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 31465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 31475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 31485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 31495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 31505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This is the xColumn method, called by SQLite to request a value from 31515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the row that the supplied cursor currently points to. 31525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 31535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3ColumnMethod( 31545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */ 31555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_context *pContext, /* Context for sqlite3_result_xxx() calls */ 31565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol /* Index of column to read value from */ 31575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 31585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return Code */ 31595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr = (Fts3Cursor *) pCursor; 31605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pCursor->pVtab; 31615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 31625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* The column value supplied by SQLite must be in range. */ 31635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( iCol>=0 && iCol<=p->nColumn+1 ); 31645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 31655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iCol==p->nColumn+1 ){ 31665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* This call is a request for the "docid" column. Since "docid" is an 31675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** alias for "rowid", use the xRowid() method to obtain the value. 31685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 31695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iRowid; 31705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3RowidMethod(pCursor, &iRowid); 31715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_int64(pContext, iRowid); 31725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( iCol==p->nColumn ){ 31735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* The extra column whose name is the same as the table. 31745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** Return a blob which is a pointer to the cursor. 31755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 31765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_blob(pContext, &pCsr, sizeof(pCsr), SQLITE_TRANSIENT); 31775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_OK; 31785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 31795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3CursorSeek(0, pCsr); 31805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 31815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_value(pContext, sqlite3_column_value(pCsr->pStmt, iCol+1)); 31825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 31835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 31845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 31855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 31865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 31875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 31885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is the implementation of the xUpdate callback used by 31895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** FTS3 virtual tables. It is invoked by SQLite each time a row is to be 31905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** inserted, updated or deleted. 31915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 31925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3UpdateMethod( 31935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab *pVtab, /* Virtual table handle */ 31945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nArg, /* Size of argument array */ 31955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_value **apVal, /* Array of arguments */ 31965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite_int64 *pRowid /* OUT: The affected (or effected) rowid */ 31975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 31985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return sqlite3Fts3UpdateMethod(pVtab, nArg, apVal, pRowid); 31995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 32005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 32025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of xSync() method. Flush the contents of the pending-terms 32035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** hash-table to the database. 32045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 32055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3SyncMethod(sqlite3_vtab *pVtab){ 32065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = sqlite3Fts3PendingTermsFlush((Fts3Table *)pVtab); 32075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3SegmentsClose((Fts3Table *)pVtab); 32085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 32095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 32105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 32125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of xBegin() method. This is a no-op. 32135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 32145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3BeginMethod(sqlite3_vtab *pVtab){ 32155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(pVtab); 32165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( ((Fts3Table *)pVtab)->nPendingData==0 ); 32175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 32185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 32195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 32215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of xCommit() method. This is a no-op. The contents of 32225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** the pending-terms hash-table have already been flushed into the database 32235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** by fts3SyncMethod(). 32245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 32255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3CommitMethod(sqlite3_vtab *pVtab){ 32265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(pVtab); 32275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( ((Fts3Table *)pVtab)->nPendingData==0 ); 32285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 32295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 32305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 32325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of xRollback(). Discard the contents of the pending-terms 32335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** hash-table. Any changes made to the database are reverted by SQLite. 32345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 32355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3RollbackMethod(sqlite3_vtab *pVtab){ 32365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3PendingTermsClear((Fts3Table *)pVtab); 32375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 32385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 32395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 32415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Load the doclist associated with expression pExpr to pExpr->aDoclist. 32425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The loaded doclist contains positions as well as the document ids. 32435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This is used by the matchinfo(), snippet() and offsets() auxillary 32445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** functions. 32455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 32465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3ExprLoadDoclist(Fts3Cursor *pCsr, Fts3Expr *pExpr){ 32475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; 32485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pExpr->eType==FTSQUERY_PHRASE && pExpr->pPhrase ); 32495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pCsr->eEvalmode==FTS3_EVAL_NEXT ); 32505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3EvalExpr(pCsr, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1); 32515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 32525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 32535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3ExprLoadFtDoclist( 32555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr, 32565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pExpr, 32575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **paDoclist, 32585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int *pnDoclist 32595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 32605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; 32615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pCsr->eEvalmode==FTS3_EVAL_NEXT ); 32625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pExpr->eType==FTSQUERY_PHRASE && pExpr->pPhrase ); 32635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->eEvalmode = FTS3_EVAL_MATCHINFO; 32645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = fts3EvalExpr(pCsr, pExpr, paDoclist, pnDoclist, 1); 32655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr->eEvalmode = FTS3_EVAL_NEXT; 32665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 32675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 32685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 32705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** After ExprLoadDoclist() (see above) has been called, this function is 32715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** used to iterate/search through the position lists that make up the doclist 32725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** stored in pExpr->aDoclist. 32735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 32745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)char *sqlite3Fts3FindPositions( 32755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Expr *pExpr, /* Access this expressions doclist */ 32765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_int64 iDocid, /* Docid associated with requested pos-list */ 32775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol /* Column of requested pos-list */ 32785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 32795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pExpr->isLoaded ); 32805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->aDoclist ){ 32815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pEnd = &pExpr->aDoclist[pExpr->nDoclist]; 32825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *pCsr; 32835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->pCurrent==0 ){ 32855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pExpr->pCurrent = pExpr->aDoclist; 32865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pExpr->iCurrent = 0; 32875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pExpr->pCurrent += sqlite3Fts3GetVarint(pExpr->pCurrent,&pExpr->iCurrent); 32885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 32895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr = pExpr->pCurrent; 32905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pCsr ); 32915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 32925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( pCsr<pEnd ){ 32935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->iCurrent<iDocid ){ 32945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3PoslistCopy(0, &pCsr); 32955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pCsr<pEnd ){ 32965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3GetDeltaVarint(&pCsr, &pExpr->iCurrent); 32975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 32985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pExpr->pCurrent = pCsr; 32995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 33005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pExpr->iCurrent==iDocid ){ 33015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iThis = 0; 33025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iCol<0 ){ 33035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* If iCol is negative, return a pointer to the start of the 33045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** position-list (instead of a pointer to the start of a list 33055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** of offsets associated with a specific column). 33065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 33075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return pCsr; 33085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) while( iThis<iCol ){ 33105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3ColumnlistCopy(0, &pCsr); 33115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( *pCsr==0x00 ) return 0; 33125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr++; 33135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pCsr += sqlite3Fts3GetVarint32(pCsr, &iThis); 33145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( iCol==iThis && (*pCsr&0xFE) ) return pCsr; 33165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 0; 33185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 33225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 0; 33235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 33245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 33255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 33265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Helper function used by the implementation of the overloaded snippet(), 33275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** offsets() and optimize() SQL functions. 33285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 33295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** If the value passed as the third argument is a blob of size 33305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** sizeof(Fts3Cursor*), then the blob contents are copied to the 33315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** output variable *ppCsr and SQLITE_OK is returned. Otherwise, an error 33325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** message is written to context pContext and SQLITE_ERROR returned. The 33335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** string passed via zFunc is used as part of the error message. 33345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 33355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3FunctionArg( 33365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_context *pContext, /* SQL function call context */ 33375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zFunc, /* Function name */ 33385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_value *pVal, /* argv[0] passed to function */ 33395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor **ppCsr /* OUT: Store cursor handle here */ 33405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 33415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pRet; 33425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( sqlite3_value_type(pVal)!=SQLITE_BLOB 33435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || sqlite3_value_bytes(pVal)!=sizeof(Fts3Cursor *) 33445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 33455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char *zErr = sqlite3_mprintf("illegal first argument to %s", zFunc); 33465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_error(pContext, zErr, -1); 33475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(zErr); 33485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_ERROR; 33495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) memcpy(&pRet, sqlite3_value_blob(pVal), sizeof(Fts3Cursor *)); 33515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *ppCsr = pRet; 33525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return SQLITE_OK; 33535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 33545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 33555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 33565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of the snippet() function for FTS3 33575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 33585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3SnippetFunc( 33595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_context *pContext, /* SQLite function call context */ 33605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nVal, /* Size of apVal[] array */ 33615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_value **apVal /* Array of arguments */ 33625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 33635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr; /* Cursor handle passed through apVal[0] */ 33645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zStart = "<b>"; 33655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zEnd = "</b>"; 33665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zEllipsis = "<b>...</b>"; 33675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int iCol = -1; 33685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nToken = 15; /* Default number of tokens in snippet */ 33695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 33705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* There must be at least one argument passed to this function (otherwise 33715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** the non-overloaded version would have been called instead of this one). 33725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 33735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nVal>=1 ); 33745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 33755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( nVal>6 ){ 33765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_error(pContext, 33775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "wrong number of arguments to function snippet()", -1); 33785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return; 33795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( fts3FunctionArg(pContext, "snippet", apVal[0], &pCsr) ) return; 33815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 33825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) switch( nVal ){ 33835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case 6: nToken = sqlite3_value_int(apVal[5]); 33845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case 5: iCol = sqlite3_value_int(apVal[4]); 33855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]); 33865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]); 33875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case 2: zStart = (const char*)sqlite3_value_text(apVal[1]); 33885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !zEllipsis || !zEnd || !zStart ){ 33905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_error_nomem(pContext); 33915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){ 33925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3Snippet(pContext, pCsr, zStart, zEnd, zEllipsis, iCol, nToken); 33935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 33945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 33955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 33965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 33975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of the offsets() function for FTS3 33985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 33995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3OffsetsFunc( 34005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_context *pContext, /* SQLite function call context */ 34015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nVal, /* Size of argument array */ 34025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_value **apVal /* Array of arguments */ 34035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 34045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr; /* Cursor handle passed through apVal[0] */ 34055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(nVal); 34075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nVal==1 ); 34095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( fts3FunctionArg(pContext, "offsets", apVal[0], &pCsr) ) return; 34105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( pCsr ); 34115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){ 34125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3Offsets(pContext, pCsr); 34135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 34145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 34155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 34175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of the special optimize() function for FTS3. This 34185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** function merges all segments in the database to a single segment. 34195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Example usage is: 34205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 34215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** SELECT optimize(t) FROM t LIMIT 1; 34225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 34235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** where 't' is the name of an FTS3 table. 34245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 34255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3OptimizeFunc( 34265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_context *pContext, /* SQLite function call context */ 34275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nVal, /* Size of argument array */ 34285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_value **apVal /* Array of arguments */ 34295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 34305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return code */ 34315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p; /* Virtual table handle */ 34325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCursor; /* Cursor handle passed through apVal[0] */ 34335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(nVal); 34355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nVal==1 ); 34375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( fts3FunctionArg(pContext, "optimize", apVal[0], &pCursor) ) return; 34385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p = (Fts3Table *)pCursor->base.pVtab; 34395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( p ); 34405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3Optimize(p); 34425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) switch( rc ){ 34445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case SQLITE_OK: 34455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_text(pContext, "Index optimized", -1, SQLITE_STATIC); 34465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 34475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) case SQLITE_DONE: 34485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_text(pContext, "Index already optimal", -1, SQLITE_STATIC); 34495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 34505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) default: 34515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_result_error_code(pContext, rc); 34525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) break; 34535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 34545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 34555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 34575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of the matchinfo() function for FTS3 34585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 34595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void fts3MatchinfoFunc( 34605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_context *pContext, /* SQLite function call context */ 34615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nVal, /* Size of argument array */ 34625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_value **apVal /* Array of arguments */ 34635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 34645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Cursor *pCsr; /* Cursor handle passed through apVal[0] */ 34655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( nVal==1 || nVal==2 ); 34665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( SQLITE_OK==fts3FunctionArg(pContext, "matchinfo", apVal[0], &pCsr) ){ 34675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zArg = 0; 34685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( nVal>1 ){ 34695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) zArg = (const char *)sqlite3_value_text(apVal[1]); 34705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 34715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3Matchinfo(pContext, pCsr, zArg); 34725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 34735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 34745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 34765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This routine implements the xFindFunction method for the FTS3 34775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** virtual table. 34785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 34795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3FindFunctionMethod( 34805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab *pVtab, /* Virtual table handle */ 34815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int nArg, /* Number of SQL function arguments */ 34825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zName, /* Name of SQL function */ 34835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) void (**pxFunc)(sqlite3_context*,int,sqlite3_value**), /* OUT: Result */ 34845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) void **ppArg /* Unused */ 34855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 34865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) struct Overloaded { 34875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zName; 34885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) void (*xFunc)(sqlite3_context*,int,sqlite3_value**); 34895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } aOverload[] = { 34905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) { "snippet", fts3SnippetFunc }, 34915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) { "offsets", fts3OffsetsFunc }, 34925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) { "optimize", fts3OptimizeFunc }, 34935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) { "matchinfo", fts3MatchinfoFunc }, 34945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }; 34955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int i; /* Iterator variable */ 34965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 34975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(pVtab); 34985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(nArg); 34995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) UNUSED_PARAMETER(ppArg); 35005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) for(i=0; i<SizeofArray(aOverload); i++){ 35025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( strcmp(zName, aOverload[i].zName)==0 ){ 35035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) *pxFunc = aOverload[i].xFunc; 35045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 1; 35055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 35065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 35075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* No function of the specified name was found. Return 0. */ 35095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return 0; 35105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 35115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 35135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Implementation of FTS3 xRename method. Rename an fts3 table. 35145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 35155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static int fts3RenameMethod( 35165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_vtab *pVtab, /* Virtual table handle */ 35175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const char *zName /* New name of table */ 35185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 35195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Table *p = (Fts3Table *)pVtab; 35205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db = p->db; /* Database connection */ 35215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc; /* Return Code */ 35225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3PendingTermsFlush(p); 35245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ){ 35255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 35265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 35275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 35295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "ALTER TABLE %Q.'%q_content' RENAME TO '%q_content';", 35305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName, zName 35315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 35325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p->bHasDocsize ){ 35335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 35345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "ALTER TABLE %Q.'%q_docsize' RENAME TO '%q_docsize';", 35355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName, zName 35365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 35375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 35385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( p->bHasStat ){ 35395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 35405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "ALTER TABLE %Q.'%q_stat' RENAME TO '%q_stat';", 35415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName, zName 35425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 35435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 35445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 35455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "ALTER TABLE %Q.'%q_segments' RENAME TO '%q_segments';", 35465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName, zName 35475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 35485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fts3DbExec(&rc, db, 35495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) "ALTER TABLE %Q.'%q_segdir' RENAME TO '%q_segdir';", 35505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) p->zDb, p->zName, zName 35515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 35525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 35535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 35545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static const sqlite3_module fts3Module = { 35565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* iVersion */ 0, 35575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xCreate */ fts3CreateMethod, 35585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xConnect */ fts3ConnectMethod, 35595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xBestIndex */ fts3BestIndexMethod, 35605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xDisconnect */ fts3DisconnectMethod, 35615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xDestroy */ fts3DestroyMethod, 35625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xOpen */ fts3OpenMethod, 35635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xClose */ fts3CloseMethod, 35645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xFilter */ fts3FilterMethod, 35655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xNext */ fts3NextMethod, 35665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xEof */ fts3EofMethod, 35675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xColumn */ fts3ColumnMethod, 35685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xRowid */ fts3RowidMethod, 35695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xUpdate */ fts3UpdateMethod, 35705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xBegin */ fts3BeginMethod, 35715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xSync */ fts3SyncMethod, 35725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xCommit */ fts3CommitMethod, 35735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xRollback */ fts3RollbackMethod, 35745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xFindFunction */ fts3FindFunctionMethod, 35755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* xRename */ fts3RenameMethod, 35765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)}; 35775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 35795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** This function is registered as the module destructor (called when an 35805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** FTS3 enabled database connection is closed). It frees the memory 35815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** allocated for the tokenizer hash table. 35825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 35835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)static void hashDestroy(void *p){ 35845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Hash *pHash = (Fts3Hash *)p; 35855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3HashClear(pHash); 35865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pHash); 35875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 35885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 35895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 35905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** The fts3 built-in tokenizers - "simple", "porter" and "icu"- are 35915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** implemented in files fts3_tokenizer1.c, fts3_porter.c and fts3_icu.c 35925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** respectively. The following three forward declarations are for functions 35935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** declared in these files used to retrieve the respective implementations. 35945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** 35955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Calling sqlite3Fts3SimpleTokenizerModule() sets the value pointed 35965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** to by the argument to point to the "simple" tokenizer implementation. 35975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** And so on. 35985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 35995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)void sqlite3Fts3SimpleTokenizerModule(sqlite3_tokenizer_module const**ppModule); 36005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)void sqlite3Fts3PorterTokenizerModule(sqlite3_tokenizer_module const**ppModule); 36015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#ifdef SQLITE_ENABLE_ICU 36025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)void sqlite3Fts3IcuTokenizerModule(sqlite3_tokenizer_module const**ppModule); 36035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 36045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)/* 36065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** Initialise the fts3 extension. If this extension is built as part 36075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** of the sqlite library, then this function is called directly by 36085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** SQLite. If fts3 is built as a dynamically loadable extension, this 36095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)** function is called by the sqlite3_extension_init() entry point. 36105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)*/ 36115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3Fts3Init(sqlite3 *db){ 36125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) int rc = SQLITE_OK; 36135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Fts3Hash *pHash = 0; 36145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const sqlite3_tokenizer_module *pSimple = 0; 36155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const sqlite3_tokenizer_module *pPorter = 0; 36165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#ifdef SQLITE_ENABLE_ICU 36185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const sqlite3_tokenizer_module *pIcu = 0; 36195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3IcuTokenizerModule(&pIcu); 36205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 36215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3InitAux(db); 36235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc!=SQLITE_OK ) return rc; 36245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3SimpleTokenizerModule(&pSimple); 36265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3PorterTokenizerModule(&pPorter); 36275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Allocate and initialise the hash-table used to store tokenizers. */ 36295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) pHash = sqlite3_malloc(sizeof(Fts3Hash)); 36305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( !pHash ){ 36315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 36325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) }else{ 36335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3HashInit(pHash, FTS3_HASH_STRING, 1); 36345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 36355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Load the built-in tokenizers into the hash table */ 36375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 36385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( sqlite3Fts3HashInsert(pHash, "simple", 7, (void *)pSimple) 36395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || sqlite3Fts3HashInsert(pHash, "porter", 7, (void *)pPorter) 36405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#ifdef SQLITE_ENABLE_ICU 36415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) || (pIcu && sqlite3Fts3HashInsert(pHash, "icu", 4, (void *)pIcu)) 36425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 36435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 36445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = SQLITE_NOMEM; 36455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 36465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 36475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#ifdef SQLITE_TEST 36495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 36505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3Fts3ExprInitTestInterface(db); 36515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 36525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 36535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Create the virtual table wrapper around the hash-table and overload 36555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** the two scalar functions. If this is successful, register the 36565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ** module with sqlite. 36575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) */ 36585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( SQLITE_OK==rc 36595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#if CHROMIUM_FTS3_CHANGES && !SQLITE_TEST 36605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* fts3_tokenizer() disabled for security reasons. */ 36615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#else 36625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer")) 36635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 36645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1)) 36655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1)) 36665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", 1)) 36675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", 2)) 36685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) && SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1)) 36695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ){ 36705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_create_module_v2( 36715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) db, "fts3", &fts3Module, (void *)pHash, hashDestroy 36725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 36735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#if CHROMIUM_FTS3_CHANGES && !SQLITE_TEST 36745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* Disable fts4 pending review. */ 36755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#else 36765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( rc==SQLITE_OK ){ 36775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) rc = sqlite3_create_module_v2( 36785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) db, "fts4", &fts3Module, (void *)pHash, 0 36795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ); 36805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 36815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 36825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 36835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 36845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /* An error has occurred. Delete the hash table and return the error code. */ 36865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) assert( rc!=SQLITE_OK ); 36875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if( pHash ){ 36885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3Fts3HashClear(pHash); 36895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3_free(pHash); 36905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) } 36915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return rc; 36925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 36935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 36945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#if !SQLITE_CORE 36955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)int sqlite3_extension_init( 36965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sqlite3 *db, 36975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) char **pzErrMsg, 36985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) const sqlite3_api_routines *pApi 36995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)){ 37005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) SQLITE_EXTENSION_INIT2(pApi) 37015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) return sqlite3Fts3Init(db); 37025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)} 37035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 37045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 37055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)#endif 3706