1324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver/* 2324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver [The "BSD licence"] 3324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver Copyright (c) 2004 Terence Parr and Loring Craymer 4324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver All rights reserved. 5324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 6324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver Redistribution and use in source and binary forms, with or without 7324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver modification, are permitted provided that the following conditions 8324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver are met: 9324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 1. Redistributions of source code must retain the above copyright 10324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver notice, this list of conditions and the following disclaimer. 11324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 2. Redistributions in binary form must reproduce the above copyright 12324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver notice, this list of conditions and the following disclaimer in the 13324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver documentation and/or other materials provided with the distribution. 14324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 3. The name of the author may not be used to endorse or promote products 15324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver derived from this software without specific prior written permission. 16324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 17324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 18324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 19324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 20324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, 21324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 22324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 23324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 24324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 25324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 26324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver*/ 28324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 29324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 30324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver/** Python does not explicitly provide begin and end nesting signals. 31324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver Rather, the indentation level indicates when you begin and end. 32324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver This is an interesting lexical problem because multiple DEDENT 33324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver tokens should be sent to the parser sometimes without a corresponding 34324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver input symbol! Consider the following example: 35324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 36324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver a=1 37324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if a>1: 38324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver print a 39324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver b=3 40324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 41324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver Here the "b" token on the left edge signals that a DEDENT is needed 42324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver after the "print a \n" and before the "b". The sequence should be 43324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 44324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver ... 1 COLON NEWLINE INDENT PRINT a NEWLINE DEDENT b ASSIGN 3 ... 45324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 46324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver For more examples, see the big comment at the bottom of this file. 47324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 48324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver This TokenStream normally just passes tokens through to the parser. 49324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver Upon NEWLINE token from the lexer, however, an INDENT or DEDENT token 50324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver may need to be sent to the parser. The NEWLINE is the trigger for 51324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this class to do it's job. NEWLINE is saved and then the first token 52324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver of the next line is examined. If non-leading-whitespace token, 53324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver then check against stack for indent vs dedent. If LEADING_WS, then 54324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver the column of the next non-whitespace token will dictate indent vs 55324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver dedent. The column of the next real token is number of spaces 56324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver in the LEADING_WS token + 1 (to move past the whitespace). The 57324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver lexer grammar must set the text of the LEADING_WS token to be 58324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver the proper number of spaces (and do tab conversion etc...). 59324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 60324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver A stack of column numbers is tracked and used to detect changes 61324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver in indent level from one token to the next. 62324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 63324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver A queue of tokens is built up to hold multiple DEDENT tokens that 64324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver are generated. Before asking the lexer for another token via 65324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver nextToken(), the queue is flushed first one token at a time. 66324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 67324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver Terence Parr and Loring Craymer 68324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver February 2004 69324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver */ 70324c4644fee44b9898524c09511bd33c3f12e2dfBen GruverPythonTokenSource = function(stream) { 71324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.stream = stream; 72324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver /** The stack of indent levels (column numbers) */ 73324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.indentStack = new Array(PythonTokenSource.MAX_INDENTS); 74324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver /** stack pointer */ 75324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.sp=-1; // grow upwards 76324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 77324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver /** The queue of tokens */ 78324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens = []; 79324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.lastTokenAddedIndex = -1; 80324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.push(PythonTokenSource.FIRST_CHAR_POSITION); 81324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver}; 82324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 83324c4644fee44b9898524c09511bd33c3f12e2dfBen GruverANTLR.lang.augmentObject(PythonTokenSource, { 84324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver MAX_INDENTS: 100, 85324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver FIRST_CHAR_POSITION: 0, 86324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver}); 87324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 88324c4644fee44b9898524c09511bd33c3f12e2dfBen GruverPythonTokenSource.prototype = { 89324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver getSourceName: function() { 90324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return this.stream.getSourceName(); 91324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver }, 92324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 93324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver /** From http://www.python.org/doc/2.2.3/ref/indentation.html 94324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 95324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver "Before the first line of the file is read, a single zero is 96324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver pushed on the stack; this will never be popped off again. The 97324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver numbers pushed on the stack will always be strictly increasing 98324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver from bottom to top. At the beginning of each logical line, the 99324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver line's indentation level is compared to the top of the 100324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver stack. If it is equal, nothing happens. If it is larger, it is 101324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver pushed on the stack, and one INDENT token is generated. If it 102324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver is smaller, it must be one of the numbers occurring on the 103324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver stack; all numbers on the stack that are larger are popped 104324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver off, and for each number popped off a DEDENT token is 105324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver generated. At the end of the file, a DEDENT token is generated 106324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver for each number remaining on the stack that is larger than 107324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver zero." 108324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 109324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver I use char position in line 0..n-1 instead. 110324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 111324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver The DEDENTS possibly needed at EOF are gracefully handled by forcing 112324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver EOF to have char pos 0 even though with UNIX it's hard to get EOF 113324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver at a non left edge. 114324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver */ 115324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver nextToken: function() { 116324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // if something in queue, just remove and return it 117324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if (this.tokens.length>0 ) { 118324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var t = this.tokens[0]; 119324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens.splice(0,1); 120324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return t; 121324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 122324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 123324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.insertImaginaryIndentDedentTokens(); 124324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 125324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return this.nextToken(); 126324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver }, 127324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 128324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver insertImaginaryIndentDedentTokens: function() 129324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver { 130324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var t = this.stream.LT(1); 131324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.stream.consume(); 132324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 133324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // if not a NEWLINE, doesn't signal indent/dedent work; just enqueue 134324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if ( t.getType()!=PythonLexer.NEWLINE ) { 135324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var hiddenTokens = this.stream.getTokens(this.lastTokenAddedIndex+1,t.getTokenIndex()-1); 136324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if ( hiddenTokens!=null ) { 137324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens = this.tokens.concat(hiddenTokens); 138324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 139324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.lastTokenAddedIndex = t.getTokenIndex(); 140324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens.push(t); 141324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return; 142324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 143324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 144324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // save NEWLINE in the queue 145324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var hiddenTokens = this.stream.getTokens(this.lastTokenAddedIndex+1,t.getTokenIndex()-1); 146324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if ( hiddenTokens!=null ) { 147324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens = this.tokens.concat(hiddenTokens); 148324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 149324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.lastTokenAddedIndex = t.getTokenIndex(); 150324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens.push(t); 151324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 152324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // grab first token of next line 153324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver t = this.stream.LT(1); 154324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.stream.consume(); 155324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 156324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver hiddenTokens = this.stream.getTokens(this.lastTokenAddedIndex+1,t.getTokenIndex()-1); 157324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if ( hiddenTokens!=null ) { 158324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens = this.tokens.concat(hiddenTokens); 159324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 160324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.lastTokenAddedIndex = t.getTokenIndex(); 161324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 162324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // compute cpos as the char pos of next non-WS token in line 163324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var cpos = t.getCharPositionInLine(); // column dictates indent/dedent 164324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if ( t.getType()==ANTLR.runtime.Token.EOF ) { 165324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver cpos = -1; // pretend EOF always happens at left edge 166324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 167324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver else if ( t.getType()==PythonLexer.LEADING_WS ) { 168324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver cpos = t.getText().length; 169324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 170324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 171324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // compare to last indent level 172324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var lastIndent = this.peek(); 173324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if ( cpos > lastIndent ) { // they indented; track and gen INDENT 174324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.push(cpos); 175324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var indent = new ANTLR.runtime.CommonToken(PythonParser.INDENT, ""); 176324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver indent.setCharPositionInLine(t.getCharPositionInLine()); 177324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver indent.setLine(t.getLine()); 178324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens.push(indent); 179324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 180324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver else if ( cpos < lastIndent ) { // they dedented 181324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // how far back did we dedent? 182324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var prevIndex = this.findPreviousIndent(cpos); 183324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // generate DEDENTs for each indent level we backed up over 184324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver for (var d=this.sp-1; d>=prevIndex; d--) { 185324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var dedent = new ANTLR.runtime.CommonToken(PythonParser.DEDENT, ""); 186324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver dedent.setCharPositionInLine(t.getCharPositionInLine()); 187324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver dedent.setLine(t.getLine()); 188324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens.push(dedent); 189324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 190324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.sp = prevIndex; // pop those off indent level 191324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 192324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if ( t.getType()!=PythonLexer.LEADING_WS ) { // discard WS 193324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.tokens.push(t); 194324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 195324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver }, 196324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 197324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver // T O K E N S T A C K M E T H O D S 198324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 199324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver push: function(i) { 200324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if (this.sp>=PythonTokenSource.MAX_INDENTS) { 201324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver throw new Error("stack overflow"); 202324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 203324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.sp++; 204324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.indentStack[this.sp] = i; 205324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver }, 206324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 207324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver pop: function() { 208324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if (this.sp<0) { 209324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver throw new Error("stack underflow"); 210324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 211324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var top = this.indentStack[this.sp]; 212324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver this.sp--; 213324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return top; 214324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver }, 215324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 216324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver peek: function() { 217324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return this.indentStack[this.sp]; 218324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver }, 219324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 220324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver /** Return the index on stack of previous indent level == i else -1 */ 221324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver findPreviousIndent: function(i) { 222324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver for (var j=this.sp-1; j>=0; j--) { 223324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver if (this.indentStack[j]==i ) { 224324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return j; 225324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 226324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 227324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return PythonTokenSource.FIRST_CHAR_POSITION; 228324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver }, 229324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 230324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver stackString: function() { 231324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver var buf = []; 232324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver for (var j=this.sp; j>=0; j--) { 233324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver buf.push(this.indentStack[j]); 234324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 235324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver return buf.join(" "); 236324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver } 237324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 238324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver} 239324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver 240324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver/* More example input / output pairs with code simplified to single chars 241324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver------- t1 ------- 242324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera a 243324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver b b 244324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver c 245324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruverd 246324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera a \n INDENT b b \n c \n DEDENT d \n EOF 247324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver------- t2 ------- 248324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera c 249324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver b 250324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruverc 251324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera c \n INDENT b \n DEDENT c \n EOF 252324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver------- t3 ------- 253324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera 254324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver b 255324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver c 256324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruverd 257324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera \n INDENT b \n INDENT c \n DEDENT DEDENT d \n EOF 258324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver------- t4 ------- 259324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera 260324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver c 261324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver d 262324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver e 263324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver f 264324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver g 265324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver h 266324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver i 267324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver j 268324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver k 269324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera \n INDENT c \n INDENT d \n DEDENT e \n f \n INDENT g \n h \n i \n INDENT j \n DEDENT DEDENT k \n DEDENT EOF 270324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver------- t5 ------- 271324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera 272324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver b 273324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver c 274324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver d 275324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver e 276324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruvera \n INDENT b \n c \n INDENT d \n e \n DEDENT DEDENT EOF 277324c4644fee44b9898524c09511bd33c3f12e2dfBen Gruver*/ 278