12da489cd246702bee5938545b18a6f710ed214bcJamie Gennis#!/usr/bin/env python 22da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# 32da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# Copyright 2011 The Closure Linter Authors. All Rights Reserved. 42da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# 52da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# Licensed under the Apache License, Version 2.0 (the "License"); 62da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# you may not use this file except in compliance with the License. 72da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# You may obtain a copy of the License at 82da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# 92da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# http://www.apache.org/licenses/LICENSE-2.0 102da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# 112da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# Unless required by applicable law or agreed to in writing, software 122da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# distributed under the License is distributed on an "AS-IS" BASIS, 132da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 142da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# See the License for the specific language governing permissions and 152da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# limitations under the License. 162da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 172da489cd246702bee5938545b18a6f710ed214bcJamie Gennis"""Contains logic for sorting goog.provide and goog.require statements. 182da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 192da489cd246702bee5938545b18a6f710ed214bcJamie GennisClosurized JavaScript files use goog.provide and goog.require statements at the 202da489cd246702bee5938545b18a6f710ed214bcJamie Gennistop of the file to manage dependencies. These statements should be sorted 212da489cd246702bee5938545b18a6f710ed214bcJamie Gennisalphabetically, however, it is common for them to be accompanied by inline 222da489cd246702bee5938545b18a6f710ed214bcJamie Genniscomments or suppression annotations. In order to sort these statements without 232da489cd246702bee5938545b18a6f710ed214bcJamie Gennisdisrupting their comments and annotations, the association between statements 242da489cd246702bee5938545b18a6f710ed214bcJamie Gennisand comments/annotations must be maintained while sorting. 252da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 262da489cd246702bee5938545b18a6f710ed214bcJamie Gennis RequireProvideSorter: Handles checking/fixing of provide/require statements. 272da489cd246702bee5938545b18a6f710ed214bcJamie Gennis""" 282da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 292da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 302da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 312da489cd246702bee5938545b18a6f710ed214bcJamie Gennisfrom closure_linter import javascripttokens 322da489cd246702bee5938545b18a6f710ed214bcJamie Gennisfrom closure_linter import tokenutil 332da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 342da489cd246702bee5938545b18a6f710ed214bcJamie Gennis# Shorthand 352da489cd246702bee5938545b18a6f710ed214bcJamie GennisType = javascripttokens.JavaScriptTokenType 362da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 372da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 382da489cd246702bee5938545b18a6f710ed214bcJamie Gennisclass RequireProvideSorter(object): 392da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Checks for and fixes alphabetization of provide and require statements. 402da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 412da489cd246702bee5938545b18a6f710ed214bcJamie Gennis When alphabetizing, comments on the same line or comments directly above a 422da489cd246702bee5938545b18a6f710ed214bcJamie Gennis goog.provide or goog.require statement are associated with that statement and 432da489cd246702bee5938545b18a6f710ed214bcJamie Gennis stay with the statement as it gets sorted. 442da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 452da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 462da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def CheckProvides(self, token): 472da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Checks alphabetization of goog.provide statements. 482da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 492da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Iterates over tokens in given token stream, identifies goog.provide tokens, 502da489cd246702bee5938545b18a6f710ed214bcJamie Gennis and checks that they occur in alphabetical order by the object being 512da489cd246702bee5938545b18a6f710ed214bcJamie Gennis provided. 522da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 532da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 542da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token: A token in the token stream before any goog.provide tokens. 552da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 562da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Returns: 572da489cd246702bee5938545b18a6f710ed214bcJamie Gennis A tuple containing the first provide token in the token stream and a list 582da489cd246702bee5938545b18a6f710ed214bcJamie Gennis of provided objects sorted alphabetically. For example: 592da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 602da489cd246702bee5938545b18a6f710ed214bcJamie Gennis (JavaScriptToken, ['object.a', 'object.b', ...]) 612da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 622da489cd246702bee5938545b18a6f710ed214bcJamie Gennis None is returned if all goog.provide statements are already sorted. 632da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 642da489cd246702bee5938545b18a6f710ed214bcJamie Gennis provide_tokens = self._GetRequireOrProvideTokens(token, 'goog.provide') 652da489cd246702bee5938545b18a6f710ed214bcJamie Gennis provide_strings = self._GetRequireOrProvideTokenStrings(provide_tokens) 662da489cd246702bee5938545b18a6f710ed214bcJamie Gennis sorted_provide_strings = sorted(provide_strings) 672da489cd246702bee5938545b18a6f710ed214bcJamie Gennis if provide_strings != sorted_provide_strings: 682da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return [provide_tokens[0], sorted_provide_strings] 692da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return None 702da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 712da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def CheckRequires(self, token): 722da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Checks alphabetization of goog.require statements. 732da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 742da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Iterates over tokens in given token stream, identifies goog.require tokens, 752da489cd246702bee5938545b18a6f710ed214bcJamie Gennis and checks that they occur in alphabetical order by the dependency being 762da489cd246702bee5938545b18a6f710ed214bcJamie Gennis required. 772da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 782da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 792da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token: A token in the token stream before any goog.require tokens. 802da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 812da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Returns: 822da489cd246702bee5938545b18a6f710ed214bcJamie Gennis A tuple containing the first require token in the token stream and a list 832da489cd246702bee5938545b18a6f710ed214bcJamie Gennis of required dependencies sorted alphabetically. For example: 842da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 852da489cd246702bee5938545b18a6f710ed214bcJamie Gennis (JavaScriptToken, ['object.a', 'object.b', ...]) 862da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 872da489cd246702bee5938545b18a6f710ed214bcJamie Gennis None is returned if all goog.require statements are already sorted. 882da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 892da489cd246702bee5938545b18a6f710ed214bcJamie Gennis require_tokens = self._GetRequireOrProvideTokens(token, 'goog.require') 902da489cd246702bee5938545b18a6f710ed214bcJamie Gennis require_strings = self._GetRequireOrProvideTokenStrings(require_tokens) 912da489cd246702bee5938545b18a6f710ed214bcJamie Gennis sorted_require_strings = sorted(require_strings) 922da489cd246702bee5938545b18a6f710ed214bcJamie Gennis if require_strings != sorted_require_strings: 932da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return (require_tokens[0], sorted_require_strings) 942da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return None 952da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 962da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def FixProvides(self, token): 972da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Sorts goog.provide statements in the given token stream alphabetically. 982da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 992da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 1002da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token: The first token in the token stream. 1012da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 1022da489cd246702bee5938545b18a6f710ed214bcJamie Gennis self._FixProvidesOrRequires( 1032da489cd246702bee5938545b18a6f710ed214bcJamie Gennis self._GetRequireOrProvideTokens(token, 'goog.provide')) 1042da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1052da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def FixRequires(self, token): 1062da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Sorts goog.require statements in the given token stream alphabetically. 1072da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1082da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 1092da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token: The first token in the token stream. 1102da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 1112da489cd246702bee5938545b18a6f710ed214bcJamie Gennis self._FixProvidesOrRequires( 1122da489cd246702bee5938545b18a6f710ed214bcJamie Gennis self._GetRequireOrProvideTokens(token, 'goog.require')) 1132da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1142da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def _FixProvidesOrRequires(self, tokens): 1152da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Sorts goog.provide or goog.require statements. 1162da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1172da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 1182da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens: A list of goog.provide or goog.require tokens in the order they 1192da489cd246702bee5938545b18a6f710ed214bcJamie Gennis appear in the token stream. i.e. the first token in this list must 1202da489cd246702bee5938545b18a6f710ed214bcJamie Gennis be the first goog.provide or goog.require token. 1212da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 1222da489cd246702bee5938545b18a6f710ed214bcJamie Gennis strings = self._GetRequireOrProvideTokenStrings(tokens) 1232da489cd246702bee5938545b18a6f710ed214bcJamie Gennis sorted_strings = sorted(strings) 1242da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1252da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # Make a separate pass to remove any blank lines between goog.require/ 1262da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # goog.provide tokens. 1272da489cd246702bee5938545b18a6f710ed214bcJamie Gennis first_token = tokens[0] 1282da489cd246702bee5938545b18a6f710ed214bcJamie Gennis last_token = tokens[-1] 1292da489cd246702bee5938545b18a6f710ed214bcJamie Gennis i = last_token 1302da489cd246702bee5938545b18a6f710ed214bcJamie Gennis while i != first_token: 1312da489cd246702bee5938545b18a6f710ed214bcJamie Gennis if i.type is Type.BLANK_LINE: 1322da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokenutil.DeleteToken(i) 1332da489cd246702bee5938545b18a6f710ed214bcJamie Gennis i = i.previous 1342da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1352da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # A map from required/provided object name to tokens that make up the line 1362da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # it was on, including any comments immediately before it or after it on the 1372da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # same line. 1382da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens_map = self._GetTokensMap(tokens) 1392da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1402da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # Iterate over the map removing all tokens. 1412da489cd246702bee5938545b18a6f710ed214bcJamie Gennis for name in tokens_map: 1422da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens_to_delete = tokens_map[name] 1432da489cd246702bee5938545b18a6f710ed214bcJamie Gennis for i in tokens_to_delete: 1442da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokenutil.DeleteToken(i) 1452da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1462da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # Re-add all tokens in the map in alphabetical order. 1472da489cd246702bee5938545b18a6f710ed214bcJamie Gennis insert_after = tokens[0].previous 1482da489cd246702bee5938545b18a6f710ed214bcJamie Gennis for string in sorted_strings: 1492da489cd246702bee5938545b18a6f710ed214bcJamie Gennis for i in tokens_map[string]: 1502da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokenutil.InsertTokenAfter(i, insert_after) 1512da489cd246702bee5938545b18a6f710ed214bcJamie Gennis insert_after = i 1522da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1532da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def _GetRequireOrProvideTokens(self, token, token_string): 1542da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Gets all goog.provide or goog.require tokens in the given token stream. 1552da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1562da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 1572da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token: The first token in the token stream. 1582da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token_string: One of 'goog.provide' or 'goog.require' to indicate which 1592da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens to find. 1602da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1612da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Returns: 1622da489cd246702bee5938545b18a6f710ed214bcJamie Gennis A list of goog.provide or goog.require tokens in the order they appear in 1632da489cd246702bee5938545b18a6f710ed214bcJamie Gennis the token stream. 1642da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 1652da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens = [] 1662da489cd246702bee5938545b18a6f710ed214bcJamie Gennis while token: 1672da489cd246702bee5938545b18a6f710ed214bcJamie Gennis if token.type == Type.IDENTIFIER: 1682da489cd246702bee5938545b18a6f710ed214bcJamie Gennis if token.string == token_string: 1692da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens.append(token) 1702da489cd246702bee5938545b18a6f710ed214bcJamie Gennis elif token.string not in ['goog.require', 'goog.provide']: 1712da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # The goog.provide and goog.require identifiers are at the top of the 1722da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # file. So if any other identifier is encountered, return. 1732da489cd246702bee5938545b18a6f710ed214bcJamie Gennis break 1742da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token = token.next 1752da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1762da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return tokens 1772da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1782da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def _GetRequireOrProvideTokenStrings(self, tokens): 1792da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Gets a list of strings corresponding to the given list of tokens. 1802da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1812da489cd246702bee5938545b18a6f710ed214bcJamie Gennis The string will be the next string in the token stream after each token in 1822da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens. This is used to find the object being provided/required by a given 1832da489cd246702bee5938545b18a6f710ed214bcJamie Gennis goog.provide or goog.require token. 1842da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1852da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 1862da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens: A list of goog.provide or goog.require tokens. 1872da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1882da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Returns: 1892da489cd246702bee5938545b18a6f710ed214bcJamie Gennis A list of object names that are being provided or required by the given 1902da489cd246702bee5938545b18a6f710ed214bcJamie Gennis list of tokens. For example: 1912da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 1922da489cd246702bee5938545b18a6f710ed214bcJamie Gennis ['object.a', 'object.c', 'object.b'] 1932da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 1942da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token_strings = [] 1952da489cd246702bee5938545b18a6f710ed214bcJamie Gennis for token in tokens: 1962da489cd246702bee5938545b18a6f710ed214bcJamie Gennis name = tokenutil.Search(token, Type.STRING_TEXT).string 1972da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token_strings.append(name) 1982da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return token_strings 1992da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2002da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def _GetTokensMap(self, tokens): 2012da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Gets a map from object name to tokens associated with that object. 2022da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2032da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Starting from the goog.provide/goog.require token, searches backwards in the 2042da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token stream for any lines that start with a comment. These lines are 2052da489cd246702bee5938545b18a6f710ed214bcJamie Gennis associated with the goog.provide/goog.require token. Also associates any 2062da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens on the same line as the goog.provide/goog.require token with that 2072da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token. 2082da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2092da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 2102da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens: A list of goog.provide or goog.require tokens. 2112da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2122da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Returns: 2132da489cd246702bee5938545b18a6f710ed214bcJamie Gennis A dictionary that maps object names to the tokens associated with the 2142da489cd246702bee5938545b18a6f710ed214bcJamie Gennis goog.provide or goog.require of that object name. For example: 2152da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2162da489cd246702bee5938545b18a6f710ed214bcJamie Gennis { 2172da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 'object.a': [JavaScriptToken, JavaScriptToken, ...], 2182da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 'object.b': [...] 2192da489cd246702bee5938545b18a6f710ed214bcJamie Gennis } 2202da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2212da489cd246702bee5938545b18a6f710ed214bcJamie Gennis The list of tokens includes any comment lines above the goog.provide or 2222da489cd246702bee5938545b18a6f710ed214bcJamie Gennis goog.require statement and everything after the statement on the same 2232da489cd246702bee5938545b18a6f710ed214bcJamie Gennis line. For example, all of the following would be associated with 2242da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 'object.a': 2252da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2262da489cd246702bee5938545b18a6f710ed214bcJamie Gennis /** @suppress {extraRequire} */ 2272da489cd246702bee5938545b18a6f710ed214bcJamie Gennis goog.require('object.a'); // Some comment. 2282da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 2292da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens_map = {} 2302da489cd246702bee5938545b18a6f710ed214bcJamie Gennis for token in tokens: 2312da489cd246702bee5938545b18a6f710ed214bcJamie Gennis object_name = tokenutil.Search(token, Type.STRING_TEXT).string 2322da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # If the previous line starts with a comment, presume that the comment 2332da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # relates to the goog.require or goog.provide and keep them together when 2342da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # sorting. 2352da489cd246702bee5938545b18a6f710ed214bcJamie Gennis first_token = token 2362da489cd246702bee5938545b18a6f710ed214bcJamie Gennis previous_first_token = tokenutil.GetFirstTokenInPreviousLine(first_token) 2372da489cd246702bee5938545b18a6f710ed214bcJamie Gennis while previous_first_token.IsAnyType(Type.COMMENT_TYPES): 2382da489cd246702bee5938545b18a6f710ed214bcJamie Gennis first_token = previous_first_token 2392da489cd246702bee5938545b18a6f710ed214bcJamie Gennis previous_first_token = tokenutil.GetFirstTokenInPreviousLine( 2402da489cd246702bee5938545b18a6f710ed214bcJamie Gennis first_token) 2412da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2422da489cd246702bee5938545b18a6f710ed214bcJamie Gennis # Find the last token on the line. 2432da489cd246702bee5938545b18a6f710ed214bcJamie Gennis last_token = tokenutil.GetLastTokenInSameLine(token) 2442da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2452da489cd246702bee5938545b18a6f710ed214bcJamie Gennis all_tokens = self._GetTokenList(first_token, last_token) 2462da489cd246702bee5938545b18a6f710ed214bcJamie Gennis tokens_map[object_name] = all_tokens 2472da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return tokens_map 2482da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2492da489cd246702bee5938545b18a6f710ed214bcJamie Gennis def _GetTokenList(self, first_token, last_token): 2502da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """Gets a list of all tokens from first_token to last_token, inclusive. 2512da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2522da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Args: 2532da489cd246702bee5938545b18a6f710ed214bcJamie Gennis first_token: The first token to get. 2542da489cd246702bee5938545b18a6f710ed214bcJamie Gennis last_token: The last token to get. 2552da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2562da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Returns: 2572da489cd246702bee5938545b18a6f710ed214bcJamie Gennis A list of all tokens between first_token and last_token, including both 2582da489cd246702bee5938545b18a6f710ed214bcJamie Gennis first_token and last_token. 2592da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2602da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Raises: 2612da489cd246702bee5938545b18a6f710ed214bcJamie Gennis Exception: If the token stream ends before last_token is reached. 2622da489cd246702bee5938545b18a6f710ed214bcJamie Gennis """ 2632da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token_list = [] 2642da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token = first_token 2652da489cd246702bee5938545b18a6f710ed214bcJamie Gennis while token != last_token: 2662da489cd246702bee5938545b18a6f710ed214bcJamie Gennis if not token: 2672da489cd246702bee5938545b18a6f710ed214bcJamie Gennis raise Exception('ran out of tokens') 2682da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token_list.append(token) 2692da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token = token.next 2702da489cd246702bee5938545b18a6f710ed214bcJamie Gennis token_list.append(last_token) 2712da489cd246702bee5938545b18a6f710ed214bcJamie Gennis 2722da489cd246702bee5938545b18a6f710ed214bcJamie Gennis return token_list 273