History log of /external/clang/lib/Lex/PTHLexer.cpp
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
ef8225444452a1486bd721f3285301fe84643b00 21-Jul-2014 Stephen Hines <srhines@google.com> Update Clang for rebase to r212749.

This also fixes a small issue with arm_neon.h not being generated always.

Includes a cherry-pick of:
r213450 - fixes mac-specific header issue
r213126 - removes a default -Bsymbolic on Android

Change-Id: I2a790a0f5d3b2aab11de596fc3a74e7cbc99081d
6bcf27bb9a4b5c3f79cb44c0e4654a6d7619ad89 29-May-2014 Stephen Hines <srhines@google.com> Update Clang for 3.5 rebase (r209713).

Change-Id: I8c9133b0f8f776dc915f270b60f94962e771bc83
651f13cea278ec967336033dd032faef0e9fc2ec 24-Apr-2014 Stephen Hines <srhines@google.com> Updated to Clang 3.5a.

Change-Id: I8127eb568f674c2e72635b639a3295381fe8af82
d2f9308220af22bfc1bcd3bc2cad118dbd8be013 19-Sep-2013 Eli Friedman <eli.friedman@gmail.com> Make Preprocessor::Lex non-recursive.

Before this patch, Lex() would recurse whenever the current lexer changed (e.g.
upon entry into a macro). This patch turns the recursion into a loop: the
various lex routines now don't return a token when the current lexer changes,
and at the top level Preprocessor::Lex() now loops until it finds a token.
Normally, the recursion wouldn't end up being very deep, but the recursion depth
can explode in edge cases like a bunch of consecutive macros which expand to
nothing (like in the testcase test/Preprocessor/macro_expand_empty.c in this


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@190980 91177308-0d34-0410-b5e6-96231b3b80d8
0fda0f75f053ad9afdb2bb8d4ea72c5e6a49d657 01-Aug-2013 Rafael Espindola <rafael.espindola@gmail.com> Use llvm::sys::fs::UniqueID for windows and unix.

This unifies the unix and windows versions of FileManager::UniqueDirContainer
and FileManager::UniqueFileContainer by using UniqueID.

We cannot just replace "struct stat" with llvm::sys::fs::file_status, since we
want to be able to construct fake ones, and file_status has different members
on unix and windows.

What the patch does is:

* Record only the information that clang is actually using.
* Use llvm::sys::fs::status instead of stat and fstat.
* Use llvm::sys::fs::UniqueID
* Delete the old windows versions of UniqueDirContainer and
UniqueFileContainer since the "unix" one now works on windows too.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@187619 91177308-0d34-0410-b5e6-96231b3b80d8
87aa2fbc75a897e7c4a4082374aaba3f50db6f0f 21-Dec-2012 Roman Divacky <rdivacky@freebsd.org> Remove duplicate includes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@170903 91177308-0d34-0410-b5e6-96231b3b80d8
e5d30e3b403539b10aaa52f03875a2243bf88904 11-Dec-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Extend stat query APIs to explicitly specify if the query is for
a file or directory, allowing just a stat call if a file descriptor
is not needed.

Doing just 'stat' is faster than 'open/fstat/close'.
This has the effect of cutting down system time for validating the input files of a PCH.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169831 91177308-0d34-0410-b5e6-96231b3b80d8
55fc873017f10f6f566b182b70f6fc22aefa3464 04-Dec-2012 Chandler Carruth <chandlerc@gmail.com> Sort all of Clang's files under 'lib', and fix up the broken headers

This required manually correcting all of the incorrect main-module
headers I could find, and running the new llvm/utils/sort_includes.py
script over the files.

I also manually added quite a few missing headers that were uncovered by
shuffling the order or moving headers up to be main-module-headers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169237 91177308-0d34-0410-b5e6-96231b3b80d8
536cebd7f81b665dbef2ad6f081dcb23349f5576 16-Sep-2012 Ted Kremenek <kremenek@apple.com> Fix dead store found by static analyzer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163994 91177308-0d34-0410-b5e6-96231b3b80d8
31ba6135375433b617a8587ea6cc836a014ebd86 06-Sep-2012 Roman Divacky <rdivacky@freebsd.org> Dont cast away const needlessly. Found by gcc48 -Wcast-qual.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163325 91177308-0d34-0410-b5e6-96231b3b80d8
c141b51d78b40c7b0b04f27adfa70ec5bbddc49c 17-Aug-2012 Richard Smith <richard-llvm@metafoo.co.uk> Switch PTH format from a 7 byte magic number to an 8 byte one, to avoid
misaligned reads throughout the file. Bump PTH format version to 10.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162076 91177308-0d34-0410-b5e6-96231b3b80d8
6f42b62b6194f53bcbc349f5d17388e1936535d7 05-Feb-2012 Dylan Noblesmith <nobled@dreamwidth.org> Basic: import OwningPtr<> into clang namespace

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149798 91177308-0d34-0410-b5e6-96231b3b80d8
d6471f7c1921c7802804ce3ff6fe9768310f72b9 26-Sep-2011 David Blaikie <dblaikie@gmail.com> Rename Diagnostic to DiagnosticsEngine as per issue 5397

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140478 91177308-0d34-0410-b5e6-96231b3b80d8
681d83dd7e36ddb8a25313c446a91c00bbc652dd 22-Sep-2011 David Blaikie <dblaikie@gmail.com> Simplify the last character check.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140287 91177308-0d34-0410-b5e6-96231b3b80d8
a64ccefdf0ea4e03ec88805d71b0af74950c7472 19-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Rename SourceLocation::getFileLocWithOffset -> getLocWithOffset.

It already works (and is useful with) macro locs as well.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140057 91177308-0d34-0410-b5e6-96231b3b80d8
7d100872341f233c81e1d7b72b40457e62c36862 04-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Support code-completion for C++ inline methods and ObjC buffering methods.

Previously we would cut off the source file buffer at the code-completion
point; this impeded code-completion inside C++ inline methods and,
recently, with buffering ObjC methods.

Have the code-completion inserted into the source buffer so that it can
be buffered along with a method body. When we actually hit the code-completion
point the cut-off lexing or parsing.

Fixes rdar://10056932&8319466

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139086 91177308-0d34-0410-b5e6-96231b3b80d8
5f9e272e632e951b1efe824cd16acb4d96077930 23-Jul-2011 Chris Lattner <sabre@nondot.org> remove unneeded llvm:: namespace qualifiers on some core types now that LLVM.h imports
them into the clang namespace.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135852 91177308-0d34-0410-b5e6-96231b3b80d8
fc8f0e14ad142ed811e90fbd9a30e419e301c717 15-Apr-2011 Chris Lattner <sabre@nondot.org> fix a bunch of comment typos found by codespell. Patch by
Luis Felipe Strano Moraes!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129559 91177308-0d34-0410-b5e6-96231b3b80d8
84021556baceb76eedf7d44be8ba71d9b8cfacce 28-Feb-2011 Peter Collingbourne <peter@pcc.me.uk> Rename tok::eom to tok::eod.

The previous name was inaccurate as this token in fact appears at
the end of every preprocessing directive, not just macro definitions.
No functionality change, except for a diagnostic tweak.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126631 91177308-0d34-0410-b5e6-96231b3b80d8
4eeebc464e1f968d9968a4786c82558f18ac2ed8 16-Dec-2010 Michael J. Spencer <bigcheesegs@gmail.com> MemoryBuffer API update.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@121956 91177308-0d34-0410-b5e6-96231b3b80d8
3a321e23f66128dbb986343927456ff6702af617 09-Dec-2010 Michael J. Spencer <bigcheesegs@gmail.com> Use error_code instead of std::string* for MemoryBuffer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@121378 91177308-0d34-0410-b5e6-96231b3b80d8
898a061f69e1145bf89a987c08203132b9922a3c 23-Nov-2010 Chris Lattner <sabre@nondot.org> change the 'is directory' indicator to be a null-or-not
pointer that is passed down through the APIs, and make
FileSystemStatCache::get be the one that filters out
directory lookups that hit files. This also paves the
way to have stat queries be able to return opened files.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120060 91177308-0d34-0410-b5e6-96231b3b80d8
d6f611198089b78e32d3a15fe8bc986204aee1aa 23-Nov-2010 Chris Lattner <sabre@nondot.org> simplify the cache miss handling code, eliminating CacheMissing.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120038 91177308-0d34-0410-b5e6-96231b3b80d8
10e286aa8d39fb51a21412850265d9dae74613ee 23-Nov-2010 Chris Lattner <sabre@nondot.org> rework the stat cache, pulling it out of FileManager.h into
its own header and giving it some more structure. No
functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120030 91177308-0d34-0410-b5e6-96231b3b80d8
681c74afe6a81161aa13291c6c114e5240b23865 23-Nov-2010 Chris Lattner <sabre@nondot.org> don't allow remapping PTH file paths with -fworking-directory, the
client should just pass in absolute paths.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120012 91177308-0d34-0410-b5e6-96231b3b80d8
39b49bcaaddb1049234fca9500c0ac02c088e23d 23-Nov-2010 Chris Lattner <sabre@nondot.org> now the FileManager has a FileSystemOpts ivar, stop threading
FileSystemOpts through a ton of apis, simplifying a lot of code.
This also fixes a latent bug in ASTUnit where it would invoke
methods on FileManager without creating one in some code paths
in cindextext.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120010 91177308-0d34-0410-b5e6-96231b3b80d8
7ad629ebefa87269cc1887debc83cfa58c285a80 23-Nov-2010 Chris Lattner <sabre@nondot.org> tidy up

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119996 91177308-0d34-0410-b5e6-96231b3b80d8
389db16c63eec6ecfa9b235155252d8da766e94e 03-Nov-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> Implement -working-directory.

When -working-directory is passed in command line, file paths are resolved relative to the specified directory.
This helps both when using libclang (where we can't require the user to actually change the working directory)
and to help reproduce test cases when the reproduction work comes along.

--FileSystemOptions is introduced which controls how file system operations are performed (currently it just contains
the working directory value if set).
--FileSystemOptions are passed around to various interfaces that perform file operations.
--Opening & reading the content of files should be done only through FileManager. This is useful in general since
file operations will be abstracted in the future for the reproduction mechanism.

FileSystemOptions is independent of FileManager so that we can have multiple translation units sharing the same
FileManager but with different FileSystemOptions.

Addresses rdar://8583824.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@118203 91177308-0d34-0410-b5e6-96231b3b80d8
2d474ba9e8ae43a1a5a9f72718c0d79092b9453f 12-Aug-2010 Douglas Gregor <dgregor@apple.com> Don't emit end-of-file diagnostics like "unterminated conditional" or
"unterminated string" when we're performing code completion.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110933 91177308-0d34-0410-b5e6-96231b3b80d8
94e3d1f0349634235606be77c08b3a48a6d563d6 27-Jul-2010 Ted Kremenek <kremenek@apple.com> Add PTHLexer::LexEndOfFile() to emit diagnostics at end-of-file similar to those by Lexer::LexEndOfFile().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109486 91177308-0d34-0410-b5e6-96231b3b80d8
700030ebddb987936d4fee14d9412821d96e4840 12-Mar-2010 Kovarththanan Rajaratnam <kovarththanan.rajaratnam@gmail.com> Switch over IdentifierInfoLookup to StringRef

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98337 91177308-0d34-0410-b5e6-96231b3b80d8
85b4521e34dcd4a0a4a1f0819e1123128e5a3125 28-Nov-2009 Benjamin Kramer <benny.kra@googlemail.com> Remove remaining VISIBILITY_HIDDEN from anonymous namespaces.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90044 91177308-0d34-0410-b5e6-96231b3b80d8
149594457da7edfea2ce537ed18ca1b412dffd54 28-Nov-2009 Kovarththanan Rajaratnam <kovarththanan.rajaratnam@gmail.com> Cleanup llvm/Support/Compiler.h include in header files

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90040 91177308-0d34-0410-b5e6-96231b3b80d8
3574f46cf495ec61618fd6864b045c5b1d0d5068 12-Nov-2009 Daniel Dunbar <daniel@zuster.org> Switch PTHManager to using diagnostics for most errors.

Also, always give errors on a token-cache PTH failure.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86939 91177308-0d34-0410-b5e6-96231b3b80d8
0f9fed70cea107b3f79df554e38bd8e98d48fe47 11-Nov-2009 Daniel Dunbar <daniel@zuster.org> Add Diagnostic::Report method for reporting diagnostics without a location.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86760 91177308-0d34-0410-b5e6-96231b3b80d8
e013d685c6689ac7ae103ee88acf573422d1ed6a 18-Oct-2009 Daniel Dunbar <daniel@zuster.org> Move clients to use IdentifierInfo::getNameStart() instead of getName()

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@84436 91177308-0d34-0410-b5e6-96231b3b80d8
2596e429a61602312bdd149786045b8a90cd2d10 18-Oct-2009 Daniel Dunbar <daniel@zuster.org> Switch to llvm::HashString.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@84375 91177308-0d34-0410-b5e6-96231b3b80d8
52e7108f51a4a9f4d6e84f33fb594d06e1d79560 16-Oct-2009 Douglas Gregor <dgregor@apple.com> Add support for a chain of stat caches in the FileManager, rather than
only supporting a single stat cache. The immediate benefit of this
change is that we can now generate a PCH/AST file when including
another PCH file; in the future, the chain of stat caches will likely
be useful with multiple levels of PCH files.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@84263 91177308-0d34-0410-b5e6-96231b3b80d8
1eb4433ac451dc16f4133a88af2d002ac26c58ef 09-Sep-2009 Mike Stump <mrs@apple.com> Remove tabs, and whitespace cleanups.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81346 91177308-0d34-0410-b5e6-96231b3b80d8
6cb7c1a43b0c8f739d1f54b7fdae5ede86033496 23-Aug-2009 Benjamin Kramer <benny.kra@googlemail.com> Replace cerr with errs().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@79854 91177308-0d34-0410-b5e6-96231b3b80d8
9378ba44b3f46d697653003c784be87746e138d2 20-Apr-2009 Douglas Gregor <dgregor@apple.com> Move the on-disk hash table code into its own header. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69580 91177308-0d34-0410-b5e6-96231b3b80d8
22f0d09298ef05136f1ef6ebc4469b482cbd73e7 22-Mar-2009 Ted Kremenek <kremenek@apple.com> PTHManager::Create():
- Make the Diagnostic::Level for PTH errors to be specified by the caller

clang (driver):
- Set the PTHManager diagnostic level to "Diagnostic::Error" for -include-pth
(a hard error) and Diagnostic::Warning for -token-cache (we can still

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67462 91177308-0d34-0410-b5e6-96231b3b80d8
783bb7494c0551d1826afa5c005cc4a9afb16b6a 21-Mar-2009 Ted Kremenek <kremenek@apple.com> Add back warning about a PTH file not containing any identifiers, but don't make
it a hard error.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67424 91177308-0d34-0410-b5e6-96231b3b80d8
8fbc44d540eb7b9dabef71ba616c9102a2eaa71a 21-Mar-2009 Ted Kremenek <kremenek@apple.com> Allow PTH files with no identifiers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67423 91177308-0d34-0410-b5e6-96231b3b80d8
1d20197cb9478f3db17cbcfc57a1f8182843fc6a 20-Mar-2009 Ted Kremenek <kremenek@apple.com> PTHManager::Create() now creates a PTHManager even if the PTH file contains no
cached tokens. This is for use with -include-pth.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67385 91177308-0d34-0410-b5e6-96231b3b80d8
68228634016f644a1164fa1f024a9ce2093656bf 19-Mar-2009 Ted Kremenek <kremenek@apple.com> Add PTHManager::getOriginalSourceFile(), a method that returns the name of the
original source file (if any) that was used to generate the PTH cache.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67343 91177308-0d34-0410-b5e6-96231b3b80d8
d578569d50adc6e9b1a2f91931add6b471b2378e 24-Feb-2009 Ted Kremenek <kremenek@apple.com> Move PTHStatCache within the anonymous namespace.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65348 91177308-0d34-0410-b5e6-96231b3b80d8
640f552acea46ce7a5679175d6028178151e20ad 19-Feb-2009 Ted Kremenek <kremenek@apple.com> Fix another PTH warning that should not be a note.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65072 91177308-0d34-0410-b5e6-96231b3b80d8
395c59c690db303fa016e64ac22151a29fbe4197 19-Feb-2009 Ted Kremenek <kremenek@apple.com> Make PTH warnings actual warnings instead of 'notes'.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65071 91177308-0d34-0410-b5e6-96231b3b80d8
ad6ce5cb0e3b5dd853d2b1e337797a6fc1091e21 13-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Cache directory and negative 'stat' calls. This gives us a 1% performance improvement on Cocoa.h (fsyntax-only+PTH).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64490 91177308-0d34-0410-b5e6-96231b3b80d8
a4b44dd9d30929a35c44d85102e5241ee847b2f2 13-Feb-2009 Ted Kremenek <kremenek@apple.com> Add some boilerplate to the PTH file to prepare for the caching of stats for directories (and negative stats too).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64477 91177308-0d34-0410-b5e6-96231b3b80d8
03a2807021acd594add9d4ab5079cc740806b983 13-Feb-2009 Eli Friedman <eli.friedman@gmail.com> Fix gcc warning: gcc correctly notes that const-qualifying the return
type doesn't do anything.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64424 91177308-0d34-0410-b5e6-96231b3b80d8
82320e94606cfb0f68c6049c070c3600f8df9081 12-Feb-2009 Daniel Dunbar <daniel@zuster.org> Fix assertion when input is an empty string.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64397 91177308-0d34-0410-b5e6-96231b3b80d8
5f747d187bf8dc779a4acc180f9c21f1dbf83f6a 12-Feb-2009 Ted Kremenek <kremenek@apple.com> Re-enable PTH stat caching. All tests pass now.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64356 91177308-0d34-0410-b5e6-96231b3b80d8
d69ab87dd98dc495fe5731a2a26e47b36a06834e 12-Feb-2009 Ted Kremenek <kremenek@apple.com> Fix bad reading of bytes in ReadUnalignedLE64() (copy-paste error).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64355 91177308-0d34-0410-b5e6-96231b3b80d8
c8b44f59979994c7a2e0865144615cb04f7fcd1d 12-Feb-2009 Ted Kremenek <kremenek@apple.com> Temporarily disable PTH stat caching as it appears to be failing on some machines.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64354 91177308-0d34-0410-b5e6-96231b3b80d8
337edcdbec05316b407d0d64865c88ff8597d910 12-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Cache stat information for files in the PTH file. Hook up FileManager
to use this stat information in the PTH file using a 'StatSysCallCache' object.

Performance impact (Cocoa.h, PTH):
- number of stat calls reduces from 1230 to 425
- fsyntax-only: time improves by 4.2%

We can reduce the number of stat calls to almost zero by caching negative stat
calls and directory stat calls in the PTH file as well.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64353 91177308-0d34-0410-b5e6-96231b3b80d8
a4bd8eb4d6d4b625f6bbb62fc180b02eab6433ed 12-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Have meta data be at the beginning of the PTH file, not the end.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64338 91177308-0d34-0410-b5e6-96231b3b80d8
7e3a004c6ed1fe87912203b9c5a113f8da89d261 11-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Replace string identifier to persistent ID lookup with a hashtable. This is
actually *slightly* slower than the binary search. Since this is algorithmically
better, further performance tuning should be able to make this faster.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64326 91177308-0d34-0410-b5e6-96231b3b80d8
f1de4649e917f891d24e0718d02ee904b9edbe9d 11-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Don't emit the PTH offset of the IdentifierInfo string data as that data is
referenced by other tables.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64304 91177308-0d34-0410-b5e6-96231b3b80d8
d8c02929fe70f03111be73e7b8c402c724238ee9 10-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Replace ad hoc 'file name' -> 'PTH data' lookup table in the PTH file with an on-disk chained hash table. This data structure is implemented using templates, and will be used to replace similar data structures. This change leads to no visibile performance impact on Cocoa.h, but now we only pay a price for the table on order with the number of files accessed and not the number in the PTH file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64245 91177308-0d34-0410-b5e6-96231b3b80d8
26555b18aa2c3b78744e77927acd3faa53ae7369 28-Jan-2009 Ted Kremenek <kremenek@apple.com> Add more PTH diagnostics for invalid PTH files, etc.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63232 91177308-0d34-0410-b5e6-96231b3b80d8
8a6aec620dbec1f292fe4116c0373ac81ab90234 28-Jan-2009 Ted Kremenek <kremenek@apple.com> Enhance PTHManager::Create() to take an optional Diagnostic* argument that can be used to report issues such as a missing PTH file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63231 91177308-0d34-0410-b5e6-96231b3b80d8
277faca30c9f8f72b79f55695cbe3395ec246e7c 27-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Use Token::setLiteralData() to directly store a pointer to cached spelling data in the PTH file. This removes a ton of code for looking up spellings using sourcelocations in the PTH file. This simplifies both PTH-generation and reading.

Performance impact for -fsyntax-only on Cocoa.h (with Cocoa.h in the PTH file):
- PTH generation time improves by 5%
- PTH reading improves by 0.3%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63072 91177308-0d34-0410-b5e6-96231b3b80d8
4adc71ae2cfc190f8d2cf58876e2a7893aa74ee0 26-Jan-2009 Ted Kremenek <kremenek@apple.com> Silence warning.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63054 91177308-0d34-0410-b5e6-96231b3b80d8
67d15050bbea16ae256e204ecd464f2e454c3c99 26-Jan-2009 Ted Kremenek <kremenek@apple.com> Add version number checking to PTH files.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63047 91177308-0d34-0410-b5e6-96231b3b80d8
e1b6498c41b94c3bc5cede17b0702282543385ef 26-Jan-2009 Ted Kremenek <kremenek@apple.com> Embed the offset of the PTH table inside the prologue of the PTH file. This will help improve gradual versioning of PTH files instead of relying that the PTH table is at a fixed offset.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63045 91177308-0d34-0410-b5e6-96231b3b80d8
de7aeefc5573d669ed476d7bda7a8940d3bcadb7 26-Jan-2009 Chris Lattner <sabre@nondot.org> Check in the long promised SourceLocation rewrite. This lays the
ground work for implementing #line, and fixes the "out of macro ID's"

There is nothing particularly tricky about the code, other than the
very performance sensitive SourceManager::getFileID() method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62978 91177308-0d34-0410-b5e6-96231b3b80d8
863c486fcb6162495a94fddf7ac8409de2638995 23-Jan-2009 Chris Lattner <sabre@nondot.org> This is a follow-up to r62675:

Refactor how the preprocessor changes a token from being an tok::identifier to a
keyword (e.g. tok::kw_for). Instead of doing this in HandleIdentifier, hoist this
common case out into the caller, so that every keyword doesn't have to go through
HandleIdentifier. This drops time in HandleIdentifier from 1.25ms to .62ms, and
speeds up clang -Eonly with PTH by about 1%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62855 91177308-0d34-0410-b5e6-96231b3b80d8
fbc33388c199d6f731170bf55719d57373a09c1f 23-Jan-2009 Chris Lattner <sabre@nondot.org> Update comment.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62819 91177308-0d34-0410-b5e6-96231b3b80d8
6f78c3b8b9343e7e9fbf2d457cccf00df6da5d47 23-Jan-2009 Chris Lattner <sabre@nondot.org> remove my gross #ifdef's, using portable abstractions now that the 32-bit
load is always aligned.

I verified that the bswap doesn't occur in the assembly code on x86.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62815 91177308-0d34-0410-b5e6-96231b3b80d8
5ff4317536dbd7f03332bb250c8b35ec04a6f5db 22-Jan-2009 Chris Lattner <sabre@nondot.org> remove Read8/Read24, which are dead. Rename Read16/Read32 to be more

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62775 91177308-0d34-0410-b5e6-96231b3b80d8
b248d53f2599d8e7b53b144b713e163ca521ffca 21-Jan-2009 Ted Kremenek <kremenek@apple.com> Fix <rdar://problem/6512717> by correctly reading the right offset in the token data in PTHLexer::getSourceLocation().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62725 91177308-0d34-0410-b5e6-96231b3b80d8
d0a69696acca62798dfc8b98f97c92bfa7fa0490 21-Jan-2009 Chris Lattner <sabre@nondot.org> merge two checks for identifiers in the pth loop into one.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62677 91177308-0d34-0410-b5e6-96231b3b80d8
6a170eb3ea6d6319277becabef68eb1a26bf8766 21-Jan-2009 Chris Lattner <sabre@nondot.org> Add a bit to IdentifierInfo that acts as a simple predicate which
tells us whether Preprocessor::HandleIdentifier needs to be called.
Because this method is only rarely needed, this saves a call and a
bunch of random checks. This drops the time in HandleIdentifier
from 3.52ms to .98ms on cocoa.h on my machine.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62675 91177308-0d34-0410-b5e6-96231b3b80d8
cdd8f2153e18796e9e2a126ebcbd4f3e1bd7135b 21-Jan-2009 Ted Kremenek <kremenek@apple.com> Don't crash on empty PTH files. This fixes <rdar://problem/6512714>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62673 91177308-0d34-0410-b5e6-96231b3b80d8
6353bebfc2d6d19e15f1b2c673507cdb6985270f 21-Jan-2009 Chris Lattner <sabre@nondot.org> really we only need on Read24!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62672 91177308-0d34-0410-b5e6-96231b3b80d8
aff6ef8e7bc3c3739f984c390e0af693e60be064 21-Jan-2009 Chris Lattner <sabre@nondot.org> revert my previous patch, it assumed endianness.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62671 91177308-0d34-0410-b5e6-96231b3b80d8
2f6572bafa2ee53e1ac288934aae8a3129ec03b1 21-Jan-2009 Chris Lattner <sabre@nondot.org> minor cleanups: now that tokens are 4-byte aligned in a PTH
file, just load them directly as ints.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62668 91177308-0d34-0410-b5e6-96231b3b80d8
ea9c26b3dbd74a1497f5609ae6e19a85f42b6073 21-Jan-2009 Ted Kremenek <kremenek@apple.com> Fix: <rdar://problem/6510344> [pth] PTH slows down regular lexer considerably (when it has substantial work)

Changes to IdentifierTable:
- High-level summary: StringMap never owns IdentifierInfos. It just
references them.
- The string map now has StringMapEntry<IdentifierInfo*> instead of
StringMapEntry<IdentifierInfo>. The IdentifierInfo object is
allocated using the same bump pointer allocator as used by the

Changes to IdentifierInfo:
- Added an extra pointer to point to the
StringMapEntry<IdentifierInfo*> in the string map. This pointer
will be null if the IdentifierInfo* is *only* used by the PTHLexer
(that is it isn't in the StringMap).

Algorithmic changes:
- Non-PTH case:
IdentifierInfo::get() will always consult the StringMap first to
see if we have an IdentifierInfo object. If that StringMapEntry
references a null pointer, we allocate a new one from the BumpPtrAllocator
and update the reference in the StringMapEntry.
- PTH case:
We do the same lookup as with the non-PTH case, but if we don't get
a hit in the StringMap we do a secondary lookup in the PTHManager for
the IdentifierInfo. If we don't find an IdentifierInfo we create a
new one as in the non-PTH case. If we do find and IdentifierInfo
in the PTHManager, we update the StringMapEntry to refer to it so
that the IdentifierInfo will be found on the next StringMap lookup.
This way we only do a binary search in the PTH file at most once
for a given IdentifierInfo. This greatly speeds things up for source
files containing a non-trivial amount of code.

Performance impact:
While these changes do add some extra indirection in
IdentifierTable to access an IdentifierInfo*, I saw speedups even
in the non-PTH case as well.

Non-PTH: For -fsyntax-only on Cocoa.h, we see a 6% speedup.
PTH (with Cocoa.h in token cache): 11% speedup.

I also did an experiment where we did -fsyntax-only on a source file
including a large header and Cocoa.h, but the token cache did not
contain the larger header. For this file, we were seeing a performance
*regression* when using PTH of 3% over non-PTH. Now we are seeing
a performance improvement of 9%!

The serialization tests are now failing. I looked at this extensively,
and I my belief is that this change is unmasking a bug rather than
introducing a new one. I have disabled the serialization tests for now.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62636 91177308-0d34-0410-b5e6-96231b3b80d8
7b78b7c6d96deb1e63f8d0655ee6fa53de0b65ef 20-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Emitted tokens now consist of 12 bytes that are loaded used 3 32-bit loads. This reduces user time but increases system time because of the slightly larger PTH file. Although there is no performance win on Cocoa.h and -Eonly, overall this seems like a good step.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62542 91177308-0d34-0410-b5e6-96231b3b80d8
77ecb3a28f21496ecfdbb3d5f5b66b0d2abf48c9 18-Jan-2009 Chris Lattner <sabre@nondot.org> rearrange GetIdentifierInfo so that the fast path can be partially inlined into PTHLexer::Lex. This speeds up the user time of PTH -Eonly by another 2ms (4.4%)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62454 91177308-0d34-0410-b5e6-96231b3b80d8
898a0bb1972efb6e03cb1151412ec7392cef07de 18-Jan-2009 Chris Lattner <sabre@nondot.org> rename some variables, only set a tokens identifierinfo if non-null.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62450 91177308-0d34-0410-b5e6-96231b3b80d8
f15674c680730c652a37a16a5d3f3ff429b0c308 18-Jan-2009 Chris Lattner <sabre@nondot.org> On i386 and x86-64, just do unaligned loads
instead of assembling from bytes. This speeds up -Eonly PTH reading
of cocoa.h by about 2ms, which is 4.2%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62447 91177308-0d34-0410-b5e6-96231b3b80d8
1b5285e1ba31975864da356b2ed927e87670e654 18-Jan-2009 Chris Lattner <sabre@nondot.org> switch PTHLexer to use Read32 and friends instead of lots of inlined
copies. I verified that this causes no performance change in PTH.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62445 91177308-0d34-0410-b5e6-96231b3b80d8
da9d61c96c412f6babc7f824152609562f302388 18-Jan-2009 Chris Lattner <sabre@nondot.org> switch PTH lexer from using "const char*"s to "const unsigned char*"s
internally. This is just a cleanup that reduces the need to cast to
unsigned char before assembling a larger integer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62442 91177308-0d34-0410-b5e6-96231b3b80d8
f056d92e182cbe4f62c8d14102544dc38066dabc 17-Jan-2009 Chris Lattner <sabre@nondot.org> simplify PTHManager::CreateLexer

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62424 91177308-0d34-0410-b5e6-96231b3b80d8
6b7b84036126a8f2a3e312cfec0a9a4f8ae9d8ed 17-Jan-2009 Chris Lattner <sabre@nondot.org> suck the call to "getSpellingLoc" that all clients do into
the implementation of PTHManager::getSpelling.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62408 91177308-0d34-0410-b5e6-96231b3b80d8
2b2453a7d8fe732561795431f39ceb2b2a832d84 17-Jan-2009 Chris Lattner <sabre@nondot.org> this massive patch introduces a simple new abstraction: it makes
"FileID" a concept that is now enforced by the compiler's type checker
instead of yet-another-random-unsigned floating around.

This is an important distinction from the "FileID" currently tracked by
SourceLocation. *That* FileID may refer to the start of a file or to a
chunk within it. The new FileID *only* refers to the file (and its
#include stack and eventually #line data), it cannot refer to a chunk.

FileID is a completely opaque datatype to all clients, only SourceManager
is allowed to poke and prod it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62407 91177308-0d34-0410-b5e6-96231b3b80d8
df7c17a8d02fe09a3466786bae3e40fc3252687a 16-Jan-2009 Chris Lattner <sabre@nondot.org> Change some terminology in SourceLocation: instead of referring to
the "physical" location of tokens, refer to the "spelling" location.
This is more concrete and useful, tokens aren't really physical objects!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62309 91177308-0d34-0410-b5e6-96231b3b80d8
e1deaac73379eeb9864215c7979f6005ebd74cef 15-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Fix termination condition in binary search.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62277 91177308-0d34-0410-b5e6-96231b3b80d8
72b1b15ee88aac0a63e2c1dc53fe22f5ab297b20 15-Jan-2009 Ted Kremenek <kremenek@apple.com> IdentifierInfo:
- IdentifierInfo can now (optionally) have its string data not be
co-located with itself. This is for use with PTH. This aspect is a
little gross, as getName() and getLength() now make assumptions
about a possible alternate representation of IdentifierInfo.
Perhaps we should make IdentifierInfo have virtual methods?

- Added class "IdentifierInfoLookup" that can be used by
IdentifierTable to perform "string -> IdentifierInfo" lookups using
an auxilliary data structure. This is used by PTH.
- Perform tests show that IdentifierTable::get() does not slow down
because of the extra check for the IdentiferInfoLookup object (the
regular StringMap lookup does enough work to mitigate the impact of
an extra null pointer check).
- The upshot is that now that some IdentifierInfo objects might be
owned by the IdentiferInfoLookup object. This should be reviewed.

- Modified PTHManager::GetIdentifierInfo to *not* insert entries in
IdentifierTable's string map, and instead create IdentifierInfo
objects on the fly when mapping from persistent IDs to
IdentifierInfos. This saves a ton of work with string copies,
hashing, and StringMap lookup and resizing. This change was
motivated because when processing source files in the PTH cache we
don't need to do any string -> IdentifierInfo lookups.
- PTHManager now subclasses IdentifierInfoLookup, allowing clients of
IdentifierTable to transparently use IdentifierInfo objects managed
by the PTH file. PTHManager resolves "string -> IdentifierInfo"
queries by doing a binary search over a sorted table of identifier
strings in the PTH file (the exact algorithm we use can be changed
as needed).

These changes lead to the following performance changes when using PTH on Cocoa.h:
- fsyntax-only: 10% performance improvement
- Eonly: 30% performance improvement

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62273 91177308-0d34-0410-b5e6-96231b3b80d8
293b4af04ca37576d4b228adf09acd6cee3ddb16 15-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Embed a persistentID side-table in the PTH file that is sorted in the
lexical order of the corresponding identifier strings. This will be used for a
forthcoming optimization. This slows down PTH generation time by 7%. We can
revert this change if the optimization proves to not be valuable.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62248 91177308-0d34-0410-b5e6-96231b3b80d8
28396608ec20d44e9d1470e1ea51689bb504d0de 14-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH:
- Use canonical FileID when using getSpelling() caching. This
addresses some cache misses we were seeing with -fsyntax-only on
- Added Preprocessor::getPhysicalCharacterAt() utility method for
clients to grab the first character at a specified sourcelocation.
This uses the PTH spelling cache.
- Modified Sema::ActOnNumericConstant() to use
Preprocessor::getPhysicalCharacterAt() instead of
SourceManager::getCharacterData() (to get PTH hits).

These changes cause -fsyntax-only to not page in any sources from
Cocoa.h. We see a speedup of 27%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62193 91177308-0d34-0410-b5e6-96231b3b80d8
7bd88aa620406ab99547169b973de085785a0ca2 13-Jan-2009 Ted Kremenek <kremenek@apple.com> Fix corner cases in PTH getSpelling() binary search.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62187 91177308-0d34-0410-b5e6-96231b3b80d8
f02f6f0ee3e05b958bcf67f00f4503671d67eccd 13-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Fix remaining cases where the spelling cache in the PTH file was being missed when it shouldn't. This shaves another 7% off PTH time for -Eonly on Cocoa.h

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62186 91177308-0d34-0410-b5e6-96231b3b80d8
5f074266cc59563036c40516c814d63825723e20 09-Jan-2009 Ted Kremenek <kremenek@apple.com> Enhance PTH 'getSpelling' caching:
- Refactor caching logic into a helper class PTHSpellingSearch
- Allow "random accesses" in the spelling cache, thus catching the remaining
cases where 'getSpelling' wasn't hitting the PTH cache

For -Eonly, PTH, Cocoa.h:
- This reduces wall time by 3% (user time unchanged, sys time reduced)
- This reduces the amount of paged source by 1112K.
The remaining 1112K still being paged in is from somewhere else

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62009 91177308-0d34-0410-b5e6-96231b3b80d8
defb7094c835998bb821e894253287625ce8c74d 09-Jan-2009 Ted Kremenek <kremenek@apple.com> Invert assertion condition.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61961 91177308-0d34-0410-b5e6-96231b3b80d8
32a8ad526f9bc00539f000a2dd1ac3e167db61c1 08-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Hook up getSpelling() caching in PTHLexer. This results in a nice
performance gain. Here's what we see for -Eonly on Cocoa.h (using PTH):

- wall time decreases by 21% (26% speedup overall)
- system time decreases by 35%
- user time decreases by 6%

These reductions are due to not paging source files just to get spellings for
literals. The solution in place doesn't appear to be 100% yet, as we still see
some of the pages for source files getting mapped in. Using -print-stats, we see
that SourceManager maps in 7179K less bytes of source text (reduction of 75%).
Will investigate why the remaining 25% are getting paged in.

With these changes, here's how PTH compares to non-PTH on Cocoa.h:
-Eonly: PTH takes 64% of the time as non-PTH (54% speedup)
-fsyntax-only: PTH takes 89% of the time as non-PTH (11% speedup)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61913 91177308-0d34-0410-b5e6-96231b3b80d8
b70e3dafb9618f34017061400dc19ac5e3539a6d 08-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH:
- Added stub PTHLexer::getSpelling() that will be used for fetching cached
spellings from the PTH file. This doesn't do anything yet.
- Added a hook in Preprocessor::getSpelling() to call PTHLexer::getSpelling()
when using a PTHLexer.
- Updated PTHLexer to read the offsets of spelling tables in the PTH file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61911 91177308-0d34-0410-b5e6-96231b3b80d8
59d08cb672136322375e5400578ee1fbd0947de2 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH: Remove some methods and simplify some conditions in PTHLexer::Lex(). No big functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61381 91177308-0d34-0410-b5e6-96231b3b80d8
18d9afb815bd8aff885dd64c5078760b3398d7be 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH: Use 3 bytes instead of 4 bytes to encode the persistent ID for a token.
- This reduces the PTH size for Cocoa.h by 7%.
- The increases PTH -Eonly speed for Cocoa.h by 0.8%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61377 91177308-0d34-0410-b5e6-96231b3b80d8
6b1c9708c8fca3786c766b6d1869721656b31322 23-Dec-2008 Ted Kremenek <kremenek@apple.com> Cosmetics: rename a variable and tighten spacing. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61375 91177308-0d34-0410-b5e6-96231b3b80d8
8f174e173c1fa195151a86a8cbd7611dde5e2b1a 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
- Encode the token length with 2 bytes instead of 4.
- This reduces the size of the .pth file for Cocoa.h by 12%.
- This speeds up PTH time (-Eonly) on Cocoa.h by 1.6%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61364 91177308-0d34-0410-b5e6-96231b3b80d8
866bdf74547efe32c320554837ffce00fcc084fe 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
- In PTHLexer::Lex read all of the token data from PTH file before
constructing the token. The idea is to enhance locality.
- Do not use Read8/Read32 in PTHLexer::Lex. Inline these operations manually.
- Change PTHManager::ReadIdentifierInfo() to PTHManager::GetIdentifierInfo().
They are functionally the same except that PTHLexer::Lex() reads the
persistent id.

These changes result in a 3.3% speedup for PTH on Cocoa.h (-Eonly).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61363 91177308-0d34-0410-b5e6-96231b3b80d8
e5680f3cd678014cf0872d34726dc804b0cbbdd4 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
- Embed 'eom' tokens in PTH file.
- Use embedded 'eom' tokens to not lazily generate them in the PTHLexer.
This means that PTHLexer can always advance to the next token after
reading a token (instead of buffering tokens using a copy).
- Moved logic of 'ReadToken' into Lex. GetToken & ReadToken no longer exist.
- These changes result in a 3.3% speedup (-Eonly) on Cocoa.h.
- The code is a little gross. Many cleanups are possible and should be done.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61360 91177308-0d34-0410-b5e6-96231b3b80d8
80d2f3059326f99ebf7c867db1c7f106ec9485f5 18-Dec-2008 Ted Kremenek <kremenek@apple.com> Use '&' to test StartOfLine flag.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61205 91177308-0d34-0410-b5e6-96231b3b80d8
74c3e6e5e95af08096aab415d1ce15f15ffff02a 18-Dec-2008 Ted Kremenek <kremenek@apple.com> Rewrite PTHLexer::DiscardToEndOfLine() to not use GetToken and instead only read the bytes needed to determine if a token is not at the start of the line.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61172 91177308-0d34-0410-b5e6-96231b3b80d8
30a12ec2a7f331d9e08acabe7cda853aaa7ba54b 18-Dec-2008 Ted Kremenek <kremenek@apple.com> Change PTHLexer::getSourceLocation() to not call GetToken() and instead just read the file offset in the token data buffer directly.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61170 91177308-0d34-0410-b5e6-96231b3b80d8
b60d7999d621fce608e03d39e82c0e7eda750054 17-Dec-2008 Chris Lattner <sabre@nondot.org> add a dropped word back

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61152 91177308-0d34-0410-b5e6-96231b3b80d8
1c13c4f87ebdf4309d9b9b75336f15301b8ec8ce 17-Dec-2008 Ted Kremenek <kremenek@apple.com> Shadow CurPtr with a local variable in ReadToken.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61145 91177308-0d34-0410-b5e6-96231b3b80d8
41a2660377d215d004fe413c03874bd066b5384c 12-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH: Added minor 'sibling jumping' optimization for iterating over the side table used for fast preprocessor block skipping. This has a minor performance improvement when preprocessing Cocoa.h, but can have some wins in pathologic cases.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60966 91177308-0d34-0410-b5e6-96231b3b80d8
268ee7016a2811803989487c0ad3799486092c63 12-Dec-2008 Ted Kremenek <kremenek@apple.com> Added PTH optimization to not process entire blocks of tokens that appear in skipped preprocessor blocks. This improves PTH speed by 6%. The code for this optimization itself is not very optimized, and will get cleaned up.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60956 91177308-0d34-0410-b5e6-96231b3b80d8
fb645b6547b75ddc2e3c7ab2126ad8beeefca62d 12-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
- Added a side-table per each token-cached file with the preprocessor conditional stack. This tracks what #if's are matched with what #endifs and where their respective tokens are in the PTH file. This will allow for quick skipping of excluded conditional branches in the Preprocessor.
- Performance testing shows the addition of this information (without actually utilizing it) leads to no performance regressions.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60911 91177308-0d34-0410-b5e6-96231b3b80d8
cd223444d1680290efe11da657faafc9a1ac14ba 11-Dec-2008 Ted Kremenek <kremenek@apple.com> PTHLexer: Keep track of the location of the last '#' token and provide the means to jump ahead in the token stream.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60905 91177308-0d34-0410-b5e6-96231b3b80d8
7415326dcbba6f333f5f7708ac1bf4982f3adb5a 11-Dec-2008 Ted Kremenek <kremenek@apple.com> Remove unused ivar CurTokenIdx.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60896 91177308-0d34-0410-b5e6-96231b3b80d8
cf58e6249c6b018508e34bcb76202caa42d2451a 10-Dec-2008 Ted Kremenek <kremenek@apple.com> Declare PerIDCache as IdentifierInfo** instead of void*. This is just cleaner. No performance change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60843 91177308-0d34-0410-b5e6-96231b3b80d8
0e50b6e7c104d00614baa3d80df62f1630a94d9c 04-Dec-2008 Ted Kremenek <kremenek@apple.com> Remove unneeded assertion.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60559 91177308-0d34-0410-b5e6-96231b3b80d8
271b7af521b784ec505a37a29ac36ec596954a08 04-Dec-2008 Ted Kremenek <kremenek@apple.com> Use 'free' to release PerIDCache since it was allocated using calloc().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60556 91177308-0d34-0410-b5e6-96231b3b80d8
6183e4815a4019e97ad01bd880f12355599b75fd 03-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
Use an array instead of a DenseMap to cache persistent IDs -> IdentifierInfo*. This leads to a 4% speedup at -fsyntax-only using PTH.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60452 91177308-0d34-0410-b5e6-96231b3b80d8
0c6a77bc1f52f282a969538f139ebde429076ed3 03-Dec-2008 Ted Kremenek <kremenek@apple.com> - Remove PTHManager.cpp. Move all of its functions to PTHLexer.cpp since some of the internal methods are used by PTHLexer (their implementations are intertwined.) This enables some important inlining opportunities at -O3.

- Don't construct an std::vector<Token> prior to feeding PTH tokens to the Preprocessor. Stream them off the PTH file directly.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60447 91177308-0d34-0410-b5e6-96231b3b80d8
82a500b141b9a8001dac69f047478a43e2aebdff 27-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer now owns the Token vector.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60136 91177308-0d34-0410-b5e6-96231b3b80d8
d2bdeed0727c291a1a0037dc4fa80379dafa0687 22-Nov-2008 Ted Kremenek <kremenek@apple.com> In PTHLexer::DiscardToEndOfLine() use Lex() instead of AdvanceToken(). This handles transitions in the preprocessor state.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59845 91177308-0d34-0410-b5e6-96231b3b80d8
cd4e2aecde5bb7810715d5d5a88ac63ce7946f34 21-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer:
- Move out logic for handling the end-of-file to LexEndOfFile (to match the Lexer) class. The logic now mirrors the Lexer class more, which allows us to pass most of the Preprocessor test cases.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59768 91177308-0d34-0410-b5e6-96231b3b80d8
89d7ee9619d2dbdfa8d956a695c612a104a92cad 20-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer:
- Move PTHLexer::GetToken() to be inside PTHLexer.cpp.
- When lexing in raw mode, null out identifiers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59744 91177308-0d34-0410-b5e6-96231b3b80d8
31aba425a01c8c957e662ccfaa71f923d0f0932a 20-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer:
- Rename 'CurToken' and 'LastToken' to 'CurTokenIdx' and 'LastTokenIdx'
- Add helper methods GetToken(), AdvanceToken(), AtLastToken() to abstract away
details of the token stream. This also allows us to easily replace their
implementation later.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59733 91177308-0d34-0410-b5e6-96231b3b80d8
d6f53dc4951aace69014619761760addac9e59ec 20-Nov-2008 Ted Kremenek <kremenek@apple.com> Rewrote PTHLexer::Lex by digging through the sources of Lexer again. Now we can do basic macro expansion using the PTHLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59724 91177308-0d34-0410-b5e6-96231b3b80d8
452e37837a48b2f0ced144784277fd4d28cbede9 20-Nov-2008 Ted Kremenek <kremenek@apple.com> - Default initialize ParsingPreprocessorDirective, ParsingFilename, and
LexingRawMode in the ctor of PreprocessorLexer.

- PTHLexer: Use "LastToken" instead of "NumToken"

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59690 91177308-0d34-0410-b5e6-96231b3b80d8
4d35da2e41941965bbee8ed7e8c30e7c21000d71 20-Nov-2008 Ted Kremenek <kremenek@apple.com> Add (untested) implementation of PTHLexer::isNextPPTokenLParen() and PTHLexer::DiscardToEndOfLine().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59687 91177308-0d34-0410-b5e6-96231b3b80d8
2f1c0243f7fc5d12446efcb006c071854c345923 19-Nov-2008 Ted Kremenek <kremenek@apple.com> Add stub for PTHLexer::isNextPPTokenLParen().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59670 91177308-0d34-0410-b5e6-96231b3b80d8
17ff58a63197b398ae52697b088dc0fb8b255519 19-Nov-2008 Ted Kremenek <kremenek@apple.com> When using a PTHLexer, use DiscardToEndOfLine() instead of ReadToEndOfLine().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59668 91177308-0d34-0410-b5e6-96231b3b80d8
41938c8493b4380df738263166b746eacb33c309 19-Nov-2008 Ted Kremenek <kremenek@apple.com> - Move static function IsNonPragmaNonMacroLexer into Preprocessor.h.
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.
- Add 'FileID' to PreprocessorLexer, and have Preprocessor query this fileid
when looking up the FileEntry for a file

Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59666 91177308-0d34-0410-b5e6-96231b3b80d8
51881c56b0fb30eac731ae0f79cf430daa8bcb4a 16-Nov-2008 Chris Lattner <sabre@nondot.org> Trivial tidying

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59424 91177308-0d34-0410-b5e6-96231b3b80d8
d6a2e7d71e9a910c45a3eb0268a5d94635003d58 13-Nov-2008 Ted Kremenek <kremenek@apple.com> Move some diagnostic handling to PreprocessorLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59191 91177308-0d34-0410-b5e6-96231b3b80d8
274b20863a728cc6a31ee75c670e3733600c1531 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Add skeleton for PTH lexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59169 91177308-0d34-0410-b5e6-96231b3b80d8