• Home
  • History
  • Annotate
  • only in /external/clang/include/clang/Lex/
History log of /external/clang/include/clang/Lex/
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
ef8225444452a1486bd721f3285301fe84643b00 21-Jul-2014 Stephen Hines <srhines@google.com> Update Clang for rebase to r212749.

This also fixes a small issue with arm_neon.h not being generated always.

Includes a cherry-pick of:
r213450 - fixes mac-specific header issue
r213126 - removes a default -Bsymbolic on Android

Change-Id: I2a790a0f5d3b2aab11de596fc3a74e7cbc99081d
6bcf27bb9a4b5c3f79cb44c0e4654a6d7619ad89 29-May-2014 Stephen Hines <srhines@google.com> Update Clang for 3.5 rebase (r209713).

Change-Id: I8c9133b0f8f776dc915f270b60f94962e771bc83
651f13cea278ec967336033dd032faef0e9fc2ec 24-Apr-2014 Stephen Hines <srhines@google.com> Updated to Clang 3.5a.

Change-Id: I8127eb568f674c2e72635b639a3295381fe8af82
1fab7c3e3bd97a909a80b1bfea1909c6e7347fc0 12-Feb-2014 Stephen Hines <srhines@google.com> Merge remote-tracking branch 'upstream/release_34' into merge-20140211


Change-Id: I17ca7161f32007272ee82036d237d051847dd02e
3347b497157d36cf280d1d9f52956faa6e702f34 12-Nov-2013 Richard Smith <richard-llvm@metafoo.co.uk> Rather than duplicating extension diagnostics to allow them to cause a
substitution failure, allow a flag to be set on the Diagnostic object,
to mark it as 'causes substitution failure'.

Refactor Diagnostic.td and the tablegen to use an enum for SFINAE behavior
rather than a bunch of flags.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@194444 91177308-0d34-0410-b5e6-96231b3b80d8
a5ba7b405dff48a5a0a63fff32c99eedb044842b 07-Nov-2013 Douglas Gregor <dgregor@apple.com> Modules: Teach the preprocessor to recognize 'import' only after an '@'.

The preprocessor currently recognizes module declarations to load a
module based on seeing the 'import' keyword followed by an
identifier. This sequence is fairly unlikely in C (one would need a
type named 'import'), but is more common in Objective-C (where a
variable named 'import' can cause problems). Since import declarations
currently require a leading '@', recognize that in the preprocessor as
well. Fixes <rdar://problem/15084587>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@194225 91177308-0d34-0410-b5e6-96231b3b80d8
527dcd47caa224b673d9af40fa422caf9d2b04fe 05-Nov-2013 Todd Fiala <tfiala@google.com> Fixes for LLDB build to work around host 4.6.2+ compiler issues.

These fixes to the clang source add manual copy constructor and operator=()
methods for classes that use member bitfields and are used in templated

The intent is to keep this change (and related LLVM and LLDB changes)
local to android only until we either fix the compiler or use a new one
for host executable builds.

Change-Id: Ib590d78a4280bce3291a0cc82275da783393181e
ee0cd37fe4a9f4e2ee73ae34cf93c410cb299a82 24-Oct-2013 Manuel Klimek <klimek@google.com> Use the same SourceManager for ModuleMaps and compilations.

This allows using virtual file mappings on the original SourceManager to
map in virtual module.map files. Without this patch, the ModuleMap
search will find a module.map file (as the FileEntry exists in the
FileManager), but will be unable to get the content from the
SourceManager (as ModuleMap previously created its own SourceManager).

Two problems needed to be fixed which this patch exposed:

1. Storing the inferred module map
When writing out a module, the ASTWriter stores the names of the files
in the main source manager; when loading the AST again, the ASTReader
errs out if such a file is found missing, unless it is overridden.
Previously CompilerInstance's compileModule method would store the
inferred module map to a temporary file; the problem with this approach
is that now that the module map is handled by the main source manager,
the ASTWriter stores the name of the temporary module map as source to
the compilation; later, when the module is loaded, the temporary file
has already been deleted, which leads to a compilation error. This patch
changes the inferred module map to instead inject a virtual file into
the source manager. This both saves some disk IO, and works with how the
ASTWriter/ASTReader handle overridden source files.

2. Changing test input in test/Modules/Inputs/*
Now that the module map file is handled by the main source manager, the
VerifyDiagnosticConsumer will not ignore diagnostics created while
parsing the module map file. The module test test/Modules/renamed.m uses
-I test/Modules/Inputs and triggers recursive loading of all module maps
in test/Modules/Inputs, some of which had conflicting names, thus
leading errors while parsing the module maps. Those diagnostics already
occur on trunk, but before this patch they would not break the test, as
they were ignored by the VerifyDiagnosticConsumer. This patch thus
changes the module maps that have been recently introduced which broke
the invariant of compatible modules maps in test/Modules/Inputs.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@193314 91177308-0d34-0410-b5e6-96231b3b80d8
c641709607d45bf97772e925647db6c94866c50a 22-Oct-2013 Daniel Jasper <djasper@google.com> Allow a header to be part of multiple modules.

This patch changes two things:

a) Allow a header to be part of multiple modules. The reasoning is that
in existing codebases that have a module-like build system, the same
headers might be used in several build targets. Simple reasons might be
that they defined different classes that are declared in the same
header. Supporting a header as a part of multiple modules will make the
transistion easier for those cases. A later step in clang can then
determine whether the two modules are actually compatible and can be
merged and error out appropriately. The later check is similar to what
needs to be done for template specializations anyway.

b) Allow modules to be stored in a directory tree separate from the
headers they describe.

Review: http://llvm-reviews.chandlerc.com/D1951

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@193151 91177308-0d34-0410-b5e6-96231b3b80d8
a0950e83cae5df77c00d02d7f77a166979cd2b8b 12-Oct-2013 Pekka Jaaskelainen <pekka.jaaskelainen@tut.fi> Callback support for OpenCL extension pragmas.

Patch from Rami Ylimäki and Mikael Lepistö!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@192531 91177308-0d34-0410-b5e6-96231b3b80d8
064c3f1ed782eb641df6bc00beec069b3c80fa8a 26-Sep-2013 Richard Smith <richard-llvm@metafoo.co.uk> Replace a bool with an enum for clarity, based on review comment from James Dennett.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@191420 91177308-0d34-0410-b5e6-96231b3b80d8
859b6227694033dd6eaf3991a2b80877a406c382 26-Sep-2013 Richard Smith <richard-llvm@metafoo.co.uk> Implement C++1y digit separator proposal (' as a digit separator). This is not
yet approved by full committee, but was unanimously supported by EWG.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@191417 91177308-0d34-0410-b5e6-96231b3b80d8
1b8840ce0d7a3383915b29dc3779dfadec817458 24-Sep-2013 Daniel Jasper <djasper@google.com> Add -fmodule-map-file option.

With this option, arbitrarily named module map files can be specified
to be loaded as required for headers in the respective (sub)directories.

This, together with the extern module declaration allows for specifying
module maps in a modular fashion without the need for files called

Among other things, this allows a directory to contain two modules that
are completely independent of one another.

Review: http://llvm-reviews.chandlerc.com/D1697.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@191284 91177308-0d34-0410-b5e6-96231b3b80d8
ddd2dfc1d3f4a36cbe8cd775c588623a17049f9f 24-Sep-2013 Daniel Jasper <djasper@google.com> Module use declarations (II)

Review: http://llvm-reviews.chandlerc.com/D1546.

I have picked up this patch form Lawrence
(http://llvm-reviews.chandlerc.com/D1063) and did a few changes.

From the original change description (updated as appropriate):
This patch adds a check that ensures that modules only use modules they
have so declared. To this end, it adds a statement on intended module
use to the module.map grammar:

use module-id

A module can then only use headers from other modules if it 'uses' them.
This enforcement is off by default, but may be turned on with the new
option -fmodules-decluse.

When enforcing the module semantics, we also need to consider a source
file part of a module. This is achieved with a compiler option


The compiler at present only applies restrictions to the module directly
being built.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@191283 91177308-0d34-0410-b5e6-96231b3b80d8
d2f9308220af22bfc1bcd3bc2cad118dbd8be013 19-Sep-2013 Eli Friedman <eli.friedman@gmail.com> Make Preprocessor::Lex non-recursive.

Before this patch, Lex() would recurse whenever the current lexer changed (e.g.
upon entry into a macro). This patch turns the recursion into a loop: the
various lex routines now don't return a token when the current lexer changes,
and at the top level Preprocessor::Lex() now loops until it finds a token.
Normally, the recursion wouldn't end up being very deep, but the recursion depth
can explode in edge cases like a bunch of consecutive macros which expand to
nothing (like in the testcase test/Preprocessor/macro_expand_empty.c in this


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@190980 91177308-0d34-0410-b5e6-96231b3b80d8
2ee042dee9f7693665e28463955401905474a284 14-Sep-2013 Reid Kleckner <reid@kleckner.net> Lex and ignore Microsoft's #pragma warning(...)

This fixes PR17145 and avoids unknown pragma warnings.

This change does not attempt to map MSVC warning numbers to clang
warning flags. Perhaps in the future we will implement a mapping for
some common subset of Microsoft warnings, but for now we don't.

Reviewers: rsmith

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D1652

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@190726 91177308-0d34-0410-b5e6-96231b3b80d8
344472ebeded2fca2ed5013b9e87f81d09bfa908 23-Aug-2013 Robert Wilhelm <robert.wilhelm@gmx.net> Use pop_back_val() instead of both back() and pop_back().
No functionality change intended.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@189112 91177308-0d34-0410-b5e6-96231b3b80d8
f0dd853bb1e8f3e59b169e6d34a8556c6003c47c 20-Aug-2013 Fariborz Jahanian <fjahanian@apple.com> ObjectiveC migrator: More work towards
insertion of ObjC audit pragmas.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@188733 91177308-0d34-0410-b5e6-96231b3b80d8
056ec12ca14ee5014ea49786f719ceaa299a7e19 05-Aug-2013 Daniel Jasper <djasper@google.com> Add option to disable module loading.

This patch was created by Lawrence Crowl and reviewed in:

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@187738 91177308-0d34-0410-b5e6-96231b3b80d8
4ac537b0f07f2efb9fcf081f60d54e6cfb1cf2d5 23-Jul-2013 Richard Smith <richard-llvm@metafoo.co.uk> C++1y literal suffix support:
* Allow ns, us, ms, s, min, h as numeric ud-suffixes
* Allow s as string ud-suffix

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@186933 91177308-0d34-0410-b5e6-96231b3b80d8
0189fd61ec72d86e008aa5615be80581f84cf703 20-Jul-2013 Enea Zaffanella <zaffanella@cs.unipr.it> Added preproc callback for pragma directives.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@186770 91177308-0d34-0410-b5e6-96231b3b80d8
6fb63ab0c74bb777772a3c3ecec09799e275599a 19-Jul-2013 John Thompson <John.Thompson.JTSoftware@gmail.com> Add SourceRange to PPCallbacks::Defined callback.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@186707 91177308-0d34-0410-b5e6-96231b3b80d8
6a04b5e8c4f15c724b8b89a9c0ea702586ad1852 18-Jul-2013 John Thompson <John.Thompson.JTSoftware@gmail.com> Add condition expression result to if and elif callbacks.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@186547 91177308-0d34-0410-b5e6-96231b3b80d8
09d19efaa147762f84aed55efa7930bb3616a4e5 04-Jul-2013 Craig Topper <craig.topper@gmail.com> Use SmallVectorImpl instead of SmallVector for iterators and references to avoid specifying the vector size unnecessarily.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@185610 91177308-0d34-0410-b5e6-96231b3b80d8
2ccf698c58f8aca1bf784998f491ef004351db64 26-Jun-2013 Nico Weber <nicolasweber@gmx.de> Fix warning: commas at the end of enumerator lists are a C++11 extension [-Wc++11-extensions]

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@184972 91177308-0d34-0410-b5e6-96231b3b80d8
11be06464a338b7c6e007898a7709498dc456db2 26-Jun-2013 Reid Kleckner <reid@kleckner.net> Match MSVC's handling of commas during macro argument expansion

This allows clang to parse the type_traits header in Visual Studio 2012,
which is included widely in practice.

This is a rework of r163022 by João Matos. The original patch broke
preprocessing of gtest headers, which this patch addresses.

Patch by Will Wilson!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@184968 91177308-0d34-0410-b5e6-96231b3b80d8
8f5d7d1d1f990f174c7f2682271a83acf64dd93d 21-Jun-2013 Douglas Gregor <dgregor@apple.com> [Modules] If a module map resides in a system header directory, treat it as a system module.

This prevents -pedantic from causing warnings in the system headers
used to create modules. Fixes <rdar://problem/14201171>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@184560 91177308-0d34-0410-b5e6-96231b3b80d8
bc3f628815b3841dc99109e7f67f9afa7793bc94 20-Jun-2013 Lawrence Crowl <crowl@google.com> This patch adds new private headers to the module map. Private
headers may be included from within the module, but not from outside
the module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@184471 91177308-0d34-0410-b5e6-96231b3b80d8
cdeba0b0127e0d97b10752a1e875901da4426810 20-Jun-2013 Lawrence Crowl <crowl@google.com> Fix English grammar error.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@184463 91177308-0d34-0410-b5e6-96231b3b80d8
671538e8a51eab5bd65a1f9f327ba7f44f84e486 12-Jun-2013 Richard Trieu <rtrieu@google.com> Introducing -Wheader-guard, a warning that checks header guards actually work
properly. This warning checks that the #ifndef and #define directives at
the beginning of a header refer to the same macro name. Includes a fix-it
hint to correct the header guard.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@183867 91177308-0d34-0410-b5e6-96231b3b80d8
49e1d9814a34e4024a347c2fce8e9567c40e6bf5 04-Jun-2013 Alexander Kornienko <alexfh@google.com> Added Lexer::getBufferEnd().

There's Lexer::getBufferStart(), and we need getBufferEnd() to access
the whole input buffer in clang::format::reformat. We don't want to
rely on the fact that the Lexer::BufferEnd always points to '\0', as there can
be embedded '\0's as well.

Reviewers: jordan_rose

Reviewed By: jordan_rose

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D916

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@183236 91177308-0d34-0410-b5e6-96231b3b80d8
a7ff62f9443efa3b13a28a1e566d4625b15b8553 04-Jun-2013 Aaron Ballman <aaron@aaronballman.com> Adding support for MSVC #pragma detect_mismatch functionality by emitting a FAILIFMISMATCH linker command into the object file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@183178 91177308-0d34-0410-b5e6-96231b3b80d8
3b7deda7137e62810a810ce25b062927a9fc7c71 24-May-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [modules] If we hit a failure while loading a PCH/module, abort parsing instead of trying to continue in an invalid state.

Also don't let libclang create a PCH with such an error.

Fixes rdar://13953768

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@182629 91177308-0d34-0410-b5e6-96231b3b80d8
440d456c5cf9613a3ee6a3297f892ddd8da5b8f8 22-May-2013 Richard Smith <richard-llvm@metafoo.co.uk> PR16094: I should have known Obj-C init-capture disambiguation couldn't be
*that* easy...

Try a bit harder to disambiguate. This is mostly straightforward, but for
=-style initializers, we actually need to know where an expression ends:

[foo = bar baz]

is a message send, whereas

[foo = bar + baz]

is a lambda-introducer. Handle this by parsing the expression eagerly, and
replacing it with an annotation token. By chance, we use the *exact same*
parsing rules in both cases (except that we need to assume we're inside a
message send for the parse, to turn off various forms of inapplicable
error recovery).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@182432 91177308-0d34-0410-b5e6-96231b3b80d8
7247c88d1e41514a41085f83ebf03dd5220e054a 15-May-2013 David Blaikie <dblaikie@gmail.com> Use only explicit bool conversion operator

The most common (non-buggy) case are where such objects are used as
return expressions in bool-returning functions or as boolean function
arguments. In those cases I've used (& added if necessary) a named
function to provide the equivalent (or sometimes negative, depending on
convenient wording) test.

DiagnosticBuilder kept its implicit conversion operator owing to the
prevalent use of it in return statements.

One bug was found in ExprConstant.cpp involving a comparison of two
PointerUnions (PointerUnion did not previously have an operator==, so
instead both operands were converted to bool & then compared). A test
is included in test/SemaCXX/constant-expression-cxx1y.cpp for the fix
(adding operator== to PointerUnion in LLVM).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181869 91177308-0d34-0410-b5e6-96231b3b80d8
30a16f1383b56cb71be251999a577b2e37db2ce0 11-May-2013 Douglas Gregor <dgregor@apple.com> [Modules] Make r180934 more efficient by only loading top-level module maps in system header directories.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181643 91177308-0d34-0410-b5e6-96231b3b80d8
f9ba851c9b5db8ea30846215e737702a1cc2c194 09-May-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH] Remove the ASTReaderListener::ReadHeaderFileInfo callback.

This made sense in pre-module era, before merging of HeaderFileInfos was introduced.

Final part of rdar://13840148.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181490 91177308-0d34-0410-b5e6-96231b3b80d8
d3220dbeeadc4ac54ceecea8cf63f8d8be291d2a 09-May-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [modules] When building a module, make sure we don't serialize out HeaderFileInfo for headers not belonging to the module.

After r180934 we may initiate module map parsing for modules not related to the module what we are building,
make sure we ignore the header file info of headers from such modules.

First part of rdar://13840148

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181489 91177308-0d34-0410-b5e6-96231b3b80d8
7adf79a620cb7fbde0608e21727425930676b7db 06-May-2013 Reid Kleckner <reid@kleckner.net> Move PragmaCommentHandler to lib/Parse in preparation for calling Sema

No functionality change. The existing tests for this pragma only verify
that we can preprocess it.

Reviewers: rsmith

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D751

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181246 91177308-0d34-0410-b5e6-96231b3b80d8
a4a90cabfa15c900016a7cfaea51a6d4e8ebf4db 04-May-2013 Douglas Gregor <dgregor@apple.com> When building a module, forward diagnostics to the outer diagnostic consumer.

Previously, we would clone the current diagnostic consumer to produce
a new diagnostic consumer to use when building a module. The problem
here is that we end up losing diagnostics for important diagnostic
consumers, such as serialized diagnostics (where we'd end up with two
diagnostic consumers writing the same output file). With forwarding,
the diagnostics from all of the different modules being built get
forwarded to the one serialized-diagnostic consumer and are emitted in
a sane way.

Fixes <rdar://problem/13663996>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181067 91177308-0d34-0410-b5e6-96231b3b80d8
dd08a0c178329ec16cb9e494e6880f3991708b93 04-May-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [Preprocessor] For the MacroExpands preprocessor callback, also pass the MacroArgs object that provides information about
the argument tokens for a function macro.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@181065 91177308-0d34-0410-b5e6-96231b3b80d8
3e25b990f1e2ba2a9a63dde5fd111492a7f6194a 30-Apr-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH] Fix memory leak related to deserialized MacroInfo objects.

Deserialized MacroInfos were not destroyed and if their SmallVector did heap allocation,
it was leaked.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@180771 91177308-0d34-0410-b5e6-96231b3b80d8
076eea20b80024fc63bbd71fb019375983680ea6 17-Apr-2013 Andy Gibbs <andyg1001@hotmail.co.uk> Implemented #pragma GCC warning/error in the same mould as #pragma message.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@179687 91177308-0d34-0410-b5e6-96231b3b80d8
85192c7fe187d5486e12dbc6960af28f16a558a0 16-Apr-2013 Tareq A. Siraj <tareq.a.sriaj@intel.com> Parser support for #pragma clang __debug captured

This patch implements parsing ‘#pragma clang __debug’ as a first step for
implementing captured statements. Captured statements are a mechanism for
doing outlining in the AST.
see http://lists.cs.uiuc.edu/pipermail/cfe-dev/2013-January/027540.html.

Currently returns StmtEmpty

Author: Andy Zhang <andy.zhang@intel.com>

Differential Revision: http://llvm-reviews.chandlerc.com/D369

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@179614 91177308-0d34-0410-b5e6-96231b3b80d8
7332ae49671762239c9986c8f30f291c3a13d27e 12-Apr-2013 Douglas Gregor <dgregor@apple.com> <rdar://problem/13643854> Only emit ambiguous-expansion warnings when at least one of the macro definitions comes from a non-system header.

This slightly weakens the heuristic introduced in r178109.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@179411 91177308-0d34-0410-b5e6-96231b3b80d8
bd25ff8632e18bb3af761369fd3b0f1b48fdf061 03-Apr-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] Allow comparing two macro definitions syntactically instead of only lexically.

Syntactically means the function macro parameter names do not need to use the same
identifiers in order for the definitions to be considered identical.

Syntactic equivalence is a microsoft extension for macro redefinitions and we'll also
use this kind of comparison to check for ambiguous macros coming from modules.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@178671 91177308-0d34-0410-b5e6-96231b3b80d8
52151fd57eaf40603fa8f1cd34dcb4ad5f7701b3 27-Mar-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [modules] Before marking the module imported macros as ambiguous, check if this is a case where
the system macro uses a not identical definition compared to a macro from the clang headers.

For example (these come from different modules):
\#define LONG_MAX __LONG_MAX__ (clang's limits.h)
\#define LONG_MAX 0x7fffffffffffffffL (system's limits.h)
in which case don't mark them ambiguous to avoid the "ambiguous macro expansion" warning.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@178109 91177308-0d34-0410-b5e6-96231b3b80d8
d560ce3b66325ff22cd3aca23c6de3143bd74d24 27-Mar-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [Preprocessor] Remove PPMutationListener.

It's not used anymore.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@178106 91177308-0d34-0410-b5e6-96231b3b80d8
c56fff7fd231aebf4b152f60f8f11ef91835c48a 26-Mar-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [Preprocessor/Modules] Separate the macro directives kinds into their own MacroDirective's subclasses.

For each macro directive (define, undefine, visibility) have a separate object that gets chained
to the macro directive history. This has several benefits:

-No need to mutate a MacroDirective when there is a undefine/visibility directive. Stuff like
PPMutationListener become unnecessary.
-No need to keep extra source locations for the undef/visibility locations for the define directive object
(which is the majority of the directives)
-Much easier to hide/unhide a section in the macro directive history.
-Easier to track the effects of the directives across different submodules.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@178037 91177308-0d34-0410-b5e6-96231b3b80d8
d44d2872b2ebe58237de4dbc350b82cab944ccc5 25-Mar-2013 Douglas Gregor <dgregor@apple.com> <rdar://problem/13434605> Periodically prune the module cache so that it does not grow forever.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177918 91177308-0d34-0410-b5e6-96231b3b80d8
fd61d6fe0aa9853f0577ca88a63901c3773e8101 25-Mar-2013 Nick Lewycky <nicholas@mxc.ca> Correct indentation. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177916 91177308-0d34-0410-b5e6-96231b3b80d8
9317ab94bb68122ba6fc728eb73c1308fb913cd1 22-Mar-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH/Modules] De/Serialize MacroInfos separately than MacroDirectives.

-Serialize the macro directives history into its own section
-Get rid of the macro updates section
-When de/serializing an identifier from a module, associate only one macro per
submodule that defined+exported it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177761 91177308-0d34-0410-b5e6-96231b3b80d8
baa74bd3968028d8e5b10ee9b50d0dceb41e85a9 22-Mar-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [modules] When a MacroInfo object is deserialized, allocate and store its submodule ID.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177760 91177308-0d34-0410-b5e6-96231b3b80d8
cdf2808c4e735a717599751dcd2b434f239e1c68 21-Mar-2013 Douglas Gregor <dgregor@apple.com> <rdar://problem/13037793> Allow the names of modules to differ from the name of their subdirectory in the include path.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177621 91177308-0d34-0410-b5e6-96231b3b80d8
81ba9d11609132d1d804f2b78d321daf97c081f5 20-Mar-2013 James Dennett <jdennett@google.com> Documentation cleanup for MacroInfo.
* Clarify what MacroInfo::isBuiltinMacro means, as it really means something
more like "isMagicalMacro" or "requiresProcessingBeforeExpansion" -- the
macros defined in "<built-in>" are not considered built-in by this function;
* Escape __LINE__ as \__LINE__ in Doxygen comments so that the underscores
don't get replaced by *bold* output;
* Turn comments in MacroInfo.cpp into non-Doxygen comments, so that they
don't result in duplicated/badly formatted Doxygen output;
* Clean up a bunch of \brief formatting, and add a \file comment for

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177581 91177308-0d34-0410-b5e6-96231b3b80d8
906d66acc5cf2679453e10a4f0a67feedd765b21 20-Mar-2013 Douglas Gregor <dgregor@apple.com> <rdar://problem/12368093> Extend module maps with a 'conflict' declaration, and warn when a newly-imported module conflicts with an already-imported module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177577 91177308-0d34-0410-b5e6-96231b3b80d8
55ea75bf61a5d76f6453513d937944ce68181c6a 13-Mar-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [Modules] Don't eagerly load and associate all the module header files.

In a module-enabled Cocoa PCH file, we spend a lot of time stat'ing the headers
in order to associate the FileEntries with their modules and support implicit
module import.

Use a more lazy scheme by enhancing HeaderInfoTable to store extra info about
the module that a header belongs to, and associate it with its module only when
there is a request for loading the header info for a particular file.

Part of rdar://13391765

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@176976 91177308-0d34-0410-b5e6-96231b3b80d8
c515978bd3a703aa733f846a0094ffa84d149074 24-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] Use MacroDirective in the preprocessor callbacks to make available the
full information about the macro (e.g if it was imported and where).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175978 91177308-0d34-0410-b5e6-96231b3b80d8
82a29af8cbc4b117af83c5c5dfa875b5699c97c4 24-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> Add comments for the MacroDirective & MacroInfo classes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175977 91177308-0d34-0410-b5e6-96231b3b80d8
0b849d34b3a9574615e98e108db4e8099e9032e0 22-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessing record] Have the MacroDefinitions map point to the MacroDefinition object instead
its index in the preprocessed entities vector.

This is because the order of the entities in the vector can change in some (uncommon) cases.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175907 91177308-0d34-0410-b5e6-96231b3b80d8
6aad4a31b35df07fe818f193fcfd3c0197aea467 21-Feb-2013 Jordan Rose <jordan_rose@apple.com> Preprocessor: preserve whitespace in -traditional-cpp mode.

Note that unlike GNU cpp we currently do not preserve whitespace in macros
(even in -traditional-cpp mode).


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175778 91177308-0d34-0410-b5e6-96231b3b80d8
66874fb18afbffb8b2ca05576851a64534be3352 21-Feb-2013 David Blaikie <dblaikie@gmail.com> Use None rather than Optional<T>() where possible.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175705 91177308-0d34-0410-b5e6-96231b3b80d8
dc84cd5efdd3430efb22546b4ac656aa0540b210 20-Feb-2013 David Blaikie <dblaikie@gmail.com> Include llvm::Optional in clang/Basic/LLVM.h

Post-commit CR feedback from Jordan Rose regarding r175594.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175679 91177308-0d34-0410-b5e6-96231b3b80d8
9818a1d443e97677dd3422305de9cc2b1fb2a8c1 20-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] Split the MacroInfo class into two separate concepts, MacroInfo class
for the data specific to a macro definition (e.g. what the tokens are), and
MacroDirective class which encapsulates the changes to the "macro namespace"
(e.g. the location where the macro name became active, the location where it was undefined, etc.)

(A MacroDirective always points to a MacroInfo object.)

Usually a macro definition (MacroInfo) is where a macro name becomes active (MacroDirective) but
splitting the concepts allows us to better model the effect of modules to the macro namespace
(also as a bonus it allows better modeling of push_macro/pop_macro #pragmas).
Modules can have their own macro history, separate from the local (current translation unit)
macro history; MacroDirectives will be used to model the macro history (changes to macro namespace).

For example, if "@import A;" imports macro FOO, there will be a new local MacroDirective created
to indicate that "FOO" became active at the import location. Module "A" itself will contain another
MacroDirective in its macro history (at the point of the definition of FOO) and both MacroDirectives
will point to the same MacroInfo object.

Introducing the separation of macro concepts is the first part towards better modeling of module macros.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175585 91177308-0d34-0410-b5e6-96231b3b80d8
0be5e567e3a48592fd6b11f88cc77efb20c76f26 19-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [modules] Const'ify some functions of ModuleMap.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175552 91177308-0d34-0410-b5e6-96231b3b80d8
959dc8475fc20ce8c3fd55021cb9f02a531cddc5 16-Feb-2013 Dmitri Gribenko <gribozavr@gmail.com> Use trailing documentation comments properly

Patch by Alexander Zinenko.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175376 91177308-0d34-0410-b5e6-96231b3b80d8
3f6f51e28231f65de9c2dd150a2d757b2162cfa3 08-Feb-2013 Jordan Rose <jordan_rose@apple.com> Excise <cctype> from Clang (except clang-tblgen) in favor of CharInfo.h.

Nearly all of these changes are one-to-one replacements; the few that
aren't have to do with custom identifier validation.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@174768 91177308-0d34-0410-b5e6-96231b3b80d8
1b58c74af272a1d8228b8161c93a8a018456098e 08-Feb-2013 Douglas Gregor <dgregor@apple.com> Teach subframework header lookup to suggest modules <rdar://problem/13176200>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@174683 91177308-0d34-0410-b5e6-96231b3b80d8
2a06085281d1b6aee628f85e8676eec04542cbc9 07-Feb-2013 Douglas Gregor <dgregor@apple.com> Introduce -fmodules-ignore-macro=NNN to ignore a macro when building/loading modules.

The use of this flag enables a modules optimization where a given set
of macros can be labeled as "ignored" by the modules
system. Definitions of those macros will be completely ignored when
building the module hash and will be stripped when actually building
modules. The overall effect is that this flag can be used to
drastically reduce the number of

Eventually, we'll want modules to tell us what set of macros they
respond to (the "configuration macros"), and anything not in that set
will be excluded. However, that requires a lot of per-module
information that must be accurate, whereas this option can be used
more readily.

Fixes the rest of <rdar://problem/13165109>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@174560 91177308-0d34-0410-b5e6-96231b3b80d8
9484c1dcf084452228aae439edcd00e1b4ce7470 01-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> Const'ify Preprocessor::getPredefinesFileID.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@174218 91177308-0d34-0410-b5e6-96231b3b80d8
5ebcb20b0331a6e64c213f0bb5f4bed9a9e8eb34 01-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> For ModuleLoader::makeModuleVisible() also pass the source location where the
module import occurred.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@174191 91177308-0d34-0410-b5e6-96231b3b80d8
95d912c994832333a998bc7d4ae0e8e830a2d38b 01-Feb-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce SourceManager::PredefinesFileID, to allow each checking of whether
a source location came from the predefines buffer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@174190 91177308-0d34-0410-b5e6-96231b3b80d8
74c2498bb9e339345ee32bdd095e76157cec3b86 30-Jan-2013 Jordan Rose <jordan_rose@apple.com> Don't warn about Unicode characters in -E mode.

People use the C preprocessor for things other than C files. Some of them
have Unicode characters. We shouldn't warn about Unicode characters
appearing outside of identifiers in this case.

There's not currently a way for the preprocessor to tell if it's in -E mode,
so I added a new flag, derived from the PreprocessorOutputOptions. This is
only used by the Unicode warnings for now, but could conceivably be used by
other warnings or even behavioral differences later.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173881 91177308-0d34-0410-b5e6-96231b3b80d8
59fd63581d6d572f23e82e81a50e0b940c8d1089 30-Jan-2013 Daniel Dunbar <daniel@zuster.org> [Frontend] Remove HeaderSearchOptions::Entry::IsInternal, which is unused.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173866 91177308-0d34-0410-b5e6-96231b3b80d8
d09436c724baa1b11804d60e286ec0b9124489ee 30-Jan-2013 Daniel Dunbar <daniel@zuster.org> [Frontend] Remove actual ImplicitExternC member variable.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173860 91177308-0d34-0410-b5e6-96231b3b80d8
ef84554239d77cc52a14b42f2bff8c2d02e7630e 30-Jan-2013 Daniel Dunbar <daniel@zuster.org> [Frontend] Add an ExternCSystem include entry group.

- The only group where it makes sense for the "ExternC" bit is System, so this
simplifies having to have the extra isCXXAware (or ImplicitExternC, depending
on what code you talk to) bit caried around.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173859 91177308-0d34-0410-b5e6-96231b3b80d8
eab6652b7e417c8e46b21c7d340f3a7d41492d6e 25-Jan-2013 Daniel Dunbar <daniel@zuster.org> [Frontend] Remove another IsUserSpecified member variable that is now unused.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173412 91177308-0d34-0410-b5e6-96231b3b80d8
1ea6bc0fd9c1ff9fa03e8a829a79c4167445d503 25-Jan-2013 Daniel Dunbar <daniel@zuster.org> [Lex] Remove DirectoryLookup.UserSpecified, which is unused.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173409 91177308-0d34-0410-b5e6-96231b3b80d8
c7629d941557f7179eb8fa8a2e2a74d749cbaf7c 24-Jan-2013 Jordan Rose <jordan_rose@apple.com> Handle universal character names and Unicode characters outside of literals.

This is a missing piece for C99 conformance.

This patch handles UCNs by adding a '\\' case to LexTokenInternal and
LexIdentifier -- if we see a backslash, we tentatively try to read in a UCN.
If the UCN is not syntactically well-formed, we fall back to the old
treatment: a backslash followed by an identifier beginning with 'u' (or 'U').

Because the spelling of an identifier with UCNs still has the UCN in it, we
need to convert that to UTF-8 in Preprocessor::LookUpIdentifierInfo.

Of course, valid code that does *not* use UCNs will see only a very minimal
performance hit (checks after each identifier for non-ASCII characters,
checks when converting raw_identifiers to identifiers that they do not
contain UCNs, and checks when getting the spelling of an identifier that it
does not contain a UCN).

This patch also adds basic support for actual UTF-8 in the source. This is
treated almost exactly the same as UCNs except that we consider stray
Unicode characters to be mistakes and offer a fixit to remove them.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173369 91177308-0d34-0410-b5e6-96231b3b80d8
dc1088f6201c3fb8f3f97f54c343f7d163fbec06 19-Jan-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH/Modules] Revert r172843, it caused a module to fail building.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172884 91177308-0d34-0410-b5e6-96231b3b80d8
f953276b6165dc7e8f4679cce4d0e7c649cd4232 18-Jan-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH/Modules] Re-apply r172620 and r172629, now with 100% less infinite loops!

Makes sure that a deserialized macro is only added to the preprocessor macro definitions only once.
Unfortunately I couldn't get a reduced test case.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172843 91177308-0d34-0410-b5e6-96231b3b80d8
d3b036efdf0bf4ec216c701183a4b990cd041cd6 18-Jan-2013 Douglas Gregor <dgregor@apple.com> Revert Clang r172620 and r172629, which caused a hang when building
complicated modules (<rdar://problem/13038265>). Unfortunately, this
un-fixes <rdar://problem/13016031>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172783 91177308-0d34-0410-b5e6-96231b3b80d8
31672b1745195d709b641f1c0f44e203742fa73b 16-Jan-2013 Aaron Ballman <aaron@aaronballman.com> No longer crashing with an assert when __has_include or __has_include_next is used outside of a preprocessor directive. This fixes PR14837.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172639 91177308-0d34-0410-b5e6-96231b3b80d8
7f69b1ab4f63ded511dddd7fb7e1cd8860757016 16-Jan-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH/Modules] Change how macro [re]definitions are de/serialized.

Previously we would serialize the macro redefinitions as a list, part of
the identifier, and try to chain them together across modules individually
without having the info that they were already chained at definition time.

Change this by serializing the macro redefinition chain and then try
to synthesize the chain parts across modules. This allows us to correctly
pinpoint when 2 different definitions are ambiguous because they came from
unrelated modules.

Fixes bogus "ambiguous expansion of macro" warning when a macro in a PCH
is redefined without undef'ing it first.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172620 91177308-0d34-0410-b5e6-96231b3b80d8
b3958476b21702a15f64f09d2862506d03dffe7b 14-Jan-2013 Dmitri Gribenko <gribozavr@gmail.com> Constify argument of Preprocessor::getMacroInfoHistory and propagate to
callers, removing unneeded const_cast

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172372 91177308-0d34-0410-b5e6-96231b3b80d8
cfa88f893915ceb8ae4ce2f17c46c24a4d67502f 12-Jan-2013 Dmitri Gribenko <gribozavr@gmail.com> Remove useless 'llvm::' qualifier from names like StringRef and others that are
brought into 'clang' namespace by clang/Basic/LLVM.h

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172323 91177308-0d34-0410-b5e6-96231b3b80d8
9946fc735d7285f2195f89635370f534afd9877e 12-Jan-2013 Dmitri Gribenko <gribozavr@gmail.com> Add missing includes and forward declarations so that headers don't depend on
other headers included before them.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172320 91177308-0d34-0410-b5e6-96231b3b80d8
ca2ab45341c448284cf93770018c717810575f86 12-Jan-2013 Douglas Gregor <dgregor@apple.com> Provide Decl::getOwningModule(), which determines the (sub)module in
which a particular declaration resides. Use this information to
customize the "definition of 'blah' must be imported from another
module" diagnostic with the module the user actually has to
import. Additionally, recover by importing that module, so we don't
complain about other names in that module.

Still TODO: coming up with decent Fix-Its for these cases, and expand
this recovery approach for other name lookup failures.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172290 91177308-0d34-0410-b5e6-96231b3b80d8
7005b907ea159c8e74e81f85269777429bc18d3c 10-Jan-2013 Douglas Gregor <dgregor@apple.com> Rework the realpath nonsense for framework lookups to deal more
uniformly with symlinks between top-level and embedded frameworks.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@172030 91177308-0d34-0410-b5e6-96231b3b80d8
d93335c43fd462145fee3ea8f4d84d430577c821 07-Jan-2013 Argyrios Kyrtzidis <akyrtzi@gmail.com> Pull the bulk of Lexer::MeasureTokenLength() out into a new function,

No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@171771 91177308-0d34-0410-b5e6-96231b3b80d8
5f5250b140544436eb3c2fbd9f43e14019ef01f2 08-Dec-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [libclang] Resolve a cursor that points to a macro name inside a #ifdef/#ifndef
directive as a macro expansion.

This is more of a "macro reference" than a macro expansion but it's close enough
for libclang's purposes. If it causes issues we can revisit and introduce a new
kind of cursor.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169666 91177308-0d34-0410-b5e6-96231b3b80d8
61c1c8ee3fa1bd1f4866d246c72af854be834634 08-Dec-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [Preprocessor] Enhance Ifdef/Ifndef/Defined preprocessor callbacks to also pass
a MacroInfo object if the identifier was a macro name.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169665 91177308-0d34-0410-b5e6-96231b3b80d8
30a2e16f6c27f888dd11eba6bbbae1e980078fcb 04-Dec-2012 Chandler Carruth <chandlerc@gmail.com> Sort #include lines for all files under include/...

This is a simpler sort, entirely automatic with the help of
llvm/utils/sort_includes.py -- no manual edits here.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169238 91177308-0d34-0410-b5e6-96231b3b80d8
37ed12720a35b7bfa1c4de73ad6f1c6c1c88ee17 04-Dec-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Refactor recording the preprocessor conditional directive regions out of
PreprocessingRecord and into its own class, PPConditionalDirectiveRecord.

Decoupling allows a client to use the functionality of PPConditionalDirectiveRecord
without needing a PreprocessingRecord.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169229 91177308-0d34-0410-b5e6-96231b3b80d8
df1059cac42d613547d86b4e44c5e364bfc03073 04-Dec-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> In the PreprocessingRecord, to identify the different conditional directive regions
use the SourceLocation at the start of the respective region, instead of a unique integer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169228 91177308-0d34-0410-b5e6-96231b3b80d8
830ea5b7c75413526c19531f0180fa6e45b98919 30-Nov-2012 Douglas Gregor <dgregor@apple.com> When an error occurs while building a module on demand, provide "While
building module 'Foo' imported from..." notes (the same we we provide
"In file included from..." notes) in the diagnostic, so that we know
how this module got included in the first place. This is part of

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169021 91177308-0d34-0410-b5e6-96231b3b80d8
463d90986ec54c62bf8fe31193ef5db701db48a5 30-Nov-2012 Douglas Gregor <dgregor@apple.com> Keep track of modules that have failed to build. If we encounter an
import of that module elsewhere, don't try to build the module again:
it won't work, and the experience is quite dreadful. We track this
information somewhat globally, shared among all of the related
CompilerInvocations used to build modules on-the-fly, so that a
particular Clang instance will only try to build a given module once.

Fixes <rdar://problem/12552849>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@168961 91177308-0d34-0410-b5e6-96231b3b80d8
97f8461a2c553f68a258612d2322e4281c3f0915 17-Nov-2012 Andy Gibbs <andyg1001@hotmail.co.uk> Made the "expected string literal" diagnostic more expressive

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@168267 91177308-0d34-0410-b5e6-96231b3b80d8
02a176871d91bba3004e4f94b2d4d588ae4b2122 17-Nov-2012 Andy Gibbs <andyg1001@hotmail.co.uk> Refactored duplicate string literal lexing code within Preprocessor, into a
common LexStringLiteral function. In doing so, some consistency problems have
been ironed out (e.g. where the first token in the string literal was lexed
with macro expansion, but subsequent ones were not) and also an erroneous
diagnostic has been corrected.

LexStringLiteral is complemented by a FinishLexStringLiteral function which
can be used in the situation where the first token of the string literal has
already been lexed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@168266 91177308-0d34-0410-b5e6-96231b3b80d8
4fa4b480270c14dfdcd0dfd4f76938e973082e3b 14-Nov-2012 Eli Friedman <eli.friedman@gmail.com> Suppress elided variadic macro argument extension diagnostic for macros using
the related comma pasting extension.

In certain cases, we used to get two diagnostics for what is essentially one
extension. This change suppresses the first diagnostic in certain cases
where we know we're going to print the second diagnostic. The
diagnostic is redundant, and it can't be suppressed in the definition
of the macro because it points at the use of the macro, so we want to
avoid printing it if possible.

The implementation works by detecting constructs which look like comma
pasting at the time of the definition of the macro; this information
is then used when the macro is used. (We can't actually detect
whether we're using the comma pasting extension until the macro is
actually used, but we can detecting constructs which will be comma
pasting if the varargs argument is elided.)


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167907 91177308-0d34-0410-b5e6-96231b3b80d8
6e649737898ffb627c377fd8fa0a437d0a42ae4a 13-Nov-2012 Daniel Dunbar <daniel@zuster.org> Revert r167801, "[preprocessor] When #including something that contributes no
tokens at all,". This change broke External/Nurbs in LLVM test-suite.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167858 91177308-0d34-0410-b5e6-96231b3b80d8
4d10b40ea8ee489c7b9194aa2b025df4ecd2ab01 13-Nov-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] When #including something that contributes no tokens at all,
don't recursively continue lexing.

This avoids a stack overflow with a sequence of many empty #includes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167801 91177308-0d34-0410-b5e6-96231b3b80d8
03e67fbf7a2646447972742b6bb97c82ce4698fb 11-Nov-2012 Nico Weber <nicolasweber@gmx.de> Delete comment reference to non-existent method

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167692 91177308-0d34-0410-b5e6-96231b3b80d8
bb23628148f555a4cf71f98c27096a7a804c085c 11-Nov-2012 Nico Weber <nicolasweber@gmx.de> s/BCPLComment/LineComment/

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167690 91177308-0d34-0410-b5e6-96231b3b80d8
82e52377bd76ed71e8c09edc5f2f452e388b16ad 06-Nov-2012 Douglas Gregor <dgregor@apple.com> Introduce inferred framework modules into the module map file,
allowing a module map to be placed one level above the '.framework'
directories to specify that all .frameworks within that directory can
be inferred as framework modules. One can also specifically exclude
frameworks known not to work.

This makes explicit (and more restricted) behavior modules have had
"forever", where *any* .framework was assumed to be able to be built
as a module. That's not necessarily true, so we white-list directories
(with exclusions) when those directories have been audited.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167482 91177308-0d34-0410-b5e6-96231b3b80d8
164fcd022185fddc0e689f61ac8ad4b3ac744c2a 06-Nov-2012 Nico Weber <nicolasweber@gmx.de> Delete an outdated comment.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167441 91177308-0d34-0410-b5e6-96231b3b80d8
4182ed686283b72736b287cbe28583cb641f8934 31-Oct-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH] Remove the stat cache from the PCH file.

The stat cache became essentially useless ever since we started
validating all file entries in the PCH.
But the motivating reason for removing it now is that it also affected
correctness in this situation:

-You have a header without include guards (using "#pragma once" or #import)
-When creating the PCH:
-The same header is referenced in an #include with different filename cases.
-In the PCH, of course, we record only one file entry for the header file
-But we cache in the PCH file the stat info for both filename cases

-Then the source files are updated and the header file is updated in a way that
its size and modification time are the same but its inode changes

-When using the PCH:
-We validate the headers, we check that header file and we create a file entry with its current inode
-There's another #include with a filename with different case than the previously created file entry
-In order to get its stat info we go through the cached stat info of the PCH and we receive the old inode
-because of the different inodes, we think they are different files so we go ahead and include its contents.

Removing the stat cache will potentially break clients that are attempting to use the stat cache
as a way of avoiding having the actual input files available. If that use case is important, patches are welcome
to bring it back in a way that will actually work correctly (i.e., emit a PCH that is self-contained, coping with
literal strings, line/column computations, etc.).

This fixes rdar://5502805

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167172 91177308-0d34-0410-b5e6-96231b3b80d8
a71a7d8a1ce4474e7bdb680658fb58b6caf391d3 24-Oct-2012 Douglas Gregor <dgregor@apple.com> (De-)serialize the preprocessor options, including macros defined,
-include'd files, etc.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@166614 91177308-0d34-0410-b5e6-96231b3b80d8
36a16498ff911a218f26c7955376bbe99ddb16df 24-Oct-2012 Douglas Gregor <dgregor@apple.com> Teach the preprocessor to hold onto the preprocessor options.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@166599 91177308-0d34-0410-b5e6-96231b3b80d8
14e71f04136de056c08ffb9ccd44b4ca391cc8a5 24-Oct-2012 Douglas Gregor <dgregor@apple.com> Move PreprocessorOptions into the Lex library, and make it intrusively

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@166587 91177308-0d34-0410-b5e6-96231b3b80d8
c042edd54face617a3b9d0b4b9d5a3ff229d0f48 24-Oct-2012 Douglas Gregor <dgregor@apple.com> Move HeaderSearchOptions into the Lex library, make it intrusively
reference-counted, and hold a reference to it in HeaderSearch.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@166583 91177308-0d34-0410-b5e6-96231b3b80d8
2b49d1f0ad790a8a5d514af1be211591a746cb73 15-Oct-2012 Douglas Gregor <dgregor@apple.com> Introduce the notion of excluded headers into the module map
description. Previously, one could emulate this behavior by placing
the header in an always-unavailable submodule, but Argyrios guilted me
into expressing this idea properly.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165921 91177308-0d34-0410-b5e6-96231b3b80d8
b43d87b0646aa04951056c7e0d1ab9a58eb09f66 12-Oct-2012 Sean Silva <silvas@purdue.edu> Remove pointless classof()'s.

Updates to llvm/Support/Casting.h have rendered these classof()'s

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165770 91177308-0d34-0410-b5e6-96231b3b80d8
e8219a655128b98d0573658a139de5d848451fda 11-Oct-2012 Douglas Gregor <dgregor@apple.com> Diagnose the expansion of ambiguous macro definitions. This can happen
only with modules, when two disjoint modules #define the same
identifier to different token sequences.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165746 91177308-0d34-0410-b5e6-96231b3b80d8
3ab50fe3fc8311fb68c4f5eb97e9bb3d0041bf1a 11-Oct-2012 Douglas Gregor <dgregor@apple.com> Introduce a simple "hint" scheme to eliminate the quadratic behavior
associated with deserializing macro history for an identifier.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165729 91177308-0d34-0410-b5e6-96231b3b80d8
6c6c54a59a6e7dbe63ff6a9bbab76f6e0c7c8462 11-Oct-2012 Douglas Gregor <dgregor@apple.com> Deserialize macro history when we deserialize an identifier that has
macro history.

When deserializing macro history, we arrange history such that the
macros that have definitions (that haven't been #undef'd) and are
visible come at the beginning of the list, which is what the
preprocessor and other clients of Preprocessor::getMacroInfo()
expect. If additional macro definitions become visible later, they'll
be moved toward the front of the list. Note that it's possible to have
ambiguities, but we don't diagnose them yet.

There is a partially-implemented design decision here that, if a
particular identifier has been defined or #undef'd within the
translation unit, that definition (or #undef) hides any macro
definitions that come from imported modules. There's still a little
work to do to ensure that the right #undef'ing happens.

Additionally, we'll need to scope the update records for #undefs, so
they only kick in when the submodule containing that update record
becomes visible.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165682 91177308-0d34-0410-b5e6-96231b3b80d8
a8235d6c4093cd38dcf742909651f867de62e55b 10-Oct-2012 Douglas Gregor <dgregor@apple.com> Rework the (de-)serialization of macros, as stored in
MacroInfo*. Instead of simply dumping an offset into the current file,
give each macro definition a proper ID with all of the standard
modules-remapping facilities. Additionally, when a macro is modified
in a subsequent AST file (e.g., #undef'ing a macro loaded from another
module or from a precompiled header), provide a macro update record
rather than rewriting the entire macro definition. This gives us
greater consistency with the way we handle declarations, and ties
together macro definitions much more cleanly.

Note that we're still not actually deserializing macro history (we
never were), but it's far easy to do properly now.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165560 91177308-0d34-0410-b5e6-96231b3b80d8
f03b888a12ccff7d54ddd8a79a0141cf23adbf67 05-Oct-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessing record] Have PPEntityID be independent of the size of the
loaded entities vector, otherwise its meaning will change when a module
is imported and the vector size changes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165278 91177308-0d34-0410-b5e6-96231b3b80d8
632dcc92f60ab7f806a89c5bca3a0951763a9219 02-Oct-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [libclang] When indexing an AST file, only deserialize the preprocessing record
entities of the current primary module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165023 91177308-0d34-0410-b5e6-96231b3b80d8
8dd927cf03e85b942f323eebb42e06c839887ebb 02-Oct-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Add info in the preprocessing record whether an inclusion directive
resulted in an automatic module import.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165022 91177308-0d34-0410-b5e6-96231b3b80d8
1ecf0e6e271f3046bc53264318c47eae0fb80afd 02-Oct-2012 Axel Naumann <Axel.Naumann@cern.ch> Enable programmatic provisioning of virtual module.map files (instead of writing out actual module.map files).
Opens up the wonders of clang::Modules to tools - though they remain as experimental as before.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165002 91177308-0d34-0410-b5e6-96231b3b80d8
1952354bd376062c3ab3d328c0fc6c36530c9309 29-Sep-2012 Dmitri Gribenko <gribozavr@gmail.com> Move the 'find macro by spelling' infrastructure to the Preprocessor class and
use it to suggest appropriate macro for __attribute__((deprecated)) in

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164892 91177308-0d34-0410-b5e6-96231b3b80d8
f8afcffe6a0213760b64c211812b1750e1e1e967 29-Sep-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> For PPCallbacks::InclusionDirective() add a parameter for the module, whenever
an inclusion directive was automatically turned into a module import, and
PPCallbacks::moduleImport() for an explicit module import.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164874 91177308-0d34-0410-b5e6-96231b3b80d8
66da0abf7ab7cd449bb1d5134b2ef97d9d34d812 29-Sep-2012 Alexander Kornienko <alexfh@google.com> Compatibility macro detection for the -Wimplicit-fallthrough diagnostic.

When issuing a diagnostic message for the -Wimplicit-fallthrough diagnostics, always try to find the latest macro, defined at the point of fallthrough, which is immediately expanded to "[[clang::fallthrough]]", and use it's name instead of the actual sequence.

Known issues:
* uses PP.getSpelling() to compare macro definition with a string (anyone can suggest a convenient way to fill a token array, or maybe lex it in runtime?);
* this can be generalized and used in other similar cases, any ideas where it should reside then?

Reviewers: doug.gregor, rsmith

Reviewed By: rsmith

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D50

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164858 91177308-0d34-0410-b5e6-96231b3b80d8
da313592441db36cf4b06be97c4bcc238ee6fa9c 27-Sep-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Per discussion in http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20120917/064551.html
have PPCallbacks::InclusionDirective pass the character range for the filename quotes or brackets.

rdar://11113134 & http://llvm.org/PR13880

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164743 91177308-0d34-0410-b5e6-96231b3b80d8
93dec51750411678bb9c5bc1b27b259f5f5a23bd 26-Sep-2012 Nico Weber <nicolasweber@gmx.de> Revert r163022, it caused PR13924.

Add a test for PR13924. Do not revert the test added in r163022,
it surprisingly still passes even after reverting the code changes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164672 91177308-0d34-0410-b5e6-96231b3b80d8
4d7e0ced7f16a04aabe2d8d91cbbb52fb1162810 25-Sep-2012 Alexander Kornienko <alexfh@google.com> Macro history (de-)serialization. Deserialization currently reads only the latest macro definition. Needs more work.

Summary: Passes all tests (+ the new one with code completion), but needs a thorough review in part related to modules.

Reviewers: doug.gregor

Reviewed By: alexfh

CC: cfe-commits, rsmith

Differential Revision: http://llvm-reviews.chandlerc.com/D41

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164610 91177308-0d34-0410-b5e6-96231b3b80d8
b09de5177ee8101818a59dcd0038c75b190a2509 25-Sep-2012 Douglas Gregor <dgregor@apple.com> Introduce builtin macros to determine whether we're building a
specific module (__building_module(modulename)) and to get the name of
the current module as an identifier (__MODULE__).

Used to help headers behave differently when they're being included as
part of building a module. Oh, the irony.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164605 91177308-0d34-0410-b5e6-96231b3b80d8
374b3837d676133fcc1eb70a25c8baf8ec4a5c4a 24-Sep-2012 Dmitri Gribenko <gribozavr@gmail.com> StringRef'ize Preprocessor::CreateString().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164555 91177308-0d34-0410-b5e6-96231b3b80d8
c12906ee78e006df6a71acf52ca9dd3171ddbf07 24-Sep-2012 Douglas Gregor <dgregor@apple.com> Make sure that we don't end up making an #undef'd macro visible after
the fact. Test cases will come when we're actually (de-)serializing
macro history.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164549 91177308-0d34-0410-b5e6-96231b3b80d8
fc97ea29b1afd9e87341bce2b0cbb0c7172b7dd8 24-Sep-2012 Dmitri Gribenko <gribozavr@gmail.com> Small cleanup of literal semantic analysis: hiding 'char *' pointers behind
StringRef makes code cleaner. Also, make the temporary buffer smaller:
512 characters is unreasonably large for integer literals.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164484 91177308-0d34-0410-b5e6-96231b3b80d8
1a1e9955b3221822f8f2cd100be9b090168bfb78 18-Sep-2012 Craig Topper <craig.topper@gmail.com> Remove an unused private field exposed by the recent LLVM_DELETED_FUNCTION changes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164103 91177308-0d34-0410-b5e6-96231b3b80d8
be2fa7ebf01259b63dc52fe46c8d101c18e72269 18-Sep-2012 Craig Topper <craig.topper@gmail.com> Mark unimplemented copy constructors and copy assignment operators with LLVM_DELETED_FUNCTION.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@164102 91177308-0d34-0410-b5e6-96231b3b80d8
f56faa01936b9cf909623d7f06e3c2569ca4a78e 15-Sep-2012 Dmitri Gribenko <gribozavr@gmail.com> Use LLVM_DELETED_FUNCTION in place of 'DO NOT IMPLEMENT' comments.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163983 91177308-0d34-0410-b5e6-96231b3b80d8
ec1b330cc309efa2f0835e8d83892ea2c89e3ae1 15-Sep-2012 Dmitri Gribenko <gribozavr@gmail.com> Make PTHManager properly noncopyable.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163982 91177308-0d34-0410-b5e6-96231b3b80d8
1824d54df85a462ada812dadda18130f951d40f3 13-Sep-2012 Dmitri Gribenko <gribozavr@gmail.com> Fix Doxygen misuse: refer to parameter names in paragraphs correctly (\arg is
not what most people want -- it starts a new paragraph).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163793 91177308-0d34-0410-b5e6-96231b3b80d8
e5f0588840b20897631cc8110344fd2745ef4caa 08-Sep-2012 Richard Smith <richard-llvm@metafoo.co.uk> When a bad UTF-8 encoding or bogus escape sequence is encountered in a
string literal, produce a diagnostic pointing at the erroneous character
range, not at the start of the literal.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163459 91177308-0d34-0410-b5e6-96231b3b80d8
9366a5a8d0c5834cffda2c31c924605fb9dffc9b 07-Sep-2012 Abramo Bagnara <abramo.bagnara@bugseng.com> Moved back getCharAndSizeNoWarn to public area.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163408 91177308-0d34-0410-b5e6-96231b3b80d8
181cd5bdebf0b88e8194c3f58da60f7f5b918b33 31-Aug-2012 Joao Matos <ripzonetriton@gmail.com> Emulate MSVC's preprocessor macro argument separator behavior by not considering commas from nested macro expansions as argument separators. Fixes parsing of VS 2012 headers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163022 91177308-0d34-0410-b5e6-96231b3b80d8
e506f8a41063410c75945ebb804758bd0202947f 31-Aug-2012 Eli Friedman <eli.friedman@gmail.com> Make a bunch of methods on Lexer private.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162970 91177308-0d34-0410-b5e6-96231b3b80d8
c30981a563a8947cb26b1e308d122fa2ef90fceb 30-Aug-2012 Richard Smith <richard-llvm@metafoo.co.uk> Make preprocessor act in a GCC-compatible fashion when a macro is redefined
within its own argument list. The original definition is used for the immediate
expansion, but the new definition is used for any subsequent occurences within
the argument list or after the expansion.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162906 91177308-0d34-0410-b5e6-96231b3b80d8
e40c4238a572bf8241a04e0005f70550cbfc1cfb 29-Aug-2012 Alexander Kornienko <alexfh@google.com> Fixed a problem with #pragma push_macro/pop_macro implementation.

The problem was with the following sequence:
#pragma push_macro("long")
#undef long
#pragma pop_macro("long")
in case when "long" didn't represent a macro.
Fixed crash and removed code duplication for #undef/pop_macro case. Added regression tests.

Reviewers: doug.gregor, klimek

Reviewed By: doug.gregor

CC: cfe-commits, chapuni

Differential Revision: http://llvm-reviews.chandlerc.com/D31

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162845 91177308-0d34-0410-b5e6-96231b3b80d8
8a64bb58c3b24d7d97895e435bbc0965c99bd3be 29-Aug-2012 Alexander Kornienko <alexfh@google.com> Keep history of macro definitions and #undefs

Summary: Keep history of macro definitions and #undefs with corresponding source locations, so that we can later find out all macros active in a specified source location. We don't save the history in PCH (no need currently). Memory overhead is about sizeof(void*)*3*<number of macro definitions and #undefs>+<in-memory size of all #undef'd macros>

I've run a test on a file composed of 109 .h files from boost 1.49 on x86-64 linux.
Stats before this patch:
*** Preprocessor Stats:
73222 directives found:
19171 #define.
4345 #undef.
5233 source files entered.
27 max include stack depth
19210 #if/#ifndef/#ifdef.
2384 #else/#elif.
6891 #endif.
408 #pragma.
14466 #if/#ifndef#ifdef regions skipped
80023/451669/1270 obj/fn/builtin macros expanded, 85724 on the fast path.
127145 token paste (##) operations performed, 11008 on the fast path.

Preprocessor Memory: 5874615B total
BumpPtr: 4399104
Macro Expanded Tokens: 417768
Predefines Buffer: 8135
Macros: 1048576
#pragma push_macro Info: 0
Poison Reasons: 1024
Comment Handlers: 8

Stats with this patch:
Preprocessor Memory: 7541687B total
BumpPtr: 6066176
Macro Expanded Tokens: 417768
Predefines Buffer: 8135
Macros: 1048576
#pragma push_macro Info: 0
Poison Reasons: 1024
Comment Handlers: 8

In my test increase in memory usage is about 1.7Mb, which is ~28% of initial preprocessor's memory usage and about 0.8% of clang's total VMM allocation.

As for CPU overhead, it should only be noticeable when iterating over all macros, and should mostly consist of couple extra dereferences and one comparison per macro + skipping of #undef'd macros. It's less trivial to measure, though, as the preprocessor consumes a very small fraction of compilation time.

Reviewers: doug.gregor, klimek, rsmith, djasper

Reviewed By: doug.gregor

CC: cfe-commits, chandlerc

Differential Revision: http://llvm-reviews.chandlerc.com/D28

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162810 91177308-0d34-0410-b5e6-96231b3b80d8
64f9b500987394c69cd9a2f5483d06b5cafb0824 28-Aug-2012 Dmitri Gribenko <gribozavr@gmail.com> Fix a -Wdocumentation warning.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162721 91177308-0d34-0410-b5e6-96231b3b80d8
70517ca5c07c4b41ff8662b94ee22047b0299f8c 23-Aug-2012 Dmitri Gribenko <gribozavr@gmail.com> Fix a bunch of -Wdocumentation warnings.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162452 91177308-0d34-0410-b5e6-96231b3b80d8
ad0fe03b897f9486191e75c8d90c3ffa9b4fd6a5 23-Aug-2012 Ted Kremenek <kremenek@apple.com> Fix an assortment of doxygen comment issues found by -Wdocumentation.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162412 91177308-0d34-0410-b5e6-96231b3b80d8
0576681bac125be07f77f66b02a3dba2c3a24557 18-Aug-2012 Richard Smith <richard-llvm@metafoo.co.uk> PR41111, PR5925, PR13210: Teach tentative parsing to annotate identifiers and
nested names as id-expressions, using the annot_primary_expr annotation, where
possible. This removes some redundant lookups, and also allows us to
typo-correct within tentative parsing, and to carry on disambiguating past an
identifier which we can determine will fail lookup as both a type and as a
non-type, allowing us to disambiguate more declarations (and thus offer
improved error recovery for such cases).

This also introduces to the parser the notion of a tentatively-declared name,
which is an identifier which we *might* have seen a declaration for in a
tentative parse (but only if we end up disambiguating the tokens as a
declaration). This is necessary to correctly disambiguate cases where a
variable is used within its own initializer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162159 91177308-0d34-0410-b5e6-96231b3b80d8
c141b51d78b40c7b0b04f27adfa70ec5bbddc49c 17-Aug-2012 Richard Smith <richard-llvm@metafoo.co.uk> Switch PTH format from a 7 byte magic number to an 8 byte one, to avoid
misaligned reads throughout the file. Bump PTH format version to 10.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162076 91177308-0d34-0410-b5e6-96231b3b80d8
bed28ac1d1463adca3ecf24fca5c30646fa9dbb2 23-Jul-2012 Sylvestre Ledru <sylvestre@debian.org> Fix a typo (the the => the)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@160622 91177308-0d34-0410-b5e6-96231b3b80d8
efce31f51d6e7e31e125f96c20f6cdab3ead0a47 22-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Primarily fixed \param commands with names not matching any actual
parameters of the documented functions. In many cases this consists
just of fixing up the parameter name in the \param to match the code,
in some it means deleting obsolete documentation and occasionally it
means documenting the parameter that has replaced the older one that
was documented, which sometimes means some simple reverse-engineering
of the docs from the implementation;
* Fixed \param ParamName [out] to the correct format with [out] before
the parameter name;
* Fixed some \brief summaries.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158980 91177308-0d34-0410-b5e6-96231b3b80d8
27289b1b8f249bd5bea46b215b5f2e72bbe1c6d4 22-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Added \file documentation for PPCallbacks.h;
* Added/formated \brief summaries;
* Deleted documentation for parameters that no longer exist;
* Used \param more systematically for documentation of parameters;
* Escaped # characters in Doxygen comments.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158978 91177308-0d34-0410-b5e6-96231b3b80d8
36be6c43533194ed5c7613f33851c7bc390f1c8e 22-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup: escape # characters in Doxygen comments as needed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158972 91177308-0d34-0410-b5e6-96231b3b80d8
dc3d8c9c7094f89438407b97795b3c993bb3d6fe 22-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup: escape # characters in Doxygen comments as needed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158969 91177308-0d34-0410-b5e6-96231b3b80d8
5f0cf083692ddc05a8c4bcf1e2e91b57272f4e97 22-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup: Fixed the format of the file header (that I broke in
a recent commit), and eliminated a Doxygen error by changing a comment inside
a function to not be a Doxygen comment.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158967 91177308-0d34-0410-b5e6-96231b3b80d8
58bf98725b6d4588338e191d2ab981b104471dab 22-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Escaped # and < characters in Doxygen comments as needed;
* Fixed up some \brief summaries;
* Marked up some parameter references with \p;
* Added \code...\endcode around code examples;
* Used \returns a little more.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158966 91177308-0d34-0410-b5e6-96231b3b80d8
198129e24b56ffcc85f66e798d52a8614e3a07a4 21-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup: escape a < character in a Doxygen comment

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158861 91177308-0d34-0410-b5e6-96231b3b80d8
206d7db051d509e836e646377d1ecaffb35ca646 20-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup: escape # character in a Doxygen comment.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158775 91177308-0d34-0410-b5e6-96231b3b80d8
80b5a2fea27561dce90c7588181185130d61e55e 20-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Escaped # characters in Doxygen comments as needed;
* Added/fixed \file and \brief markup to add summaries to Doxygen output.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158773 91177308-0d34-0410-b5e6-96231b3b80d8
aa0cd85838f2a024e589ea4e8c2094130065af21 20-Jun-2012 Dmitri Gribenko <gribozavr@gmail.com> Structured comment parsing, first step.

* Retain comments in the AST
* Serialize/deserialize comments
* Find comments attached to a certain Decl
* Expose raw comment text and SourceRange via libclang

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158771 91177308-0d34-0410-b5e6-96231b3b80d8
a24bd5d9ad79be50cec0e25364d8267e7623c33f 17-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Escaped # characters in Doxygen comments as needed;
* Exposed summaries using the \brief command.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158636 91177308-0d34-0410-b5e6-96231b3b80d8
071662d971e51ac64e0fe16d3baf9d85894b09a5 17-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Escaped # characters in Doxygen comments as needed;
* Added/reformatted \brief docs;
* Used a \file comment to document the file (MultipleIncludeOpt.h).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158635 91177308-0d34-0410-b5e6-96231b3b80d8
3d7e24e1887fade254cf8b304834f30de2f8d662 17-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup: add parameter name to \param

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158633 91177308-0d34-0410-b5e6-96231b3b80d8
a1e92ace653fb15742dc6d35198a9da433d90e15 17-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Escaped # characters in Doxygen comments;
* Added \brief markup to provide summaries.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158632 91177308-0d34-0410-b5e6-96231b3b80d8
7e6f1a23fe26748a4eafc5c64ff6c319f6d932f3 17-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup: escaping #define in Doxygen comments

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158629 91177308-0d34-0410-b5e6-96231b3b80d8
63333619fa68c8d1d8219f6d7f2d3c36f4356346 15-Jun-2012 James Dennett <jdennett@google.com> Documentation cleanup:
* Escaped the # of #define in Doxygen comments;
* Formatting: Annotated __VA_ARGS__ with \c;
* Converted docs to use \brief to provide summaries;
* Fixed a typo: disbles -> disables.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158553 91177308-0d34-0410-b5e6-96231b3b80d8
260328c27ddbad6e6d81a84e68a9a4aedccbfbf3 15-Jun-2012 James Dennett <jdennett@google.com> Documentation fix: Add a missing parameter name for a \param command

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158512 91177308-0d34-0410-b5e6-96231b3b80d8
809d1be9820039b4cf6efa48246a0d70ffa13394 14-Jun-2012 James Dennett <jdennett@google.com> More doxygen/documentation cleanups.

This reduces the number of warnings generated by Doxygen by about 100
(roughly 10%). Issues addressed:
(1) Primarily, backslash-escaped "@foo" and "#bah" in Doxygen comments
when they're not supposed to be Doxygen commands or links, and
similarly for "<baz>" when it's not intended as as HTML tag;
(2) Changed some \t commands (which don't exist) to \c ("to refer to a
word of code", as the Doxygen manual says);
(3) \precondition becomes \pre;
(4) When touching comments, deleted a couple of spurious spaces in them;
(5) Changed some \n and \r to \\n and \\r;
(6) Fixed one tiny typo: #pragms -> #pragma.

This patch touches documentation/comments only.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158422 91177308-0d34-0410-b5e6-96231b3b80d8
f122a138e39dbb29162abfa9a3d44091d8efa7af 13-Jun-2012 Richard Smith <richard-llvm@metafoo.co.uk> Add -isystem-prefix and -ino-system-prefix arguments, which can be used to
override whether headers are system headers by checking for prefixes of the
header name specified in the #include directive.

This allows warnings to be disabled for third-party code which is found in
specific subdirectories of include paths.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158418 91177308-0d34-0410-b5e6-96231b3b80d8
092bf67e5ca560d2fc6aa70be1f172b8b3a5ff96 09-Jun-2012 Dmitri Gribenko <gribozavr@gmail.com> Correct method name in comment: from LexRawToken to LexFromRawLexer, according
to a change done long ago in r57393.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158243 91177308-0d34-0410-b5e6-96231b3b80d8
6fe6a49c4058211ff4489023c78615ec0266c5ff 08-Jun-2012 Jordan Rose <jordan_rose@apple.com> Disable _Pragma during HTML macro rewriting to keep from crashing.

The preprocessor's handling of diagnostic push/pops is stateful, so
encountering pragmas during a re-parse causes problems. HTMLRewrite
already filters out normal # directives including #pragma, so it's
clear it's not expected to be interpreting pragmas in this mode.

This fix adds a flag to Preprocessor to explicitly disable pragmas.
The "right" fix might be to separate pragma lexing from pragma
parsing so that we can throw away pragmas like we do preprocessor
directives, but right now it's important to get the fix in.

Note that this has nothing to do with the "hack" of re-using the
input preprocessor in HTMLRewrite. Even if we someday copy the
preprocessor instead of re-using it, the copy would (and should) include
the diagnostic level tables and have the same problems.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158214 91177308-0d34-0410-b5e6-96231b3b80d8
d880b3aa6d594d1a7f2d307c29378c6f59b216ff 07-Jun-2012 Jordan Rose <jordan_rose@apple.com> Insert a space if necessary when suggesting CFBridgingRetain/Release.

This was a problem for people who write 'return(result);'

Also fix ARCMT's corresponding code, though there's no test case for this
because implicit casts like this are rejected by the migrator for being
ambiguous, and explicit casts have no problem.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158130 91177308-0d34-0410-b5e6-96231b3b80d8
8c0b3787e7ccc7978b42dfbb84da2b802c743a5d 06-Jun-2012 David Blaikie <dblaikie@gmail.com> Add a -rewrite-includes option, which is similar to -rewrite-macros, but only expands #include directives.

Patch contributed by Lubos Lunak (l.lunax@suse.cz).
Review by Matt Beaumont-Gay (matthewbg@google.com).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158093 91177308-0d34-0410-b5e6-96231b3b80d8
3093b20d824a953d8bc7a786dd952414898f8d6d 18-May-2012 Benjamin Kramer <benny.kra@googlemail.com> Lexer::ReadToEndOfLine: Only build the string if it's actually used and do so in a less malloc-intensive way.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@157064 91177308-0d34-0410-b5e6-96231b3b80d8
66c44e700fb3f244b2c443bfbd6a05b5f1843ec0 10-May-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] Make sure that MacroExpands callbacks are always in source order.

Fixes assertion hit in the preprocessing record. rdar://11426523

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@156557 91177308-0d34-0410-b5e6-96231b3b80d8
95da18142abeec5099dedc50d6f418917bd6167d 04-May-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Add more comments for Lexer::makeFileCharRange.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@156188 91177308-0d34-0410-b5e6-96231b3b80d8
3144749f8bf9bbf7c027f2161a930bff80ad6f72 03-May-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> In StringLiteralParser::init, make sure we emit an error when
failing to lex the string, as suggested by Eli.

Part of rdar://11305263.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@156081 91177308-0d34-0410-b5e6-96231b3b80d8
85ff9693b178658f9d8af7be30a086fb1ab81fdd 05-Apr-2012 Daniel Dunbar <daniel@zuster.org> [Lex] Add support for 'user specified system frameworks' (see test case).
- Developers of system frameworks need a way for their framework to be treated as a "system framework" during development. Otherwise, they are unable to properly test how their framework behaves when installed because of the semantic changes (in warning behavior) applied to system frameworks.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@154105 91177308-0d34-0410-b5e6-96231b3b80d8
9ee35f9f35452dec05c81fd1bbdd2f700872ea7f 05-Apr-2012 Daniel Dunbar <daniel@zuster.org> [Lex] HeaderSearch: Introduce a FrameworkCacheEntry structure to hold the FrameworkMap items.
- No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@154104 91177308-0d34-0410-b5e6-96231b3b80d8
14e645557ae91c6770d62beb00a1c522e0bfd5d6 03-Apr-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Correct handling of _Pragma macro inside a macro argument.

If we are pre-expanding a macro argument don't actually "activate"
the pragma at that point, activate the pragma whenever we encounter
it again in the token stream.
This ensures that we will activate it in the correct location
or that we will ignore it if it never enters the token stream, e.g:

\#define EMPTY(x)
\#define INACTIVE(x) EMPTY(x)
INACTIVE(_Pragma("clang diagnostic ignored \"-Wconversion\""))

This also fixes the crash in rdar://11168596.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@153959 91177308-0d34-0410-b5e6-96231b3b80d8
e2e8768c290ceceb3ac2e369b5243f40d6c0e3e2 31-Mar-2012 Abramo Bagnara <abramo.bagnara@gmail.com> Fixed MacroInfo definition range for function like macros with empty body.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@153819 91177308-0d34-0410-b5e6-96231b3b80d8
db81d38d9cd468a9eeffe8ab092be4d48e43888e 27-Mar-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] Handle correctly inclusion directives that have macro expansions, e.g
"#include MACRO(STUFF)".

-As an inclusion position for the included file, use the file location of the file where it
was included but *after* the macro expansions. We want the macro expansions to be considered
as before-in-translation-unit for everything in the included file.

-In the preprocessing record take into account that only inclusion directives can be encountered
as "out-of-order" (by comparing the start of the range which for inclusions is the hash location)
and use binary search if there is an extreme number of macro expansions in the include directive.

Fixes rdar://11111779

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@153527 91177308-0d34-0410-b5e6-96231b3b80d8
4207edaf03da28fa917a94b31c5c1eef0e6417dc 18-Mar-2012 Aaron Ballman <aaron@aaronballman.com> Turns #import in MS Mode into an error.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@153009 91177308-0d34-0410-b5e6-96231b3b80d8
e55329d6834647ba0e06f8a319e5d84c77310035 16-Mar-2012 Axel Naumann <Axel.Naumann@cern.ch> From Vassil Vassilev:
Enable incremental parsing by the Preprocessor,
where more code can be provided after an EOF.
It mainly prevents the tearing down of the topmost lexer.
To be used like this:
while (getMoreSource()) {
while (Parser.ParseTopLevelDecl(ADecl)) {...}

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152914 91177308-0d34-0410-b5e6-96231b3b80d8
4e4d08403ca5cfd4d558fa2936215d3a4e5a528d 11-Mar-2012 David Blaikie <dblaikie@gmail.com> Unify naming of LangOptions variable/get function across the Clang stack (Lex to AST).

The member variable is always "LangOpts" and the member function is always "getLangOpts".

Reviewed by Chris Lattner

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152536 91177308-0d34-0410-b5e6-96231b3b80d8
aa49a7d70e58dac2aeb40664ba16d2ea571b8c95 09-Mar-2012 Daniel Dunbar <daniel@zuster.org> [AST/etc] Mark {getSourceRange(),getStartLoc(),getEndLoc()} as LLVM_READONLY.
- The theory here is that we have these functions sprinkled in all over the
place. This should allow the optimizer to at least realize it can still do
load CSE across these calls.
- I blindly marked all instances as such, even though the optimizer can infer
this attribute in some instances (some of the inline ones) as that was easier
and also, when given the choice between thinking and not thinking, I prefer
the latter.

You might think this is mere frivolity, but actually this is good for a .7 -
1.1% speedup on 403.gcc/combine.c, JSC/Interpreter.cpp,

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152426 91177308-0d34-0410-b5e6-96231b3b80d8
b453ad3214d00acc51c9aa702c76c58354d84b84 08-Mar-2012 Richard Smith <richard-llvm@metafoo.co.uk> Add support for cooked forms of user-defined-integer-literal and
user-defined-floating-literal. Support for raw forms of these literals
to follow.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152302 91177308-0d34-0410-b5e6-96231b3b80d8
dd66be718f23c8149d74ae8b011b002e11e8d5ba 08-Mar-2012 Richard Smith <richard-llvm@metafoo.co.uk> User-defined literal support for character literals.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152277 91177308-0d34-0410-b5e6-96231b3b80d8
99831e4677a7e2e051af636221694d60ba31fcdb 06-Mar-2012 Richard Smith <richard-llvm@metafoo.co.uk> User-defined literals: reject string and character UDLs in all places where the
grammar requires a string-literal and not a user-defined-string-literal. The
two constructs are still represented by the same TokenKind, in order to prevent
a combinatorial explosion of different kinds of token. A flag on Token tracks
whether a ud-suffix is present, in order to prevent clients from needing to look
at the token's spelling.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152098 91177308-0d34-0410-b5e6-96231b3b80d8
3ebdd8548f97dc5b77172122ab7d6b65de8d4609 06-Mar-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Add some doxygen comments.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152075 91177308-0d34-0410-b5e6-96231b3b80d8
c6c54521f95760a5eaf29b668d4bf41fe2af49d7 05-Mar-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] Enhance PreprocessingRecord to keep track of locations of conditional directives.

Introduce PreprocessingRecord::rangeIntersectsConditionalDirective() which returns
true if a given range intersects with a conditional directive block.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152018 91177308-0d34-0410-b5e6-96231b3b80d8
bb660666883de8b32999c1e5fbe3c32e2cafbdf6 05-Mar-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> [preprocessor] Enhance the preprocessor callbacks:

-Add location parameter for the directives callbacks
-Skip callbacks if the directive is inside a skipped range.
-Make sure the directive callbacks are invoked in source order.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152017 91177308-0d34-0410-b5e6-96231b3b80d8
5cc2c6eb67b6e5361bbe96f79b519fd62ec666d6 05-Mar-2012 Richard Smith <richard-llvm@metafoo.co.uk> Lexing support for user-defined literals. Currently these lex as the same token
kinds as the underlying string literals, and we silently drop the ud-suffix;
those issues will be fixed by subsequent patches.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152012 91177308-0d34-0410-b5e6-96231b3b80d8
4c55c54db8e676aa3e042188773d9d82d59fff91 02-Mar-2012 Aaron Ballman <aaron@aaronballman.com> Adding support for #pragma include_alias in MS compatibility mode. This implements PR 10705.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@151949 91177308-0d34-0410-b5e6-96231b3b80d8
10285d9113c14d1e523f86a55b193eb752638ea5 01-Mar-2012 Richard Smith <richard-llvm@metafoo.co.uk> Revert r151800, which was committed without review and has correctness issues.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@151804 91177308-0d34-0410-b5e6-96231b3b80d8
701e3366828adf740734ae5bf4424ecac059bc9a 01-Mar-2012 Aaron Ballman <aaron@aaronballman.com> Fixing a buildbot complaint about nested templates.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@151803 91177308-0d34-0410-b5e6-96231b3b80d8
7abe1666f225b6d1a11aa7ed19d9a0dcc49391cb 01-Mar-2012 Aaron Ballman <aaron@aaronballman.com> Implements support for #pragma include_alias in ms compatibility mode. Fixes PR10705.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@151800 91177308-0d34-0410-b5e6-96231b3b80d8
f4fb07ed4dfd144a39d904fc77b3b0bab580eb2e 29-Feb-2012 Nico Weber <nicolasweber@gmx.de> Allow operator keywords to be #defined in ms-ext mode.

Fixes PR10606.

I'm not sure if this is the best way to go about it, but
I locally enabled this code path without the msext conditional,
and all tests pass, except for test/Preprocessor/cxx_oper_keyword.cpp
which explicitly checks that operator keywords can't be redefined.

I also parsed chromium/win with a clang with and without this patch.
It introduced no new errors, but removes 43 existing errors.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@151768 91177308-0d34-0410-b5e6-96231b3b80d8
e1d4330adaaa7faf093e725c9c993207eb2d778a 25-Feb-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Don't record nested macro expansions in the preprocessing record,
it can only bring pain when dealing with preprocessor abuse (see: boost).


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@151427 91177308-0d34-0410-b5e6-96231b3b80d8
c93dc7889644293e318e19d82830ea2acc45b678 20-Feb-2012 Dylan Noblesmith <nobled@dreamwidth.org> Basic: import IntrusiveRefCntPtr<> into clang namespace

The class name is long enough without the llvm:: added.
Also bring in RefCountedBase and RefCountedBaseVPTR.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@150958 91177308-0d34-0410-b5e6-96231b3b80d8
91359302b822d829afa93c0dadf5f7ce6e19fbc6 11-Feb-2012 Eli Friedman <eli.friedman@gmail.com> Implement warning for non-wide string literals with an unexpected encoding. Downgrade error for non-wide character literals with an unexpected encoding to a warning for compatibility with gcc and older versions of clang. <rdar://problem/10837678>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@150295 91177308-0d34-0410-b5e6-96231b3b80d8
f94d392847fb183b3012dbcb2e372b586c80b9fb 09-Feb-2012 Benjamin Kramer <benny.kra@googlemail.com> Kill the brief and full explanation fields from StaticDiagInfoRec. They were unused and wasted space for nothing.

- per PR11952.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@150199 91177308-0d34-0410-b5e6-96231b3b80d8
d7a3e2c5f61cd4893f95b69a424fe4def3aa0f69 07-Feb-2012 Benjamin Kramer <benny.kra@googlemail.com> Revert my patches which removed Diagnostic.h includes by moving some operator overloads out of line.

This seems to negatively affect compile time onsome ObjC tests
(which use a lot of partial diagnostics I assume). I have to come
up with a way to keep them inline without including Diagnostic.h
everywhere. Now adding a new diagnostic requires a full rebuild
of e.g. the static analyzer which doesn't even use those diagnostics.

This reverts commit 6496bd10dc3a6d5e3266348f08b6e35f8184bc99.
This reverts commit 7af19b817ba964ac560b50c1ed6183235f699789.
This reverts commit fdd15602a42bbe26185978ef1e17019f6d969aa7.
This reverts commit 00bd44d5677783527d7517c1ffe45e4d75a0f56f.
This reverts commit ef9b60ffed980864a8db26ad30344be429e58ff5.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@150006 91177308-0d34-0410-b5e6-96231b3b80d8
f7ccbad5d9949e7ddd1cbef43d482553b811e026 05-Feb-2012 Dylan Noblesmith <nobled@dreamwidth.org> Basic: import SmallString<> into clang namespace

(I was going to fix the TODO about DenseMap too, but
that would break self-host right now. See PR11922.)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149799 91177308-0d34-0410-b5e6-96231b3b80d8
6f42b62b6194f53bcbc349f5d17388e1936535d7 05-Feb-2012 Dylan Noblesmith <nobled@dreamwidth.org> Basic: import OwningPtr<> into clang namespace

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149798 91177308-0d34-0410-b5e6-96231b3b80d8
8fe83e1df954d72c0f4ffc15d20a5222ec151c21 04-Feb-2012 Benjamin Kramer <benny.kra@googlemail.com> Move a method from IdentifierTable.h out of line and remove the SmallString include.

Fix all the transitive include users.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149783 91177308-0d34-0410-b5e6-96231b3b80d8
fdd15602a42bbe26185978ef1e17019f6d969aa7 04-Feb-2012 Benjamin Kramer <benny.kra@googlemail.com> Remove Diagnostic.h include from Preprocessor.h.

- Move the offending methods out of line and fix transitive includers.
- This required changing an enum in the PPCallback API into an unsigned.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149782 91177308-0d34-0410-b5e6-96231b3b80d8
a83f4d2315dbeb3914868f1ccb8e74fb2ccdbb0c 03-Feb-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Change Lexer::makeFileCharRange() to have it accept a CharSourceRange
instead of a SourceRange, and handle the case where the range is
a char (not token) range.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149677 91177308-0d34-0410-b5e6-96231b3b80d8
2f04f1843ca0ffca13b8b0d4dadd1f50dffb38b8 02-Feb-2012 Douglas Gregor <dgregor@apple.com> Back out my heinous hack that tricked the module generation mechanism
into using non-absolute system includes (<foo>)...

... and introduce another hack that is simultaneously more heineous
and more effective. We whitelist Clang-supplied headers that augment
or override system headers (such as float.h, stdarg.h, and
tgmath.h). For these headers, Clang does not provide a module
mapping. Instead, a system-supplied module map can refer to these
headers in a system module, and Clang will look both in its own
include directory and wherever the system-supplied module map
suggests, then adds either or both headers. The end result is that
Clang-supplied headers get merged into the system-supplied module for
the C standard library.

As a drive-by, fix up a few dependencies in the _Builtin_instrinsics

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149611 91177308-0d34-0410-b5e6-96231b3b80d8
dc58aa71026cce539ca9b5c2c52cc4efc7bd77fe 30-Jan-2012 Douglas Gregor <dgregor@apple.com> Thread a TargetInfo through to the module map; we'll need it for
target-specific module requirements.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149224 91177308-0d34-0410-b5e6-96231b3b80d8
c5b2e58840748145d1706c1d1481369d1863fabf 29-Jan-2012 Douglas Gregor <dgregor@apple.com> Implement code completion support for module import declarations, e.g.,

@import <complete with module names here>


@import std.<complete with submodule names here>

Addresses <rdar://problem/10710117>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149199 91177308-0d34-0410-b5e6-96231b3b80d8
e434ec71fccfe078906403affd641f709702d598 29-Jan-2012 Douglas Gregor <dgregor@apple.com> Rework HeaderSearch's interface for getting a module from a name and
for getting the name of the module file, unifying the code for
searching for a module with a given name (into lookupModule()) and
separating out the mapping to a module file (into
getModuleFileName()). No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149197 91177308-0d34-0410-b5e6-96231b3b80d8
a1f1fad8b60e1cb9d21a40a37f2e03150bcbeb6f 27-Jan-2012 Douglas Gregor <dgregor@apple.com> Introduce module attributes into the module map grammar, along with a
single attribute ("system") that allows us to mark a module as being a
"system" module. Each of the headers that makes up a system module is
considered to be a system header, so that we (for example) suppress
warnings there.

If a module is being inferred for a framework, and that framework
directory is within a system frameworks directory, infer it as a
system framework.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149143 91177308-0d34-0410-b5e6-96231b3b80d8
5d5051f3e79bf754134ccdf7a1dc7778cd80c73e 24-Jan-2012 Douglas Gregor <dgregor@apple.com> Only mark an IdentifierInfo as having changed since deserialization
when it actually has changed (and not, e.g., when we've simply attached a
deserialized macro definition). Good for ~1.5% reduction in module
file size, mostly in the identifier table.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148808 91177308-0d34-0410-b5e6-96231b3b80d8
e64d9037658a1b95c79ea275af6167a110b3c563 19-Jan-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce Lexer::getSourceText() that returns a string for the source
that the given source range encompasses.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148481 91177308-0d34-0410-b5e6-96231b3b80d8
11b652d41d0d97380ab321a1dba48ecb044f9de8 19-Jan-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce Lexer::makeFileCharRange() that accepts a token source range
and returns a character range with file locations.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148480 91177308-0d34-0410-b5e6-96231b3b80d8
69bda4c027671df7163619f215209529eb236620 19-Jan-2012 Argyrios Kyrtzidis <akyrtzi@gmail.com> For Lexer's isAt[Start/End]OfMacroExpansion add an out parameter for the macro
start/end location.

It is commonly needed after calling the function; with this way we avoid
recalculating it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148479 91177308-0d34-0410-b5e6-96231b3b80d8
c2a8d6cee01fc4845f5409bf5c021a64616ac8c3 18-Jan-2012 Anna Zaks <ganna@apple.com> Refactor: Pull getImmediateMacroName() out of DiagnosticRenderer and
into Lexer and Preprocessor; making it widely available.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148410 91177308-0d34-0410-b5e6-96231b3b80d8
d6aba06861c41ccbc4926e5fe3cecd97b20410c0 04-Jan-2012 Douglas Gregor <dgregor@apple.com> Don't treat 'import' as a contextual keyword when we're in a caching lexer, or when modules are disabled.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147524 91177308-0d34-0410-b5e6-96231b3b80d8
c13a34b690d2dc2a03c2fea75a0a1438636c19ce 03-Jan-2012 Douglas Gregor <dgregor@apple.com> Eliminate the uglified keyword __import_module__ for importing
modules. This leaves us without an explicit syntax for importing
modules in C/C++, because such a syntax needs to be discussed
first. In Objective-C/Objective-C++, the @import syntax is used to
import modules.

Note that, under -fmodules, C/C++ programs can import modules via the
#include mechanism when a module map is in place for that header. This
allows us to work with modules in C/C++ without committing to a syntax.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147467 91177308-0d34-0410-b5e6-96231b3b80d8
94ad28b31433058445a27db722f60402ee820bea 03-Jan-2012 Douglas Gregor <dgregor@apple.com> Under -fmodules, accept #public <macroname> and #private <macroname>
to make a macro public (the default for headers) or private,

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147455 91177308-0d34-0410-b5e6-96231b3b80d8
5948ae1021122164b22f74353bb7fe325a64f616 03-Jan-2012 Douglas Gregor <dgregor@apple.com> Introduce a non-uglified syntax for module imports in Objective-C:

@import identifier [. identifier]* ;

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147452 91177308-0d34-0410-b5e6-96231b3b80d8
752c74d99b647710a495c2ff5f815c30a30c3264 03-Jan-2012 Douglas Gregor <dgregor@apple.com> Add a "Modules" language option, which subsumes the previous
"AutoModuleImport" preprocessor option and is tied to -fmodules.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147448 91177308-0d34-0410-b5e6-96231b3b80d8
51f564f80d9f71e175635b452ffeeeff899e9bf1 31-Dec-2011 Douglas Gregor <dgregor@apple.com> Implement support for module requirements, which indicate the language
features needed for a particular module to be available. This allows
mixed-language modules, where certain headers only work under some
language variants (e.g., in C++, std.tuple might only be available in
C++11 mode).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147387 91177308-0d34-0410-b5e6-96231b3b80d8
99ba9e3bd70671f3441fb974895f226a83ce0e66 20-Dec-2011 David Blaikie <dblaikie@gmail.com> Unweaken vtables as per http://llvm.org/docs/CodingStandards.html#ll_virtual_anch

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@146959 91177308-0d34-0410-b5e6-96231b3b80d8
cb381eac84e5a14a8c7e7654eadbe1d3d54d795c 09-Dec-2011 Chandler Carruth <chandlerc@gmail.com> Move a free function from the Frontend library into the Lex library as
part of HeaderSearch. This function just normalizes filenames for use
inside of a synthetic include directive, but it is used in both the
Frontend and Serialization libraries so it needs a common home.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@146227 91177308-0d34-0410-b5e6-96231b3b80d8
77d029f6a24dbf70d97e61757945df53fb250ea0 08-Dec-2011 Douglas Gregor <dgregor@apple.com> Implement umbrella directories for modules, which are similar to
umbrella headers in the sense that all of the headers within that
directory (and eventually its subdirectories) are considered to be
part of the module with that umbrella directory. However, unlike
umbrella headers, which are expected to include all of the headers
within their subdirectories, Clang will automatically include all of
the headers it finds in the named subdirectory.

The intent here is to allow a module map to trivially turn a
subdirectory into a module, where the module's structure can mimic the
directory structure.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@146165 91177308-0d34-0410-b5e6-96231b3b80d8
5e3f9223db88227d6d21679c613b139d8160186d 08-Dec-2011 Douglas Gregor <dgregor@apple.com> Keep track of import dependencies between submodules within the module
that's currently being built. This is important for supporting
transitive dependencies ("export *" in the module map) completely.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@146156 91177308-0d34-0410-b5e6-96231b3b80d8
ac252a3b0f8101a7274309e4a5cf2d5f0fdba675 06-Dec-2011 Douglas Gregor <dgregor@apple.com> When inferring a module map for a framework, infer subframework
modules for each of its subframeworks.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145957 91177308-0d34-0410-b5e6-96231b3b80d8
e209e5026892cb07294f733c72bd51359c0f0e72 06-Dec-2011 Douglas Gregor <dgregor@apple.com> Implement inferred submodules support, which (when requested)
implicitly generates submodules corresponding to the headers that fall
within a module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145887 91177308-0d34-0410-b5e6-96231b3b80d8
55988680ece66b8e505ee136b35e74fcb1173aee 05-Dec-2011 Douglas Gregor <dgregor@apple.com> When writing a module file, keep track of the set of (sub)modules that
it imports, establishing dependencies at the (sub)module
granularity. This is not a user-visible change (yet).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145808 91177308-0d34-0410-b5e6-96231b3b80d8
93ebfa6139bbca4d446c7343e3afc8e5ec777484 03-Dec-2011 Douglas Gregor <dgregor@apple.com> When we treat an #include or #import as a module import, create an
implicit ImportDecl in the translation unit to record the presence of
the import.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145727 91177308-0d34-0410-b5e6-96231b3b80d8
90db26000aefe9335370013eec64c85232d80227 02-Dec-2011 Douglas Gregor <dgregor@apple.com> Implementing parsing and resolution of module export declarations
within module maps, which will (eventually) be used to re-export a
module from another module. There are still some pieces missing,

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145665 91177308-0d34-0410-b5e6-96231b3b80d8
5e35693721364673f8196e4f5a370f56b92e6053 01-Dec-2011 Douglas Gregor <dgregor@apple.com> Introduce the notion of name visibility into modules. For a given
(sub)module, all of the names may be hidden, just the macro names may
be exposed (for example, after the preprocessor has seen the import of
the module but the parser has not), or all of the names may be
exposed. Importing a module makes its names, and the names in any of
its non-explicit submodules, visible to name lookup (transitively).

This commit only introduces the notion of name visible and marks
modules and submodules as visible when they are imported. The actual
name-hiding logic in the AST reader will follow (along with test cases).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145586 91177308-0d34-0410-b5e6-96231b3b80d8
1a4761edca58c6b559de825b9abfb66f7f1ba94a 01-Dec-2011 Douglas Gregor <dgregor@apple.com> Promote ModuleMap::Module to a namespace-scope class in the Basic
library, since modules cut across all of the libraries. Rename
serialization::Module to serialization::ModuleFile to side-step the
annoying naming conflict. Prune a bunch of ModuleMap.h includes that
are no longer needed (most files only needed the Module type).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145538 91177308-0d34-0410-b5e6-96231b3b80d8
392ed2b717d86ebdd202cb9bb58d18d8b3b4cd87 30-Nov-2011 Douglas Gregor <dgregor@apple.com> Implement (de-)serialization of the description of a module and its
submodules. This information will eventually be used for name hiding
when dealing with submodules. For now, we only use it to ensure that
the module "key" returned when loading a module will always be a
module (rather than occasionally being a FileEntry).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145497 91177308-0d34-0410-b5e6-96231b3b80d8
b514c792821a8f053027d88444e13bfaa8efef76 30-Nov-2011 Douglas Gregor <dgregor@apple.com> Teach the preprocessor how to handle module import declarations that
involve submodules (e.g., importing std.vector), rather than always
importing the top-level module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145478 91177308-0d34-0410-b5e6-96231b3b80d8
3d3589db579f7695667b913c5043dd264ebe546f 30-Nov-2011 Douglas Gregor <dgregor@apple.com> Switch the module-loading interfaces and parser from a simple
top-level module name to a module path (e.g., std.vector). We're still
missing a number of pieces for this actually to do something.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145462 91177308-0d34-0410-b5e6-96231b3b80d8
f9e357d8a66c606a86a6e1aef678898b8843bd30 29-Nov-2011 Douglas Gregor <dgregor@apple.com> Teach the module import mechanism how to rebuild modules expressed via
module map, rather than assuming that there is an umbrella
header. This allows us to automatically build umbrella-less modules.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145415 91177308-0d34-0410-b5e6-96231b3b80d8
804c3bfee22076f232dddf4839439119cfdee2b6 29-Nov-2011 Douglas Gregor <dgregor@apple.com> Expose the printing of module maps as part of the ModuleMap::Module
interface. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145411 91177308-0d34-0410-b5e6-96231b3b80d8
a4d36a6dd00c1495cfe3b64f949d70ba9f778391 29-Nov-2011 Douglas Gregor <dgregor@apple.com> When attempting to load a module that is not in the module cache,
return the module itself (in the module map) rather than returning the
umbrella header used to build the module. While doing this, make sure
that we're inferring modules for frameworks to build that module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145310 91177308-0d34-0410-b5e6-96231b3b80d8
1c2e9332fa69727425a3a2b912e36e2ab62083f8 20-Nov-2011 Douglas Gregor <dgregor@apple.com> Allow preprocessor callbacks to recover from a "file not found" error,
from Jason Haslam!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145012 91177308-0d34-0410-b5e6-96231b3b80d8
c69c42e939e6bdaa56d162cc36da4f6b6c53e8db 17-Nov-2011 Douglas Gregor <dgregor@apple.com> When making a suggestion regarding which module to load rather than
preprocess/parse a header, report back with an actual module (which
may be a submodule) rather than just the name of the module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144925 91177308-0d34-0410-b5e6-96231b3b80d8
a865405e4155e8ea83d7ff1a1d8316907c300897 17-Nov-2011 Douglas Gregor <dgregor@apple.com> Add the notion of "framework" modules to module maps. Framework
modules (obviously) describe frameworks, and understand the header
layout of frameworks.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144921 91177308-0d34-0410-b5e6-96231b3b80d8
09fe1bb696847e6f1b482e5ac40029d53a2402df 17-Nov-2011 Douglas Gregor <dgregor@apple.com> Actually free memory for the module maps

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144880 91177308-0d34-0410-b5e6-96231b3b80d8
2821c7f8870629b56b9c41e1c50c7a091edd544d 17-Nov-2011 Douglas Gregor <dgregor@apple.com> When we're loading a framework header, first try to turn the framework
into a module. This module can either be loaded from a module map in
the framework directory (which isn't quite working yet) or inferred
from an umbrella header (which does work, and replaces the existing

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144877 91177308-0d34-0410-b5e6-96231b3b80d8
adb979924ade3e25342c38a5b564400b4e0540c1 17-Nov-2011 Douglas Gregor <dgregor@apple.com> A module with an umbrella header assumes that all of the headers in
the umbrella header's directory and its subdirectories are part of the
module (that's why it's an umbrella). Make sure that these headers are
considered to be part of the module for lookup purposes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144859 91177308-0d34-0410-b5e6-96231b3b80d8
db1cde7dc7bb3aaf48118bd9605192ab94a93635 16-Nov-2011 Douglas Gregor <dgregor@apple.com> Add support for building a module from a module map to the -cc1
interface. This is currently limited to modules with umbrella

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144736 91177308-0d34-0410-b5e6-96231b3b80d8
590ad93bf59f4e5f6adcba0d78efc3a58cac15ce 13-Nov-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH] When chaining a PCH and serializing HeaderSearch, make sure the HeaderFileInfos
from the primary PCH is deserialized, otherwise we lose info that headers were
already #imported/#included.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144510 91177308-0d34-0410-b5e6-96231b3b80d8
26697979fb0a4e2b720a0c8d062047edca92bc92 12-Nov-2011 Douglas Gregor <dgregor@apple.com> Implement a minor optimization when loading module maps to satisfy a
module import: don't re-check for a loaded module unless we've
actually loaded a new module map file. Already-loaded module map files
aren't interesting.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144435 91177308-0d34-0410-b5e6-96231b3b80d8
cf70d7873fe3098bdac72e7628f4e832d14d5143 12-Nov-2011 Douglas Gregor <dgregor@apple.com> When searching for a module, speculatively load module maps to see if
the module is described in one of the module maps in a search path or
in a subdirectory off the search path that has the same name as the
module we're looking for.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144433 91177308-0d34-0410-b5e6-96231b3b80d8
484535e45b4d301847a157e943c7823da5d40884 12-Nov-2011 Douglas Gregor <dgregor@apple.com> Teach the search for modules to consider modules described by a module
map, so long as they have an umbrella header. This makes it possible
to introduce a module map + umbrella header for a given set of
headers, to turn it into a module.

There are two major deficiencies here: first, we don't go hunting for
module map files when we just see a module import (so we won't know
about the modules described therein). Second, we don't yet have a way
to build modules that don't have umbrella headers, or have incomplete
umbrella headers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144424 91177308-0d34-0410-b5e6-96231b3b80d8
65f3b5e99009f49d51eb00a859dbd2c2ee660718 11-Nov-2011 Douglas Gregor <dgregor@apple.com> Wire up the mapping from header files mentioned in module maps over to
the corresponding (top-level) modules. This isn't actually useful yet,
because we don't yet have a way to build modules out of module maps.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144410 91177308-0d34-0410-b5e6-96231b3b80d8
a30cfe5026b12c28b7b575b48176e0a3543ce939 11-Nov-2011 Douglas Gregor <dgregor@apple.com> Introduce basic support for parsing module map files.

Module map files provide a way to map between headers and modules, so
that we can layer a module system on top of existing headers without
changing those headers at all.

This commit introduces the module map file parser and the module map
that it generates, and wires up the module map file parser so that
we'll automatically find module map files as part of header
search. Note that we don't yet use the information stored in the
module map.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144402 91177308-0d34-0410-b5e6-96231b3b80d8
8e23806863721495f9e1f84aed614f7afba774a3 11-Nov-2011 Douglas Gregor <dgregor@apple.com> I predict that HeaderSearch will need the ability to generate
diagnostics in the future. Make it so.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144347 91177308-0d34-0410-b5e6-96231b3b80d8
ba243b59a1074e0962f6abfa3bb9aa984eac1245 09-Nov-2011 David Blaikie <dblaikie@gmail.com> Fixing 80 col violations (& removing any trailing whitespace on files I was touching anyway)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144171 91177308-0d34-0410-b5e6-96231b3b80d8
64f45a24b19eb89ff88f7c3ff0df9be8e861ac97 01-Nov-2011 Eli Friedman <eli.friedman@gmail.com> Fix the representation of wide strings in the AST and IR so that it uses the native representation of integers for the elements. This fixes a bunch of nastiness involving
treating wide strings as a series of bytes.

Patch by Seth Cantrell.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@143417 91177308-0d34-0410-b5e6-96231b3b80d8
f74a4587629615ffd13bd0724868f86ba8c8f27b 01-Nov-2011 Eli Friedman <eli.friedman@gmail.com> Perform proper conversion for strings encoded in the source file as UTF-8. (For now, we are assuming the source character set is always UTF-8; this can be easily extended if necessary.)

Tests will be coming up in a subsequent commit.

Patch by Seth Cantrell.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@143416 91177308-0d34-0410-b5e6-96231b3b80d8
eee242ff426bf79149f221798966e58688383c1e 27-Oct-2011 Douglas Gregor <dgregor@apple.com> Make the loading of information attached to an IdentifierInfo from an
AST file more lazy, so that we don't eagerly load that information for
all known identifiers each time a new AST file is loaded. The eager
reloading made some sense in the context of precompiled headers, since
very few identifiers were defined before PCH load time. With modules,
however, a huge amount of code can get parsed before we see an
@import, so laziness becomes important here.

The approach taken to make this information lazy is fairly simple:
when we load a new AST file, we mark all of the existing identifiers
as being out-of-date. Whenever we want to access information that may
come from an AST (e.g., whether the identifier has a macro definition,
or what top-level declarations have that name), we check the
out-of-date bit and, if it's set, ask the AST reader to update the
IdentifierInfo from the AST files. The update is a merge, and we now
take care to merge declarations before/after imports with declarations
from multiple imports.

The results of this optimization are fairly dramatic. On a small
application that brings in 14 non-trivial modules, this takes modules
from being > 3x slower than a "perfect" PCH file down to 30% slower
for a full rebuild. A partial rebuild (where the PCH file or modules
can be re-used) is down to 7% slower. Making the PCH file just a
little imperfect (e.g., adding two smallish modules used by a bunch of
.m files that aren't in the PCH file) tips the scales in favor of the
modules approach, with 24% faster partial rebuilds.

This is just a first step; the lazy scheme could possibly be improved
by adding versioning, so we don't search into modules we already
searched. Moreover, we'll need similar lazy schemes for all of the
other lookup data structures, such as DeclContexts.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@143100 91177308-0d34-0410-b5e6-96231b3b80d8
f226ff9fe8c8db6c5b74a61ce649eda1491c3502 25-Oct-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH] When visiting preprocessed entities, make it possible to avoid deserializing
preprocessed entities that are #included in the range that we are interested.

This is useful when we are interested in preprocessed entities of a specific file, e.g
when we are annotating tokens. There is also an optimization where we cache the last
result of PreprocessingRecord::getPreprocessedEntitiesInRange and we re-use it if
there is a call with the same range as before.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@142887 91177308-0d34-0410-b5e6-96231b3b80d8
aa93a875605536d72a10359a0098396192b7d4ec 17-Oct-2011 Douglas Gregor <dgregor@apple.com> For modules, all macros that aren't include guards are implicitly
public. Add a __private_macro__ directive to hide a macro, similar to
the __module_private__ declaration specifier.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@142188 91177308-0d34-0410-b5e6-96231b3b80d8
3d5f955855cab5a831cc6b9339dc77a7d46f1a30 14-Oct-2011 Douglas Gregor <dgregor@apple.com> Add a preprocessor callback that is invoked every time the 'defined'
operator is seen, from Jason Haslam!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141926 91177308-0d34-0410-b5e6-96231b3b80d8
d768150ef57f617c8d9fef48f3c92e8f21698024 12-Oct-2011 Ted Kremenek <kremenek@apple.com> Implement built-in macro '__has_warning', which allows one to query if a warning flag is valid. Fixes <rdar://problem/10263428>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141802 91177308-0d34-0410-b5e6-96231b3b80d8
d5e1d606f8c22ebda17c6fbf952f8c1696428758 12-Oct-2011 Richard Smith <richard-llvm@metafoo.co.uk> Handle Perforce-style conflict markers like normal conflict markers. Perforce
swaps over the <<<< and >>>> markers, and uses shorter markers than traditional

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141751 91177308-0d34-0410-b5e6-96231b3b80d8
98d86b98b3fd0bd9c546123b16fd9995509aaae1 11-Oct-2011 Richard Smith <richard-llvm@metafoo.co.uk> Add a -Wc++0x-compat warning for C++11 keywords used as identifiers when in
C++98 mode. Only the first occurrence of each keyword will produce a warning.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141700 91177308-0d34-0410-b5e6-96231b3b80d8
c892c5fa92db137ff68d95afb13fe969d17f4fb2 11-Oct-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> For the FileChanged Preprocessor callback, when exiting a file, pass its FileID.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141681 91177308-0d34-0410-b5e6-96231b3b80d8
6ee5b9384533d5b3f8c18b578fccd3935e1b892f 06-Oct-2011 Peter Collingbourne <peter@pcc.me.uk> Clang-side build system infrastructure for multiple tblgens.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141267 91177308-0d34-0410-b5e6-96231b3b80d8
a08529cc3f00e0b47a3c028823634129ac46847b 03-Oct-2011 Abramo Bagnara <abramo.bagnara@gmail.com> Fixed exapnsion range for # and ##.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141012 91177308-0d34-0410-b5e6-96231b3b80d8
8dfac0baaf0f81d3945bcb306480e358ba8d1f08 30-Sep-2011 John McCall <rjmccall@apple.com> Add explicit attributes to mark functions as having had their
CoreFoundation object-transfer properties audited, and add a #pragma
to cause them to be automatically applied to functions in a particular
span of code. This has to be implemented largely in the preprocessor
because of the requirement that the region be entirely contained in
a single file; that's hard to impose from the parser without registering
for a ton of callbacks.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140846 91177308-0d34-0410-b5e6-96231b3b80d8
4213df3b5da21ce25a4541ca5c447eeb28b515a3 29-Sep-2011 Daniel Dunbar <daniel@zuster.org> Basic/Diagnostics: Split out the default warning "no-Werror" and
"show-in-system-header" bits, which is part of teasing them apart from the
diagnostic mapping kind.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140742 91177308-0d34-0410-b5e6-96231b3b80d8
6b4ff04dd22e7159887ea6c1bb8b1c01fc9c70a9 27-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce a callback to PPCallbacks for lines skipped by the preprocessor.

Patch by Jason Haslam!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140612 91177308-0d34-0410-b5e6-96231b3b80d8
d6471f7c1921c7802804ce3ff6fe9768310f72b9 26-Sep-2011 David Blaikie <dblaikie@gmail.com> Rename Diagnostic to DiagnosticsEngine as per issue 5397

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140478 91177308-0d34-0410-b5e6-96231b3b80d8
5555bf78e0a76337f7149669723bf4f3f9ddbc1c 21-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Remove PreprocessingDirectiveKind since it's not necessary.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140191 91177308-0d34-0410-b5e6-96231b3b80d8
de4e0a8e57e643bbe78ad37ad6023c45a8a9f7e2 21-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> The location of the name in MacroDefinition is the beginning of its range,
don't store an extra location for it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140190 91177308-0d34-0410-b5e6-96231b3b80d8
92ddef1bf843e1e18c040d69f48a6bf0bc7c776a 19-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> In libclang, when visiting preprocessed entities in a source range, use
PreprocessingRecord's getPreprocessedEntitiesInRange.

Also remove all the stuff that were added in ASTUnit that are unnecessary now
that we do a binary search for preprocessed entities and deserialize only
what is necessary.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140063 91177308-0d34-0410-b5e6-96231b3b80d8
b6441ef9b7285bd1aa77b05b10f473f7a3f413e7 19-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce local_begin()/local_end() methods in PreprocessingRecord which
return iterators for local, non-loaded, preprocessed entities.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140062 91177308-0d34-0410-b5e6-96231b3b80d8
2dbaca748bc3eb6539f417bd8354c930bdf88fa4 19-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce PreprocessingRecord::getPreprocessedEntitiesInRange()
which will do a binary search and return a pair of iterators
for preprocessed entities in the given source range.

Source ranges of preprocessed entities are stored twice currently in
the PCH/Module file but this will be fixed in a subsequent commit.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140058 91177308-0d34-0410-b5e6-96231b3b80d8
fba18aa8f2cd1994dc65e8cb9f4be201c560dc0b 16-Sep-2011 Douglas Gregor <dgregor@apple.com> Add an experimental flag -fauto-module-import that automatically turns
#include or #import direcctives of framework headers into module
imports of the corresponding framework module.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139860 91177308-0d34-0410-b5e6-96231b3b80d8
e24692b30adbe8144597678a0e3354912e99c747 15-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> [PCH] Overhaul how preprocessed entities are [de]serialized.

-Use an array of offsets for all preprocessed entities
-Get rid of the separate array of offsets for just macro definitions;
for references to macro definitions use an index inside the preprocessed
entities array.
-Deserialize each preprocessed entity lazily, at first request; not in bulk.

Paves the way for binary searching of preprocessed entities that will offer
efficiency and will simplify things on the libclang side a lot.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139809 91177308-0d34-0410-b5e6-96231b3b80d8
6e975c4517958bcc11c834336d340797356058db 14-Sep-2011 Douglas Gregor <dgregor@apple.com> For modules, use a hash of the compiler version, language options, and
target triple to separate modules built under different
conditions. The hash is used to create a subdirectory in the module
cache path where other invocations of the compiler (with the same
version, language options, etc.) can find the precompiled modules.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139662 91177308-0d34-0410-b5e6-96231b3b80d8
21cae2059a06f7d89eee169409c9266def1b1aca 13-Sep-2011 Douglas Gregor <dgregor@apple.com> When an import statement fails to find a module in the module cache,
but there is a corresponding umbrella header in a framework, build the
module on-the-fly so it can be immediately loaded at the import
statement. This is very much proof-of-concept code, with details to be
fleshed out over time.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139558 91177308-0d34-0410-b5e6-96231b3b80d8
9a6da6930644b4f2dbe4885b0eb4fc1274ff56a4 12-Sep-2011 Douglas Gregor <dgregor@apple.com> Introduce a cc1-level option to provide the path to the module cache,
where the compiler will look for module files. Eliminates the
egregious hack where we looked into the header search paths for

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139538 91177308-0d34-0410-b5e6-96231b3b80d8
8f7c540ac42370c40ebcdc4b69018c938faf94ec 08-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> [libclang] Fix annotation and getting a "macro expansion" cursor
for a builtin macro expansion.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139298 91177308-0d34-0410-b5e6-96231b3b80d8
b8db7cd9ac05c522855631670ec2e97255384f5a 08-Sep-2011 Douglas Gregor <dgregor@apple.com> Optimize the preprocessor's handling of the __import_module__
keyword. We now handle this keyword in HandleIdentifier, making a note
for ourselves when we've seen the __import_module__ keyword so that
the next lexed token can trigger a module import (if needed). This
greatly simplifies Preprocessor::Lex(), and completely erases the 5.5%
-Eonly slowdown Argiris noted when I originally implemented
__import_module__. Big thanks to Argiris for noting that horrible

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139265 91177308-0d34-0410-b5e6-96231b3b80d8
a3e9a969fd25022805a9e0ad827f3729286f6d58 07-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Revert r139222, operator->() in PreprocessingRecord::iterator. It is useful
to meet the requirements of the InputIterator concept.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139261 91177308-0d34-0410-b5e6-96231b3b80d8
a1e99cc7e9b89b27ea0a0351ef9f1421f62156e5 07-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> operator->() in PreprocessingRecord::iterator is useless since we
are returning a pointer to pointer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139222 91177308-0d34-0410-b5e6-96231b3b80d8
7d100872341f233c81e1d7b72b40457e62c36862 04-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Support code-completion for C++ inline methods and ObjC buffering methods.

Previously we would cut off the source file buffer at the code-completion
point; this impeded code-completion inside C++ inline methods and,
recently, with buffering ObjC methods.

Have the code-completion inserted into the source buffer so that it can
be buffered along with a method body. When we actually hit the code-completion
point the cut-off lexing or parsing.

Fixes rdar://10056932&8319466

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139086 91177308-0d34-0410-b5e6-96231b3b80d8
998b3d3e8528ebd9d2c5d78d3a82edd90a8953a4 02-Sep-2011 Douglas Gregor <dgregor@apple.com> Allow the preprocessor to be constructed without performing target-
and language-specific initialization. Use this to allow ASTUnit to
create a preprocessor object *before* loading the AST file. No actual
functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138983 91177308-0d34-0410-b5e6-96231b3b80d8
e2ca828119b8bff4a5c25c6db8ee4fec558451e7 01-Sep-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Don't try keeping a 'LeadingEmptyMacroLoc' in NullStmt. This fails
in the face of buffering C++/ObjC method bodies.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138972 91177308-0d34-0410-b5e6-96231b3b80d8
3e3cd93b2fd9644e970c389e715c13883faf68b6 01-Sep-2011 Douglas Gregor <dgregor@apple.com> Teach ASTContext and Preprocessor to hold on to references to the same
LangOptions, rather than making distinct copies of
LangOptions. Granted, LangOptions doesn't actually get modified, but
this will eventually make it easier to construct ASTContext and
Preprocessor before we know all of the LangOptions.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138959 91177308-0d34-0410-b5e6-96231b3b80d8
7143aab97c6e849a5a5005b7853b8c7d5af008ed 01-Sep-2011 Douglas Gregor <dgregor@apple.com> Modules hide macro definitions by default, so that silly things like
include guards don't show up as macro definitions in every translation
unit that imports a module. Macro definitions can, however, be
exported with the intentionally-ugly #__export_macro__
directive. Implement this feature by not even bothering to serialize
non-exported macros to a module, because clients of that module need
not (should not) know that these macros even exist.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138943 91177308-0d34-0410-b5e6-96231b3b80d8
65030af6526748ce11534e92f0ccefc44091ba13 31-Aug-2011 Douglas Gregor <dgregor@apple.com> Switch __import__ over to __import_module__, so we don't conflict with
existing practice with Python extension modules. Not that Python
extension modules should be using a double-underscored identifier
anyway, but...

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138870 91177308-0d34-0410-b5e6-96231b3b80d8
f84139a1331c63c998e8b7d54148c75ac0b48ccd 31-Aug-2011 Eli Friedman <eli.friedman@gmail.com> Change err_pp_file_not_found back to an Error; when it's a Warning, we suppress it in system headers. And it is not a good idea to suppress it in system headers. (This was originally changed in r134996 to implement -MG.)

Fixes <rdar://10041960>. And also brings down the number of warnings without a flag by one :)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138842 91177308-0d34-0410-b5e6-96231b3b80d8
6be16fe900bdd1e5f677d23ae34fffead5bcfc77 27-Aug-2011 Douglas Gregor <dgregor@apple.com> Take an entirely different approach to handling the "parsing" of
__import__ within the preprocessor, since the prior one foolishly
assumed that Preprocessor::Lex() was re-entrant. We now handle
__import__ at the top level (only), after macro expansion. This should
fix the buildbot failures.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138704 91177308-0d34-0410-b5e6-96231b3b80d8
6aa52ec6b969faabf3764baf79d89810b8249a7e 27-Aug-2011 Douglas Gregor <dgregor@apple.com> Introduce support for a simple module import declaration, which
loads the named module. The syntax itself is intentionally hideous and
will be replaced at some later point with something more
palatable. For now, we're focusing on the semantics:
- Module imports are handled first by the preprocessor (to get macro
definitions) and then the same tokens are also handled by the parser
(to get declarations). If both happen (as in normal compilation),
the second one is redundant, because we currently have no way to
hide macros or declarations when loading a module. Chris gets credit
for this mad-but-workable scheme.
- The Preprocessor now holds on to a reference to a module loader,
which is responsible for loading named modules. CompilerInstance is
the only important module loader: it now knows how to create and
wire up an AST reader on demand to actually perform the module load.
- We search for modules in the include path, using the module name
with the suffix ".pcm" (precompiled module) for the file name. This
is a temporary hack; we hope to improve the situation in the

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138679 91177308-0d34-0410-b5e6-96231b3b80d8
2ea054fbc59ed19b8e3304e7e7cbdd56a5a5120f 27-Aug-2011 Douglas Gregor <dgregor@apple.com> Teach the ASTReader how to avoid cycles when loading declarations that
are lexically within a particular DeclContext. Test forthcoming.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138668 91177308-0d34-0410-b5e6-96231b3b80d8
03c107a42fae79e89d0016999a1a04c07d65591a 25-Aug-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Make Lexer::ComputePreamble accept a LangOptions parameter, otherwise it may be
out-of-sync how a file is compiled. Patch by Matthias Kleine!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138580 91177308-0d34-0410-b5e6-96231b3b80d8
499ea5550d6e2fc5cfbd33b47f06d92ce25d7a13 23-Aug-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce SourceManager::isInSLocAddrSpace and use it in TokenLexer instead of isInFileID
since it is a bit more efficient.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138379 91177308-0d34-0410-b5e6-96231b3b80d8
d9d2b679d0728ea7f539f38aaea38e26b8b08043 22-Aug-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Boost the efficiency of SourceManager::getMacroArgExpandedLocation.

Currently getMacroArgExpandedLocation is very inefficient and for the case
of a location pointing at the main file it will end up checking almost all of
the SLocEntries. Make it faster:

-Use a map of macro argument chunks to their expanded source location. The map
is for a single source file, it's stored in the file's ContentCache and lazily
computed, like the source lines cache.
-In SLocEntry's FileInfo add an 'unsigned NumCreatedFIDs' field that keeps track
of the number of FileIDs (files and macros) that were created during preprocessing
of that particular file SLocEntry. This is useful when computing the macro argument
map in skipping included files while scanning for macro arg FileIDs that lexed from
a specific source file. Due to padding, the new field does not increase the size
of SLocEntry.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138225 91177308-0d34-0410-b5e6-96231b3b80d8
d60a34a4e514ec0dfddd05ef2744be104e111f45 20-Aug-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> For assigning SourceLocations to macro arg tokens, reserve a single SLocEntry
for tokens that are lexed consecutively from the same FileID, instead of creating
a SLocEntry for each token. e.g for

assert(foo == bar);

there will be a single SLocEntry for the "foo == bar" chunk and locations
for the 'foo', '==', 'bar' tokens will point inside that chunk.

For parsing SemaExpr.cpp, this reduced the number of SLocEntries by 25%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138129 91177308-0d34-0410-b5e6-96231b3b80d8
ec3b5389a5032433116ec2f2ee266b9666ad2eb4 20-Aug-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Rename TokenLexer::getMacroExpansionLocation -> getExpansionLocForMacroDefLoc, no functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138128 91177308-0d34-0410-b5e6-96231b3b80d8
5c5f03e4020e90b9760ec547962ba02b029cc359 18-Aug-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> [libclang] Support code-completion inside macro arguments.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137973 91177308-0d34-0410-b5e6-96231b3b80d8
1b2d53647dda5e4a70cea54ec3bedbf167081120 18-Aug-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> For the MacroExpands preprocessor callback, also pass the SourceRange
of expansion (for function macros it includes the right paren).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137909 91177308-0d34-0410-b5e6-96231b3b80d8
03720fce6e709661af020f3e4e6dfd08a96e8044 11-Aug-2011 Craig Topper <craig.topper@gmail.com> Raw string followup. Pass a couple StringRefs by value.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137301 91177308-0d34-0410-b5e6-96231b3b80d8
2fa4e86b4fdada3b9ecbbbd99965b83ed879f69b 11-Aug-2011 Craig Topper <craig.topper@gmail.com> Add support for C++0x raw string literals.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137298 91177308-0d34-0410-b5e6-96231b3b80d8
88710f22be1acb04b7d812aebe87bea56394c07c 03-Aug-2011 Eli Friedman <eli.friedman@gmail.com> A couple fixes for preprocessor expressions:

1. Be more tolerant of comments in -CC (comment-preserving) mode. We were missing a few cases.

2. Make sure to expand the second FOO in "#if defined FOO FOO". (See also
r97253, which addressed the case of "#if defined(FOO FOO".)

Fixes PR10286.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136748 91177308-0d34-0410-b5e6-96231b3b80d8
65e02fa80e1c185f18e5f81cefc30d75383a7301 28-Jul-2011 Douglas Gregor <dgregor@apple.com> Introduce the "-index-header-map" option, to give special semantics
for quoted header lookup when dealing with not-yet-installed
frameworks. Fixes <rdar://problem/9824020>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136331 91177308-0d34-0410-b5e6-96231b3b80d8
aca25bccefe56121b686706afc84c8cb5d46e65b 27-Jul-2011 Anna Zaks <ganna@apple.com> Add a utility function to the Lexer, which makes it easier to find a token after the given location. (It is a generalized version of trans::findLocationAfterSemi from ArcMigrate, which will be changed to use the Lexer utility).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136268 91177308-0d34-0410-b5e6-96231b3b80d8
5cee1195584fa8672253139c86e922daeda69b9e 27-Jul-2011 Douglas Gregor <dgregor@apple.com> Add support for C++0x unicode string and character literals, from Craig Topper!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136210 91177308-0d34-0410-b5e6-96231b3b80d8
d1194fbbf65374bfa3578eb40a547e4f97b497d1 27-Jul-2011 Ted Kremenek <kremenek@apple.com> clang_getCXTUResourceUsage: report memory used by HeaderSearch.

This required converting the StringMaps to use a BumpPtrAllocator. I measured the
compile time and saw no observable regression.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136190 91177308-0d34-0410-b5e6-96231b3b80d8
91d1bd6ede1d101a2e49719250c33154b39e0016 26-Jul-2011 Ted Kremenek <kremenek@apple.com> Report more memory using in Preprocessor::getTotalMemory() and PreprocessingRecord::getTotalMemory().

Most of the memory was already reported; but now we report more memory from side data structures.

Fixes <rdar://problem/9379717>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136150 91177308-0d34-0410-b5e6-96231b3b80d8
2d3ba4f5a923a90c3fc290ddfba5e36c2d0a9b46 23-Jul-2011 Chris Lattner <sabre@nondot.org> Move ArrayRef to LLVM.h and eliminate now-redundant qualifiers, patch by Jon Mulder!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135855 91177308-0d34-0410-b5e6-96231b3b80d8
d47d3b0cfeb7e8564ff77f48130fe63282b6d127 23-Jul-2011 Chris Lattner <sabre@nondot.org> clean up forward declarations of raw_ostream to use the new LLVM.h
patch by Jon Mulder!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135851 91177308-0d34-0410-b5e6-96231b3b80d8
1a23f12e1e283384b76e768a83f01bfcbbd61ca0 21-Jul-2011 Douglas Gregor <dgregor@apple.com> One last RandomAccessIterator operator for PreprocessingRecord::iterator

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135680 91177308-0d34-0410-b5e6-96231b3b80d8
67029567f7d39d9356fbc505fffc2c1e21bf06ce 21-Jul-2011 Douglas Gregor <dgregor@apple.com> Add the remaining RandomAccessIterator operations to
PreprocessingRecord::iterator. Where's concept_map when I need it?

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135679 91177308-0d34-0410-b5e6-96231b3b80d8
8d366c0e347ab600ac09e2ba9b676d12017a448a 21-Jul-2011 Francois Pichet <pichet2000@gmail.com> For some reason I don't fully comprehend, the MSVC debug build will fail with a huge 50+ lines template error message if PreprocessingRecord::iterator has no operator<()

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135670 91177308-0d34-0410-b5e6-96231b3b80d8
4c30bb148b53c8063e940ca3e049ba4d270dc9d5 21-Jul-2011 Douglas Gregor <dgregor@apple.com> Rework the detailed preprocessing record to separate preprocessing
entities generated directly by the preprocessor from those loaded from
the external source (e.g., the ASTReader). By separating these two
sets of entities into different vectors, we allow both to grow
independently, and eliminate the need for preallocating all of the
loaded preprocessing entities. This is similar to the way the recent
SourceManager refactoring treats FileIDs and the source location
address space.

As part of this, switch over to building a continuous range map to
track preprocessing entities.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135646 91177308-0d34-0410-b5e6-96231b3b80d8
686775deca8b8685eb90801495880e3abdd844c2 20-Jul-2011 Chris Lattner <sabre@nondot.org> now that we have a centralized place to do so, add some using declarations for
some common llvm types: stringref and smallvector. This cleans up the codebase
quite a bit.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135576 91177308-0d34-0410-b5e6-96231b3b80d8
9e5bb85ac899eeab7c21b5ff9030c3da6ff4837b 14-Jul-2011 Chandler Carruth <chandlerc@gmail.com> Move the rest of the preprocessor terminology from 'instantiate' and
variants to 'expand'. This changed a couple of public APIs, including
one public type "MacroInstantiation" which is now "MacroExpansion". The
rest of the codebase was updated to reflect this, especially the
libclang code. Two of the C++ (and thus easily changed) libclang APIs
were updated as well because they pertained directly to the old
MacroInstantiation class.

No functionality changed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135139 91177308-0d34-0410-b5e6-96231b3b80d8
433db06b614f26dc6829e86d6ff469e2cca7d4f9 14-Jul-2011 Chandler Carruth <chandlerc@gmail.com> Convert terminology in the Lexer from 'instantiate' and variants to
'expand'. Also update the public API it provides to the new term, and
propagate that update to the various clients.

No functionality changed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135138 91177308-0d34-0410-b5e6-96231b3b80d8
d0f5e443ceed37e1a72015bd109db8e5fc6206cf 14-Jul-2011 Chandler Carruth <chandlerc@gmail.com> Switch the TokenLexer's terminology from various forms of 'instantiate'
to 'expand' for macros. Only comments and uses local to the TokenLexer
are updated.

No functionality changed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135137 91177308-0d34-0410-b5e6-96231b3b80d8
7a759606d93975866051f67104ae58446e55f404 07-Jul-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Move SourceManager::isAt[Start/End]OfMacroInstantiation functions to the Lexer, since they depend on it now.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134644 91177308-0d34-0410-b5e6-96231b3b80d8
b73377eeb3eff76be134203aebb6068244b177f3 07-Jul-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Make the Preprocessor more memory efficient and improve macro instantiation diagnostics.

When a macro instantiation occurs, reserve a SLocEntry chunk with length the
full length of the macro definition source. Set the spelling location of this chunk
to point to the start of the macro definition and any tokens that are lexed directly
from the macro definition will get a location from this chunk with the appropriate offset.

For any tokens that come from argument expansion, '##' paste operator, etc. have their
instantiation location point at the appropriate place in the instantiated macro definition
(the argument identifier and the '##' token respectively).
This improves macro instantiation diagnostics:


t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
t.c:5:11: note: instantiated from:
int y = M(/);


t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
t.c:3:20: note: instantiated from:
\#define M(op) (foo op 3);
~~~ ^ ~
t.c:5:11: note: instantiated from:
int y = M(/);

The memory savings for a candidate boost library that abuses the preprocessor are:

- 32% less SLocEntries (37M -> 25M)
- 30% reduction in PCH file size (900M -> 635M)
- 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134587 91177308-0d34-0410-b5e6-96231b3b80d8
5b3284a9275a27f5c4410e25eb8933be540601d6 30-Jun-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce a caching mechanism for macro expanded tokens.

Previously macro expanded tokens were added to Preprocessor's bump allocator and never released,
even after the TokenLexer that were lexing them was finished, thus they were wasting memory.
A very "useful" boost library was causing clang to eat 1 GB just for the expanded macro tokens.

Introduce a special cache that works like a stack; a TokenLexer can add the macro expanded tokens
in the cache, and when it finishes, the tokens are removed from the end of the cache.

Now consumed memory by expanded tokens for that library is ~ 1.5 MB.

Part of rdar://9327049.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134105 91177308-0d34-0410-b5e6-96231b3b80d8
c5c5e92ec53f7e6ac7ebbbf77c6d8e4b7d88daec 30-Jun-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce Preprocessor::getTotalMemory() and use it in CIndex.cpp, no functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134103 91177308-0d34-0410-b5e6-96231b3b80d8
4c4cbe5794897442e7cd5a57932f6aa91b6dff55 23-Jun-2011 Douglas Gregor <dgregor@apple.com> Remove superfluous comment

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133757 91177308-0d34-0410-b5e6-96231b3b80d8
7f88c0133272129111ec32d7aa5eacf6e685c6fc 23-Jun-2011 Douglas Gregor <dgregor@apple.com> Bump Token::Kind from an unsigned char to an unsigned short, from Anton Lokhmotov

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133750 91177308-0d34-0410-b5e6-96231b3b80d8
c09ce1224dedc470fce9747e5936ff83cc6762eb 22-Jun-2011 Douglas Gregor <dgregor@apple.com> Copy diagnostic pragmas to the preprocessed output, from Richard Osborne!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133633 91177308-0d34-0410-b5e6-96231b3b80d8
65aa6885818d4b4eea2e5a9d12085b2398148662 21-Jun-2011 Jay Foad <jay.foad@gmail.com> Make more use of llvm::StringRef in various APIs. In particular, don't
use the deprecated forms of llvm::StringMap::GetOrCreateValue().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133515 91177308-0d34-0410-b5e6-96231b3b80d8
74a5fd8bcc68b540b58f6fcd2d80e6e926966e71 24-May-2011 Nico Weber <nicolasweber@gmx.de> Make it possible for external tools to distinguish between paths that come from -I and paths that come from -system. Patch from Paul Holden!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@131955 91177308-0d34-0410-b5e6-96231b3b80d8
403de3f932b5d1d3e4e58f69960000911d04dd2a 18-May-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Fix a nasty bug where inside StringLiteralParser:

1. We would assume that the length of the string literal token was at least 2
2. We would allocate a buffer with size length-2

And when the stars aligned (one of which would be an invalid source location due to stale PCH)
The length would be 0 and we would try to allocate a 4GB buffer.

Add checks for this corner case and a bunch of asserts.
(We really really should have had an assert for 1.).

Note that there's no test case since I couldn't get one (it was major PITA to reproduce),
maybe later.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@131492 91177308-0d34-0410-b5e6-96231b3b80d8
c1b5fa41f09512c74030b9a2a0d1564535e22a76 13-May-2011 Peter Collingbourne <peter@pcc.me.uk> Introduce __has_extension macro

__has_extension is a function-like macro which takes the same set
of feature identifiers as __has_feature. It evaluates to 1 if the
feature is supported by Clang in the current language (either as a
language extension or a standard language feature) or 0 if not.

At the same time, add support for the C1X feature identifiers
c_generic_selections (renamed from generic_selections) and
c_static_assert, and document them.

Patch by myself and Jean-Daniel Dupas.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@131308 91177308-0d34-0410-b5e6-96231b3b80d8
dca8ee8b7bc86076916a3a80f553f7a4e98c14af 06-May-2011 Douglas Gregor <dgregor@apple.com> Introduce a new libclang parsing flag,
CXTranslationUnit_NestedMacroInstantiations, which indicates whether
we want to see "nested" macro instantiations (e.g., those that occur
inside other macro instantiations) within the detailed preprocessing
record. Many clients (e.g., those that only care about visible tokens)
don't care about this information, and in code that uses preprocessor
metaprogramming, this information can have a very high cost.

Addresses <rdar://problem/9389320>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130990 91177308-0d34-0410-b5e6-96231b3b80d8
5e1db6a434d0e3fe0fbde0bca2ec44552818fb22 04-May-2011 Ted Kremenek <kremenek@apple.com> Enhance clang_getCXTUResourceUsage() to return the amount of memory used by the Preprocessor's bump allocator as well as those from the PreprocessingRecord.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130823 91177308-0d34-0410-b5e6-96231b3b80d8
dd3e5549e3c11e217078938aacf72f042eea5343 04-May-2011 Douglas Gregor <dgregor@apple.com> Introduce a new libclang API, clang_isFileMultipleIncludeGuarded(),
which determines whether a particular file is actually a header that
is intended to be guarded from multiple inclusions within the same
translation unit.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130808 91177308-0d34-0410-b5e6-96231b3b80d8
28bbe4b8acc338476fe0825769b41fb32b423c72 28-Apr-2011 John Wiegley <johnw@boostpro.com> Parsing/AST support for Structured Exception Handling

Patch authored by Sohail Somani.

Provide parsing and AST support for Windows structured exception handling.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130366 91177308-0d34-0410-b5e6-96231b3b80d8
b7d98d35ea723624345f06e5895ddce2e0388ef0 27-Apr-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> If a null statement was preceded by an empty macro keep its instantiation source location
in NullStmt.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130289 91177308-0d34-0410-b5e6-96231b3b80d8
7412494982c8b50c90961302c3a718633b2c3ab7 26-Apr-2011 Manuel Klimek <klimek@google.com> To be able to replay compilations we need to accurately remodel how
includes get resolved, especially when they are found relatively to
another include file. We also try to get it working for framework
includes, but that part of the code is untested, as I don't have a code
base that uses it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130246 91177308-0d34-0410-b5e6-96231b3b80d8
c69a505cfa318d571ce8a0cd038c8d958585a735 23-Apr-2011 Jay Foad <jay.foad@gmail.com> Remove unused STL header includes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130068 91177308-0d34-0410-b5e6-96231b3b80d8
7d2b8c1fcc2b707be78b09930a7767477822462f 16-Apr-2011 Douglas Gregor <dgregor@apple.com> Initial work to improve documentation for Clang's diagnostics, from Matthieu Monrocq

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129614 91177308-0d34-0410-b5e6-96231b3b80d8
fc8f0e14ad142ed811e90fbd9a30e419e301c717 15-Apr-2011 Chris Lattner <sabre@nondot.org> fix a bunch of comment typos found by codespell. Patch by
Luis Felipe Strano Moraes!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129559 91177308-0d34-0410-b5e6-96231b3b80d8
4f32786ac45210143654390177105eb749b614e9 21-Mar-2011 Ted Kremenek <kremenek@apple.com> Improve crash recovery cleanup to recovery CompilerInstances during crash recovery. This was a huge resource "root" during crashes.

This change requires making a bunch of fundamental Clang structures (optionally) reference counted to allow correct
ownership semantics of these objects (e.g., ASTContext) to play out between an active ASTUnit and CompilerInstance

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@128011 91177308-0d34-0410-b5e6-96231b3b80d8
b5142bb7af5c70fffd09f05172a1379a35a9c29a 16-Mar-2011 Chandler Carruth <chandlerc@gmail.com> Add a 'RawPath' parameter to the PPCallbacks interface. This allows
clients to observe the exact path through which an #included file was
located. This is very useful when trying to record and replay inclusion
operations without it beind influenced by the aggressive caching done
inside the FileManager to avoid redundant system calls and filesystem

The work to compute and return this is only done in the presence of
callbacks, so it should have no effect on normal compilation.

Patch by Manuel Klimek.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127742 91177308-0d34-0410-b5e6-96231b3b80d8
834e3f6c77d9ac03997a3f0c56934edcf406a355 08-Mar-2011 John McCall <rjmccall@apple.com> Fix my earlier commit to work with escaped newlines and leave breadcrumbs
in case we want to make a world where we can check intermediate instantiations
for this kind of breadcrumb.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127221 91177308-0d34-0410-b5e6-96231b3b80d8
30900dafbe972f4802c2663262b4393edab0d1cd 08-Mar-2011 John McCall <rjmccall@apple.com> Add an API call to retrieve the spelling data of a token from its SourceLocation.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127216 91177308-0d34-0410-b5e6-96231b3b80d8
ec356c320a81062b1843f0dbe7fedf29ed947ece 06-Mar-2011 Douglas Gregor <dgregor@apple.com> xpose HeaderSearch::SearchDirs to tools,s, from Paul Holden

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127122 91177308-0d34-0410-b5e6-96231b3b80d8
84021556baceb76eedf7d44be8ba71d9b8cfacce 28-Feb-2011 Peter Collingbourne <peter@pcc.me.uk> Rename tok::eom to tok::eod.

The previous name was inaccurate as this token in fact appears at
the end of every preprocessing directive, not just macro definitions.
No functionality change, except for a diagnostic tweak.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126631 91177308-0d34-0410-b5e6-96231b3b80d8
a5ef584fd3d18da0c98342b4b6453948b7eb30d3 22-Feb-2011 Peter Collingbourne <peter@pcc.me.uk> Reimplement __pragma support using a TokenLexer

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126221 91177308-0d34-0410-b5e6-96231b3b80d8
b2eb53d9fd973a1a02e05e67a3307b3efd12eff2 22-Feb-2011 Peter Collingbourne <peter@pcc.me.uk> Make TokenLexer capable of storing preprocessor directive tokens

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126220 91177308-0d34-0410-b5e6-96231b3b80d8
c8da1ecaf58ff41b652dd53331aace948027039b 20-Feb-2011 Oscar Fuentes <ofv@wanadoo.es> New function for tablegenning: clang_tablegen.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126093 91177308-0d34-0410-b5e6-96231b3b80d8
6c948378154dc37e772e50f95033530d506fa912 19-Feb-2011 Peter Collingbourne <peter@pcc.me.uk> Reimplement Token::isAnnotation() using TokenKinds.def. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126045 91177308-0d34-0410-b5e6-96231b3b80d8
9c97ca091200311229ce3cb070f0ed8846687eda 16-Feb-2011 Douglas Gregor <dgregor@apple.com> Teach PPChainedCallbacks to forward the InclusionDirective() callback.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125669 91177308-0d34-0410-b5e6-96231b3b80d8
9d3f5f7550a2fab4178ed01425758c349b73a609 14-Feb-2011 Peter Collingbourne <peter@pcc.me.uk> Make LexOnOffSwitch a Preprocessor member function

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125473 91177308-0d34-0410-b5e6-96231b3b80d8
0a480293f726508ad9aed0be39e8c9779e84f2f2 11-Feb-2011 Douglas Gregor <dgregor@apple.com> Rename the operation that loads a preprocessed entity from a given offset to indicate that we're loading from an offset, not an index, lest one be confused. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125394 91177308-0d34-0410-b5e6-96231b3b80d8
cfbf1c7536e016dc275139dd842d4a5f059a749f 10-Feb-2011 Douglas Gregor <dgregor@apple.com> Implement two related optimizations that make de-serialization of
AST/PCH files more lazy:
- Don't preload all of the file source-location entries when reading
the AST file. Instead, load them lazily, when needed.
- Only look up header-search information (whether a header was already
#import'd, how many times it's been included, etc.) when it's needed
by the preprocessor, rather than pre-populating it.

Previously, we would pre-load all of the file source-location entries,
which also populated the header-search information structure. This was
a relatively minor performance issue, since we would end up stat()'ing
all of the headers stored within a AST/PCH file when the AST/PCH file
was loaded. In the normal PCH use case, the stat()s were cached, so
the cost--of preloading ~860 source-location entries in the Cocoa.h
case---was relatively low.

However, the recent optimization that replaced stat+open with
open+fstat turned this into a major problem, since the preloading of
source-location entries would now end up opening those files. Worse,
those files wouldn't be closed until the file manager was destroyed,
so just opening a Cocoa.h PCH file would hold on to ~860 file
descriptors, and it was easy to blow through the process's limit on
the number of open file descriptors.

By eliminating the preloading of these files, we neither open nor stat
the headers stored in the PCH/AST file until they're actually needed
for something. Concretely, we went from

*** HeaderSearch Stats:
835 files tracked.
364 #import/#pragma once files.
823 included exactly once.
6 max times a file is included.
3 #include/#include_next/#import.
0 #includes skipped due to the multi-include optimization.
1 framework lookups.
0 subframework lookups.

*** Source Manager Stats:
835 files mapped, 3 mem buffers mapped.
37460 SLocEntry's allocated, 11215575B of Sloc address space used.
62 bytes of files mapped, 0 files with line #'s computed.

with a trivial program that uses a chained PCH including a Cocoa PCH

*** HeaderSearch Stats:
4 files tracked.
1 #import/#pragma once files.
3 included exactly once.
2 max times a file is included.
3 #include/#include_next/#import.
0 #includes skipped due to the multi-include optimization.
1 framework lookups.
0 subframework lookups.

*** Source Manager Stats:
3 files mapped, 3 mem buffers mapped.
37460 SLocEntry's allocated, 11215575B of Sloc address space used.
62 bytes of files mapped, 0 files with line #'s computed.

for the same program.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125286 91177308-0d34-0410-b5e6-96231b3b80d8
418df343bb50802586d20aae3b83e2eb44c6c828 27-Jan-2011 Douglas Gregor <dgregor@apple.com> Separate the access-control diagnostics from other diagnostics that do not have SFINAE behavior.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@124441 91177308-0d34-0410-b5e6-96231b3b80d8
3437f1f1294499d4ef306c1089fcb3e29ec2aa68 03-Jan-2011 Argyrios Kyrtzidis <akyrtzi@gmail.com> Speed up code-completion by skipping function bodies.

When we are in code-completion mode, skip parsing of all function bodies except the one where the
code-completion point resides.

For big .cpp files like 'SemaExpr.cpp' the improvement makes a huge difference, in some cases cutting down
code-completion time -62% !

We don't get diagnostics for the bodies though, so modify the code-completion tests that check for errors.

See rdar://8814203.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@122765 91177308-0d34-0410-b5e6-96231b3b80d8
c4bf2b9afb7d47445a9dc6bc848657098a4e3851 22-Dec-2010 Abramo Bagnara <abramo.bagnara@gmail.com> Introduced raw_identifier token kind.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@122394 91177308-0d34-0410-b5e6-96231b3b80d8
403ba3522d1b1c97ae5fad81c1a2c4b3a754e1c1 19-Dec-2010 Nick Lewycky <nicholas@mxc.ca> Add missing standard includes. Patch by Joerg Sonnenberger!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@122194 91177308-0d34-0410-b5e6-96231b3b80d8
0827408865e32789e0ec4b8113a302ccdc531423 15-Dec-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> Fix diagnostic pragmas.

Diagnostic pragmas are broken because we don't keep track of the diagnostic state changes and we only check the current/latest state.
Problems manifest if a diagnostic is emitted for a source line that has different diagnostic state than the current state; this can affect
a lot of places, like C++ inline methods, template instantiations, the lexer, etc.

Fix the issue by having the Diagnostic object keep track of the source location of the pragmas so that it is able to know what is the diagnostic state at any given source location.

Fixes rdar://8365684.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@121873 91177308-0d34-0410-b5e6-96231b3b80d8
89d9980bbc2e4a4ac86673e6ec16fb9f5babb63b 30-Nov-2010 Douglas Gregor <dgregor@apple.com> When using a precompiled preamble with detailed preprocessing records,
trap the serialized preprocessing records (macro definitions, macro
instantiations, macro definitions) from the generation of the
precompiled preamble, then replay those when walking the list of
preprocessed entities. This eliminates a bug where clang_getCursor()
wasn't able to find preprocessed-entity cursors in the preamble.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120396 91177308-0d34-0410-b5e6-96231b3b80d8
03013fa9a0bf1ef4b907f5fec006c8f4000fdd21 29-Nov-2010 Michael J. Spencer <bigcheesegs@gmail.com> Merge System into Support.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120297 91177308-0d34-0410-b5e6-96231b3b80d8
10e286aa8d39fb51a21412850265d9dae74613ee 23-Nov-2010 Chris Lattner <sabre@nondot.org> rework the stat cache, pulling it out of FileManager.h into
its own header and giving it some more structure. No
functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120030 91177308-0d34-0410-b5e6-96231b3b80d8
681c74afe6a81161aa13291c6c114e5240b23865 23-Nov-2010 Chris Lattner <sabre@nondot.org> don't allow remapping PTH file paths with -fworking-directory, the
client should just pass in absolute paths.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120012 91177308-0d34-0410-b5e6-96231b3b80d8
39b49bcaaddb1049234fca9500c0ac02c088e23d 23-Nov-2010 Chris Lattner <sabre@nondot.org> now the FileManager has a FileSystemOpts ivar, stop threading
FileSystemOpts through a ton of apis, simplifying a lot of code.
This also fixes a latent bug in ASTUnit where it would invoke
methods on FileManager without creating one in some code paths
in cindextext.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@120010 91177308-0d34-0410-b5e6-96231b3b80d8
44aa1f397855f130e88e62ffc1029f7f83bb5d2e 20-Nov-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> Revert r119838 "Don't warn for empty 'if' body if there is a macro that expands to nothing"
and use a better and more general approach, where NullStmt has a flag to indicate whether it was preceded by an empty macro.

Thanks to Abramo Bagnara for the hint!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119887 91177308-0d34-0410-b5e6-96231b3b80d8
2aa9267cf193e5eceb9fd24a51b51c031b606fe9 19-Nov-2010 Craig Silverstein <csilvers2000@yahoo.com> Several PPCallbacks take an SourceLocation + IdentifierInfo, rather
than a Token that holds the same information all in one easy-to-use
package. There's no technical reason to prefer the former -- the
information comes from a Token originally -- and it's clumsier to use,
so I've changed the code to use tokens everywhere.

Approved by clattner

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119845 91177308-0d34-0410-b5e6-96231b3b80d8
a25b6a4b43e8b9611f7506e5fe1b448833b10a46 19-Nov-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> Don't warn for empty 'if' body if there is a macro that expands to nothing, e.g:

if (condition)
CALL(0); // empty macro but don't warn for empty body.

Fixes rdar://8436021.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119838 91177308-0d34-0410-b5e6-96231b3b80d8
33e4e70c8c0a17e0ccb7465d96556b077a68ecb1 18-Nov-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> Refactoring of Diagnostic class.

-Move the stuff of Diagnostic related to creating/querying diagnostic IDs into a new DiagnosticIDs class.
-DiagnosticIDs can be shared among multiple Diagnostics for multiple translation units.
-The rest of the state in Diagnostic object is considered related and tied to one translation unit.
-Have Diagnostic point to the SourceManager that is related with. Diagnostic can now accept just a
SourceLocation instead of a FullSourceLoc.
-Reflect the changes to various interfaces.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119730 91177308-0d34-0410-b5e6-96231b3b80d8
b0607279cb98bbf2bbfe0db170aed39ef91e86a2 17-Nov-2010 Chris Lattner <sabre@nondot.org> move getSpelling from Preprocessor to Lexer, which it is more conceptually related to.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119479 91177308-0d34-0410-b5e6-96231b3b80d8
0833dd0675c25cbb35671c7a2006d511d5c77ce3 17-Nov-2010 Chris Lattner <sabre@nondot.org> propagate preprocessor out of StringLiteralParser. It is now
possible to create one without a preprocessor.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119476 91177308-0d34-0410-b5e6-96231b3b80d8
a95880d6513c617bb96634bcc1f16c6bdb80dedc 17-Nov-2010 Chris Lattner <sabre@nondot.org> push the preprocessor out of EncodeUCNEscape

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119475 91177308-0d34-0410-b5e6-96231b3b80d8
7ef5c27eb6e8ebe58b52013246c06753c3613263 17-Nov-2010 Chris Lattner <sabre@nondot.org> move AdvanceToTokenCharacter and getLocForEndOfToken from
Preprocessor to Lexer where they make more sense.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119474 91177308-0d34-0410-b5e6-96231b3b80d8
872a45e91778eb0b706ff57272fe547d4512eb19 17-Nov-2010 Chris Lattner <sabre@nondot.org> add a static version of PP::AdvanceToTokenCharacter.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119472 91177308-0d34-0410-b5e6-96231b3b80d8
6c66f07854c1334a1ce9eae1428d61d54182a6e1 17-Nov-2010 Chris Lattner <sabre@nondot.org> push use of Preprocessor out farther.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119471 91177308-0d34-0410-b5e6-96231b3b80d8
ca1475ea0e76da6b852796610139ed9b49c8d4a6 17-Nov-2010 Chris Lattner <sabre@nondot.org> push use of Preprocessor out of getOffsetOfStringByte

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119470 91177308-0d34-0410-b5e6-96231b3b80d8
48cf9824fbad42995f4d91d59d08d2620effd683 17-Nov-2010 Chris Lattner <sabre@nondot.org> add a static form of the efficient PP::getSpelling method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119469 91177308-0d34-0410-b5e6-96231b3b80d8
91f54ce93bec136fb9e18740b895cf1c1339524b 17-Nov-2010 Chris Lattner <sabre@nondot.org> refactor the interface to StringLiteralParser::getOffsetOfStringByte,
pushing the dependency on the preprocessor out a bit.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119468 91177308-0d34-0410-b5e6-96231b3b80d8
b5b38adbe29cd82711595bd184d6123125bd9a9b 08-Nov-2010 Craig Silverstein <csilvers2000@yahoo.com> 1) Fix a typo in PPCallbacks: It's elif, not elfif. :-) This is
contentful, since the typo was in the method-name...

2) Clarify some comments in RecursiveASTVisitor.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@118448 91177308-0d34-0410-b5e6-96231b3b80d8
08985b91e5a87e7731ba24c675f2dc9f9cb645d4 06-Nov-2010 Craig Silverstein <csilvers2000@yahoo.com> Add PPCallbacks for #if/#ifdef/etc.

The callback info for #if/#elif is not great -- ideally it would give
us a list of tokens in the #if, or even better, a little parse tree.
But that's a lot more work. Instead, clients can retokenize using

Reviewed by nlewycky.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@118318 91177308-0d34-0410-b5e6-96231b3b80d8
389db16c63eec6ecfa9b235155252d8da766e94e 03-Nov-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> Implement -working-directory.

When -working-directory is passed in command line, file paths are resolved relative to the specified directory.
This helps both when using libclang (where we can't require the user to actually change the working directory)
and to help reproduce test cases when the reproduction work comes along.

--FileSystemOptions is introduced which controls how file system operations are performed (currently it just contains
the working directory value if set).
--FileSystemOptions are passed around to various interfaces that perform file operations.
--Opening & reading the content of files should be done only through FileManager. This is useful in general since
file operations will be abstracted in the future for the reproduction mechanism.

FileSystemOptions is independent of FileManager so that we can have multiple translation units sharing the same
FileManager but with different FileSystemOptions.

Addresses rdar://8583824.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@118203 91177308-0d34-0410-b5e6-96231b3b80d8
4ab829c2a229dc20ecb9f35e7337bbfa95bf4026 01-Nov-2010 Douglas Gregor <dgregor@apple.com> Plug a leak in the preprocessing record's handling of inclusion
directives. We had a std::string in an object that was allocated via a

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@117912 91177308-0d34-0410-b5e6-96231b3b80d8
295a2a617ac335f590e430ab7fcd98f8ce109251 30-Oct-2010 Douglas Gregor <dgregor@apple.com> Make the deserialization of macro definitions lazy, so that we can
load identifiers without loading their corresponding macro
definitions. This is likely to improve PCH performance slightly, and
reduces deserialization stack depth considerably when using
preprocessor metaprogramming.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@117750 91177308-0d34-0410-b5e6-96231b3b80d8
ecdcb883cbc6bb4a2445dc6f02d58d9bdb54a0ed 21-Oct-2010 Douglas Gregor <dgregor@apple.com> Extend the preprocessing record and libclang with support for
inclusion directives, keeping track of every #include, #import,
etc. in the translation unit. We keep track of the source location and
kind of the inclusion, how the file name was spelled, and the
underlying file to which the inclusion resolved.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@116952 91177308-0d34-0410-b5e6-96231b3b80d8
cae5095c116c29a4e52c91fa0ad88df09dd2b45f 20-Oct-2010 Anders Carlsson <andersca@mac.com> Add a __has_attribute macro that works much like __has_feature and __has_builtin.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@116906 91177308-0d34-0410-b5e6-96231b3b80d8
9714a2385cb66b6efa373fc668641de602dd9adb 20-Oct-2010 Ted Kremenek <kremenek@apple.com> Really^2 fix <rdar://problem/8361834>, this time without crashing.

Now MICache is a linked list (per the FIXME), where we tradeoff between MacroInfo objects being in MICache
and MIChainHead. MacroInfo objects in the MICache chain are already "Destroy()'ed", so they can be reused. When
inserting into MICache, we need to remove them from the regular linked list so that they aren't destroyed more than

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@116869 91177308-0d34-0410-b5e6-96231b3b80d8
af8fa25c0d4e0540952a50bbd06dc1558954ccd9 19-Oct-2010 Ted Kremenek <kremenek@apple.com> Simplify lifetime management of MacroInfo objects in Preprocessor by having the Preprocessor maintain them in a linked
list of allocated MacroInfos. This requires only 1 extra pointer per MacroInfo object, and allows us to blow them
away in one place. This fixes an elusive memory leak with MacroInfos (whose exact location I couldn't still figure
out despite substantial digging).

Fixes <rdar://problem/8361834>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@116842 91177308-0d34-0410-b5e6-96231b3b80d8
6f2d1b111ed6d3c686303746e3949e3cbc9f3870 18-Oct-2010 Douglas Gregor <dgregor@apple.com> Add iteration over the preprocessor conditional stack to PreprocessorLexer

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@116703 91177308-0d34-0410-b5e6-96231b3b80d8
b57a624f22b1efa873e0abac78128b69fba3d486 28-Sep-2010 Sebastian Redl <sebastian.redl@getdesigned.at> When chaining PCHs, only write PPRecords that don't come from PCH, and give them the correct IDs. Fixes a crash in XCode.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@114913 91177308-0d34-0410-b5e6-96231b3b80d8
80c60f72848896f867f6b7e664e7060d9e78f019 10-Sep-2010 Douglas Gregor <dgregor@apple.com> When we parse a pragma, keep track of how that pragma was originally
spelled (#pragma, _Pragma, __pragma). In -E mode, use that information
to add appropriate newlines when translating _Pragma and __pragma into
#pragma, like GCC does. Fixes <rdar://problem/8412013>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@113553 91177308-0d34-0410-b5e6-96231b3b80d8
afbc68177cc11b8bfa47464b20e15d5f8fb21d4e 03-Sep-2010 Fariborz Jahanian <fjahanian@apple.com> Use getSpelling to get original text of the
c++ operator token. (radar 8328250).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112977 91177308-0d34-0410-b5e6-96231b3b80d8
be74740cc246ce08d42804a684385a42eb814edb 03-Sep-2010 Fariborz Jahanian <fjahanian@apple.com> Patch to allow alternative representation of c++
operators (and, or, etc.) to be used as selectors
to match g++'s behavior.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112935 91177308-0d34-0410-b5e6-96231b3b80d8
6cf750298d3621d8a10a6dd07fcee8e274b9d94d 30-Aug-2010 Sean Hunt <scshunt@csclub.uwaterloo.ca> Revert my user-defined literal commits - r1124{58,60,67} pending
some issues being sorted out.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112493 91177308-0d34-0410-b5e6-96231b3b80d8
75b8508380e6d44276a8fafe5cafcedb9cf532d3 30-Aug-2010 Sean Hunt <scshunt@csclub.uwaterloo.ca> Two minor fixes to user-defined literals:

- Zero-initialize UDLData so that crashes stop
- Stop complaining that we can't emit them (we most certainly can)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112467 91177308-0d34-0410-b5e6-96231b3b80d8
e425368eb59687b66620a2c6adc3e426c997d796 30-Aug-2010 Sean Hunt <scshunt@csclub.uwaterloo.ca> Fix some test-breaking that snuck into my previous commit

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112460 91177308-0d34-0410-b5e6-96231b3b80d8
0016d519b831859526b79405cdae4c64c73731c8 29-Aug-2010 Sean Hunt <scshunt@csclub.uwaterloo.ca> Implement C++0x user-defined string literals.

The extra data stored on user-defined literal Tokens is stored in extra
allocated memory, which is managed by the PreprocessorLexer because there isn't
a better place to put it that makes sure it gets deallocated, but only after
it's used up. My testing has shown no significant slowdown as a result, but
independent testing would be appreciated.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112458 91177308-0d34-0410-b5e6-96231b3b80d8
1ef8a2e7675f3d8b6e8d9963b00378086e1dcdc7 29-Aug-2010 John McCall <rjmccall@apple.com> Add support for Microsoft's __pragma in the preprocessor.
Patch by Francois Pichet!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112391 91177308-0d34-0410-b5e6-96231b3b80d8
55817afdf9d453a443262a733f6caf6692dca118 25-Aug-2010 Douglas Gregor <dgregor@apple.com> Introduce a preprocessor code-completion hook for contexts where we
expect "natural" language and should not provide any completions,
e.g., comments, string literals, #error.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112054 91177308-0d34-0410-b5e6-96231b3b80d8
f29c5233085a5af795c3c01b94d319e5b3235d56 25-Aug-2010 Douglas Gregor <dgregor@apple.com> Implement code completion for preprocessor expressions and in macro

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111976 91177308-0d34-0410-b5e6-96231b3b80d8
1fbb447e9d43c2c676e94081fbfee7eb6cbe933b 24-Aug-2010 Douglas Gregor <dgregor@apple.com> Implement preprocessor code completion where a macro name is expected,
e.g., after #ifdef/#ifndef or #undef, or inside a defined <macroname>
expression in a preprocessor conditional.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111954 91177308-0d34-0410-b5e6-96231b3b80d8
f44e854ed1e3aa86d2ed6d615ccd109d50ddcff9 24-Aug-2010 Douglas Gregor <dgregor@apple.com> Introduce basic code-completion support for preprocessor directives,
e.g., after a "#" we'll suggest #if, #ifdef, etc.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111943 91177308-0d34-0410-b5e6-96231b3b80d8
3c7f4134603d04b44f997b43c0a9def270f25386 19-Aug-2010 Sebastian Redl <sebastian.redl@getdesigned.at> More PCH -> AST renaming.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111472 91177308-0d34-0410-b5e6-96231b3b80d8
c43b54cbc10654ed59de797898042e1a05265246 19-Aug-2010 Sebastian Redl <sebastian.redl@getdesigned.at> Rename PCHReader to ASTReader.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111467 91177308-0d34-0410-b5e6-96231b3b80d8
2c1ab9079cb117dc0470ab423fe0bc5177546339 18-Aug-2010 Chris Lattner <sabre@nondot.org> no need to pass bumppointer allocator into macroinfo::destroy

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111364 91177308-0d34-0410-b5e6-96231b3b80d8
f47724bf78299c7a50f008e0443c5f9f9f279ddc 17-Aug-2010 Chris Lattner <sabre@nondot.org> Implement #pragma push_macro, patch by Francois Pichet!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111234 91177308-0d34-0410-b5e6-96231b3b80d8
df95a13ec73d2cdaea79555cb412d767f4963120 09-Aug-2010 Douglas Gregor <dgregor@apple.com> Use precompiled preambles for in-process code completion.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110596 91177308-0d34-0410-b5e6-96231b3b80d8
10fe93d57c11f068aa4d78eb4ca7f60329818306 08-Aug-2010 Douglas Gregor <dgregor@apple.com> Revert r110440, the fix for PR4897. Chris claims to have a better way.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110544 91177308-0d34-0410-b5e6-96231b3b80d8
2f05449fa1c9015a84b74c3308b856fc704662be 08-Aug-2010 Benjamin Kramer <benny.kra@googlemail.com> Push location through the MacroUndefined PPCallback and use it to print #undefs in -dD mode. (PR7818)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110523 91177308-0d34-0410-b5e6-96231b3b80d8
91db30ad4a7bad37b57748c31f8b592b7bf7cbeb 06-Aug-2010 Douglas Gregor <dgregor@apple.com> Fix the #include search path when reading from stdin, from Jon Simons!
Fixes PR4897.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110440 91177308-0d34-0410-b5e6-96231b3b80d8
083abdf67f157e9d2ab5a8c9d5e71240479d3c99 28-Jul-2010 Sebastian Redl <sebastian.redl@getdesigned.at> Record macros in dependent PCHs. Also add various info tables to dependent PCHs; tests for this to follow.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109554 91177308-0d34-0410-b5e6-96231b3b80d8
94e3d1f0349634235606be77c08b3a48a6d563d6 27-Jul-2010 Ted Kremenek <kremenek@apple.com> Add PTHLexer::LexEndOfFile() to emit diagnostics at end-of-file similar to those by Lexer::LexEndOfFile().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109486 91177308-0d34-0410-b5e6-96231b3b80d8
56eb1ec6e54080e47bbc62412737c25afb5211ed 27-Jul-2010 Ted Kremenek <kremenek@apple.com> Fix predicate in 'InCachingLexMode' to include 'CurPTHLexer'.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109485 91177308-0d34-0410-b5e6-96231b3b80d8
f4f6c9db68465b886ec2e596feaa6ecc782395a4 26-Jul-2010 Douglas Gregor <dgregor@apple.com> Introduce basic support for loading a precompiled preamble while
reparsing an ASTUnit. When saving a preamble, create a buffer larger
than the actual file we're working with but fill everything from the
end of the preamble to the end of the file with spaces (so the lexer
will quickly skip them). When we load the file, create a buffer of the
same size, filling it with the file and then spaces. Then, instruct
the lexer to start lexing after the preamble, therefore continuing the
parse from the spot where the preamble left off.

It's now possible to perform a simple preamble build + parse (+
reparse) with ASTUnit. However, one has to disable a bunch of checking
in the PCH reader to do so. That part isn't committed; it will likely
be handled with some other kind of flag (e.g., -fno-validate-pch).

As part of this, fix some issues with null termination of the memory
buffers created for the preamble; we were trying to explicitly
NULL-terminate them, even though they were also getting implicitly
NULL terminated, leading to excess warnings about NULL characters in
source files.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109445 91177308-0d34-0410-b5e6-96231b3b80d8
a8e5c5bdbe387b2552c1c23b828f54abcf085a40 22-Jul-2010 Douglas Gregor <dgregor@apple.com> Improve performance during cursor traversal when a region of interest
is present.

Rather than using clang_getCursorExtent(), which requires
us to lex the token at the ending position to determine its
length. Then, we'd be comparing [a, b) source ranges that cover the
characters in the range rather than the normal behavior for Clang's
source ranges, which covers the tokens in the range. However, relexing
causes us to read the source file (which may come from a precompiled
header), which is rather unfortunate and affects performance.

In the new scheme, we only use Clang-style source ranges that cover
the tokens in the range. At the entry points where this matters
(clang_annotateTokens, clang_getCursor), we make sure to move source
locations to the start of the token.

Addresses most of <rdar://problem/8049381>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109134 91177308-0d34-0410-b5e6-96231b3b80d8
f033f1da4a34f8df6e95e9929dc04ff54bb8fb01 20-Jul-2010 Douglas Gregor <dgregor@apple.com> Introduce a new lexer function to compute the "preamble" of a file,
which is the part of the file that contains all of the initial
comments, includes, and preprocessor directives that occur before any
of the actual code. Added a new -print-preamble cc1 action that is
only used for testing.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108913 91177308-0d34-0410-b5e6-96231b3b80d8
9b36c3f0de0105e903130bbda3c4aea7d792c0af 13-Jul-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> Modify the pragma handlers to accept and use StringRefs instead of IdentifierInfos.

When loading the PCH, IdentifierInfos that are associated with pragmas cause declarations that use these identifiers to be deserialized (e.g. the "clang" pragma causes the "clang" namespace to be loaded).
We can avoid this if we just use StringRefs for the pragmas.

As a bonus, since we don't have to create and pass IdentifierInfos, the pragma interfaces get a bit more simplified.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108237 91177308-0d34-0410-b5e6-96231b3b80d8
03569eaabceb14a20e23d043135fdccbc7309a96 12-Jul-2010 Argyrios Kyrtzidis <akyrtzi@gmail.com> If we are past tok::eof and in caching lex mode, avoid caching repeated tok::eofs.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108175 91177308-0d34-0410-b5e6-96231b3b80d8
abfe094ce71c42656dcb84a3bdc3e79cb3c16fc3 26-Jun-2010 Chris Lattner <sabre@nondot.org> Implement support for #pragma message, patch by Michael Spencer!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@106950 91177308-0d34-0410-b5e6-96231b3b80d8
170e5080011acc60e33ec9b53f96c569a1078aa9 17-Jun-2010 Gabor Greif <ggreif@gmail.com> fix some more gcc3.4 constness warnings

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@106216 91177308-0d34-0410-b5e6-96231b3b80d8
c54539cd173e20d8b179b9488fabfc5eb4cf5ee6 15-Jun-2010 Chris Lattner <sabre@nondot.org> fix the various buildbot failures by ensuring that tokens are really completely initialized.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@106043 91177308-0d34-0410-b5e6-96231b3b80d8
c72cc5072cdc1a1a6e05f9d0f962f293a69248c4 11-Jun-2010 Daniel Dunbar <daniel@zuster.org> Preprocessor: Ignore unknown pragmas in -E -dM and -Eonly modes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@105830 91177308-0d34-0410-b5e6-96231b3b80d8
1c573cb0e06bac4e557123703069da7dd45d3dc0 08-Jun-2010 Benjamin Kramer <benny.kra@googlemail.com> Token is POD-like.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@105604 91177308-0d34-0410-b5e6-96231b3b80d8
b90f4b3fb94056609da9cca5eef7358d95a363b2 26-May-2010 Douglas Gregor <dgregor@apple.com> Tell the string literal parser when it's not permitted to emit
diagnostics. That would be while we're parsing string literals for the
sole purpose of producing a diagnostic about them. Fixes

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@104684 91177308-0d34-0410-b5e6-96231b3b80d8
321c2688198f6f1a4086964e505f239683423cec 14-May-2010 Chris Lattner <sabre@nondot.org> fit in 80 cols, remove prototypes for handling #assert since apparently noone cares.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@103782 91177308-0d34-0410-b5e6-96231b3b80d8
27b0f510d1d4f9ed5668eec1195f04b4a74a4426 04-May-2010 Chris Lattner <sabre@nondot.org> add the ability to associate 'category' names with diagnostics
and diagnostic groups. This allows the compiler to group
diagnostics together (e.g. "Logic Warning",
"Format String Warning", etc) like the static analyzer does.
This is not exposed through anything in the compiler yet.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@103051 91177308-0d34-0410-b5e6-96231b3b80d8
e127a0d80155b45dafe77f2b4380e5fa111a3345 20-Apr-2010 Chris Lattner <sabre@nondot.org> push some source location information down through the compiler,
into ContentCache::getBuffer. This allows it to produce
diagnostics on the broken #include line instead of without a

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@101939 91177308-0d34-0410-b5e6-96231b3b80d8
6fbe3ebeaef08665a37423f8425314c90b8b5bcf 19-Apr-2010 Chris Lattner <sabre@nondot.org> add a PPCallback handler for a skipped #include, patch by
Zhanyong Wan!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@101813 91177308-0d34-0410-b5e6-96231b3b80d8
8877321ca66b2887c2f377a7f724a62f34fdf1cd 14-Apr-2010 Chris Lattner <sabre@nondot.org> make the token paste avoidance logic turn "..." into ".. ." instead of ". . ."
when avoiding paste. Patch by David Peixotto!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@101218 91177308-0d34-0410-b5e6-96231b3b80d8
0cc70f1586874aed3337c57cb255b112cdac0948 30-Mar-2010 Daniel Dunbar <daniel@zuster.org> Fix FileEntry declaration.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@99896 91177308-0d34-0410-b5e6-96231b3b80d8
dbd8209b33e6c9f151e4913a9c095d64a95439c4 23-Mar-2010 Daniel Dunbar <daniel@zuster.org> PPCallbacks: Add hook for reaching the end of the main file, and fix DependencyFile to not do work in its destructor.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@99257 91177308-0d34-0410-b5e6-96231b3b80d8
1b058e8956ea29a7fb7649fdee2b80581dcbdf39 19-Mar-2010 Douglas Gregor <dgregor@apple.com> Robustify PreprocessingRecord slightly, by only creating macro
instantiations when we have the corresponding macro definition and by
removing macro definition information from our table when the macro is

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@99004 91177308-0d34-0410-b5e6-96231b3b80d8
6a5a23f8e7fb65e028c8092bc1d1a1d9dfe2e9bc 19-Mar-2010 Douglas Gregor <dgregor@apple.com> Implement serialization and lazy deserialization of the preprocessing
record (which includes all macro instantiations and definitions). As
with all lay deserialization, this introduces a new external source
(here, an external preprocessing record source) that loads all of the
preprocessed entities prior to iterating over the entities.

The preprocessing record is an optional part of the precompiled header
that is disabled by default (enabled with
-detailed-preprocessing-record). When the preprocessor given to the
PCH writer has a preprocessing record, that record is written into the
PCH file. When the PCH reader is given a PCH file that contains a
preprocessing record, it will be lazily loaded (which, effectively,
implicitly adds -detailed-preprocessing-record). This is the first
case where we have sections of the precompiled header that are
added/removed based on a compilation flag, which is
unfortunate. However, this data consumes ~550k in the PCH file for
Cocoa.h (out of ~9.9MB), and there is a non-trivial cost to gathering
this detailed preprocessing information, so it's too expensive to turn
on by default. In the future, we should investigate a better encoding
of this information.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@99002 91177308-0d34-0410-b5e6-96231b3b80d8
b9e1b75772db2c7db566c6034ba90a07f22e35eb 19-Mar-2010 Douglas Gregor <dgregor@apple.com> Make the preprocessing record a PPCallbacks subclass itself,
eliminating the extra PopulatePreprocessingRecord object. This will
become useful once we start writing the preprocessing record to
precompiled headers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98966 91177308-0d34-0410-b5e6-96231b3b80d8
94dc8f640ebea52241412512ed48601626edbc58 19-Mar-2010 Douglas Gregor <dgregor@apple.com> Optionally store a PreprocessingRecord in the preprocessor itself, and
tie its creation to a CC1 flag -detailed-preprocessing-record.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98963 91177308-0d34-0410-b5e6-96231b3b80d8
bf7efa2742dc94363d3561d284c9d634fc5a780f 18-Mar-2010 Douglas Gregor <dgregor@apple.com> Explicitly link macro instantiations to macro definitions in the
preprocessing record. Use that link with clang_getCursorReferenced()
and clang_getCursorDefinition() to match instantiations of a macro to
the definition of the macro.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98842 91177308-0d34-0410-b5e6-96231b3b80d8
572feb2a190b5e8b04fb06c4ac50ee0f61e93ff0 18-Mar-2010 Douglas Gregor <dgregor@apple.com> Expose macro definitions as CIndex cursors. These can still only be
generated by clang_annotateTokens().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98837 91177308-0d34-0410-b5e6-96231b3b80d8
4ae8f298b1ea51b4c2234f9148e2e4349c9bdd23 18-Mar-2010 Douglas Gregor <dgregor@apple.com> Introduce the notion of a "preprocessing record", which keeps track of
the macro definitions and macro instantiations that are found
during preprocessing. Preprocessing records are *not* generated by
default; rather, we provide a PPCallbacks subclass that hooks into the
existing callback mechanism to record this activity.

The only client of preprocessing records is CIndex, which keeps track
of macro definitions and instantations so that they can be exposed via
cursors. At present, only token annotation uses these facilities, and
only for macro instantiations; both will change in the near
future. However, with this change, token annotation properly annotates
macro instantiations that do not produce any tokens and instantiations
of macros that are later undef'd, improving our consistency.

Preprocessing directives that are not macro definitions are still
handled by clang_annotateTokens() via re-lexing, so that we don't have
to track every preprocessing directive in the preprocessing record.

Performance impact of preprocessing records is still TBD, although it
is limited to CIndex and therefore out of the path of the main compiler.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98836 91177308-0d34-0410-b5e6-96231b3b80d8
dbf8ee630e4c86e5150492eaf8dbceea3c718ee1 17-Mar-2010 Douglas Gregor <dgregor@apple.com> Entering the main source file in the preprocessor can fail if the
source file has been changed. Handle that failure more gracefully.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98727 91177308-0d34-0410-b5e6-96231b3b80d8
a543016fe07030f695d6d56fd22c8c8da617e0d7 16-Mar-2010 Douglas Gregor <dgregor@apple.com> Audit all callers of SourceManager::getCharacterData(); update some of
them to recover more gracefully on failure.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98672 91177308-0d34-0410-b5e6-96231b3b80d8
12fab31aa5868b1a6b52246b5a87daa48a338fe2 16-Mar-2010 Douglas Gregor <dgregor@apple.com> Fix header-search problems with precompiled headers, where the
presence or absence of header map arguments when using the precompiled
header would cause Clang to get confused about which headers had
already been included/imported, along with their controlling
macros. The fundamental problem is that the serialization of the
header search information was relying on the UIDs of FileEntry objects
at PCH generation time and PCH load time to be equivalent, which
effectively means that we had to probe the same files in the same
order. Differing header map arguments caused an extra FileEntry
lookup, but it's easy to imagine other minor command-line arguments
triggering this problem.

Header-search information is now encoded along with the
source-location entry for a file, so that we register information
about a file's properties as a header at the same time we create the
FileEntry for that file.

Fixes <rdar://problem/7743243>.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98636 91177308-0d34-0410-b5e6-96231b3b80d8
50f6af7a6d6951a63f3da7d4c5a7d3965bf73b63 16-Mar-2010 Douglas Gregor <dgregor@apple.com> Introduce optional "Invalid" parameters to routines that invoke the
SourceManager's getBuffer() and, therefore, could fail, along with
Preprocessor::getSpelling(). Use the Invalid parameters in the literal
parsers (string, floating point, integral, character) to make them
robust against errors that stem from, e.g., PCH files that are not
consistent with the underlying file system.

I still need to audit every use caller to all of these routines, to
determine which ones need specific handling of error conditions.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98608 91177308-0d34-0410-b5e6-96231b3b80d8
193575455e00eca03fd7177f60e3f2e6263cb661 13-Mar-2010 Kovarththanan Rajaratnam <kovarththanan.rajaratnam@gmail.com> Use SmallString instead of SmallVector

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98436 91177308-0d34-0410-b5e6-96231b3b80d8
700030ebddb987936d4fee14d9412821d96e4840 12-Mar-2010 Kovarththanan Rajaratnam <kovarththanan.rajaratnam@gmail.com> Switch over IdentifierInfoLookup to StringRef

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98337 91177308-0d34-0410-b5e6-96231b3b80d8
a5d10c4df435964600e104ebef6a96b106e416b7 07-Mar-2010 Kovarththanan Rajaratnam <kovarththanan.rajaratnam@gmail.com> Rename to addPPCallbacks since we're effectively adding a callback and maybe chaining it to an existing one

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@97913 91177308-0d34-0410-b5e6-96231b3b80d8
51f5fe3f2527cd1640d798d8d134268b14de3e86 27-Feb-2010 Benjamin Kramer <benny.kra@googlemail.com> Move method out-of-line. I thought this would be a candidate for inlining but I was wrong.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@97330 91177308-0d34-0410-b5e6-96231b3b80d8
a15e7eaea9e63f58d1a7e72ed167cbb244da438a 27-Feb-2010 Benjamin Kramer <benny.kra@googlemail.com> Fix thinko.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@97323 91177308-0d34-0410-b5e6-96231b3b80d8
ddeea5644367c9c153c9fee9e51bdea85ce43cbd 27-Feb-2010 Benjamin Kramer <benny.kra@googlemail.com> Add an overload of Preprocessor::getSpelling which takes a SmallVector and
returns a StringRef. Use it to simplify some repetitive code.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@97322 91177308-0d34-0410-b5e6-96231b3b80d8
39d67117f896c6e2faa727671ef64b3c04b0e3fe 08-Feb-2010 Sebastian Redl <sebastian.redl@getdesigned.at> When placing an annotation token over an existing annotation token, make sure that the new token's range extends to the end of the old token. Assert that in AnnotateCachedTokens. Fixes PR6248.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@95555 91177308-0d34-0410-b5e6-96231b3b80d8
fc8ea23eb6cbaaa5046f2abb4c033e24c8659efd 26-Jan-2010 Douglas Gregor <dgregor@apple.com> Introduce a CIndex API for lexing the raw tokens within a given source
range. The token-annotation function does nothing, yet.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@94551 91177308-0d34-0410-b5e6-96231b3b80d8
33e9abd21083a0191a7676a04b497006d2da184d 22-Jan-2010 Douglas Gregor <dgregor@apple.com> Teach CIndex's cursor visitor to restrict its traversal to a specific
region of interest (if provided). Implement clang_getCursor() in terms
of this traversal rather than using the Index library; the unified
cursor visitor is more complete, and will be The Way Forward.

Minor other tweaks needed to make this work:
- Extend Preprocessor::getLocForEndOfToken() to accept an offset
from the end, making it easy to move to the last character in the
token (rather than just past the end of the token).
- In Lexer::MeasureTokenLength(), the length of whitespace is zero.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@94200 91177308-0d34-0410-b5e6-96231b3b80d8
f45b646244705410866d62f1d8bf017a047ed662 22-Jan-2010 Chris Lattner <sabre@nondot.org> revert my patch for rdar://7520940 that warns when a published header
is #included with "foo.h" style syntax instead of framework syntax.
It produced too much noise.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@94120 91177308-0d34-0410-b5e6-96231b3b80d8
046c2277dcbcc8eb89dbb5b1b8c5226b7cb81635 18-Jan-2010 Chris Lattner <sabre@nondot.org> allow the HandlerComment callback to push tokens into the
preprocessor. This could be used by an OpenMP implementation
or something. Patch by Abramo Bagnara!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93795 91177308-0d34-0410-b5e6-96231b3b80d8
a139481e62fdb209d9d87a54a5733f989d2e8d51 10-Jan-2010 Chris Lattner <sabre@nondot.org> stringref'ize a bunch of filename handling logic. Much
nicer than passing around two const char*'s.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93094 91177308-0d34-0410-b5e6-96231b3b80d8
804f65271953f358dc01bfcf74a93e8c93c5b2d6 10-Jan-2010 Chris Lattner <sabre@nondot.org> implement rdar://7520940: published framework headers should
import other headers within the same framework with the full
framework path, not with a relative include.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93083 91177308-0d34-0410-b5e6-96231b3b80d8
f8c381fff1a3c10e0f7f0c9f223f08e8d7dce0e3 04-Jan-2010 Daniel Dunbar <daniel@zuster.org> Add missing newline (which breaks MSVC build???)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92522 91177308-0d34-0410-b5e6-96231b3b80d8
88a35862fbe473f2a4f0c19f24dbe536937e1dc6 04-Jan-2010 Douglas Gregor <dgregor@apple.com> Teach Preprocessor::macro_begin/macro_end to lazily load all macro
definitions from a precompiled header. This ensures that
code-completion with macro names behaves the same with or without
precompiled headers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92497 91177308-0d34-0410-b5e6-96231b3b80d8
94c939dc1d4958b62ea5a89294dd8b2905f3191f 24-Dec-2009 John McCall <rjmccall@apple.com> Diagnose out-of-bounds floating-point constants. Fixes rdar://problem/6974641

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92127 91177308-0d34-0410-b5e6-96231b3b80d8
4718e67ca9018342483dccb40d43e339101ac386 23-Dec-2009 Chris Lattner <sabre@nondot.org> update comments

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92022 91177308-0d34-0410-b5e6-96231b3b80d8
23f77e59718385512984d4e2a021bef52b9f6ddf 15-Dec-2009 Chris Lattner <sabre@nondot.org> set up the machinery for a MacroArgs cache hanging off Preprocessor.
We creating and free thousands of MacroArgs objects (and the related
std::vectors hanging off them) for the testcase in PR5610 even though
there are only ~20 live at a time. This doesn't actually use the
cache yet.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91391 91177308-0d34-0410-b5e6-96231b3b80d8
34f349da38a7bd99103e12d8ea6c73bc8d025193 14-Dec-2009 Chris Lattner <sabre@nondot.org> teach clang to recover gracefully from conflict markers left in source
files: PR5238.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91270 91177308-0d34-0410-b5e6-96231b3b80d8
1d9c54df56391ac4740db27d551782e81189cb51 14-Dec-2009 Chris Lattner <sabre@nondot.org> formatting changes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91263 91177308-0d34-0410-b5e6-96231b3b80d8
63ceaa32a2371e38d1f912080fe471285e6b6e56 06-Dec-2009 Daniel Dunbar <daniel@zuster.org> Change Preprocessor::EnterSourceFile to make ErrorStr non-optional, clients should be forced to deal with error conditions.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90700 91177308-0d34-0410-b5e6-96231b3b80d8
109ae73ec71a1d27358510ce049b59637b4a6b40 03-Dec-2009 Douglas Gregor <dgregor@apple.com> Minor cleanup to the code-completion-point logic suggested by Chris.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90459 91177308-0d34-0410-b5e6-96231b3b80d8
2968442603b029949246467253eeac8139a5b6d8 02-Dec-2009 Douglas Gregor <dgregor@apple.com> Extend the source manager with the ability to override the contents of
files with the contents of an arbitrary memory buffer. Use this new
functionality to drastically clean up the way in which we handle file
truncation for code-completion: all of the truncation/completion logic
is now encapsulated in the preprocessor where it belongs

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90300 91177308-0d34-0410-b5e6-96231b3b80d8
39d9841ed4c0568d4b44dfbc12ac04491f60a374 01-Dec-2009 Chris Lattner <sabre@nondot.org> pass the reason for failure up from MemoryBuffer and report it
in diagnostics when we fail to open a file. This allows us to
report things like:

$ clang test.c -I.
test.c:2:10: fatal error: error opening file './foo.h': Permission denied
#include "foo.h"

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90276 91177308-0d34-0410-b5e6-96231b3b80d8
6e2901407bff59aeb4cc301cc58b034723d0eb49 30-Nov-2009 Chris Lattner <sabre@nondot.org> Fix PR5633 by making the preprocessor handle the case where we can
stat a file but where mmaping it fails. In this case, we emit an
error like:
t.c:1:10: fatal error: error opening file '../../foo.h'

instead of "cannot find file".

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90110 91177308-0d34-0410-b5e6-96231b3b80d8
0ff1042ddaad1419264be0de6da17f3b378482a4 14-Nov-2009 Daniel Dunbar <daniel@zuster.org> Add static version of Preprocessor::getSpelling.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@88732 91177308-0d34-0410-b5e6-96231b3b80d8
444be7366d0a1e172c0290a1ea54c1cb16b5947c 13-Nov-2009 Daniel Dunbar <daniel@zuster.org> Wherein the TargetInfo argument to Preprocessor is made 'const' and propogated.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@87087 91177308-0d34-0410-b5e6-96231b3b80d8
3574f46cf495ec61618fd6864b045c5b1d0d5068 12-Nov-2009 Daniel Dunbar <daniel@zuster.org> Switch PTHManager to using diagnostics for most errors.

Also, always give errors on a token-cache PTH failure.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86939 91177308-0d34-0410-b5e6-96231b3b80d8
5814e657c9ad9ef6049a2a4af0d2aad248a8a15c 11-Nov-2009 Daniel Dunbar <daniel@zuster.org> Allow Preprocessor to take ownership of the HeaderSearch object. I think it should probably always own the header search object, but I'm not sure...

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86882 91177308-0d34-0410-b5e6-96231b3b80d8
314b97f8c564b465af605efaee23f91ec18a982b 10-Nov-2009 Douglas Gregor <dgregor@apple.com> Improve parsing of template arguments to lay the foundation for
handling template template parameters properly. This refactoring:

- Parses template template arguments as id-expressions, representing
the result of the parse as a template name (Action::TemplateTy)
rather than as an expression (lame!).

- Represents all parsed template arguments via a new parser-specific
type, ParsedTemplateArgument, which stores the kind of template
argument (type, non-type, template) along with all of the source
information about the template argument. This replaces an ad hoc
set of 3 vectors (one for a void*, which was either a type or an
expression; one for a bit telling whether the first was a type or
an expression; and one for a single source location pointing at
the template argument).

- Moves TemplateIdAnnotation into the new Parse/Template.h. It never
belonged in the Basic library anyway.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86708 91177308-0d34-0410-b5e6-96231b3b80d8
c3222091e1ffa35d0264ca6b680a88c9dc84ede2 05-Nov-2009 Daniel Dunbar <daniel@zuster.org> Make LookUpIdentifierInfo const. This makes the Identifiers table mutable and is
a little fuzzy, but conceptually it's just uniquing the identifier.

Chris, please review. I debated splitting into const/non-const versions where
the const one propogated constness to the resulting IdentifierInfo*.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86106 91177308-0d34-0410-b5e6-96231b3b80d8
3da736c1143126be19b253804b3b135ebcd3d6ff 05-Nov-2009 Daniel Dunbar <daniel@zuster.org> StringRefize Preprocessor::getIdentifierInfo.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86105 91177308-0d34-0410-b5e6-96231b3b80d8
90b1827c1c1cf075266b96b416eefcf37924333b 05-Nov-2009 Daniel Dunbar <daniel@zuster.org> Kill PreprocessorFactory, which was both morally repugnant and totally unused.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86076 91177308-0d34-0410-b5e6-96231b3b80d8
082fb9ae304a25aa6be1f74d9b9720c52a96231e 04-Nov-2009 Daniel Dunbar <daniel@zuster.org> Use unsigned char instead of unsigned : 8 to make the optimizer happier.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@85985 91177308-0d34-0410-b5e6-96231b3b80d8
014e88d94ff83e3aad4e33b16413a2d1817ec208 04-Nov-2009 Douglas Gregor <dgregor@apple.com> Parsing and semantic analysis for template-ids that name overloaded
operators, e.g.,


which now works in declarators, id-expressions, and member access
expressions. This commit only implements the non-dependent case, where
we can resolve the template-id to an actual declaration.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@85966 91177308-0d34-0410-b5e6-96231b3b80d8
92bd8c70a6837b647a6c55964f8d0a50bf561dbc 02-Nov-2009 John Thompson <John.Thompson.JTSoftware@gmail.com> Added __has_include and __has_include_next.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@85834 91177308-0d34-0410-b5e6-96231b3b80d8
a28cc090f265d5bdaef9ebb24fb9d579928d3a18 30-Oct-2009 John Thompson <John.Thompson.JTSoftware@gmail.com> Re-arranged some internal functions for coming __has_include changes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@85589 91177308-0d34-0410-b5e6-96231b3b80d8
9f8eb2032030482b1d3de86e9bee725d93564302 26-Oct-2009 Chandler Carruth <chandlerc@gmail.com> Update location of DataTypes.h to reflect move in LLVM with r85086.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@85087 91177308-0d34-0410-b5e6-96231b3b80d8
b79fe2d28777652a4df4f49dc876cbec060ca90e 09-Oct-2009 Mike Stump <mrs@apple.com> This fixes support for complex literals, reworked to avoid a goto, and
to add a flag noting the presence of a Microsoft extension suffix (i8,
i16, i32, i64). Patch by John Thompson.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@83591 91177308-0d34-0410-b5e6-96231b3b80d8
b657f115c3b4e262e72906a28cbcf3eaccd9460c 22-Sep-2009 Douglas Gregor <dgregor@apple.com> Replace the -code-completion-dump option with


which performs code completion at the specified location by truncating
the file at that position and enabling code completion. This approach
makes it possible to run multiple tests from a single test file, and
gives a more natural command-line interface.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82571 91177308-0d34-0410-b5e6-96231b3b80d8
0369297684c13d5495ab490d46ed9d14ef235ce3 18-Sep-2009 Chris Lattner <sabre@nondot.org> allow clearing this value.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82271 91177308-0d34-0410-b5e6-96231b3b80d8
81b747b7fcc91c2fba9a3183d8fac80adbfc1d3e 17-Sep-2009 Douglas Gregor <dgregor@apple.com> Initial implementation of a code-completion interface in Clang. In
essence, code completion is triggered by a magic "code completion"
token produced by the lexer [*], which the parser recognizes at
certain points in the grammar. The parser then calls into the Action
object with the appropriate CodeCompletionXXX action.

Sema implements the CodeCompletionXXX callbacks by performing minimal
translation, then forwarding them to a CodeCompletionConsumer
subclass, which uses the results of semantic analysis to provide
code-completion results. At present, only a single, "printing" code
completion consumer is available, for regression testing and
debugging. However, the design is meant to permit other
code-completion consumers.

This initial commit contains two code-completion actions: one for
member access, e.g., "x." or "p->", and one for
nested-name-specifiers, e.g., "std::". More code-completion actions
will follow, along with improved gathering of code-completion results
for the various contexts.

[*] In the current -code-completion-dump testing/debugging mode, the
file is truncated at the completion point and EOF is translated into
"code completion".

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82166 91177308-0d34-0410-b5e6-96231b3b80d8
f4d5eb4866a27d497f0bb75b12c2ffd48ad4d9c0 12-Sep-2009 Benjamin Kramer <benny.kra@googlemail.com> SmallVectorize preprocessor's token cache. Testing shows there is almost never
more than one token in the cache.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81612 91177308-0d34-0410-b5e6-96231b3b80d8
1eb4433ac451dc16f4133a88af2d002ac26c58ef 09-Sep-2009 Mike Stump <mrs@apple.com> Remove tabs, and whitespace cleanups.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81346 91177308-0d34-0410-b5e6-96231b3b80d8
79d39f92590cf2e91bf81486b02cd1156d13ca54 09-Sep-2009 Mike Stump <mrs@apple.com> Remove tab characters.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81340 91177308-0d34-0410-b5e6-96231b3b80d8
0c5920c3159e6a8c83a871f26f7d7ea621c29860 05-Sep-2009 Daniel Dunbar <daniel@zuster.org> Add missing include.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81059 91177308-0d34-0410-b5e6-96231b3b80d8
ba1e898c64048e25cb65afec3807ad463e41914b 23-Jul-2009 Argyrios Kyrtzidis <akyrtzi@gmail.com> Change Preprocessor to keep a copy of LangOptions instead of reference, like ASTContext.

Now when creating a Preprocessor we can pass it a temporary LangOptions object instead of having to remember to keep it around.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@76815 91177308-0d34-0410-b5e6-96231b3b80d8
2e22253e03e175144aeb9d13350a12fd83f858be 02-Jul-2009 Douglas Gregor <dgregor@apple.com> Add support for retrieving the Doxygen comment associated with a given
declaration in the AST.

The new ASTContext::getCommentForDecl function searches for a comment
that is attached to the given declaration, and returns that comment,
which may be composed of several comment blocks.

Comments are always available in an AST. However, to avoid harming
performance, we don't actually parse the comments. Rather, we keep the
source ranges of all of the comments within a large, sorted vector,
then lazily extract comments via a binary search in that vector only
when needed (which never occurs in a "normal" compile).

Comments are written to a precompiled header/AST file as a blob of
source ranges. That blob is only lazily loaded when one requests a
comment for a declaration (this never occurs in a "normal" compile).

The indexer testbed now supports comment extraction. When the
-point-at location points to a declaration with a Doxygen-style
comment, the indexer testbed prints the associated comment
block(s). See test/Index/comments.c for an example.

Some notes:
- We don't actually attempt to parse the comment blocks themselves,
beyond identifying them as Doxygen comment blocks to associate them
with a declaration.
- We won't find comment blocks that aren't adjacent to the
declaration, because we start our search based on the location of
the declaration.
- We don't go through the necessary hops to find, for example,
whether some redeclaration of a declaration has comments when our
current declaration does not. Similarly, we don't attempt to
associate a \param Foo marker in a function body comment with the
parameter named Foo (although that is certainly possible).
- Verification of my "no performance impact" claims is still "to be

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@74704 91177308-0d34-0410-b5e6-96231b3b80d8
5e9f35c7cb61aea46f56d46c77cbcf47c0cf28ba 14-Jun-2009 Douglas Gregor <dgregor@apple.com> Update LLVM.

Implement support for C++ Substitution Failure Is Not An Error
(SFINAE), which says that errors that occur during template argument
deduction do *not* produce diagnostics and do not necessarily make a
program ill-formed. Instead, template argument deduction silently
fails. This is currently implemented for template argument deduction
during matching of class template partial specializations, although
the mechanism will also apply to template argument deduction for
function templates. The scheme is simple:

- If we are in a template argument deduction context, any diagnostic
that is considered a SFINAE error (or warning) will be
suppressed. The error will be propagated up the call stack via the
normal means.
- By default, all warnings and errors are SFINAE errors. Add the
NoSFINAE class to a diagnostic in the .td file to make it a hard
error (e.g., for access-control violations).

Note that, to make this fully work, every place in Sema that emits an
error *and then immediately recovers* will need to check
Sema::isSFINAEContext() to determine whether it must immediately
return an error rather than recovering.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@73332 91177308-0d34-0410-b5e6-96231b3b80d8
1b63e4f732dbc73d90abf886b4d21f8e3a165f6d 14-Jun-2009 Chris Lattner <sabre@nondot.org> Sink the BuiltinInfo object from ASTContext into the
preprocessor and initialize it early in clang-cc. This
ensures that __has_builtin works in all modes, not just
when ASTContext is around.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@73319 91177308-0d34-0410-b5e6-96231b3b80d8
148772a841cae6f32db16d890e788b92a763bb3f 13-Jun-2009 Chris Lattner <sabre@nondot.org> implement and document a new __has_feature and __has_builtin magic
builtin preprocessor macro. This appears to work with two caveats:
1) builtins are registered in -E mode, and 2) target-specific builtins
are unconditionally registered even if they aren't supported by the
target (e.g. SSE4 builtin when only SSE1 is enabled).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@73289 91177308-0d34-0410-b5e6-96231b3b80d8
2a1c363f38e59a5044fc349aa7e538a50954c244 01-Jun-2009 Eli Friedman <eli.friedman@gmail.com> PR4283: Don't truncate multibyte character constants in the

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72686 91177308-0d34-0410-b5e6-96231b3b80d8
7633ab80454147ab4892b8de5fd16582a1f221e8 14-May-2009 Douglas Gregor <dgregor@apple.com> Add forward declaration of Token. Thanks to Martin Doucha for pointing this out

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@71772 91177308-0d34-0410-b5e6-96231b3b80d8
e5393fb93eb879d9ebbef102ae9311fa77e023cc 03-May-2009 Daniel Dunbar <daniel@zuster.org> PR4063, with feeling: Chain PP callbacks by default.
- This is somewhat cleaner and also fixes PR4063 for real, I had the
order wrong so we were just creating an empty dependency file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@70687 91177308-0d34-0410-b5e6-96231b3b80d8
8c5a760b82e73ed90b560090772db97e2ae27b09 26-Apr-2009 Douglas Gregor <dgregor@apple.com> Lazily load the controlling macros for all of the headers known in the
PCH file. In the Cocoa-prefixed "Hello, World" benchmark, this takes
us from reading 503 identifiers down to 37 and from 470 macros down to
4. It also results in an 8% performance improvement.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@70094 91177308-0d34-0410-b5e6-96231b3b80d8
83d63c78810556d26b62ac4cbae2eda6cdd2570c 24-Apr-2009 Steve Naroff <snaroff@apple.com> Add PCH support for #import.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69987 91177308-0d34-0410-b5e6-96231b3b80d8
2451b528fe114595d0f10ef2c05047928558ab0f 21-Apr-2009 Chris Lattner <sabre@nondot.org> improve MacroInfo to track the source range of the macro definition,
patch by Alexei Svitkine!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69659 91177308-0d34-0410-b5e6-96231b3b80d8
41c17473e3ece9e60d97c5d9397866b7730cf7ee 21-Apr-2009 Chris Lattner <sabre@nondot.org> add a preprocessor callback function for #undef, patch by
Alexei Svitkine!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69656 91177308-0d34-0410-b5e6-96231b3b80d8
4bc11af9bed1d4a247e3db1fcb754d410ad99099 21-Apr-2009 Sanjiv Gupta <sanjiv.gupta@microchip.com> Use an APInt of target int size to detect overflow while parsing multichars.
So 'abc' on i16 platforms will warn but not on i32 platforms.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69653 91177308-0d34-0410-b5e6-96231b3b80d8
033749571f8d4c804eeb357c70b06424aa24503b 19-Apr-2009 Chris Lattner <sabre@nondot.org> add a new Lexer::SkipEscapedNewLines method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69483 91177308-0d34-0410-b5e6-96231b3b80d8
24f0e48c0aa62f2268e061aad70f9b19a59e7b52 19-Apr-2009 Chris Lattner <sabre@nondot.org> factor escape newline measuring out into its own helper function.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69482 91177308-0d34-0410-b5e6-96231b3b80d8
ab82f41b217ce588a9456c0b4411f219d3ed0df8 18-Apr-2009 Chris Lattner <sabre@nondot.org> #line is allowed to have macros that expand to nothing after them.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69401 91177308-0d34-0410-b5e6-96231b3b80d8
19cbb44e3c4f2181dafa2ab92d3e3a26619b71d9 16-Apr-2009 Chris Lattner <sabre@nondot.org> tblgen is now passing diagnostic group information in the .inc file, ignore it everywhere.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69269 91177308-0d34-0410-b5e6-96231b3b80d8
4ac072aceb9c68f7a023050e2ab9ccacb4fe6e5b 15-Apr-2009 Chris Lattner <sabre@nondot.org> Tblgen now passes the default mapping explicitly, instead of having it
be tied to the diag class. This requires an LLVM tree update.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69175 91177308-0d34-0410-b5e6-96231b3b80d8
2c78b873f4f3823ae859c15674cb3d76c8554113 15-Apr-2009 Chris Lattner <sabre@nondot.org> Change Lexer::MeasureTokenLength to take a LangOptions reference.
This allows it to accurately measure tokens, so that we get:

t.cpp:8:13: error: unknown type name 'X'
static foo::X P;

instead of the woefully inferior:

t.cpp:8:13: error: unknown type name 'X'
static foo::X P;
~~~~ ^

Most of this is just plumbing to push the reference around.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69099 91177308-0d34-0410-b5e6-96231b3b80d8
c1f9d828c733ec1eba06d01070735d1f36fda733 13-Apr-2009 Chris Lattner <sabre@nondot.org> implement the microsoft/gnu "__COUNTER__" macro: rdar://4329310

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68933 91177308-0d34-0410-b5e6-96231b3b80d8
f4a72b07747cd6ae1a9f6974e148ae504c879349 12-Apr-2009 Chris Lattner <sabre@nondot.org> add a ppcallback hook for macro definitions.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68883 91177308-0d34-0410-b5e6-96231b3b80d8
e1d918e9fe55e3b34401fd5d420c47ea0f9572c9 11-Apr-2009 Douglas Gregor <dgregor@apple.com> Compare the predefines buffer in the PCH file with the predefines
buffer generated for the current translation unit. If they are
different, complain and then ignore the PCH file. This effectively
checks for all compilation options that somehow would affect
preprocessor state (-D, -U, -include, the dreaded -imacros, etc.).

When we do accept the PCH file, throw away the contents of the
predefines buffer rather than parsing them, since all of the results
of that parsing are already stored in the PCH file. This eliminates
the ugliness with the redefinition of __builtin_va_list, among other

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68838 91177308-0d34-0410-b5e6-96231b3b80d8
531cc8355608295caa01f31fa4e078d77a3d6470 11-Apr-2009 Chris Lattner <sabre@nondot.org> do a dance with predefines, and finally enable reading of macros from
PCH. This works now, except for limitations not being able to do things
with identifiers. The basic example in the testcase works though.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68832 91177308-0d34-0410-b5e6-96231b3b80d8
f908c519c1625e81d9e33d8a2a306a92834fe317 10-Apr-2009 Chris Lattner <sabre@nondot.org> make a method public

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68827 91177308-0d34-0410-b5e6-96231b3b80d8
7c5d24efcd2e505b5739f7def08dfe25ce59a1b2 10-Apr-2009 Chris Lattner <sabre@nondot.org> emit function-like and object-like macros to the PCH file.
Note that we don't do anything useful with identifier infos yet
and don't emit the tokens that the macros are defined to.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68797 91177308-0d34-0410-b5e6-96231b3b80d8
14f79002e58556798e86168c63e48d533287eda5 10-Apr-2009 Douglas Gregor <dgregor@apple.com> PCH serialization/deserialization of the source manager. With this
improvement, source locations read from the PCH file will properly
resolve to the source files that were used to build the PCH file

Once we have the preprocessor state stored in the PCH file, source
locations that refer to macro instantiations that occur in the PCH
file should have the appropriate instantiation information.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68758 91177308-0d34-0410-b5e6-96231b3b80d8
de076650fa79a02743b5fe57a72f991ba24779f9 08-Apr-2009 Chris Lattner <sabre@nondot.org> reject the #__include_macros directive unless it comes from the
predefines buffer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@68627 91177308-0d34-0410-b5e6-96231b3b80d8
22f0d09298ef05136f1ef6ebc4469b482cbd73e7 22-Mar-2009 Ted Kremenek <kremenek@apple.com> PTHManager::Create():
- Make the Diagnostic::Level for PTH errors to be specified by the caller

clang (driver):
- Set the PTHManager diagnostic level to "Diagnostic::Error" for -include-pth
(a hard error) and Diagnostic::Warning for -token-cache (we can still

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67462 91177308-0d34-0410-b5e6-96231b3b80d8
880dcf21dfdb3ff763c60195b6794bab0d913095 20-Mar-2009 Ted Kremenek <kremenek@apple.com> Add accessor Preprocessor::getPTHManager().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67351 91177308-0d34-0410-b5e6-96231b3b80d8
4d7a089144f336b11e5e8ce437b3b27bce2b310d 20-Mar-2009 Sebastian Redl <sebastian.redl@getdesigned.at> Bindir and Win32 builds work, so switch to .inc files. Leave the .def files in the tree for a day or so longer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67346 91177308-0d34-0410-b5e6-96231b3b80d8
68228634016f644a1164fa1f024a9ce2093656bf 19-Mar-2009 Ted Kremenek <kremenek@apple.com> Add PTHManager::getOriginalSourceFile(), a method that returns the name of the
original source file (if any) that was used to generate the PTH cache.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67343 91177308-0d34-0410-b5e6-96231b3b80d8
e89b6b272f9f3b15afa56a701a4d7a6b1001ed34 14-Mar-2009 Sebastian Redl <sebastian.redl@getdesigned.at> Revert the switch to the tablegen diags. It fails for seperate objdir builds and cmake builds, and I have no clue what to do about it. Revisit this after someone with a clue about the build systems has looked at it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67009 91177308-0d34-0410-b5e6-96231b3b80d8
3801d71fbeb6e74a22cdd6a858e10d887bc29c7a 14-Mar-2009 Sebastian Redl <sebastian.redl@getdesigned.at> Switch diagnostics from .def to tablegen files. Please validate the Windows build.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67007 91177308-0d34-0410-b5e6-96231b3b80d8
7c175fb196a2bc3dbc86ea3865c713e1875f3f6d 13-Mar-2009 Chris Lattner <sabre@nondot.org> fix PR3798 by ignoring all diagnostics generated while repreprocessing a file in rewrite macros.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66961 91177308-0d34-0410-b5e6-96231b3b80d8
836040f9eafe862fb1607df5c30cd3df0c22c832 13-Mar-2009 Chris Lattner <sabre@nondot.org> make Preprocessor::Diags be a pointer instead of a reference.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66955 91177308-0d34-0410-b5e6-96231b3b80d8
ba9eee326434ba62b180271d3cc2999d94ee0de4 12-Mar-2009 Chris Lattner <sabre@nondot.org> add a callback for macro expansion, based on a patch by Paolo Bolzoni!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66799 91177308-0d34-0410-b5e6-96231b3b80d8
1fa495304c81e03f07f278a47b5efe9317104aab 08-Mar-2009 Chris Lattner <sabre@nondot.org> simplify some logic by making ScratchBuffer handle the application of trailing
\0's to created tokens instead of making all clients do it. No functionality

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66373 91177308-0d34-0410-b5e6-96231b3b80d8
32fca722dd974b8202d0fb9c71b6c185c0767da6 04-Mar-2009 Chris Lattner <sabre@nondot.org> make the token lexer allocate its temporary token buffers for
preexpanded macro arguments from the preprocessor's bump pointer.
This reduces # mallocs from 12444 to 11792.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66025 91177308-0d34-0410-b5e6-96231b3b80d8
b2fb6de9070fea9abc56c8e8d5469066e964cefe 27-Feb-2009 Douglas Gregor <dgregor@apple.com> Clean up and document code modification hints.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65641 91177308-0d34-0410-b5e6-96231b3b80d8
dbb1ecc32ca122b07b7c98fd0a8f6f53985adacc 27-Feb-2009 Chris Lattner <sabre@nondot.org> fix some sema problems with wide strings and hook up basic codegen for them.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65582 91177308-0d34-0410-b5e6-96231b3b80d8
ac1afdc58e6ad5c969f45fd2ff6d140d3b4dd862 25-Feb-2009 Douglas Gregor <dgregor@apple.com> Include the appropriate header for malloc

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65471 91177308-0d34-0410-b5e6-96231b3b80d8
39a8de10c18365bde7062d8959b7ed525449c561 25-Feb-2009 Douglas Gregor <dgregor@apple.com> Implement parsing of nested-name-specifiers that involve template-ids, e.g.,


When we parse a template-id that names a type, it will become either a
template-id annotation (which is a parsed representation of a
template-id that has not yet been through semantic analysis) or a
typename annotation (where semantic analysis has resolved the
template-id to an actual type), depending on the context. We only
produce a type in contexts where we know that we only need type
information, e.g., in a type specifier. Otherwise, we create a
template-id annotation that can later be "upgraded" by transforming it
into a typename annotation when the parser needs a type. This occurs,
for example, when we've parsed "std::vector<int>" above and then see
the '::' after it. However, it means that when writing something like

template<> class Outer::Inner<int> { ... };

We have two tokens to represent Outer::Inner<int>: one token for the
nested name specifier Outer::, and one template-id annotation token
for Inner<int>, which will be passed to semantic analysis to define
the class template specialization.

Most of the churn in the template tests in this patch come from an
improvement in our error recovery from ill-formed template-ids.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65467 91177308-0d34-0410-b5e6-96231b3b80d8
685befeb5f6472585bae473a6389e47cab9eac67 20-Feb-2009 Chris Lattner <sabre@nondot.org> switch the macroinfo argument lists from being allocated off the heap
to being allocated from the same bumpptr that the MacroInfo objects
themselves are.

This speeds up -Eonly cocoa.h pth by ~4%, fsyntax-only is barely measurable.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65195 91177308-0d34-0410-b5e6-96231b3b80d8
cf29e0716bb3ecbbc15b74cd648367d6b075fdf0 20-Feb-2009 Chris Lattner <sabre@nondot.org> detemplatify setArgumentList and some other cleanups.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65187 91177308-0d34-0410-b5e6-96231b3b80d8
0301b3ff132a4d986c092d161cb77d74b04cd2a6 20-Feb-2009 Chris Lattner <sabre@nondot.org> require the MAcroInfo objects are explcitly destroyed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@65179 91177308-0d34-0410-b5e6-96231b3b80d8
719e61573f27c11057ecfe0dd8f141621602c571 18-Feb-2009 Chris Lattner <sabre@nondot.org> Next step toward making string diagnostics correct: handle
escapes in the string for subtoken positioning. This gives
us working examples like:

t.m:5:16: warning: field width should have type 'int', but argument has type 'unsigned int'
printf("\n\n%*d", (unsigned) 1, 1);
^ ~~~~~~~~~~~~

where before the caret pointed two spaces to the left.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64940 91177308-0d34-0410-b5e6-96231b3b80d8
e7fb48466afcbf2c4ccdfa658824282fdc3c512c 15-Feb-2009 Chris Lattner <sabre@nondot.org> track "just a little more" location information for macro instantiations.
Now instead of just tracking the expansion history, also track the full
range of the macro that got replaced. For object-like macros, this doesn't
change anything. For _Pragma and function-like macros, this means we track
the locations of the ')'.

This is required for PR3579 because apparently GCC uses the line of the ')'
of a function-like macro as the location to expand __LINE__ to.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64601 91177308-0d34-0410-b5e6-96231b3b80d8
a4b44dd9d30929a35c44d85102e5241ee847b2f2 13-Feb-2009 Ted Kremenek <kremenek@apple.com> Add some boilerplate to the PTH file to prepare for the caching of stats for directories (and negative stats too).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64477 91177308-0d34-0410-b5e6-96231b3b80d8
d7038e18ef540a78fe6ce2a43125ce9b08fdbbc5 13-Feb-2009 Chris Lattner <sabre@nondot.org> factor token concatenation avoidance logic out of
PrintPreprocessedOutput into its own file. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64418 91177308-0d34-0410-b5e6-96231b3b80d8
337edcdbec05316b407d0d64865c88ff8597d910 12-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Cache stat information for files in the PTH file. Hook up FileManager
to use this stat information in the PTH file using a 'StatSysCallCache' object.

Performance impact (Cocoa.h, PTH):
- number of stat calls reduces from 1230 to 425
- fsyntax-only: time improves by 4.2%

We can reduce the number of stat calls to almost zero by caching negative stat
calls and directory stat calls in the PTH file as well.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64353 91177308-0d34-0410-b5e6-96231b3b80d8
a4bd8eb4d6d4b625f6bbb62fc180b02eab6433ed 12-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Have meta data be at the beginning of the PTH file, not the end.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64338 91177308-0d34-0410-b5e6-96231b3b80d8
7e3a004c6ed1fe87912203b9c5a113f8da89d261 11-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Replace string identifier to persistent ID lookup with a hashtable. This is
actually *slightly* slower than the binary search. Since this is algorithmically
better, further performance tuning should be able to make this faster.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64326 91177308-0d34-0410-b5e6-96231b3b80d8
f1de4649e917f891d24e0718d02ee904b9edbe9d 11-Feb-2009 Ted Kremenek <kremenek@apple.com> PTH: Don't emit the PTH offset of the IdentifierInfo string data as that data is
referenced by other tables.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64304 91177308-0d34-0410-b5e6-96231b3b80d8
b99b0c0e292544eb1f4feeaf9c2664d12329d558 10-Feb-2009 Ted Kremenek <kremenek@apple.com> Bump PTH version.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64249 91177308-0d34-0410-b5e6-96231b3b80d8
f73903a1ded46748e1dfda151f5d037b7b3d31f9 06-Feb-2009 Chris Lattner <sabre@nondot.org> Add an implementation of -dM that follows GCC closely enough to permit
diffing the output of:
clang -dM -o - -E -x c foo.c | sort

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63926 91177308-0d34-0410-b5e6-96231b3b80d8
09b5c1d08a33ecf5d9c61b922fbe679867336684 06-Feb-2009 Chris Lattner <sabre@nondot.org> add interface for walking macro table.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63925 91177308-0d34-0410-b5e6-96231b3b80d8
19e8e2cffc19606d0f44e7c2897cd126ffd3f9b0 29-Jan-2009 Chris Lattner <sabre@nondot.org> next round of diagnostics cleanups, moving some
diags around, eliminating #defines, etc. Patch by
Anders Johnsen!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63318 91177308-0d34-0410-b5e6-96231b3b80d8
88eccaf06f9d88191723e71bdf5ca68409393be6 29-Jan-2009 Chris Lattner <sabre@nondot.org> Fix -Wimplicit-function-declaration, which required some refactoring and
changes in various diagnostics code.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63282 91177308-0d34-0410-b5e6-96231b3b80d8
500d3297d2a21edeac4d46cbcbe21bc2352c2a28 29-Jan-2009 Chris Lattner <sabre@nondot.org> move library-specific diagnostic headers into library private dirs. Reduce
redundant #includes. Patch by Anders Johnsen!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63271 91177308-0d34-0410-b5e6-96231b3b80d8
8a6aec620dbec1f292fe4116c0373ac81ab90234 28-Jan-2009 Ted Kremenek <kremenek@apple.com> Enhance PTHManager::Create() to take an optional Diagnostic* argument that can be used to report issues such as a missing PTH file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63231 91177308-0d34-0410-b5e6-96231b3b80d8
277faca30c9f8f72b79f55695cbe3395ec246e7c 27-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Use Token::setLiteralData() to directly store a pointer to cached spelling data in the PTH file. This removes a ton of code for looking up spellings using sourcelocations in the PTH file. This simplifies both PTH-generation and reading.

Performance impact for -fsyntax-only on Cocoa.h (with Cocoa.h in the PTH file):
- PTH generation time improves by 5%
- PTH reading improves by 0.3%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63072 91177308-0d34-0410-b5e6-96231b3b80d8
7216dc9cb49f47254595120cf15a737cee53f0bd 26-Jan-2009 Chris Lattner <sabre@nondot.org> rename getSpelledCharacterAt to getSpellingOfSingleCharacterNumericConstant,
optimize it to use the LiteralData when possible.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63060 91177308-0d34-0410-b5e6-96231b3b80d8
468edb509346e00a3228536b9f81064eb3baba19 26-Jan-2009 Ted Kremenek <kremenek@apple.com> Add version number to PTH files.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63052 91177308-0d34-0410-b5e6-96231b3b80d8
47246be8ac5b0ddde6c402b8fc6946b6135487b5 26-Jan-2009 Chris Lattner <sabre@nondot.org> This change refactors some of the low-level lexer interfaces a bit.
Token now has a class of kinds for "literals", which include
numeric constants, strings, etc. These tokens can optionally have
a pointer to the start of the token in the lexer buffer. This
makes it faster to get spelling and do other gymnastics, because we
don't have to go through source locations.

This change is performance neutral, but will make other changes
more feasible down the road.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63028 91177308-0d34-0410-b5e6-96231b3b80d8
478a18ec47fdb7e0e580a2635648456e9db9ad4f 26-Jan-2009 Chris Lattner <sabre@nondot.org> add parsing and constraint enforcement for GNU line marker directives.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63003 91177308-0d34-0410-b5e6-96231b3b80d8
359cc4475487ea0a660b3e8918334a2866e71c66 26-Jan-2009 Chris Lattner <sabre@nondot.org> parse and enforce required constraints on #line directives. Right now
we just discard them.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62999 91177308-0d34-0410-b5e6-96231b3b80d8
de7aeefc5573d669ed476d7bda7a8940d3bcadb7 26-Jan-2009 Chris Lattner <sabre@nondot.org> Check in the long promised SourceLocation rewrite. This lays the
ground work for implementing #line, and fixes the "out of macro ID's"

There is nothing particularly tricky about the code, other than the
very performance sensitive SourceManager::getFileID() method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62978 91177308-0d34-0410-b5e6-96231b3b80d8
17d527b051fbc3927b8a1b4ce4607a9b2ed445ee 23-Jan-2009 Chris Lattner <sabre@nondot.org> Preprocessor doesn't require and IdentifierInfoLookup object.
Patch by Axel Naumann!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62854 91177308-0d34-0410-b5e6-96231b3b80d8
bcc2a67e5180612417727cbdd8afd0f79fdf726d 19-Jan-2009 Chris Lattner <sabre@nondot.org> Make SourceLocation::getFileLoc private to reduce the API exposure of
SourceLocation. This requires making some cleanups to token pasting
and _Pragma expansion.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62490 91177308-0d34-0410-b5e6-96231b3b80d8
77ecb3a28f21496ecfdbb3d5f5b66b0d2abf48c9 18-Jan-2009 Chris Lattner <sabre@nondot.org> rearrange GetIdentifierInfo so that the fast path can be partially inlined into PTHLexer::Lex. This speeds up the user time of PTH -Eonly by another 2ms (4.4%)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62454 91177308-0d34-0410-b5e6-96231b3b80d8
3a874a405d1913b132e4ecfa52e22c4e2264b05d 18-Jan-2009 Chris Lattner <sabre@nondot.org> Avoid malloc thrashing on the std::vector for ConditionalStack. Because
there is one of these per header, this almost always gets alloc/free'd for
each #ifdef.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62451 91177308-0d34-0410-b5e6-96231b3b80d8
da9d61c96c412f6babc7f824152609562f302388 18-Jan-2009 Chris Lattner <sabre@nondot.org> switch PTH lexer from using "const char*"s to "const unsigned char*"s
internally. This is just a cleanup that reduces the need to cast to
unsigned char before assembling a larger integer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62442 91177308-0d34-0410-b5e6-96231b3b80d8
42e00d19b0dac64732eb5449d52a076282fcbf77 17-Jan-2009 Chris Lattner <sabre@nondot.org> Change the Lexer ctor used to lex _Pragma directives into a static factory
method. This lets us clean up the interface and make it more obvious that
this method is *really really* _Pragma specific.

Note that _Pragma handling uglifies the Lexer in the critical path. It would
be very interesting to consider making _Pragma remapping be a new special
lexer class of its own.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62425 91177308-0d34-0410-b5e6-96231b3b80d8
f056d92e182cbe4f62c8d14102544dc38066dabc 17-Jan-2009 Chris Lattner <sabre@nondot.org> simplify PTHManager::CreateLexer

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62424 91177308-0d34-0410-b5e6-96231b3b80d8
88d3ac1341aa016cabd966c5b113a95ac05ea43f 17-Jan-2009 Chris Lattner <sabre@nondot.org> Change the Lexer ctor used in the non _Pragma case to take a FileID instead
of a SourceLocation. This should speed it up and definitely simplifies it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62422 91177308-0d34-0410-b5e6-96231b3b80d8
0770dabb1ae81a2a9c2e7199262067103062a0b3 17-Jan-2009 Chris Lattner <sabre@nondot.org> More simplifications to the lexer ctors.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62419 91177308-0d34-0410-b5e6-96231b3b80d8
de96c0f29c4cacabe6ea577c61db87c2a85aea6c 17-Jan-2009 Chris Lattner <sabre@nondot.org> make the verbose raw-lexer ctor fully explicit instead of having
embedded magic.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62417 91177308-0d34-0410-b5e6-96231b3b80d8
025c3a66402fb713c2d9bf5dc174ff264765379a 17-Jan-2009 Chris Lattner <sabre@nondot.org> add a simplified lexer ctor that sets up the lexer to raw-lex an
entire file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62414 91177308-0d34-0410-b5e6-96231b3b80d8
22d91ca8d7c134eac5cc6a4869e6a84c461ad624 17-Jan-2009 Chris Lattner <sabre@nondot.org> refactor some common initialization code out of the two lexer ctors.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62411 91177308-0d34-0410-b5e6-96231b3b80d8
6b7b84036126a8f2a3e312cfec0a9a4f8ae9d8ed 17-Jan-2009 Chris Lattner <sabre@nondot.org> suck the call to "getSpellingLoc" that all clients do into
the implementation of PTHManager::getSpelling.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62408 91177308-0d34-0410-b5e6-96231b3b80d8
2b2453a7d8fe732561795431f39ceb2b2a832d84 17-Jan-2009 Chris Lattner <sabre@nondot.org> this massive patch introduces a simple new abstraction: it makes
"FileID" a concept that is now enforced by the compiler's type checker
instead of yet-another-random-unsigned floating around.

This is an important distinction from the "FileID" currently tracked by
SourceLocation. *That* FileID may refer to the start of a file or to a
chunk within it. The new FileID *only* refers to the file (and its
#include stack and eventually #line data), it cannot refer to a chunk.

FileID is a completely opaque datatype to all clients, only SourceManager
is allowed to poke and prod it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62407 91177308-0d34-0410-b5e6-96231b3b80d8
a9d9145741ef77db45890911674705b81605b10b 16-Jan-2009 Chris Lattner <sabre@nondot.org> Improve #pragma comment support by building the string argument and
notifying PPCallbacks about it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62333 91177308-0d34-0410-b5e6-96231b3b80d8
bbee00b6456e90a09f63c83c20233e6c5ad6000a 16-Jan-2009 Chris Lattner <sabre@nondot.org> minor cleanups to StringLiteralParser: no need to pass target info
into its ctor. Also, make it handle validity checking of pascal
strings instead of making clients do it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62332 91177308-0d34-0410-b5e6-96231b3b80d8
636c5ef6572e899d36cec1b0023fb28ba65189e1 16-Jan-2009 Chris Lattner <sabre@nondot.org> Implement basic support for parsing #pragma comment, a microsoft extension
documented here:

This is according to my understanding reading the docs, I don't know if it
really agrees fully with what VC++ allows.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62317 91177308-0d34-0410-b5e6-96231b3b80d8
0c21e84b82191ae1c4d04444a84008c464433868 16-Jan-2009 Chris Lattner <sabre@nondot.org> rename PP::getPhysicalCharacterAt -> PP::getSpelledCharacterAt.
Slightly speed up sema of numbers like '1' by going directly to
TargetInfo instead of through ASTContext.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62314 91177308-0d34-0410-b5e6-96231b3b80d8
df7c17a8d02fe09a3466786bae3e40fc3252687a 16-Jan-2009 Chris Lattner <sabre@nondot.org> Change some terminology in SourceLocation: instead of referring to
the "physical" location of tokens, refer to the "spelling" location.
This is more concrete and useful, tokens aren't really physical objects!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62309 91177308-0d34-0410-b5e6-96231b3b80d8
72b1b15ee88aac0a63e2c1dc53fe22f5ab297b20 15-Jan-2009 Ted Kremenek <kremenek@apple.com> IdentifierInfo:
- IdentifierInfo can now (optionally) have its string data not be
co-located with itself. This is for use with PTH. This aspect is a
little gross, as getName() and getLength() now make assumptions
about a possible alternate representation of IdentifierInfo.
Perhaps we should make IdentifierInfo have virtual methods?

- Added class "IdentifierInfoLookup" that can be used by
IdentifierTable to perform "string -> IdentifierInfo" lookups using
an auxilliary data structure. This is used by PTH.
- Perform tests show that IdentifierTable::get() does not slow down
because of the extra check for the IdentiferInfoLookup object (the
regular StringMap lookup does enough work to mitigate the impact of
an extra null pointer check).
- The upshot is that now that some IdentifierInfo objects might be
owned by the IdentiferInfoLookup object. This should be reviewed.

- Modified PTHManager::GetIdentifierInfo to *not* insert entries in
IdentifierTable's string map, and instead create IdentifierInfo
objects on the fly when mapping from persistent IDs to
IdentifierInfos. This saves a ton of work with string copies,
hashing, and StringMap lookup and resizing. This change was
motivated because when processing source files in the PTH cache we
don't need to do any string -> IdentifierInfo lookups.
- PTHManager now subclasses IdentifierInfoLookup, allowing clients of
IdentifierTable to transparently use IdentifierInfo objects managed
by the PTH file. PTHManager resolves "string -> IdentifierInfo"
queries by doing a binary search over a sorted table of identifier
strings in the PTH file (the exact algorithm we use can be changed
as needed).

These changes lead to the following performance changes when using PTH on Cocoa.h:
- fsyntax-only: 10% performance improvement
- Eonly: 30% performance improvement

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62273 91177308-0d34-0410-b5e6-96231b3b80d8
28396608ec20d44e9d1470e1ea51689bb504d0de 14-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH:
- Use canonical FileID when using getSpelling() caching. This
addresses some cache misses we were seeing with -fsyntax-only on
- Added Preprocessor::getPhysicalCharacterAt() utility method for
clients to grab the first character at a specified sourcelocation.
This uses the PTH spelling cache.
- Modified Sema::ActOnNumericConstant() to use
Preprocessor::getPhysicalCharacterAt() instead of
SourceManager::getCharacterData() (to get PTH hits).

These changes cause -fsyntax-only to not page in any sources from
Cocoa.h. We see a speedup of 27%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62193 91177308-0d34-0410-b5e6-96231b3b80d8
f02f6f0ee3e05b958bcf67f00f4503671d67eccd 13-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Fix remaining cases where the spelling cache in the PTH file was being missed when it shouldn't. This shaves another 7% off PTH time for -Eonly on Cocoa.h

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62186 91177308-0d34-0410-b5e6-96231b3b80d8
5f074266cc59563036c40516c814d63825723e20 09-Jan-2009 Ted Kremenek <kremenek@apple.com> Enhance PTH 'getSpelling' caching:
- Refactor caching logic into a helper class PTHSpellingSearch
- Allow "random accesses" in the spelling cache, thus catching the remaining
cases where 'getSpelling' wasn't hitting the PTH cache

For -Eonly, PTH, Cocoa.h:
- This reduces wall time by 3% (user time unchanged, sys time reduced)
- This reduces the amount of paged source by 1112K.
The remaining 1112K still being paged in is from somewhere else

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62009 91177308-0d34-0410-b5e6-96231b3b80d8
32a8ad526f9bc00539f000a2dd1ac3e167db61c1 08-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH: Hook up getSpelling() caching in PTHLexer. This results in a nice
performance gain. Here's what we see for -Eonly on Cocoa.h (using PTH):

- wall time decreases by 21% (26% speedup overall)
- system time decreases by 35%
- user time decreases by 6%

These reductions are due to not paging source files just to get spellings for
literals. The solution in place doesn't appear to be 100% yet, as we still see
some of the pages for source files getting mapped in. Using -print-stats, we see
that SourceManager maps in 7179K less bytes of source text (reduction of 75%).
Will investigate why the remaining 25% are getting paged in.

With these changes, here's how PTH compares to non-PTH on Cocoa.h:
-Eonly: PTH takes 64% of the time as non-PTH (54% speedup)
-fsyntax-only: PTH takes 89% of the time as non-PTH (11% speedup)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61913 91177308-0d34-0410-b5e6-96231b3b80d8
b70e3dafb9618f34017061400dc19ac5e3539a6d 08-Jan-2009 Ted Kremenek <kremenek@apple.com> PTH:
- Added stub PTHLexer::getSpelling() that will be used for fetching cached
spellings from the PTH file. This doesn't do anything yet.
- Added a hook in Preprocessor::getSpelling() to call PTHLexer::getSpelling()
when using a PTHLexer.
- Updated PTHLexer to read the offsets of spelling tables in the PTH file.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61911 91177308-0d34-0410-b5e6-96231b3b80d8
79ed16e2e605d67a12cccdcf9ad1b231175da1a6 06-Jan-2009 Chris Lattner <sabre@nondot.org> Make Token::setLength assert that the token is not an annotation token.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61791 91177308-0d34-0410-b5e6-96231b3b80d8
b31757b68afe06ba442a05775d08fe7aa0f6f889 06-Jan-2009 Chris Lattner <sabre@nondot.org> rename tok::annot_qualtypename -> tok::annot_typename, which is both
shorter and more accurate. The type name might not be qualified.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61788 91177308-0d34-0410-b5e6-96231b3b80d8
83cf05a3b0e655dc8ea1cb4c4e1eef541b770992 05-Jan-2009 Chris Lattner <sabre@nondot.org> Fix a bug where we'd try to look beyond the current cached tokens when
not in backtracking mode. This was just using the wrong predicate.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61666 91177308-0d34-0410-b5e6-96231b3b80d8
59d08cb672136322375e5400578ee1fbd0947de2 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH: Remove some methods and simplify some conditions in PTHLexer::Lex(). No big functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61381 91177308-0d34-0410-b5e6-96231b3b80d8
866bdf74547efe32c320554837ffce00fcc084fe 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
- In PTHLexer::Lex read all of the token data from PTH file before
constructing the token. The idea is to enhance locality.
- Do not use Read8/Read32 in PTHLexer::Lex. Inline these operations manually.
- Change PTHManager::ReadIdentifierInfo() to PTHManager::GetIdentifierInfo().
They are functionally the same except that PTHLexer::Lex() reads the
persistent id.

These changes result in a 3.3% speedup for PTH on Cocoa.h (-Eonly).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61363 91177308-0d34-0410-b5e6-96231b3b80d8
e5680f3cd678014cf0872d34726dc804b0cbbdd4 23-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
- Embed 'eom' tokens in PTH file.
- Use embedded 'eom' tokens to not lazily generate them in the PTHLexer.
This means that PTHLexer can always advance to the next token after
reading a token (instead of buffering tokens using a copy).
- Moved logic of 'ReadToken' into Lex. GetToken & ReadToken no longer exist.
- These changes result in a 3.3% speedup (-Eonly) on Cocoa.h.
- The code is a little gross. Many cleanups are possible and should be done.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61360 91177308-0d34-0410-b5e6-96231b3b80d8
e36dabb2d9567c98734a1063ac7748dc37a999ce 20-Dec-2008 Chris Lattner <sabre@nondot.org> avoid using a typedef that isn't always included from headers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61269 91177308-0d34-0410-b5e6-96231b3b80d8
d6fb7ef028d9aa0b3e8943b7bc049c524437b407 18-Dec-2008 Douglas Gregor <dgregor@apple.com> Ultrasimplistic sketch for the parsing of C++ template-ids. This won't
become useful or correct until we (1) parse template arguments
correctly, (2) have some way to turn template-ids into types,
declarators, etc., and (3) have a real representation of templates.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61208 91177308-0d34-0410-b5e6-96231b3b80d8
30a12ec2a7f331d9e08acabe7cda853aaa7ba54b 18-Dec-2008 Ted Kremenek <kremenek@apple.com> Change PTHLexer::getSourceLocation() to not call GetToken() and instead just read the file offset in the token data buffer directly.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61170 91177308-0d34-0410-b5e6-96231b3b80d8
daeee815169f37f2a49f80af7bd104722672ad97 18-Dec-2008 Ted Kremenek <kremenek@apple.com> PTHLexer::isNextPPTokenLParen() no longer calls GetToken() and just reads the token kind from the token data buffer. This results in a minor speedup and reduces the dependency on GetToken().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61168 91177308-0d34-0410-b5e6-96231b3b80d8
0ea76727ae91bca918a8414ed85b530eddcfedeb 15-Dec-2008 Ted Kremenek <kremenek@apple.com> Preprocessor: Allocate MacroInfo objects using a BumpPtrAllocator instead using new/delete. This speeds up -Eonly on Cocoa.h using the regular lexer by 1.8% and the PTHLexer by 3%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@61042 91177308-0d34-0410-b5e6-96231b3b80d8
268ee7016a2811803989487c0ad3799486092c63 12-Dec-2008 Ted Kremenek <kremenek@apple.com> Added PTH optimization to not process entire blocks of tokens that appear in skipped preprocessor blocks. This improves PTH speed by 6%. The code for this optimization itself is not very optimized, and will get cleaned up.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60956 91177308-0d34-0410-b5e6-96231b3b80d8
cd223444d1680290efe11da657faafc9a1ac14ba 11-Dec-2008 Ted Kremenek <kremenek@apple.com> PTHLexer: Keep track of the location of the last '#' token and provide the means to jump ahead in the token stream.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60905 91177308-0d34-0410-b5e6-96231b3b80d8
7415326dcbba6f333f5f7708ac1bf4982f3adb5a 11-Dec-2008 Ted Kremenek <kremenek@apple.com> Remove unused ivar CurTokenIdx.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60896 91177308-0d34-0410-b5e6-96231b3b80d8
bc0f6bc0391ecdff331885cdc769c20b2cb628a6 11-Dec-2008 Ted Kremenek <kremenek@apple.com> PreprocessorLexer (and subclasses):
- Added virtual method 'getSourceLocation()' (no arguments) that gets the location of the next "observable" location (e.g., next character, next token).
- Implemented FIXME by using PreprocessorLexer::getSourceLocation() to get the location in the file we are returning to after lexing a #included file. This appears to be slightly faster than having the branch (i.e., 'if(CurLexer)'). It's also not a really hot part of the Preprocessor.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60860 91177308-0d34-0410-b5e6-96231b3b80d8
cf58e6249c6b018508e34bcb76202caa42d2451a 10-Dec-2008 Ted Kremenek <kremenek@apple.com> Declare PerIDCache as IdentifierInfo** instead of void*. This is just cleaner. No performance change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60843 91177308-0d34-0410-b5e6-96231b3b80d8
6183e4815a4019e97ad01bd880f12355599b75fd 03-Dec-2008 Ted Kremenek <kremenek@apple.com> PTH:
Use an array instead of a DenseMap to cache persistent IDs -> IdentifierInfo*. This leads to a 4% speedup at -fsyntax-only using PTH.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60452 91177308-0d34-0410-b5e6-96231b3b80d8
0c6a77bc1f52f282a969538f139ebde429076ed3 03-Dec-2008 Ted Kremenek <kremenek@apple.com> - Remove PTHManager.cpp. Move all of its functions to PTHLexer.cpp since some of the internal methods are used by PTHLexer (their implementations are intertwined.) This enables some important inlining opportunities at -O3.

- Don't construct an std::vector<Token> prior to feeding PTH tokens to the Preprocessor. Stream them off the PTH file directly.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60447 91177308-0d34-0410-b5e6-96231b3b80d8
6137dc99ef0c2b14050631367057758b0d596cb3 02-Dec-2008 Ted Kremenek <kremenek@apple.com> Preprocessor:
- Added method "setPTHManager" that will be called by the driver to install
a PTHManager for the Preprocessor.
- Fixed some comments.
- Added EnterSourceFileWithPTH to mirror EnterSourceFileWithLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60437 91177308-0d34-0410-b5e6-96231b3b80d8
be1ee79d20db9cec7ef81d7a22ba2eaddc9c95b5 02-Dec-2008 Ted Kremenek <kremenek@apple.com> Added PTHManager, a utility class that will be used by Preprocessor to lazily create PTHLexer objects for pre-tokenized files.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60436 91177308-0d34-0410-b5e6-96231b3b80d8
82a500b141b9a8001dac69f047478a43e2aebdff 27-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer now owns the Token vector.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60136 91177308-0d34-0410-b5e6-96231b3b80d8
63db2a78010f4d0f7a6783ea1e2d1469c9b154ad 26-Nov-2008 Ted Kremenek <kremenek@apple.com> Add setter method PreprocessorLexer::setParsingPreprocessorDirective(). This will be used by the mechanism to generate cached tokens.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60070 91177308-0d34-0410-b5e6-96231b3b80d8
ac92d829111bc19d1cc97cd85c3c04bc39b969d1 22-Nov-2008 Chris Lattner <sabre@nondot.org> remove the NumericLiteralParser::Diag helper method, inlining it into
its call sites. This makes it more explicit when the hasError flag is
getting set and removes a confusing difference in behavior between
PP.Diag and Diag in this code.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59863 91177308-0d34-0410-b5e6-96231b3b80d8
adc4eeb08042a35ae914fc557ffec0cef3df2374 22-Nov-2008 Chris Lattner <sabre@nondot.org> Move the Preprocessor::Diag methods inline. This has the interesting
(and carefully calculated) effect of allowing the compiler to reason
about the aliasing properties of DiagnosticBuilder object better,
allowing the whole thing to be promoted to registers instead of
resulting in a ton of stack traffic.

While I'm not very concerned about the performance of the Diag() method
invocations, I *am* more concerned about their code size and impact on the
non-diagnostic code. This patch shrinks the clang executable (in
release-asserts mode with gcc-4.2) from 14523980 to 14519816 bytes. This
isn't much, but it shrinks the lexer from 38192 to 37776, PPDirectives.o
from 31116 to 28868 bytes, etc.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59862 91177308-0d34-0410-b5e6-96231b3b80d8
9efe9709287801562a3cbe1e5f3ab5dd8175c25b 22-Nov-2008 Chris Lattner <sabre@nondot.org> inline a method into its only two call sites.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59860 91177308-0d34-0410-b5e6-96231b3b80d8
30c6476e8ef00caeb3a10ea0228165013a4eb3f3 22-Nov-2008 Chris Lattner <sabre@nondot.org> actually, this version isn't really needed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59859 91177308-0d34-0410-b5e6-96231b3b80d8
306fda76b0818eb7c215094123f2e16729a44aed 22-Nov-2008 Chris Lattner <sabre@nondot.org> remove a sneaky version of Diag hiding in PreprocessorLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59858 91177308-0d34-0410-b5e6-96231b3b80d8
74d15dfd183b2082e8a5d4dfbf66bd861b220901 22-Nov-2008 Chris Lattner <sabre@nondot.org> Change the Lexer::Diag method to not magically silence warnings,
force the caller to check instead. This eliminates the need (and the
risk!) of weird null DiagnosticBuilder's floating around.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59856 91177308-0d34-0410-b5e6-96231b3b80d8
3cbfe2c4159e0a219ae660d50625c013aa4afbd0 22-Nov-2008 Chris Lattner <sabre@nondot.org> Split the DiagnosticInfo class into two disjoint classes:
one for building up the diagnostic that is in flight (DiagnosticBuilder)
and one for pulling structured information out of the diagnostic when
formatting and presenting it.

There is no functionality change with this patch.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59849 91177308-0d34-0410-b5e6-96231b3b80d8
3296ccea50e7b6649638bd666181e0825145e898 22-Nov-2008 Ted Kremenek <kremenek@apple.com> Fix predicate: we're not in caching mode if CurPPLexer == 0, not CurLexer == 0.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59848 91177308-0d34-0410-b5e6-96231b3b80d8
34eaa24e6c2dc72bbbd5ad09694368cfd09c5f14 21-Nov-2008 Ted Kremenek <kremenek@apple.com> Add comment to IsFileLexer, clean up indentation, and tighten how it's written.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59773 91177308-0d34-0410-b5e6-96231b3b80d8
cd4e2aecde5bb7810715d5d5a88ac63ce7946f34 21-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer:
- Move out logic for handling the end-of-file to LexEndOfFile (to match the Lexer) class. The logic now mirrors the Lexer class more, which allows us to pass most of the Preprocessor test cases.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59768 91177308-0d34-0410-b5e6-96231b3b80d8
89d7ee9619d2dbdfa8d956a695c612a104a92cad 20-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer:
- Move PTHLexer::GetToken() to be inside PTHLexer.cpp.
- When lexing in raw mode, null out identifiers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59744 91177308-0d34-0410-b5e6-96231b3b80d8
31aba425a01c8c957e662ccfaa71f923d0f0932a 20-Nov-2008 Ted Kremenek <kremenek@apple.com> PTHLexer:
- Rename 'CurToken' and 'LastToken' to 'CurTokenIdx' and 'LastTokenIdx'
- Add helper methods GetToken(), AdvanceToken(), AtLastToken() to abstract away
details of the token stream. This also allows us to easily replace their
implementation later.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59733 91177308-0d34-0410-b5e6-96231b3b80d8
81d24e135c60dcbe421864e6a4334ba99b8fe7ec 20-Nov-2008 Ted Kremenek <kremenek@apple.com> Rename IsNonPragmaNonMacroLexer to IsFileLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59731 91177308-0d34-0410-b5e6-96231b3b80d8
b28d6de75e0cb27ce3106ce6052f87ad0ab276d7 20-Nov-2008 Daniel Dunbar <daniel@zuster.org> De-unionize fields in Token class.
- This is fairly gross but although the code is conceptually the
same, introducting the union causes gcc 4.2 on x86 (darwin, if that
matters) to pessimize LexTokenInternal which is critical to our
preprocessor performance.

This speeds up -Eonly lexing of Cocoa.h by ~4.7% in my timings and
reduces the code size of LexTokenInternal by 8.6%.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59725 91177308-0d34-0410-b5e6-96231b3b80d8
ac2dda65d2d39c0a83d95bdf6237edbf4685bc58 20-Nov-2008 Ted Kremenek <kremenek@apple.com> Added virtual method 'IndirectLex' to PTHLexer. This will likely get removed in the future when we correctly handle #include processing.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59722 91177308-0d34-0410-b5e6-96231b3b80d8
7062d9e9fc3ed89c4e4ea7055efea6585b7bac8d 20-Nov-2008 Ted Kremenek <kremenek@apple.com> Preprocessor::isCurrentLexer() now takes a PreprocessorLexer* argument to match against CurPPLexer instead of CurLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59721 91177308-0d34-0410-b5e6-96231b3b80d8
68e48e4a81767997ef0231e47eca4f665102c95e 20-Nov-2008 Ted Kremenek <kremenek@apple.com> Preprocessor::getCurrentFileLexer() now returns a PreprocessorLexer* instead of
a Lexer*. This means it will either return the current (normal) file Lexer or a

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59694 91177308-0d34-0410-b5e6-96231b3b80d8
452e37837a48b2f0ced144784277fd4d28cbede9 20-Nov-2008 Ted Kremenek <kremenek@apple.com> - Default initialize ParsingPreprocessorDirective, ParsingFilename, and
LexingRawMode in the ctor of PreprocessorLexer.

- PTHLexer: Use "LastToken" instead of "NumToken"

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59690 91177308-0d34-0410-b5e6-96231b3b80d8
9f1384f7db76e24068f6c9d7d881714addb6c029 19-Nov-2008 Ted Kremenek <kremenek@apple.com> Fix comment.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59673 91177308-0d34-0410-b5e6-96231b3b80d8
2f1c0243f7fc5d12446efcb006c071854c345923 19-Nov-2008 Ted Kremenek <kremenek@apple.com> Add stub for PTHLexer::isNextPPTokenLParen().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59670 91177308-0d34-0410-b5e6-96231b3b80d8
17ff58a63197b398ae52697b088dc0fb8b255519 19-Nov-2008 Ted Kremenek <kremenek@apple.com> When using a PTHLexer, use DiscardToEndOfLine() instead of ReadToEndOfLine().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59668 91177308-0d34-0410-b5e6-96231b3b80d8
41938c8493b4380df738263166b746eacb33c309 19-Nov-2008 Ted Kremenek <kremenek@apple.com> - Move static function IsNonPragmaNonMacroLexer into Preprocessor.h.
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.
- Add 'FileID' to PreprocessorLexer, and have Preprocessor query this fileid
when looking up the FileEntry for a file

Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59666 91177308-0d34-0410-b5e6-96231b3b80d8
08b2c3743a29a2dddcf72e95f747760e213cdde7 19-Nov-2008 Argyrios Kyrtzidis <akyrtzi@gmail.com> Fix this:

With this snippet:
void f(a::b);

An assert is hit:
Assertion failed: CachedTokens[CachedLexPos-1].getLocation() == Tok.getAnnotationEndLoc() && "The annotation should be until the most recent cached token", file ..\..\lib\Lex\PPCaching.cpp, line 98

Introduce Preprocessor::RevertCachedTokens that reverts a specific number of tokens when backtracking is enabled.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59636 91177308-0d34-0410-b5e6-96231b3b80d8
fad03b2b38a3baea4b67e79e676fee15078e3258 19-Nov-2008 Argyrios Kyrtzidis <akyrtzi@gmail.com> Remove Preprocessor::CacheTokens boolean data member. The same functionality can be provided by using Preprocessor::isBacktrackEnabled().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59631 91177308-0d34-0410-b5e6-96231b3b80d8
2df37b8eaecc5382e0e511a738f88918f48d9454 19-Nov-2008 Ted Kremenek <kremenek@apple.com> Revert 59574 (caused tests to fail).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59579 91177308-0d34-0410-b5e6-96231b3b80d8
0a3626444039396802fa35db24733c1963aebfac 19-Nov-2008 Ted Kremenek <kremenek@apple.com> - Move static function IsNonPragmaNonMacroLexer into Preprocessor.h.
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.

Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59574 91177308-0d34-0410-b5e6-96231b3b80d8
204b2fed909b1eabea7aeb6caadd7cff718edee5 18-Nov-2008 Chris Lattner <sabre@nondot.org> Remove the last of the old-style Preprocessor::Diag methods.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59554 91177308-0d34-0410-b5e6-96231b3b80d8
56b05c8829efd13b7e5b333f8f587c71d025c67d 18-Nov-2008 Chris Lattner <sabre@nondot.org> remove one more Preprocessor::Diag method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59512 91177308-0d34-0410-b5e6-96231b3b80d8
3692b09faa9fe346f39bc922db6dce48cdcc3f63 18-Nov-2008 Chris Lattner <sabre@nondot.org> Convert the lexer and start converting the PP over to using canonical Diag methods.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59511 91177308-0d34-0410-b5e6-96231b3b80d8
f89963b1e819d5e73796a4f9d7cd70129846e2ec 18-Nov-2008 Ted Kremenek <kremenek@apple.com> Preprocessor::PushIncludeMacroStack() should always zero out CurPPLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59490 91177308-0d34-0410-b5e6-96231b3b80d8
4b391087f9c5582d448ab66ccbc7028f7673f380 18-Nov-2008 Ted Kremenek <kremenek@apple.com> - Add Lexer::isPragma() accessor for clients of Lexer that aren't friends.
- Add static method to test if the current lexer is a non-macro/non-pragma
- Refactor some code in PPLexerChange to use this static method.
- No performance change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59486 91177308-0d34-0410-b5e6-96231b3b80d8
f6452c5421c5db5a7ceff581525f286931d97f1a 18-Nov-2008 Ted Kremenek <kremenek@apple.com> Add hooks to use PTHLexer::Lex instead of Lexer::Lex when CurLexer is null.
Performance tests on Cocoa.h (using the regular Lexer) shows no performance

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59479 91177308-0d34-0410-b5e6-96231b3b80d8
9c1b750c59d510e6c9eccb1f37bccc46ccfe6844 18-Nov-2008 Ted Kremenek <kremenek@apple.com> - Add 'CurPPLexer' to Preprocessor to keep track of the current
PreprocessorLexer, which will either be a 'Lexer' or 'PTHLexer'.
- Added stub field 'CurPTHLexer' to keep track of the current PTHLexer.
- Modified IncludeStackInfo to track both the current PTHLexer and
current PreprocessorLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59472 91177308-0d34-0410-b5e6-96231b3b80d8
2e1cd4264d363ca869bf37ef160902f211d21b8c 17-Nov-2008 Douglas Gregor <dgregor@apple.com> Introduction the DeclarationName class, as a single, general method of
representing the names of declarations in the C family of
languages. DeclarationName is used in NamedDecl to store the name of
the declaration (naturally), and ObjCMethodDecl is now a NamedDecl.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59441 91177308-0d34-0410-b5e6-96231b3b80d8
caaa7df2c78bbd40197823034c0275f3dcbd63e7 13-Nov-2008 Ted Kremenek <kremenek@apple.com> Using llvm::OwningPtr<> for CurLexer and CurTokenLexer. This makes both the ownership semantics of these objects explicit within the Preprocessor and also tightens up the code (explicit deletes not needed).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59249 91177308-0d34-0410-b5e6-96231b3b80d8
d6a2e7d71e9a910c45a3eb0268a5d94635003d58 13-Nov-2008 Ted Kremenek <kremenek@apple.com> Move some diagnostic handling to PreprocessorLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59191 91177308-0d34-0410-b5e6-96231b3b80d8
2c71d0f00e40b7b0a85561d430f52247b9c6e7f2 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Add virtual dtor to PreprocessorLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59188 91177308-0d34-0410-b5e6-96231b3b80d8
eb41c28b78634ba2165b4aa04e33f3a12e590b0e 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Add LexIncludeFilename.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59187 91177308-0d34-0410-b5e6-96231b3b80d8
ab91b7086cceadce2cfd6a69cc90cebcdfcea07f 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Move LexIncludeFilename from Lexer to PreprocessorLexer.
PreprocessorLexer now has a virtual method "IndirectLex" which allows it to call the lex method of its subclasses. This is not for performance intensive operations.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59185 91177308-0d34-0410-b5e6-96231b3b80d8
0deae4b0eb085d1a34e0cb983750b5ed835c16a4 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Unbreak last commit.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59180 91177308-0d34-0410-b5e6-96231b3b80d8
d038def6e3f33bebf8c12bb3a95b2492c154a5dd 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Add Preprocessor::PushIncludeMacroStack() and Preprocessor::PopIncludeMacroStack(), two utility methods for manipulating the Preprocessor stack. These will be used to remove manually manipulation of IncludeMacroStack from the rest of the Preprocessor implementation.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59179 91177308-0d34-0410-b5e6-96231b3b80d8
274b20863a728cc6a31ee75c670e3733600c1531 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Add skeleton for PTH lexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59169 91177308-0d34-0410-b5e6-96231b3b80d8
0e977de1eacfbc143bdedad87c14b53814159023 12-Nov-2008 Ted Kremenek <kremenek@apple.com> Move pieces of Lexer that the Preprocessor mutates to a new base class 'PreprocessorLexer'. This will also be the base class of the new Preprocessed-Token-Header (PTH) lexer. No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59168 91177308-0d34-0410-b5e6-96231b3b80d8
3604e3895ecd850291b518e5a82246c888ce9d0f 08-Nov-2008 Argyrios Kyrtzidis <akyrtzi@gmail.com> Introduce annotation tokens, a special kind of token, created and used only by the parser to replace a group of tokens with a single token encoding semantic information.
Will be fully utilized later for C++ nested-name-specifiers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@58911 91177308-0d34-0410-b5e6-96231b3b80d8
21ec0e4c5db85e45b59cafabca66d640e0ee2dcc 28-Oct-2008 Chris Lattner <sabre@nondot.org> clarify comment, rename argument to avoid a subtle conflict
with an ivar that wasn't a bug but was confusing.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@58311 91177308-0d34-0410-b5e6-96231b3b80d8
9d72851fec9e9c62570a027d42701562bbf29751 27-Oct-2008 Chris Lattner <sabre@nondot.org> Rename Characteristic_t to CharacteristicKind

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@58224 91177308-0d34-0410-b5e6-96231b3b80d8
ef3b215c946d4813408b5fe872fe2baa3b246b00 21-Oct-2008 Ted Kremenek <kremenek@apple.com> Added method to access the raw flags of Token.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57877 91177308-0d34-0410-b5e6-96231b3b80d8
9e6293d4dfd688429f77ee3b6edba9dfd7ada3a2 12-Oct-2008 Chris Lattner <sabre@nondot.org> Change FormTokenWithChars to take the token kind to form, since all clients
were setting a kind and then forming it. This is just a minor API cleanup,
no functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57404 91177308-0d34-0410-b5e6-96231b3b80d8
d88dc48e33d71732708960170e57a3d1bdc8f847 12-Oct-2008 Chris Lattner <sabre@nondot.org> Add a new mode to the lexer which enables it to return all characters,
even whitespace, as tokens from the file. This is enabled with
L->SetKeepWhitespaceMode(true) on a raw lexer. In this mode, you too
can use clang as a really complex version of 'cat' with code like this:

Lexer RawLex(SourceLocation::getFileLoc(SM.getMainFileID(), 0),
PP.getLangOptions(), File.first, File.second);


Token RawTok;
while (RawTok.isNot(tok::eof)) {
std::cout << PP.getSpelling(RawTok);

This will emit exactly the input file, with no canonicalization or other
translation. Realistic clients actually do something with the tokens of
course :)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57401 91177308-0d34-0410-b5e6-96231b3b80d8
fa95a019da00b926d64ff83358ba73bbc6ae1e37 12-Oct-2008 Chris Lattner <sabre@nondot.org> add a new inKeepCommentMode() accessor to abstract the KeepCommentMode

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57397 91177308-0d34-0410-b5e6-96231b3b80d8
0af574270d3be2b0e73a3379dfaa633746f8fc6f 12-Oct-2008 Chris Lattner <sabre@nondot.org> Simplify raw mode lexing by treating an unterminate /**/ comment the
same we we do an unterminated string or character literal. This makes
it so we can guarantee that the lexer never calls into the
preprocessor (which would be suicide for a raw lexer).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57395 91177308-0d34-0410-b5e6-96231b3b80d8
590f0cc643274267d4d41125b62557e1d87886c3 12-Oct-2008 Chris Lattner <sabre@nondot.org> Change how raw lexers are handled: instead of creating them and then
using LexRawToken, create one and use LexFromRawLexer. This avoids
twiddling the RawLexer flag around and simplifies some code (even
speeding raw lexing up a tiny bit).

This change also improves the token paster to use a Lexer on the stack
instead of new/deleting it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57393 91177308-0d34-0410-b5e6-96231b3b80d8
4095080aff204008eefb26b100906c6ca2bc4bb6 04-Oct-2008 Daniel Dunbar <daniel@zuster.org> Add Preprocessor::RemovePragmaHandler.
- No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57065 91177308-0d34-0410-b5e6-96231b3b80d8
0b9e736308af5397f558ffc8e780c438c2fdb563 26-Sep-2008 Chris Lattner <sabre@nondot.org> clean up a bunch of fixme's I added, by moving
DirectoryLookup::DirType into SourceManager.h

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@56692 91177308-0d34-0410-b5e6-96231b3b80d8
721818304ac462d8c6ce05eecd02884033db78f1 26-Sep-2008 Chris Lattner <sabre@nondot.org> Fix the rest of rdar://6243860 hopefully. This requires changing FileIDInfo
to whether the fileid is a 'extern c system header' in addition to whether it
is a system header, most of this is spreading plumbing around. Once we have that,
PPLexerChange bases its "file enter/exit" notifications to PPCallbacks to
base the system header state on FileIDInfo instead of HeaderSearch. Finally,
in Preprocessor::HandleIncludeDirective, mirror logic in GCC: the system headerness
of a file being entered can be set due to the #includer or the #includee.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@56688 91177308-0d34-0410-b5e6-96231b3b80d8
e9eb3f88b82176d8a43f7856250efded80cb2504 26-Sep-2008 Daniel Dunbar <daniel@zuster.org> Comment fix.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@56634 91177308-0d34-0410-b5e6-96231b3b80d8
ed5c38682c056c147c8a4abb748b4f285de206dd 24-Aug-2008 Argyrios Kyrtzidis <akyrtzi@gmail.com> Rename Preprocessor::DisableBacktrack -> Preprocessor::CommitBacktrackedTokens.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@55281 91177308-0d34-0410-b5e6-96231b3b80d8
e671e1bc73615eda155059a772266ed2882d758c 24-Aug-2008 Chris Lattner <sabre@nondot.org> Comment tweak.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@55272 91177308-0d34-0410-b5e6-96231b3b80d8
a9e274c01ebae45629d93aaa07be450fb77dd3cb 22-Aug-2008 Argyrios Kyrtzidis <akyrtzi@gmail.com> Allow nested backtracks.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@55204 91177308-0d34-0410-b5e6-96231b3b80d8
662e8b5647adbb1bc9eeceece7b64600cfa87471 15-Aug-2008 Daniel Dunbar <daniel@zuster.org> Change Parser & Sema to use interned "super" for comparions.
- Added as private members for each because it is not clear where to
put the common definition. Perhaps the IdentifierInfos all of these
"pseudo-keywords" should be collected into one place (this would
KnownFunctionIDs and Objective-C property IDs, for example).

Remove Token::isNamedIdentifier.
- There isn't a good reason to use strcmp when we have interned
strings, and there isn't a good reason to encourage clients to do

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@54794 91177308-0d34-0410-b5e6-96231b3b80d8
7bfaaaecb3113f955db31e8d8a51acffd1bc0c27 10-Aug-2008 Nico Weber <nicolasweber@gmx.de> * Remove isInSystemHeader() from DiagClient, move it to SourceManager
* Move FormatError() from TextDiagnostic up to DiagClient, remove now
empty class TextDiagnostic
* Make DiagClient optional for Diagnostic

This fixes the following problems:

* -html-diags (and probably others) does now output the same set of
warnings as console clang does
* nothing crashes if one forgets to call setHeaderSearch() on
* some code duplication is removed

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@54620 91177308-0d34-0410-b5e6-96231b3b80d8
03db1b31dd926409b7defc1c90b66549464652c0 10-Aug-2008 Argyrios Kyrtzidis <akyrtzi@gmail.com> Allow the preprocessor to cache the lexed tokens, so that we can do efficient lookahead and backtracking.

1) New public methods added:

2) LookAhead() implementation is replaced with a more efficient one.
3) LookNext() is removed.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@54611 91177308-0d34-0410-b5e6-96231b3b80d8
e3d5e3ae5bd8028774f07d7c3751d4db82118942 10-Jul-2008 Chris Lattner <sabre@nondot.org> Add an accessor, patch by Csaba Hruska.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@53391 91177308-0d34-0410-b5e6-96231b3b80d8
9e0ed0bd5a3a7bac73973980ff32132a7724e674 10-Jul-2008 Argyrios Kyrtzidis <akyrtzi@gmail.com> Add Preprocessor::LookNext method, which implements an efficient way to 'take a peek' at the next token without consuming it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@53375 91177308-0d34-0410-b5e6-96231b3b80d8
368328c88bd46f471bbf85f05438b4f2eb95df5b 30-Jun-2008 Chris Lattner <sabre@nondot.org> refactor some code out into a new method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@52889 91177308-0d34-0410-b5e6-96231b3b80d8
8ed3044a33679cbfa0617d465a50ec557d671ed7 05-May-2008 Chris Lattner <sabre@nondot.org> Neil pointed out that clang doesn't generate ranges from diagnostics
related to pp-expressions. Doing so is pretty simple and this
patch implements it, yielding nice diagnostics like:

t.c:2:7: error: division by zero in preprocessor expression
#if 1 / (0 + 0)
~ ^ ~~~~~~~
t.c:5:14: error: expected ')' in preprocessor expression
#if (412 + 42
t.c:5:5: error: to match this '('
#if (412 + 42
t.c:10:10: warning: left side of operator converted from negative value to unsigned: -42 to 18446744073709551574
#if (-42 + 0U) / -2
~~~ ^ ~~
t.c:10:16: warning: right side of operator converted from negative value to unsigned: -2 to 18446744073709551614
#if (-42 + 0U) / -2
~~~~~~~~~~ ^ ~~
5 diagnostics generated.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@50638 91177308-0d34-0410-b5e6-96231b3b80d8
aa39197431a0a0b1326ecf6b3be6a11f6e2f8503 20-Apr-2008 Chris Lattner <sabre@nondot.org> simplify ownership of the predefines buffer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49973 91177308-0d34-0410-b5e6-96231b3b80d8
339b9c27759d7b6a53e2370f83f66e78b3254595 18-Apr-2008 Ted Kremenek <kremenek@apple.com> class Preprocessor: Now owns the "predefines" char*; it deletes [] it in its dstor.

clang.cpp: InitializePreprocessor now makes a copy of the contents of PredefinesBuffer and
passes it to the preprocessor object.

clang.cpp: DriverPreprocessorFactory now calls "InitializePreprocessor" instead of this being done in main().

html::HighlightMacros() now takes a PreprocessorFactory, allowing it to conjure up a new
Preprocessor to highlight macros.

class HTMLDiagnostics now takes a PreprocessorFactory* that it can use for html::HighlightMacros().
Updated clients of HTMLDiagnostics to use this new interface.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49875 91177308-0d34-0410-b5e6-96231b3b80d8
ec6c574478a22008847d7ebc2498ef3336752096 17-Apr-2008 Ted Kremenek <kremenek@apple.com> Added "PreprocessorFactory", an interface for lazily creating Preprocessor objects on-demand.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49868 91177308-0d34-0410-b5e6-96231b3b80d8
678c6358c8d4e368c78629099142397c63c1ee35 16-Apr-2008 Chris Lattner <sabre@nondot.org> reenable highlighting of (the first line of) comments

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49816 91177308-0d34-0410-b5e6-96231b3b80d8
eb50ed88c2aa040fac08bf2a50bde4dd3da6eb19 14-Mar-2008 Chris Lattner <sabre@nondot.org> Make the preprocessor own its PPCallbacks, fixing a memory leak.
Patch by Sam Bishop!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48357 91177308-0d34-0410-b5e6-96231b3b80d8
6b884508c3bc97cc9df9516adb92fbf88dd0a2e4 10-Mar-2008 Chris Lattner <sabre@nondot.org> implement simple support for arbitrary token lookahead. Change the
objc @try parser to use it, fixing a FIXME. Update the
objc-try-catch-1.m file to pass now that we get more reasonable

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48129 91177308-0d34-0410-b5e6-96231b3b80d8
fde2bf9befede63e3f01f84519784c17b4c81480 09-Mar-2008 Chris Lattner <sabre@nondot.org> rename HandleEndOfMacro -> HandleEndOfTokenLexer

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48076 91177308-0d34-0410-b5e6-96231b3b80d8
e5c8ffe09f8fec836b90f23c2a3b996bfc5b09c4 09-Mar-2008 Chris Lattner <sabre@nondot.org> split the MacroArgs class out of TokenLexer.cpp/h into

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48075 91177308-0d34-0410-b5e6-96231b3b80d8
6cfe7594a46b5d270142cfcb688a9c1a3a487a48 09-Mar-2008 Chris Lattner <sabre@nondot.org> rename some MacroExpander-related ivars to TokenLexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48073 91177308-0d34-0410-b5e6-96231b3b80d8
5d75de0f821023f4ed4815825bf3aea8a0b5e40d 09-Mar-2008 Chris Lattner <sabre@nondot.org> Rename MacroExpander.cpp/h -> TokenLexer.cpp/h

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48072 91177308-0d34-0410-b5e6-96231b3b80d8
1543e9c69202b8e128c7fe34784ae7aa90964889 09-Mar-2008 Chris Lattner <sabre@nondot.org> rename the MacroExpander class to TokenLexer. It handles both
token streams and macro lexing, so a more generic name is useful.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48071 91177308-0d34-0410-b5e6-96231b3b80d8
8d896438031f5990cb42eb95a7bae233db2eb333 09-Mar-2008 Chris Lattner <sabre@nondot.org> rename MacroTokens -> Tokens. When this is a token stream, there is no macro

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48070 91177308-0d34-0410-b5e6-96231b3b80d8
42e6737f2efb113563140ad794c21c7709250402 05-Mar-2008 Chris Lattner <sabre@nondot.org> Remove the first layer of support for "portability" warnings. This is
theoretically useful, but not useful in practice. It adds a bunch of
complexity, and not much value. It's best to nuke it. One big advantage
is that it means the target interfaces will soon lose their SLoc arguments
and target queries can never emit diagnostics anymore (yay). Removing this
also simplifies some of the core preprocessor which should make it slightly

Ted, I didn't simplify TripleProcessor, which can now have at most one
triple, and can probably just be removed. Please poke at it when you have

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@47930 91177308-0d34-0410-b5e6-96231b3b80d8
0184cc719f06d2c5da6ae0725847ef875c6128cd 25-Feb-2008 Lauro Ramos Venancio <lauro.venancio@gmail.com> Fix PR2086.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@47551 91177308-0d34-0410-b5e6-96231b3b80d8
ca63fa00786e51c207c829f4182f11a6c6b552be 24-Feb-2008 Ted Kremenek <kremenek@apple.com> Two more Windows-related fixes:

- More enum signeness bitfield fixes (MSVC treats enums as signed).

- Fixed in Lex/HeaderSearch.cpp an unsafe copy between two
HeaderSearch::PerFileInfo entries in a common vector. The copy involved two
calls to getFileInfo() within the assignment; these calls could have
side-effects that enlarged the internal vector, and with MSVC this would
invalidate one of the values in the assignment.

Patch by Argiris Kirtzidis!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@47536 91177308-0d34-0410-b5e6-96231b3b80d8
8e748ab52395328f2905855b295a22e33dc800b2 23-Feb-2008 Ted Kremenek <kremenek@apple.com> Change encoding of TokenKind in IdentifierTable to be of type "unsigned"
instead of TokenKind because of signedness issues with MSVC and enums.

Patch from Argiris Kirtzidis.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@47515 91177308-0d34-0410-b5e6-96231b3b80d8
3f1cc838f9caf469990f23fccd0940263c0c61ff 07-Feb-2008 Chris Lattner <sabre@nondot.org> Implement support for the extremely atrocious MS /##/ extension,
which pastes together a comment. This is only enabled with
-fms-extensions of course.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@46845 91177308-0d34-0410-b5e6-96231b3b80d8
49c1f4aa2a6c360d25d605004ec3c4affd62db77 02-Feb-2008 Steve Naroff <snaroff@apple.com> Cleanup previous patch (based on feedback from Ted).

Since this behavior is useful for most classes, we might consider adding a simple 3 method class that implements the behavior. Ted said that Boost has such a class.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@46654 91177308-0d34-0410-b5e6-96231b3b80d8
27e2eb40b9adc876c05e0260d1100385c9f0e5f5 02-Feb-2008 Steve Naroff <snaroff@apple.com> Make sure SourceManager/HeaderSearch don't support default copy constructors (since they result in bad runtime behavior).

I'm sure there are other classes that might need this "guard", however I was bitten by these 2 recently (so I thought I'd fix them).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@46653 91177308-0d34-0410-b5e6-96231b3b80d8
12fe558612303cb2188c321a77a21f7df23b6ae8 29-Jan-2008 Chris Lattner <sabre@nondot.org> fix comment typo

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@46505 91177308-0d34-0410-b5e6-96231b3b80d8
94b3cdb57fd5d245963e597626e0dfd88d479795 08-Jan-2008 Chris Lattner <sabre@nondot.org> readability improvement suggested by Sam Bishop, thanks!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45735 91177308-0d34-0410-b5e6-96231b3b80d8
4d730461735cc5cb8ecbd2f6106da9ec234064ad 07-Jan-2008 Chris Lattner <sabre@nondot.org> Fix a nasty corner case that Neil noticed in PR1900, where we would
incorrectly apply the multiple include optimization to files with
guards like:

#if !defined(x) MACRO

where MACRO could expand to different things in different contexts.
Thanks Neil!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45716 91177308-0d34-0410-b5e6-96231b3b80d8
0bc735ffcfb223c0186419547abaa5c84482663e 29-Dec-2007 Chris Lattner <sabre@nondot.org> Don't attribute in file headers anymore. See llvmdev for the
discussion of this change.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45410 91177308-0d34-0410-b5e6-96231b3b80d8
7360f9f2a5445064ae9728d19245104600886235 20-Dec-2007 Seo Sanghyeon <sanxiyn@gmail.com> Add newline

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45257 91177308-0d34-0410-b5e6-96231b3b80d8
95041a2029a069386ee67439f6d0fb524a9d184f 19-Dec-2007 Ted Kremenek <kremenek@apple.com> Interned MainFileID within SourceManager. Since SourceManager is referenced by
both Preprocessor and ASTContext, we no longer need to explicitly pass
MainFileID around in function calls that also pass either Preprocessor or
ASTContext. This resulted in some nice cleanups in the ASTConsumers and the

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45228 91177308-0d34-0410-b5e6-96231b3b80d8
7dcc968f17a6ff9088c9651dddccc8d4025a1271 19-Dec-2007 Ted Kremenek <kremenek@apple.com> Typo fix.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45227 91177308-0d34-0410-b5e6-96231b3b80d8
2243449253475574fc6f14986ff8f7fce5d46799 19-Dec-2007 Chris Lattner <sabre@nondot.org> Add support for #pragma mark, which shouldn't warn about bogus tokens.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45212 91177308-0d34-0410-b5e6-96231b3b80d8
1adbf6349d4701771a814542008386ad39e3d614 17-Dec-2007 Chris Lattner <sabre@nondot.org> implement enough helper functions to successfully dump out the
contents of the header map. Look ma, no assumptions about input data
here (aka, corrupt header maps can't crash the compiler - crazy thought).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45122 91177308-0d34-0410-b5e6-96231b3b80d8
98751314c4ba596860b574c3d3551030f01ff7d8 17-Dec-2007 Chris Lattner <sabre@nondot.org> Finish hooking up the scaffolding for headermaps. They can now do everything
except resolve lookups.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45111 91177308-0d34-0410-b5e6-96231b3b80d8
1bfd4a6313ea8ebf710c46c10111732cc65d51f6 17-Dec-2007 Chris Lattner <sabre@nondot.org> Start reading the headermap header, drop the 'errorstr' argument to
the create method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45109 91177308-0d34-0410-b5e6-96231b3b80d8
3af66a9335966e9114e660cf7aafbb9272190ec2 17-Dec-2007 Chris Lattner <sabre@nondot.org> Sink getName into DirectoryLookup to simplify the client in clang.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45106 91177308-0d34-0410-b5e6-96231b3b80d8
0f441ab20ccd8d21fd3d034de866bfcaf6cb72b2 17-Dec-2007 Chris Lattner <sabre@nondot.org> add headermap.cpp

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45095 91177308-0d34-0410-b5e6-96231b3b80d8
b09e71fd52d0e7fdf3e88b1df72ea0cee5d9b37b 17-Dec-2007 Chris Lattner <sabre@nondot.org> finish stubbing out support for HeaderMap. Now we just need an implementation!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45094 91177308-0d34-0410-b5e6-96231b3b80d8
afded5bbb85607023c710c3d6a96c372feb84d7f 17-Dec-2007 Chris Lattner <sabre@nondot.org> refactor an better comment framework lookup code. This moves it from
HeaderSearch into DirectoryLookup, as a particular framework lookup is
specific to the directory we are currently querying.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45093 91177308-0d34-0410-b5e6-96231b3b80d8
df772336655fd84ee2c0ce514c93cef0b29d60d4 17-Dec-2007 Chris Lattner <sabre@nondot.org> as it turns out, frameworks and headermaps are orthogonal. Make this so in
the internal representation. This also fixes a bug where -I foo -F foo would
not search foo as both a normal and framework include dir.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45092 91177308-0d34-0410-b5e6-96231b3b80d8
b94c707350ee2099996a80c8d97f28b61ff98c7b 17-Dec-2007 Chris Lattner <sabre@nondot.org> teach RemoveDuplicates about header maps.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45090 91177308-0d34-0410-b5e6-96231b3b80d8
822da61b74ce14e89b3fa8774db18c833aa5748b 17-Dec-2007 Chris Lattner <sabre@nondot.org> Step #1 in adding headermap support to clang.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45089 91177308-0d34-0410-b5e6-96231b3b80d8
53702cd401b8fdca985aede7732c2f6a82ad9b1c 13-Dec-2007 Chris Lattner <sabre@nondot.org> add a helper method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@44976 91177308-0d34-0410-b5e6-96231b3b80d8
e0579f08d4a4ae57971b74ca1863106df4c6399f 12-Dec-2007 Ted Kremenek <kremenek@apple.com> constified getFullLoc().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@44951 91177308-0d34-0410-b5e6-96231b3b80d8
2a2da1d0522b59c8c56f97a8607106be977a14af 12-Dec-2007 Ted Kremenek <kremenek@apple.com> Renamed getFullSourceLoc() -> getFullLoc().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@44949 91177308-0d34-0410-b5e6-96231b3b80d8
ea8646993741739d8a04d67396fe466dcc3a104f 12-Dec-2007 Ted Kremenek <kremenek@apple.com> Added method: Preprocessor::getFullSourceLoc. Used by clients of Preprocessor
to get a FullSourceLoc from a SourceLocation.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@44948 91177308-0d34-0410-b5e6-96231b3b80d8
c3d8d57b010e2ed15a2a7685d5761db14f5d2252 09-Dec-2007 Chris Lattner <sabre@nondot.org> Add dumping support for locations, make -dumptokens print out the location
info of each token.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@44741 91177308-0d34-0410-b5e6-96231b3b80d8
427d5af5b601985093b6b4b33ba1e30fc24d86dc 27-Nov-2007 Ted Kremenek <kremenek@apple.com> Added optional pass-by-reference argument "isExact" to
NumericLiteralParser::GetFloatValue(). Upon method return, this flag has the value
true if the returned APFloat can exactly represent the number in the parsed text,
and false otherwise.

Modified the implementation of GetFloatValue() to parse literals using APFloat's
convertFromString method (which allows us to set the value of isExact).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@44339 91177308-0d34-0410-b5e6-96231b3b80d8
25bfcb927d9169ea675ce6e98d8992efceeb0e42 15-Nov-2007 Chris Lattner <sabre@nondot.org> add header file I forgot to check in

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@44179 91177308-0d34-0410-b5e6-96231b3b80d8
fd1f344f874799ce8a320d81043a431b65fd1fe2 17-Oct-2007 Chris Lattner <sabre@nondot.org> Fix assertion for raw lexer.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43091 91177308-0d34-0410-b5e6-96231b3b80d8
9a6119437672f42be5f50c3fe89fe843b1bfa5b5 17-Oct-2007 Chris Lattner <sabre@nondot.org> Move token length calculation out of the diagnostics machinery into
the lexer, where it can be shared.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43090 91177308-0d34-0410-b5e6-96231b3b80d8
168ae2d44a443da75ea85db5f3b5081eb0bce113 17-Oct-2007 Chris Lattner <sabre@nondot.org> Add a new type of lexer: a raw lexer, which does not require a preprocessor
object in order to do its thing.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43084 91177308-0d34-0410-b5e6-96231b3b80d8
e5956bd2730c051835f9acd9e957c5d79f99e7c3 17-Oct-2007 Chris Lattner <sabre@nondot.org> add some comments.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43079 91177308-0d34-0410-b5e6-96231b3b80d8
ee98ac5a1330db432b188dd2d38b6631aac47bf1 15-Oct-2007 Anders Carlsson <andersca@mac.com> Add support for Pascal strings.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42974 91177308-0d34-0410-b5e6-96231b3b80d8
53b0dabbe52219a8057659b90539837394ef0fa1 10-Oct-2007 Chris Lattner <sabre@nondot.org> Make a significant change to invert the control flow handling
predefined macros. Previously, these were handled by the driver,
now they are handled by the preprocessor.

Some fallout of this:

1. Instead of preprocessing two buffers (the predefines, then the
main source file) we now start preprocessing the main source
file and inject the predefines as a "psuedo #include" from the
main source file.
2. #1 allows us to nuke the Lexer::IsMainFile flag and simplify
3. The driver doesn't have to know about standard #defines, the
preprocessor knows, which is nice for people wanting to define
their own drivers.
4. This allows us to put normal tokens in the predefine buffer,
for example a definition for __builtin_va_list that is
target-specific, and a typedef for id in objc.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42818 91177308-0d34-0410-b5e6-96231b3b80d8
000732226610650837478cba97843d19b75f648e 09-Oct-2007 Chris Lattner <sabre@nondot.org> Add two new Token helper functions, "is" and "isNot". This allows us to write
stuff like this:

// If we don't have a comma, it is either the end of the list (a ';') or
// an error, bail out.
if (Tok.isNot(tok::comma))
instead of:
// If we don't have a comma, it is either the end of the list (a ';') or
// an error, bail out.
if (Tok.getKind() != tok::comma)

There is obviously no functionality change, but the code reads a bit better and is
more terse.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42795 91177308-0d34-0410-b5e6-96231b3b80d8
c7229c338c21ef26b01ef3ecf9eec4fd373fa9ec 07-Oct-2007 Chris Lattner <sabre@nondot.org> move IdentifierTable.h from liblex to libbasic.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42730 91177308-0d34-0410-b5e6-96231b3b80d8
cc1a875f94630e58d24a55577ffbf0e89b7da8c7 07-Oct-2007 Chris Lattner <sabre@nondot.org> improve layering:
Now instead of IdentifierInfo knowing anything about MacroInfo,
only the preprocessor knows. This makes MacroInfo truly private
to the Lex library (and its direct clients) instead of being
accessed in the Basic library.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42727 91177308-0d34-0410-b5e6-96231b3b80d8
9c46de446d18f4a28446cb798d4131bd05515699 07-Oct-2007 Chris Lattner <sabre@nondot.org> add a hasMacroDefinition() method to IdentifierInfo, strength reduce a
call to getMacroInfo to call it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42725 91177308-0d34-0410-b5e6-96231b3b80d8
6d9a3e648d6bf6b347174152f191bd1377528f8c 07-Oct-2007 Chris Lattner <sabre@nondot.org> update comment.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42724 91177308-0d34-0410-b5e6-96231b3b80d8
387b98d37e08f7aa9ddf23e067e1d27e39ce16f3 07-Oct-2007 Chris Lattner <sabre@nondot.org> Remove the PPID bitfield from IdentifierInfo, shrinking it by a word
(because all bitfields now fit in 32 bits). This shrinks the identifier
table for carbon.h from 1634428 to 1451424 bytes (12%) and has no impact
on compile time.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42723 91177308-0d34-0410-b5e6-96231b3b80d8
4365a7e46822700357a272d839ee2656d9603d5a 07-Oct-2007 Chris Lattner <sabre@nondot.org> First step to fixing a long lived layering violation: this
moves the MacroInfo pointer to a side hash table (which currently
lives in IdentifierTable.cpp). This removes a pointer from
Identifier info, but doesn't shrink it, as it requires a new bit
be added. This strange approach with the 'hasmacro' bit is needed
to not lose preprocessor performance.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42722 91177308-0d34-0410-b5e6-96231b3b80d8
adb83b41cb00acac7a9c035996318d88ecf2e6c8 07-Oct-2007 Chris Lattner <sabre@nondot.org> Shrink the builtinID down by 3 bits, allowing all the bitfields to
fit in 32-bits, shrinking IdentifierInfo by a word.

This shrinks the total size of the identifier pool from
1817264 to 1634428 bytes (11%) on carbon.h.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42719 91177308-0d34-0410-b5e6-96231b3b80d8
ff38491c18b060526d754765b952f4a497a89416 07-Oct-2007 Chris Lattner <sabre@nondot.org> simplify the interfaces to create selectors: getSelector can take any
number of arguments now and does the right thing, but the nullary/unary
accessors are preserved as convenience functions. This allows us to
slightly simplify clients.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42716 91177308-0d34-0410-b5e6-96231b3b80d8
f836e3fea2c77fdbb18170fb313ee0d45551320b 07-Oct-2007 Chris Lattner <sabre@nondot.org> simplify some Selector interfaces.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42715 91177308-0d34-0410-b5e6-96231b3b80d8
85994260c41a54cab061a434ed378fc448333a4e 05-Oct-2007 Chris Lattner <sabre@nondot.org> Implement DenseMapInfo for Selector, allowing use of DenseMap/DenseSet of
Selector's instead of requiring void* to be used. I converted one use of
DenseSet<void*> over to use DenseSet<Selector> but the others should change
as well.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42645 91177308-0d34-0410-b5e6-96231b3b80d8
29238a0bf7cbf5b396efb451a0adb5fe4aa037ca 05-Oct-2007 Steve Naroff <snaroff@apple.com> Layering refinements for selectors (suggested by Chris). Specifics...

- Add SelectorTable, which enables us to remove MultiKeywordSelector from the public header.
- Remove FoldingSet from IdentifierInfo.h and Preprocessor.h.
- Remove Parser::ObjcGetUnarySelector and Parser::ObjcGetKeywordSelector, they are subsumed by SelectorTable.
- Add MultiKeywordSelector to IdentifierInfo.cpp.
- Move a bunch of selector related methods from ParseObjC.cpp to IdentifierInfo.cpp.
- Added some comments.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42643 91177308-0d34-0410-b5e6-96231b3b80d8
0e2dfd3963465d238e791eb8996e4d49fa47b615 04-Oct-2007 Chris Lattner <sabre@nondot.org> minor cleanups, make code more defensive, less branchy in Selector ctor.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42603 91177308-0d34-0410-b5e6-96231b3b80d8
1b386424dfbea0175430345172b5306fda03a3ff 04-Oct-2007 Chris Lattner <sabre@nondot.org> fix an incorrect assertion

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42602 91177308-0d34-0410-b5e6-96231b3b80d8
1f945f632ae993bbb9f9e69d908f7526924c1a88 30-Sep-2007 Chris Lattner <sabre@nondot.org> Add a new getLength() method to IdentifierInfo, which relies on a newly added
method to StringMapEntry. Steve, please use this to remove the
strlen calls in selector processing.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42481 91177308-0d34-0410-b5e6-96231b3b80d8
5b6b72f53ad164497cf62484b60cdbb4361f1978 29-Sep-2007 Steve Naroff <snaroff@apple.com> Add some comments to MultiKeywordSelector, make all methods private, add a friend, move some methods around.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42456 91177308-0d34-0410-b5e6-96231b3b80d8
bcfb06ac6da1aa3c74ac1ef7a49c2807522366e7 29-Sep-2007 Steve Naroff <snaroff@apple.com> Yesterday I discovered that 78% of all selectors in "Cocoa.h" take 0/1 argument.

This motivated implementing a devious clattner inspired solution:-)

This approach uses a small value "Selector" class to point to an IdentifierInfo for the 0/1 case. For multi-keyword selectors, we instantiate a MultiKeywordSelector object (previously known as SelectorInfo). Now, the incremental cost for selectors is only 24,800 for Cocoa.h! This saves 156,592 bytes, or 86%!! The size reduction is also the result of getting rid of the AST slot, which was not strictly necessary (we will associate a selector with it's method using another table...most likely in Sema).

This change was critical to make now, before we have too many clients.

I still need to add some comments to the Selector class...will likely add later today/tomorrow.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42452 91177308-0d34-0410-b5e6-96231b3b80d8
4cbcb897fe7c4ff1ed5322ee4dfd490c581e33d5 27-Sep-2007 Steve Naroff <snaroff@apple.com> Fix bug in SelectorInfo::getName() - method buffer needs to be passed by reference.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42411 91177308-0d34-0410-b5e6-96231b3b80d8
68d331a78e655d97294e94fcfa63f92cc1f40578 27-Sep-2007 Steve Naroff <snaroff@apple.com> Add SelectorInfo (similar in spirit to IdentifierInfo). The key difference is SelectorInfo is not string-oriented, it is a unique aggregate of IdentifierInfo's (using a folding set). SelectorInfo also has a richer API that simplifies the parser/action interface. 3 noteworthy benefits:

#1: It is cleaner. I never "liked" storing keyword selectors (i.e. foo:bar:baz) in the IdentifierTable.

#2: It is more space efficient. Since Cocoa keyword selectors can be quite long, this technique is space saving. For Cocoa.h, pulling the keyword selectors out saves ~180k. The cost of the SelectorInfo data is ~100k. Saves ~80k, or 43%.

#3: It results in many API simplifications. Here are some highlights:

- Removed 3 actions (ActOnKeywordMessage, ActOnUnaryMessage, & one flavor of ObjcBuildMethodDeclaration that was specific to unary messages).
- Removed 3 funky structs from DeclSpec.h (ObjcKeywordMessage, ObjcKeywordDecl, and ObjcKeywordInfo).
- Removed 2 ivars and 2 constructors from ObjCMessageExpr (fyi, this space savings has not been measured).

I am happy with the way it turned out (though it took a bit more hacking than I expected). Given the central role of selectors in ObjC, making sure this is "right" will pay dividends later.

Thanks to Chris for talking this through with me and suggesting this approach.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42395 91177308-0d34-0410-b5e6-96231b3b80d8
525a05093a4816af961fe2bc6b8a81c17e2e26c2 22-Sep-2007 Chris Lattner <sabre@nondot.org> Use APFloat for the representation of FP immediates, ask the target
for *which* apfloat to use for a particular type.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42234 91177308-0d34-0410-b5e6-96231b3b80d8
21d5a95b8f50be3e0fdd8c53f92fc201f665a4b4 19-Sep-2007 Steve Naroff <snaroff@apple.com> Remove SelectorTable/SelectorInfo, simply store all selectors in the central IdentifierTable.


We currently have a separate table to unique ObjC selectors. Since I don't need all the instance data in IdentifierInfo, I thought this would save space (and make more sense conceptually).

It turns out the cost of having duplicate entries for unary selectors (i.e. names without colons) outweighs the cost difference between the IdentifierInfo & SelectorInfo structures. Here is the data:

Two tables:

*** Selector/Identifier Stats:
# Selectors/Identifiers: 51635
Bytes allocated: 1999824

One table:

*** Identifier Table Stats:
# Identifiers: 49500
Bytes allocated: 1990316

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42139 91177308-0d34-0410-b5e6-96231b3b80d8
3f128ad2691d299b96663da85a9e069c4081ea7c 17-Sep-2007 Steve Naroff <snaroff@apple.com> Add support for ObjC keyword selectors.

- Add SelectorInfo/SelectorTable classes, modeled after IdentifierInfo/IdentifierTable.
- Add SelectorTable instance to ASTContext, created lazily through ASTContext::getSelectorInfo().
- Add SelectorInfo slot to ObjcMethodDecl.
- Add helper function to derive a SelectorInfo from ObjcKeywordInfo.

Misc: Got the Decl stats stuff up and running again...it was missing support for ObjC AST's.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42023 91177308-0d34-0410-b5e6-96231b3b80d8
4d33275624e53bf0c16e188d9b1f6a08b2068fcc 08-Sep-2007 Chris Lattner <sabre@nondot.org> Don't rely on ADL to find this member, patch by Justin Handville

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@41783 91177308-0d34-0410-b5e6-96231b3b80d8
b8128140956c6f8f0ab143818775a81f4b4aa477 03-Sep-2007 Chris Lattner <sabre@nondot.org> Eliminate some VC++ warnings, patch by Hartmut Kaiser!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@41685 91177308-0d34-0410-b5e6-96231b3b80d8
6e400c286b485e28d04a742ea87860ddfefa672e 26-Aug-2007 Chris Lattner <sabre@nondot.org> 1.0 is double, 1.0F is a float.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@41412 91177308-0d34-0410-b5e6-96231b3b80d8
506b8dec4ed3db3c60bf9e0dd37901f0cf3d6749 26-Aug-2007 Chris Lattner <sabre@nondot.org> 1) refactor some code.
2) Add support for lexing imaginary constants (a GCC extension):

t.c:5:10: warning: imaginary constants are an extension
A = 1.0iF;

3) Make the 'invalid suffix' diagnostic pointer more accurate:

t.c:6:10: error: invalid suffix 'qF' on floating constant
A = 1.0qF;

instead of:

t.c:6:10: error: invalid suffix 'qF' on floating constant
A = 1.0qF;

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@41411 91177308-0d34-0410-b5e6-96231b3b80d8
861cf3effdc0fbc97d401539bc3050da76b2476f 23-Aug-2007 Steve Naroff <snaroff@apple.com> Add helper functions Token::isObjCAtKeyword() and Token::getObjCKeywordID().

Convert all clients to the new cleaner, more robust API.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@41330 91177308-0d34-0410-b5e6-96231b3b80d8
d8e3083840fef752d11ca183f42786470ed061e3 24-Jul-2007 Chris Lattner <sabre@nondot.org> Use a smallstring instead of an std::string in FileChanged to avoid some malloc traffic.

This speeds up -E on xalancbmk by 2.4%

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40461 91177308-0d34-0410-b5e6-96231b3b80d8
f1c99acc544a4e70f308db4e7200ca04cd5a06d2 23-Jul-2007 Chris Lattner <sabre@nondot.org> refactor the interface to Preprocessor::GetIncludeFilenameSpelling,
no functionality changes.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40414 91177308-0d34-0410-b5e6-96231b3b80d8
448cec4c1c3705f6f49ffdefb58a7329942a2dd8 22-Jul-2007 Chris Lattner <sabre@nondot.org> Change hte lexer to start a start pointer to the underlying
memorybuffer instead of a pointer to the memorybuffer itself. This
reduces coupling and eliminates a pointer dereference on a hot path.
This speeds up -Eonly on 483.xalancbmk by 2.1%

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40394 91177308-0d34-0410-b5e6-96231b3b80d8
9960ae8ecfa2c4278dac708a02e463f83fdf17e8 22-Jul-2007 Chris Lattner <sabre@nondot.org> Implement a simple cache in headersearch. This speeds up
preprocessing 483.xalancbmk by about 10%, reducing the number
of file lookup queries from 2139411 to 199466 (over 10x)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40390 91177308-0d34-0410-b5e6-96231b3b80d8
9c683062752a26014197df1c8792a19efa9e93cf 22-Jul-2007 Chris Lattner <sabre@nondot.org> Fix a really subtle bug in the macro expander caching code, where
redefinition of a macro could cause invalid memory to be deleted.
Found preprocessing 253.perlbmk.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40380 91177308-0d34-0410-b5e6-96231b3b80d8
d217773f106856a11879ec79dc468efefaf2ee75 20-Jul-2007 Chris Lattner <sabre@nondot.org> At one point there were going to be lexer and parser tokens.
Since that point is now long gone, we should rename LexerToken to
Token, as it is the only kind of token we have.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40105 91177308-0d34-0410-b5e6-96231b3b80d8
25bdb51276d3bfc180a68688082071505a00ed27 20-Jul-2007 Chris Lattner <sabre@nondot.org> simplify the lexer ctor to take a SLoc instead of a sloc and a redundant buffer*.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40104 91177308-0d34-0410-b5e6-96231b3b80d8
9dc1f530c086d2c16f8cba758b0f59a5bf41323a 20-Jul-2007 Chris Lattner <sabre@nondot.org> Reimplement SourceLocation. Instead of having a
fileid/offset pair, it now contains a bit discriminating between
mapped locations and file locations. This separates the tables for
macros and files in SourceManager, and allows better separation of
concepts in the rest of the compiler. This allows us to have *many*
macro instantiations before running out of 'addressing space'.

This is also more efficient, because testing whether something is a
macro expansion is now a bit test instead of a table lookup (which
also used to require having a srcmgr around, now it doesn't).

This is fully functional, but there are several refinements and
optimizations left.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40103 91177308-0d34-0410-b5e6-96231b3b80d8
97ba77cf09bf7b83b679165ce67ad7d49ffd568c 16-Jul-2007 Chris Lattner <sabre@nondot.org> Add a new Preprocessor::AdvanceToTokenCharacter method which, given a sloc
specifying the start of a token and a logical (phase 3) character number,
returns a sloc representing the input character corresponding to it.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39905 91177308-0d34-0410-b5e6-96231b3b80d8
f66d3e390fac1dce6b0e3ff1a3d0180638e560f6 16-Jul-2007 Chris Lattner <sabre@nondot.org> factor a common predicate into a static method.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39903 91177308-0d34-0410-b5e6-96231b3b80d8
9594acf32de2939b15eafa8fe818607bfc56bf66 15-Jul-2007 Chris Lattner <sabre@nondot.org> Cache macro expander objects to avoid thrashing malloc in heavy expansion situations.
This doesn't significantly improve carbon.h, but it does speed up
INPUTS/macro_pounder_obj.c by 48%

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39864 91177308-0d34-0410-b5e6-96231b3b80d8
25c9648909193d380a4e135d2e3d25394ba12922 15-Jul-2007 Chris Lattner <sabre@nondot.org> switch function-like macros from using a vector for their arguments to an
explicitly new'd array. The array never mutates once created, so a vector
is overkill.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39862 91177308-0d34-0410-b5e6-96231b3b80d8
f46f68b5587b6933a92260220567ea7c36924a80 15-Jul-2007 Chris Lattner <sabre@nondot.org> switch from using a vector to a smallvector for macro replacement tokens
This speeds up parsing carbon.h by 3.3% by avoiding some malloc traffic for
small macros.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39861 91177308-0d34-0410-b5e6-96231b3b80d8
c215bd659d8266a1d6b66ce231a63405a4c61daf 15-Jul-2007 Chris Lattner <sabre@nondot.org> expose an iterator interface to getReplacementTokens instead of the datastructure itself.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39860 91177308-0d34-0410-b5e6-96231b3b80d8
6fa5f0943a84233b2e1ec9716eae55643225bfd4 12-Jul-2007 Chris Lattner <sabre@nondot.org> Fix "no newline at end of file" warnings. Patch contributed by
Benoit Boissinot!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39780 91177308-0d34-0410-b5e6-96231b3b80d8
9dc62f044a6ba21f503bd56607d94b32704e7945 12-Jul-2007 Chris Lattner <sabre@nondot.org> Improve portability to compilers where <cassert> is not implicitly
included. Patch contributed by Benoit Boissinot!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39779 91177308-0d34-0410-b5e6-96231b3b80d8
5f016e2cb5d11daeb237544de1c5d59f20fe1a6e 11-Jul-2007 Reid Spencer <rspencer@reidspencer.com> Stage two of getting CFE top correct.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39734 91177308-0d34-0410-b5e6-96231b3b80d8