cpp_token_stream.h File Reference

This file defines the C++ lexical analysis tools. It defines the concept of a C++ 'token', also a template which parses tokens from character sources (anything with an iterator), and a few concrete instances of token streams. The token sources defined below use the lexer template, CPP_Token_Stream, to instantiate CPP_Token_Source's. That is, you can write your code using CPP_Tokens to analyze the parsed source. Instead of referring directly to one of the concrete lexers (CPP_Stream_Token_Source, CPP_File_Token_Source, or CPP_String_Token_Source) you can write your code using the abstract lexer base class, CPP_Token_Source. The virtual method call overhead is likely to be small compared to the looping constructs typically found inside the lexer (CPP_Token_Stream<>). More...

#include <fstream>
#include <string>
#include <portable_strstream.h>

Include dependency graph for cpp_token_stream.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	CPP_Buffer_Token_Source
	A source of tokens extracted from a character array. More...
class	CPP_File_Token_Source
	A source of CPP_Token's which comes from a named file. More...
class	CPP_Stream_Token_Source
	A source of CPP_Token's which comes from an istream. More...
class	CPP_String_Token_Source
struct	CPP_Token
	An object that represents a C++ lexical token. CPP_Token objects are usually obtained from token streams. See CPP_Token_Stream's for an example use. More...
struct	CPP_Token_Source
	A generic representation of a stream of CPP_Token. More...
class	CPP_Token_Stream< Iterator >
	C++ token stream template (ie a source of CPP_Token objects). More...
class	CPP_Token_Stream_Prep
	A CPP_Token_Stream, does not include preprocessor directives. Doing so would greatly complicate any grammer. Instead, preprocessor directives are handled only if desired in an out of band style. To handle preprocessor directives, derive a class from this one (CPP_Token_Stream_Prep) and implement a virtual method which will be invoked by the stream as preprocessor directives are encountered. Note that the '#' and any spaces surrounding it will not be included in the text passed to the virtual method. More...
Namespaces
namespace	cxxtls
	The namespace that encapsulates most of the functionality in the CXX toolkit.
Functions
std::ostream &	operator<< (std::ostream &o, CPP_Token const &r)
	output a token to a stream

---

Detailed Description

This file defines the C++ lexical analysis tools. It defines the concept of a C++ 'token', also a template which parses tokens from character sources (anything with an iterator), and a few concrete instances of token streams. The token sources defined below use the lexer template, CPP_Token_Stream, to instantiate CPP_Token_Source's. That is, you can write your code using CPP_Tokens to analyze the parsed source. Instead of referring directly to one of the concrete lexers (CPP_Stream_Token_Source, CPP_File_Token_Source, or CPP_String_Token_Source) you can write your code using the abstract lexer base class, CPP_Token_Source. The virtual method call overhead is likely to be small compared to the looping constructs typically found inside the lexer (CPP_Token_Stream<>).

Definition in file cpp_token_stream.h.