C++ token stream template (ie a source of CPP_Token objects). More...
#include <cpp_token_stream.h>
Public Member Functions | |
CPP_Token_Stream (Iterator const &next, Iterator const &end, std::string filename="", CPP_Token_Stream_Prep const *handler=0) | |
void | operator() (CPP_Token &token) |
Read one token out of the stream defined by a pair of iterators. Do not read past the end of the token stream. When end of stream has occurred, the returned token type will be CPP_Token::eof. | |
int | token_type (std::string const &s) |
Given an arbitrary string, determine what kind of token it is. The return value will be one of CPP_Token's types. The token is assumed to be aligned at the beginning of the string with no leading blanks, tabs, etc. | |
Private Member Functions | |
void | initialize_char_type () |
initialize the array of character types so that the cpp_token function can combine tokens of similar types without having to do a lot of if then elses | |
void | parse_preprocessor_directive () |
parse any of the following: | |
bool | scan_till_first_nonblank () |
Skip characters in the stream until the first nonblank character is found. Handle line number changes in blank sequences. | |
Private Attributes | |
char | char_type_ [256] |
equivalent of ctype() | |
Iterator | cur_ |
current position in the stream | |
Iterator | end_ |
end of stream position | |
std::string | file_ |
name of current file in the stream | |
bool | initialized_ |
has char_type_ been initilaized | |
int | line_ |
line number in the stream | |
CPP_Token_Stream_Prep const * | preprocessor_handler_ |
handler for pre-processor directives | |
Friends | |
class | CPP_Buffer_Token_Source |
C++ token stream template (ie a source of CPP_Token objects).
A CPP_Token_Stream<Iterator> is a sequence of CPP_Token objects. A CPP_Token_Stream is a functor object. Use its function call operator to get the next CPP_Token out of the stream.
This class is a template, whose parameter is the kind of iterator to use when fetching characters out of the real stream. You can use any of: char* std::string::const_iterator std::istreambuf_iterator other iterators which dereference to char
Here is an example use:
#include <cxxtls/cpp_token_stream.h> std::istreambuf_iterator<char> cur(std::cin); std::istreambuf_iterator<char> end; CPP_Token_Stream< std::istreambuf_iterator<char> > ts(cur,end); CPP_Token token_one; ts(token_one); // declare and read tokens CPP_Token token_two; ts(token_two);
When end of stream has occurred, the returned token will have a type which is CPP_Token::eof. Multiple attempts to fetch the next token will all return this type without performing any subsequent parsing. The only language features recognized by this lexer are: the C++ token set line directives which are properly interpreted all other # directives are removed from the source and ignored
Definition at line 210 of file cpp_token_stream.h.
CPP_Token_Stream | ( | Iterator const & | next, | |
Iterator const & | end, | |||
std::string | filename = "" , |
|||
CPP_Token_Stream_Prep const * | handler = 0 | |||
) |
< Construct a token stream from a pair of iterators -- with optional settings to define the name of the file being parsed and a handler object that handles preprocessor directives.
Definition at line 264 of file cpp_token_stream.h.
void initialize_char_type | ( | ) | [private] |
initialize the array of character types so that the cpp_token function can combine tokens of similar types without having to do a lot of if then elses
Definition at line 911 of file cpp_token_stream.h.
void operator() | ( | CPP_Token & | token | ) |
Read one token out of the stream defined by a pair of iterators. Do not read past the end of the token stream. When end of stream has occurred, the returned token type will be CPP_Token::eof.
token | is the place to store the output token. |
Definition at line 284 of file cpp_token_stream.h.
void parse_preprocessor_directive | ( | ) | [private] |
parse any of the following:
If a preprocessor_handler_ is specified, use its function call operator method interpret the directive (other than line number directives which are consumed here)
Definition at line 963 of file cpp_token_stream.h.
bool scan_till_first_nonblank | ( | ) | [private] |
Skip characters in the stream until the first nonblank character is found. Handle line number changes in blank sequences.
Definition at line 1171 of file cpp_token_stream.h.
int token_type | ( | std::string const & | s | ) |
Given an arbitrary string, determine what kind of token it is. The return value will be one of CPP_Token's types. The token is assumed to be aligned at the beginning of the string with no leading blanks, tabs, etc.
s | the input string to check on. |
Definition at line 660 of file cpp_token_stream.h.
friend class CPP_Buffer_Token_Source [friend] |
Definition at line 259 of file cpp_token_stream.h.
char char_type_[256] [private] |
equivalent of ctype()
Definition at line 258 of file cpp_token_stream.h.
Iterator cur_ [private] |
current position in the stream
Definition at line 252 of file cpp_token_stream.h.
Iterator end_ [private] |
end of stream position
Definition at line 253 of file cpp_token_stream.h.
std::string file_ [private] |
name of current file in the stream
Definition at line 255 of file cpp_token_stream.h.
bool initialized_ [private] |
has char_type_ been initilaized
Definition at line 256 of file cpp_token_stream.h.
int line_ [private] |
line number in the stream
Definition at line 254 of file cpp_token_stream.h.
CPP_Token_Stream_Prep const* preprocessor_handler_ [private] |
handler for pre-processor directives
Definition at line 257 of file cpp_token_stream.h.