CPP_Token_Stream< Iterator > Class Template Reference

C++ token stream template (ie a source of CPP_Token objects). More...

#include <cpp_token_stream.h>

Inheritance diagram for CPP_Token_Stream< Iterator >:

[legend]

Collaboration diagram for CPP_Token_Stream< Iterator >:

[legend]

List of all members.

Public Member Functions
	CPP_Token_Stream (Iterator const &next, Iterator const &end, std::string filename="", CPP_Token_Stream_Prep const *handler=0)
void	operator() (CPP_Token &token)
	Read one token out of the stream defined by a pair of iterators. Do not read past the end of the token stream. When end of stream has occurred, the returned token type will be CPP_Token::eof.
int	token_type (std::string const &s)
	Given an arbitrary string, determine what kind of token it is. The return value will be one of CPP_Token's types. The token is assumed to be aligned at the beginning of the string with no leading blanks, tabs, etc.
Private Member Functions
void	initialize_char_type ()
	initialize the array of character types so that the cpp_token function can combine tokens of similar types without having to do a lot of if then elses
void	parse_preprocessor_directive ()
	parse any of the following:
bool	scan_till_first_nonblank ()
	Skip characters in the stream until the first nonblank character is found. Handle line number changes in blank sequences.
Private Attributes
char	char_type_ [256]
	equivalent of ctype()
Iterator	cur_
	current position in the stream
Iterator	end_
	end of stream position
std::string	file_
	name of current file in the stream
bool	initialized_
	has char_type_ been initilaized
int	line_
	line number in the stream
CPP_Token_Stream_Prep const *	preprocessor_handler_
	handler for pre-processor directives
Friends
class	CPP_Buffer_Token_Source

Detailed Description

template<class Iterator>
class cxxtls::CPP_Token_Stream< Iterator >

C++ token stream template (ie a source of CPP_Token objects).

A CPP_Token_Stream<Iterator> is a sequence of CPP_Token objects. A CPP_Token_Stream is a functor object. Use its function call operator to get the next CPP_Token out of the stream.

This class is a template, whose parameter is the kind of iterator to use when fetching characters out of the real stream. You can use any of: char* std::string::const_iterator std::istreambuf_iterator other iterators which dereference to char

Here is an example use:

       #include <cxxtls/cpp_token_stream.h>
  
       std::istreambuf_iterator<char>  cur(std::cin);
       std::istreambuf_iterator<char>  end;
  
       CPP_Token_Stream< std::istreambuf_iterator<char> >  ts(cur,end);
  
       CPP_Token token_one;  ts(token_one); // declare and read tokens
       CPP_Token token_two;  ts(token_two);

When end of stream has occurred, the returned token will have a type which is CPP_Token::eof. Multiple attempts to fetch the next token will all return this type without performing any subsequent parsing. The only language features recognized by this lexer are: the C++ token set line directives which are properly interpreted all other # directives are removed from the source and ignored

Definition at line 210 of file cpp_token_stream.h.

Constructor & Destructor Documentation

CPP_Token_Stream	(	Iterator const &	next,
		Iterator const &	end,
		std::string	filename = `""`,
		CPP_Token_Stream_Prep const *	handler = `0`
	)

< Construct a token stream from a pair of iterators -- with optional settings to define the name of the file being parsed and a handler object that handles preprocessor directives.

Definition at line 264 of file cpp_token_stream.h.

Member Function Documentation

void initialize_char_type ( ) [private]

initialize the array of character types so that the cpp_token function can combine tokens of similar types without having to do a lot of if then elses

Definition at line 911 of file cpp_token_stream.h.

void operator() ( CPP_Token & token )

Read one token out of the stream defined by a pair of iterators. Do not read past the end of the token stream. When end of stream has occurred, the returned token type will be CPP_Token::eof.

Parameters:

token

is the place to store the output token.

Definition at line 284 of file cpp_token_stream.h.

void parse_preprocessor_directive ( ) [private]

parse any of the following:

"#line number [name]"
"# number [name]"
"#pragma ..."
"#define ..."

If a preprocessor_handler_ is specified, use its function call operator method interpret the directive (other than line number directives which are consumed here)

Definition at line 963 of file cpp_token_stream.h.

bool scan_till_first_nonblank ( ) [private]

Skip characters in the stream until the first nonblank character is found. Handle line number changes in blank sequences.

Definition at line 1171 of file cpp_token_stream.h.

int token_type ( std::string const & s )

Given an arbitrary string, determine what kind of token it is. The return value will be one of CPP_Token's types. The token is assumed to be aligned at the beginning of the string with no leading blanks, tabs, etc.

Returns:: the type of token held in the input string

Parameters:

s

the input string to check on.

Definition at line 660 of file cpp_token_stream.h.

Friends And Related Function Documentation

friend class CPP_Buffer_Token_Source [friend]

Definition at line 259 of file cpp_token_stream.h.

Member Data Documentation

char char_type_[256] [private]

equivalent of ctype()

Definition at line 258 of file cpp_token_stream.h.

Iterator cur_ [private]

current position in the stream

Definition at line 252 of file cpp_token_stream.h.

Iterator end_ [private]

end of stream position

Definition at line 253 of file cpp_token_stream.h.

std::string file_ [private]

name of current file in the stream

Definition at line 255 of file cpp_token_stream.h.

bool initialized_ [private]

has char_type_ been initilaized

Definition at line 256 of file cpp_token_stream.h.

int line_ [private]

line number in the stream

Definition at line 254 of file cpp_token_stream.h.

CPP_Token_Stream_Prep const* preprocessor_handler_ [private]

handler for pre-processor directives

Definition at line 257 of file cpp_token_stream.h.

The documentation for this class was generated from the following file:

include/cxxtls/cpp_token_stream.h

CPP_Token_Stream< Iterator > Class Template Reference

Public Member Functions

Private Member Functions

Private Attributes