Tokenizer.h File Reference

Go to the source code of this file.

Classes
class	CTokenizer
	The class CTokenizer acts as a base class in order to implement tokenizers. Sub-classes must implement the methods has_next(), next_token_idx() and get_copy(). More...