Tokenizer.h 文件参考

浏览源代码.

类
singleton	SGVector< T >
	shogun vector 更多...
class	CTokenizer
	The class CTokenizer acts as a base class in order to implement tokenizers. Sub-classes must implement the methods has_next(), next_token_idx() and get_copy(). 更多...