|
FreeLing
3.1
|
Class tokenizer implements a token splitter, which converts a string into a sequence of word objects, according to a set of tokenization rules read from aconfiguration file. More...
#include <tokenizer.h>
Public Member Functions | |
| tokenizer (const std::wstring &) | |
| Constructor. | |
| void | tokenize (const std::wstring &, std::list< word > &) const |
| tokenize string | |
| std::list< word > | tokenize (const std::wstring &) const |
| tokenize string, return result as list | |
| void | tokenize (const std::wstring &, unsigned long &, std::list< word > &) const |
| tokenize string, tracking offset | |
| std::list< word > | tokenize (const std::wstring &, unsigned long &) const |
| tokenize string, tracking offset, return result as list | |
Private Attributes | |
| std::set< std::wstring > | abrevs |
| abreviations set (Dr. Mrs. etc. period is not separated) | |
| std::list< std::pair < std::wstring, freeling::regexp > > | rules |
| tokenization rules | |
| std::map< std::wstring, int > | matches |
| substrings to convert into tokens in each rule | |
Class tokenizer implements a token splitter, which converts a string into a sequence of word objects, according to a set of tokenization rules read from aconfiguration file.
| freeling::tokenizer::tokenizer | ( | const std::wstring & | tokFile | ) |
Constructor.
Create a tokenizer, using the abreviation and patterns file indicated in given options.
References freeling::config_file::add_section(), freeling::config_file::close(), ERROR_CRASH, freeling::config_file::get_content_line(), freeling::config_file::get_section(), freeling::config_file::open(), and TRACE.
| void freeling::tokenizer::tokenize | ( | const std::wstring & | , |
| std::list< word > & | |||
| ) | const |
tokenize string
| std::list<word> freeling::tokenizer::tokenize | ( | const std::wstring & | ) | const |
tokenize string, return result as list
| void freeling::tokenizer::tokenize | ( | const std::wstring & | , |
| unsigned long & | , | ||
| std::list< word > & | |||
| ) | const |
tokenize string, tracking offset
| std::list<word> freeling::tokenizer::tokenize | ( | const std::wstring & | , |
| unsigned long & | |||
| ) | const |
tokenize string, tracking offset, return result as list
std::set<std::wstring> freeling::tokenizer::abrevs [private] |
abreviations set (Dr. Mrs. etc. period is not separated)
std::map<std::wstring,int> freeling::tokenizer::matches [private] |
substrings to convert into tokens in each rule
std::list<std::pair<std::wstring, freeling::regexp> > freeling::tokenizer::rules [private] |
tokenization rules
1.7.6.1