Subword tokenization
4 methods in the atlas attack this one problem. They are rivals: each wins something the others do not.
Phrasings that mean this problem
Subword tokenizationLanguage-agnostic tokenization
information-retrieval-nlp
- Byte pair encodingFrequency-merge vocabularycanoninformation-retrieval-nlp
- WordPieceLikelihood-merge vocabularystandardinformation-retrieval-nlp
- Unigram language model tokenizerEM subword pruningspecialistinformation-retrieval-nlp
- SentencePiecestandalonespecialistinformation-retrieval-nlp