4件 （「token」）

Sudachi: a Japanese Tokenizer for Business 論文

Kazuma Takaoka・Sorami Hisamoto・Kawahara Noriko・Miho Sakamoto・Yoshitaka Uchida・Yūji Matsumoto ·2018-05-01

doi:10.63317/2hsnuu4nxsnq

Byte Pair Encoding is Suboptimal for Language Model Pretraining 論文

Kaj Bostrom・Greg Durrett ·2020-01-01 ·pp. 4617-4624

doi:10.18653/v1/2020.findings-emnlp.414

SpanAlign: Sentence Alignment Method based on Cross-Language Span Prediction and ILP 論文

Katsuki Chousa・Masaaki Nagata・Masaaki Nishino ·2020-01-01 ·pp. 4750-4761

doi:10.18653/v1/2020.coling-main.418

Optimizing Word Segmentation for Downstream Tasks by Weighting Text Vector 論文

Tatsuya Hiraoka・Sho Takase・Kei Uchiumi・Atsushi Keyaki・Naoaki Okazaki ·2021-01-01 ·『Journal of Natural Language Processing』　28(2)　pp. 479-507

doi:10.5715/jnlp.28.479