What we need is word, not morpheme; constructing word analyzer for Japanese

要旨

This paper presents our work on building a Japanese word analyzer, SNOWMAN that is not so-called a “morphological analyzer.” Although there are some morphological analyzers still available, they all produce morphemes, not words as output. That is, they are insufficient to recognize word consisting of multiple morphemes, such as idioms. Moreover, it is quite important in Japanese processing to reduce orthographical variants, that is considered partially in the current analyzers. Our analyzer strives to solve both problems. We have produced an analyzer in which 320 thousand morphemes can be merged into 287 thousand words, and 28 thousand words with multiple morphemes can be recognized as word.

主題