HMM Parameter Learning for Japanese Morphological Analyzer

要旨

This paper presents a method to apply Hidden Markov Model (HMM) to parameter learning for Japanese morphological analyzer.We especially emphasize how the following two information sources affect the results of the parameter learning: 1) The initial value of parameters, i.e., the initial probabilities and 2) some grammatical constraints that hold in Japanese sentences independently of any domain.First and foremost, a simple application of HMM to Japanese corpus does not give a satisfactory results since word boundaries are not clear in Japanese texts because of lack of word separators.The first results of the experiments show that initial probabilities learned from correct tagged corpus affects greatly to the results and that a small tagged corpus is enough for the initial probabilities.The second result is that the incorporation of simple grammatical constraints works well in the improvements of the results.The final result gives that the total performance of the HMM-based parameter learning achieves almost the same level as the human developed rule-based Japanese morphological analyzer.

主題