Corpus-based Japanese morphological analysis

要旨

The goal of this study is to improve corpus-based Japanese morphological analysis which is composed by word segmentation and part-of-speech (below POS) tagging.We divide the problem of Japanese morphological analysis into three subproblems: models for known word, models for unknown word and corpus maintenance schema.Firstly, we discuss Markov model-based approaches for known word processing.We point phenomena which are difficult to be analyzed by a simple Markov model.Special transactions are necessary for these phenomena.Therefore, we introduce three extensions for Markov model: lexicalized POS, position-wise grouping and selective trigram.Secondly, we discuss unknown word processing.We newly propose an offline model for unknown word based on a pattern recognition approach.Unknown words are extracted from the text by chunking in advance.Next, the POSs for the extracted words are estimated by a word sense disambiguation-like approach.Thirdly, we discuss maintenance schema for word segmented and POS tagged corpus.The corpus maintenance is a crucial issue for corpus-based models.We propose a relational database usage to keep consistency in the corpora.The relational database enables us synchronous transaction between the lexicon and the corpora.Therefore, the risk of discrepancy in the corpus is reduced by the proposed method.As side issues, we discuss Japanese named entity extraction and filler filtering.Japanese named entity extraction is an application in information extraction.We propose two extensions for the application.One is a character-based chunking method which solves a word boundary discrepancy problem.The other is use of point-wise n-best answers of Japanese morphological analyzer which makes the model robust.The proposed method achieves the best accuracy in the preceding works.Filler filtering is a preprocessing for Japanese morphological analysis.Many fillers and disfluencies appear in transcriptions of spoken language.These phenomena are factors of the errors in Japanese morphological analysis.We introduce a pattern recognition method for filler and disfluency filtering from the transcription.

主題