Identification of Chinese Unknown Word Based on Finite Multi-List Method

Abstract:

Unknown word is a kind of word that is not included in the sub_word vocabulary, but must be cut out by the word segmentation program. Peoples’ names, place names and translated names are the major unknown words.Unknown Chinese words is a difficult problem in natural language processing, and also contributed to the low rate of correct segmention. This paper introduces the finite multi-list method that using the word fragments’ capability to composite a word and the location in the word tree to process the unknown Chinese words.The experiment recall is 70.67% ,the correct rate is 43.65% .The result of the experiment shows that unknown Chinese word identification based on the finite multi-list method is feasible.

• Periodical: Key Engineering Materials (Volumes 474-476)
• Pages: 460-465
• DOI: https://doi.org/10.4028/www.scientific.net/KEM.474-476.460
• Online since: April 2011
• Authors: Bo Sun, Sheng Hui Huang, Xiao Hua Liu
• Keywords: Finite Multi-List Method, Natural Language Processing, Unknown Words Identification

References

[1] P. Zhang , finite multi-listing" the discussion about the of "finite multi-listing, for automatic segmentation methods in Modern Chinese: language and computer, 61-64(1986)No. 3(In Chinese).

[2] K.K. He,H. Xu,B. Sun, Design Principle of Expert System for Automatic Words Segmentation in Written Chinese: Journal of Chinese Information Processing, 38-47(1991)No. 5(In Chinese).

[3] X.H. Chen, A Package Scheme for Identifying Unlisted Words in Chinese Segmentation : Applied Linguistics(In Chinese).

[4] C.F. Yuan C.L. Huang, Chinese morpheme database construction and application : Communication of COLIPS, (1997. )(In Chinese).