Research on the Chinese Word Segmentation System Based on Incremental Learning

Abstract:

In order to cope with the increasing size of the training corpus and adapt to the requirements of incremental learning, this paper introduces a feature selection algorithm of maximum entropy model into the research of Chinese word segmentation technology, designs and implements a Chinese word segmentation system based on incremental learning. The experimental results show that the system gradually improves the segmentation accuracy in the incremental learning process which without wasting time to restudy.

• Periodical: Applied Mechanics and Materials (Volumes 602-605)
• Pages: 3469-3473
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.602-605.3469
• Online since: August 2014
• Authors: Fan Jin Mai*, Shi Tong Wu, Lai Yue Wang
• Keywords: Chinese Word Segmentation, Feature Selection, Incremental Learning, Maximum Entropy

References

[1] Ying Qin, Xiaojie Wang, Suxiang Zhang, in: The research of combinatorial ambiguity in Chinese word segmentation. Journal of Chinese Information Processing, China(2007).

[2] Chengqing Zong, in: Statistical natural language processing. Tsinghua University press, Beijing, China(2008).

[3] Fanjin Mai, Ting Wang, in: Word segmentation disambiguation model based on the bidirectional maximum matching and HMM. Modern Library and Information Technology, China(2008).

[4] Rong Zeng, Jiancheng Liu, Xinhua Jiang, in: An incremental learning algorithm based on support vector machine (SVM). Journal of Railway Science and Engineering, China(2005).

[5] Wenhua Zeng, Jian Ma, in: A new support vector machine (SVM) incremental learning algorithm. Natural Science , Journal of Xiamen University, China (2002).