An Analysis of Statistical Techniques Applying to Multi-Feature Similarity Comparison between Corpora

Abstract:

Statistical techniques applying to multi-feature similarity comparison belong to the type of goodness-of-fit test which include chi-square test, rank correlation test and Kolmogorov-Smirnov test (K-S test). Experiments show that both chi-square independence test and rank correlation test are subject to the variation of sample size. With the expansion of sample size, the former test achieves the results of significant difference and the latter achieves the results of significant correlation easily. However, both results fail to reveal the actual situation of multi-feature similarity comparison between corpora. Only K-S test, which quantifies a distance between the empirical distribution functions of two samples, can achieve the highest statistical effectiveness.

• Periodical: Applied Mechanics and Materials (Volumes 66-68)
• Pages: 2323-2329
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.66-68.2323
• Online since: July 2011
• Authors: Xiao Xiao Chen, Shi Li Ge, Min Lin
• Keywords: Corpus, K-S Test, Multi-Feature Similarity Comparison, Nonparametric Statistics

References

[1] K. Hofland & S. Johansson. Word Frequencies in British and American English [M]. Bergen: The Norwegian Computing Centre for the Humanities, (1982).

[2] L.F. Wang & H.J. Ma. A Corpus-based Research on Stance Construction in Chinese Students' English Speeches [J]. Foreign Language Teaching and Research, 2009, 41(5): 365-370.

[3] R Larson & B. Farber. Fundamental Statistics [M]. Beijing: Qinghua University Press, (2003).

[4] M.P. Oakes. Statistics for Corpus Linguistics [M]. Edinburgh: Edinburgh University Press, (1998).

[5] K. Church and W. Gale Poisson mixtures [J]. Journal of Natural Language Engineering, 1995, 1(2): 163–190.

[6] B.H. Cohen. Explaining Psychological Statistics [M]. NJ: John Wiley & Sons, (2008).

[7] C. Butler. Statistics in linguistics [M]. B. Blackwell, (1985).

[8] A. Kilgarriff. Comparing Corpora [J]. International Journal of Corpus Linguistics, 2001, 6(1): 97–133.

[9] X.Z. Wu and Z.J. Wang. Nonparametric Statistical Methods[M]. Beijing: Higher Education Press, (1996).

[10] W.J. Conover. Practical Nonparametric Statistics [M]. Beijing: Posts & Telecom Press, (2006).

[11] S.C. Gui and H.Z. Yang. Chinese Learner English Corpus [M]. Shaihai: Shanghai Foreign Language Education Press, (2002).