Seven Factors in Evaluating Recommender System

Abstract:

Recommender system (RS) has been evaluated in many but incomparable ways beyond accuracy and thus proposing an evaluation framework to synthesize the existing strategies seems a solution. However, few scholars did it so far. Through literature review, user interview and expert assessment, this study proposed a theoretical evaluation model of RS and then formed the assessment tool, RS Evaluation Questionnaire (RSE). The results showed that RSE was an effective tool to evaluate a recommender system, with its reliability (Cronbachs α=0.803) and validity meeting the requirements of psychometrics. Seven factors such as Perceived Quality and Perceived Ease of Use were generated by factor analysis, accounting for 63.126% of the variance. Furthermore, regression analysis indicated that different combinations of RSE factors could significantly predict User Satisfaction, Reuse Intention and positive Word-Of-Mouth (WOM) spreading willingness. Enlightenments for future research and practice were discussed as well in the end.

• Periodical: Applied Mechanics and Materials (Volume 472)
• Pages: 443-449
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.472.443
• Online since: January 2014
• Authors: Li Ting Kang, Yong Wang
• Keywords: Evaluation, Personalized Recommender System, Psychometric Questionnaire, User Interview, User-Centered

References

[1] Herlocker, J. L., J. A. Konstan, K. Terveen and J. T. Riedl: Evaluating collaborative filtering recommender systems. Acm Transactions on Information Systems Vol. 22, 1(2004), P. 5-53.

DOI: 10.1145/963770.963772

[2] Liang, T. P., H. J. Lai and Y. C. Ku: Personalized content recommendation and user satisfaction: Theoretical synthesis and empirical findings. Journal of Management Information Systems Vol. 23, 3(2006), P. 45-70.

DOI: 10.2753/mis0742-1222230303

[3] Pu, P., L. Chen and R. Hu: A user-centric evaluation framework for recommender systems. (Proc. RecSys 2010 workshop on User-Centric Evaluation of RSs and Their Interfaces (UCERSTI), Barcelona, Spain, 2011).

DOI: 10.1145/2043932.2044020

[4] Resnick, P., N. Iacovou, M. Suchak, P. Bergstrom and J. Riedl: GroupLens: an open architecture for collaborative filtering of netnews. (Proceedings of the 1994 ACM conference on Computer supported cooperative work, Chapel Hill, North Carolina, USA, 1994).

DOI: 10.1145/192844.192905

[5] McNee, S. M., J. Riedl and J. A. Konstan: Being accurate is not enough: how accuracy metrics have hurt recommender systems. (CHI '06 extended abstracts on Human factors in computing systems, Montral, Qubec, Canada, 2006).

DOI: 10.1145/1125451.1125659

[6] Knijnenburg, B., M. Willemsen, Z. Gantner, H. Soncu and C. Newell: Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction Vol. 22, 4(2012), P. 441-504.

DOI: 10.1007/s11257-011-9118-4

[7] Ozok, A. A., Q. Y. Fan and A. F. Norcio: Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: results from a college student population. Behaviour & Information Technology Vol. 29, 1(2010).

DOI: 10.1080/01449290903004012

[8] Pu, P., L. Chen and R. Hu: Evaluating recommender systems from the user's perspective: survey of the state of the art. User Modeling and User-Adapted Interaction Vol. 22, 4(2012a), P. 317-355.

DOI: 10.1007/s11257-011-9115-7

[9] Rashmi, S. and S. Kirsten: Comparing recommendations made by online systems and friends. (Proceedings of the DELOS-NSF Workshop on Personalization and Recommender Systems in Digital Libraries, Dublin, Ireland., 2001).

[10] Iaquinta, L., M. de Gemmis, P. Lops, G. Semeraro, M. Filannino and P. Molino: Introducing serendipity in a content-based recommender system. (Proceedings of the 2008 8th International conference on Hybrid Intelligent Systems, Barcelona, Spain, 2008).

DOI: 10.1109/his.2008.25

[11] Ge, M., C. Delgado-Battenfeld and D. Jannach: Beyond accuracy: evaluating recommender systems by coverage and serendipity. (Proceedings of the fourth ACM conference on Recommender systems, Barcelona, Spain, 2010).

DOI: 10.1145/1864708.1864761

[12] Herlocker, J. L., J. A. Konstan and J. Riedl: Explaining collaborative filtering recommendations. (Proceedings of the 2000 ACM conference on Computer supported cooperative work, Philadelphia, Pennsylvania, United States, 2000).

DOI: 10.1145/358916.358995

[13] Tintarev, N. and J. Masthoff: A survey of explanations in recommender systems. (Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering Workshop, Istanbul, Turkey, 2007).

DOI: 10.1109/icdew.2007.4401070

[14] Lathia, N., S. Hailes, L. Capra and X. Amatriain: Temporal Diversity in Recommender Systems. (Proceedings of the 33rd Annual International Acm Sigir Conference on Research Development in Information Retrieval, Geneva, Switzerland, 2010).

DOI: 10.1145/1835449.1835486

[15] Jiang, Y. C., J. Shang and Y. Z. Liu: Maximizing customer satisfaction through an online recommendation system: A novel associative classification model. Decision Support Systems Vol. 48, 3(2010), P. 470-479.

DOI: 10.1016/j.dss.2009.06.006

[16] Billsus, D. and M. J. Pazzani: A personal news agent that talks, learns and explains. (Proceedings of the third annual conference on Autonomous Agents, Seattle, Washington, USA, 1999).

DOI: 10.1145/301136.301208

[17] Ardissono, L., A. Goy, G. Petrone, M. Segnan and P. Torasso: Intrigue: Personalized recommendation of tourist attractions for desktop and hand held devices. Applied Artificial Intelligence Vol. 17, 8-9(2003), P. 687-714.

DOI: 10.1080/713827254

[18] Zeithaml, V. A., L. L. Berry and A. Parasuraman: The behavioral consequences of service quality. The Journal of Marketing Vol., (1996), P. 31-46.

DOI: 10.1177/002224299606000203

[19] Knijnenburg, B. P., M. C. Willemsen and S. Hirtbach: Receiving recommendations and providing feedback: The user-experience of a recommender system E-Commerce and Web Technologies Vol. 61, (2010), P. 207-216.

DOI: 10.1007/978-3-642-15208-5_19

[20] Legris, P., J. Ingham and P. Collerette: Why do people use information technology? A critical review of the technology acceptance model. Information & Management Vol. 40, 3(2003), P. 191-204.

DOI: 10.1016/s0378-7206(01)00143-4

[21] Lee, Y., K. A. Kozar and K. R. T. Larsen: The technology acceptance model: Past, present, and future. Communications of the Association for Information Systems Vol. 12, 50(2003), P. 752-780.

DOI: 10.17705/1cais.01250