A MapReduce Telecommunication Data Center Analysis Model

Abstract:

With the widely use of smart phone in China, all inputs and routes packets streams to the Content Distribution Service (CDS) switching centers. Each produces up to 1.5 terabytes arriving every day. Normally, the job of the switch is to transmit data. Obviously, the ordinary database cannot handle the massive dataset and complex ad-hoc query. In this paper, we propose DeepMR, a MapReduce deep service analysis system based on Hive/Hadoop frameworks. A distributed file system HDFS is used in DeepMR for fast data sharing and query. DeepMR also optimizes scheduling for switch analysis jobs and supports fault tolerance for the entire workflow. Our results show that the model achieves a higher efficiency.

• Periodical: Advanced Materials Research (Volumes 734-737)
• Pages: 2863-2866
• DOI: https://doi.org/10.4028/www.scientific.net/AMR.734-737.2863
• Online since: August 2013
• Authors: Wen Chuan Yang*, Jiang Yong Wang, Hao Yu Zeng
• Keywords: Big Data, Cloud Computing, Hadoop, MapReduce, Workflow

References

[1] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the ACM Symposium on Operating Systems Principles, 2003.

DOI: 10.1145/945445.945462

[2] A. AuYoung, L. Grit, J. Wiener, and J. Wilkes. Service contracts and aggregate utility functions. In Proceedings of the IEEE International Symposium on High Performance Distributed Computing (HPDC), June 2006.

DOI: 10.1109/hpdc.2006.1652143

[3] R. Avnur and J. M. Hellerstein. Eddies: Continuously adaptive query processing. In ACM SIGMOD: International Conference on Management of Data, 2007.

DOI: 10.1145/342009.335420

[4] R. E. Bryant. Data-intensive supercomputing: The case for DISC. Technical Report CMU-CS-07-128, Carnegie Mellon University, (2007)

[5] K. Cardona, J. Secretan, M. Georgiopoulos, and G.Anagnostopoulos. A grid based system for data mining using MapReduce. Technical Report TR-2007-02, AMALTHEA, 2007.

[6] B. N. Chun, P. Buonadonna, A. AuYoung, C. Ng, D. C. Parkes, J. Shneidman, A. C. Snoeren, and A. Vahdat. Mirage: A microeconomic resource allocation system for SensorNet testbeds. In Proceedings of the 2nd IEEE Workshop on Embedded Networked Sensors, (2005)

DOI: 10.1109/emnets.2005.1469095

[7] B. N. Chun and D. E. Culler. Market-based proportional resource sharing for clusters. Technical Report CSD-1092, University of California at Berkeley, Computer Science Division, January 2000.

[8] B. N. Chun and D. E. Culler. User-centric performance analysis of market-based cluster batch schedulers. In Proceedings of the 2nd IEEE International Symposium on Cluster Computing and the Grid, (2002)

DOI: 10.1109/ccgrid.2002.1017109