A MapReduce Iteration Framework in Local Parallel and Message Synchronization

Abstract:

With the development of large-scale distributed computing, Stand-alone operating environment to meet the demand of the time and space overhead of massive data based on. There is more attention to how to design the distributed algorithm for efficient cloud computing environment. The MapReduce model cannot solve the issue. In this paper, the redesign of the computing model of MapReduce, ensure the existing calculation models compatible with the old MapReduce operation. At the same time, the framework used the message synchronization mechanism to implement state data changing interaction tasks in Parallel Layer. Compared to the original MapReduce operation, greatly reduces the processing time of the MapReduce iterative algorithm.

• Periodical: Applied Mechanics and Materials (Volumes 380-384)
• Pages: 2237-2241
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.380-384.2237
• Online since: August 2013
• Authors: Ting Wang, Hua Liang Zhang, Peng Zeng
• Keywords: Barrier Grid, Local Parallel, MapReduce, Message Synchronization, Parallel Layer

References

[1] Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters" Operating System Design and Implementation (OSDI, 04), 2004, pp.137-150.

[2] Bu Y, Howe B, Balazinska M, Ernst MD. HaLoop: Efficient iterative data processing on large clusters/ Proceedings of the Conference on Very Large Databases(VLDB'10). Singapore, 2010: 285~296.

DOI: 10.14778/1920841.1920881

[3] Ekanayake J, Li H, Zhang B, Gunarathne T, Bae SH, Qiu J, Fox G. Twister: A runtime for iterative MapReduce, International Symposium on High Performance Distributed Computing, 2010, pp.810-818.

DOI: 10.1145/1851476.1851593

[4] Chen R, Weng X, He B, Yang M. Large graph processing in the cloud" International Conference on Management of Data(SIGMOD, 10), 2010, p.1123~1126.

[5] Lin J, Schatz M. Design patterns for efficient graph algorithms in MapReduce" Mining and Learning with Graphs (MLG, 10), 2010, pp.78-85.

DOI: 10.1145/1830252.1830263

[6] Lin J, Dyer C. Data Intensive Text Processing with MapReduce, (2010).

[7] Seo S, Yoon EJ, Kim J, Jin S, Kim JS, Maeng S. HAMA: An efficient matrix computation with the MapReduce framework" Cloud Computing Technology and Science Conference (CloudCom, 10), 2010, pp.721-726.

DOI: 10.1109/cloudcom.2010.17

[8] Low Y, Bickson D, Gonzalez J, Guestrin C, Kyrola A, Hellerstein J. GraphLab: A new framework for parallel machine learning" Uncertainty in Artificial Intelligence (UAI, 10), (2010).

DOI: 10.14778/2212351.2212354

[9] Hoefler T, Lumsdaine A, Dongarra J. Towards efficient MapReduce using MPI" European PVM/MPI Users, Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, 2009, pp.240-249.

DOI: 10.1007/978-3-642-03770-2_30

[10] Plimpton Steven J, Devine Karen D. MapReduce in MPI for large scale graph algorithms, Parallel Computing, 2011, 37 (9), pp.610-632.

DOI: 10.1016/j.parco.2011.02.004

[11] Albrecht Jeannie, Tuttle Christopher, Snoeren Alex C, Vahdat Amin. Loose synchronization for large scale networked systems" the Annual Conference on USENIX'06 Annual Technical Conference (ATEC, 06), 2006, p.28.

[12] Karloff H, Suri S, Vassilvitskii S. A model of computation for MapReduce" the 21st Annual ACMSIAM Symposium on Discrete Algorithms (SODA, 10), 2010, pp.938-948.

DOI: 10.1137/1.9781611973075.76