Effective ACPS-Based Rescheduling of Parallel Batch Processing Machines with MapReduce

Abstract:

MapReduce is a highly efficient distributed and parallel computing framework, allowing users to readily manage large clusters in parallel computing. For Big data search problem in the distributed computing environment based on MapReduce architecture, in this paper we propose an Ant colony parallel search algorithm (ACPSMR) for Big data. It take advantage of the group intelligence of ant colony algorithm for global parallel search heuristic scheduling capabilities to solve problem of multi-task parallel batch scheduling with low efficiency in the MapReduce. And we extended HDFS design in MapReduce architecture, which make it to achieve effective integration with MapReduce. Then the algorithm can make the best of the scalability, high parallelism of MapReduce. The simulation experiment result shows that, the new algorithm can take advantages of cloud computing to get good efficiency when mining Big data.

• Periodical: Applied Mechanics and Materials (Volume 575)
• Pages: 820-824
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.575.820
• Online since: June 2014
• Authors: Bin Zhang*, Jia Jin Le, Mei Wang
• Keywords: Ant Colony Parallel Search, Big Data, Hadoop, MapReduce

References

[1] Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters [C] Proc of 6th OSDI. SanFrancisco: USENIX Association, 2004: 137- 150.

[2] Dean J, Ghemawat S. Experiences with MapReduce: an abstract ion for large scale computation [C] Proc 15th Inter Conf on PACT. Washingt on DC, 2006: 1- 2.

[3] Dean J, Ghemawat S. MapReduce: a flexible data processing tool [J]. Communications of the ACM, 2010, 53: 72- 77.

DOI: 10.1145/1629175.1629198

[4] SHI Jin-gang, BAO Yu-bin, LENG Fang-ling, Y U Ge. Parallel Query for a Data Warehouse Utilizing MapReduce [J]. Journal of Northeastern University(Natural Science), 2011, 32(5) : 626-629.

[5] RONG Xiang , LI Ling juan. A method for frequent set mining based on MapReduce [J]. JOURNAL of XI AN UNIVERSITY of POSTS and TELECOMMUNICATIONS, 2011, 16(4) : 37-39.

[6] Dorigo M, Maniezzo V, Colorni A. Ant system: Optimization by a colony of cooperating agents [J]. Systems, Man and Cybernetics, Part B, IEEE Transactions on, 1996, 26(1) : 29-41.

DOI: 10.1109/3477.484436

[7] LIANG Jing , QIAN Xingsan, MA Liang. Two -level ant algorithm for the furnace batch scheduling in semi-conductor furnace operation [J]. Systems Engineering Theory & Practice, 2005, 25(12): 96-101.

[8] Cheng B Y, Chen H P, Shao H, et al. A chaotic ant colony optimization method for scheduling a single batch processing machine with non-identical job sizes[C]. IEEE Congress on Evolutionary Computation. Hong Kong : IEEE, 2008: 40-43.

DOI: 10.1109/cec.2008.4630773

[9] Li L, Qiao F, Wu Q D. ACO-based multi-objective rescheduling of parallel batch processing machines with advanced process control constraints [J]. International Journal of Advanced Manufacturing Technology, 2009, 44( 9-10) : 985-994.

DOI: 10.1007/s00170-008-1904-8

[10] White T. Hadoop the definitive guide [M]. Sebastpopol: Reilly Media Inc, 2009: 1- 13.

[11] Borthakur D. HDFS architecture guide [EB/OL]. (2010-08-17)[2010-12-01]. http: /hadoop. apache. org/hdfs/docs/r0. 21. 0/hdfs-design. pdf.

[12] Inmon W H. Building the Data Warehouse [M]. New York: John Wiley & Sons, Inc, 1993: 126- 133.

[13] Yang H C, Dasdan A, Hsiao R L. MapReduce merge simplified relational data processing on large clusters [C] Proc of 26th SIGMOD. Beijing, 2007: 1029- 1040.

DOI: 10.1145/1247480.1247602