Research on Hadoop Greedy Scheduler Based on the Fair

Abstract:

Job scheduling technology is one of the Hadoop platform’s key technologies, whose main function is to control the execute sequence of job and the distribution of computing resources, which directly relates to the Hadoop platform’s overall performance and system resources, usage. However, the existing job scheduling algorithms such as FIFO Scheduler, Fair Scheduler and Capacity Scheduler all have some defects. To overcome theses defects, this paper proposed a new algorithm Hadoop Greedy Scheduler Based on the Fair (HGSF). Firstly, the job pools are sorted by priority from high to low; pools in the same priority are sorted by their minimum requirements. Then their minimum requirements are met in turn. If have spare computing resources, it will be assigned to the job pool which has the highest priority and the minimum difference between the ideal requirement and the minimum requirement. Finally, the algorithm uses delay allocation strategy to improve the localization of the data for computing tasks. The experimental results testified the effectiveness of the proposed algorithm.

• Periodical: Applied Mechanics and Materials (Volume 145)
• Pages: 460-464
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.145.460
• Online since: December 2011
• Authors: Rui Chen, Wen Hua Zeng, Kai Ji Fan
• Keywords: FAIR, Greedy, Hadoop, Job Scheduling

References

[1] Information on http: /svn. apache. org/repos/asf/hadoop/mapreduce/trunk/src/contrib/fairscheduler/designdoc/fair_scheduler_design_doc. pdf.

[2] Information on http: /hadoop. apache. org/common/docs/r0. 19. 1/capacity_scheduler. html.

[3] Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmelegy, Scott Shenker, and Ion Stoica. Technical Report UCB/EECS-2009-55, EECS Department, University of California, Berkeley (2009).

[4] Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, Ion Stoica, in: Proceedings of the 5th European conference on Computer systems, Paris, France (2010).

DOI: 10.1145/1755913.1755940