Paper Title:
A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS
  Abstract

With the cloud computing is becoming mature, many of its characteristics for meteorological data processing is extremely important. Since HDFS is designed for reading and writing large files, it’s difficult to be taken advantage of small meteorological files. In this paper, an improved approach on HDFS is proposed for small meteorological files, small files are to be merged, indexed, and blocks are compressed, the pressure of memory on master node occupied by metadata is relieved, the speed of reading and writing small files is increased, read speed is increased by 50%, and write speed is up to 3-4 times of the original, saving about 2/3 of storage space and computing performance has also been improved. Thus, meteorological data processing can make use of cloud computing platform more closely.

  Info
Periodical
Chapter
Chapter 8: Material Processing Technology
Edited by
Huixuan Zhang, Ye Han, Fuxiao Chen and Jiuba Wen
Pages
1759-1765
DOI
10.4028/www.scientific.net/AMM.117-119.1759
Citation
S. J. Xue, W. B. Pan, W. Fang, "A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS", Applied Mechanics and Materials, Vols. 117-119, pp. 1759-1765, 2012
Online since
October 2011
Export
Price
$32.00
Share

In order to see related information, you need to Login.

In order to see related information, you need to Login.

Authors: De Jiao Niu, Tao Cai, Yong Zhao Zhan, Shi Guang Ju
Chapter 10: Computer-Aided Design and Simulation
Abstract:The efficiency of metadata indexing is important to the performance of distributed file system. Time and space spending of current metadata...
864
Authors: Li Qun Luo, Si Jin He
Chapter 5: Information Processing and Computational Science
Abstract:The advent of cloud is drastically changing the High Performance Computing (HPC) application scenarios. Current virtual machine-based IaaS...
677