A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS

Abstract:

Article Preview

With the cloud computing is becoming mature, many of its characteristics for meteorological data processing is extremely important. Since HDFS is designed for reading and writing large files, it’s difficult to be taken advantage of small meteorological files. In this paper, an improved approach on HDFS is proposed for small meteorological files, small files are to be merged, indexed, and blocks are compressed, the pressure of memory on master node occupied by metadata is relieved, the speed of reading and writing small files is increased, read speed is increased by 50%, and write speed is up to 3-4 times of the original, saving about 2/3 of storage space and computing performance has also been improved. Thus, meteorological data processing can make use of cloud computing platform more closely.

Info:

Periodical:

Edited by:

Huixuan Zhang, Ye Han, Fuxiao Chen and Jiuba Wen

Pages:

1759-1765

DOI:

10.4028/www.scientific.net/AMM.117-119.1759

Citation:

S. J. Xue et al., "A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS", Applied Mechanics and Materials, Vols. 117-119, pp. 1759-1765, 2012

Online since:

October 2011

Export:

Price:

$35.00

In order to see related information, you need to Login.

In order to see related information, you need to Login.