Research on Storage Strategy of Unstructured Small Files in HDFS

Long Tao Wu; Tie Ning Wang; Hai Rong Hu

doi:10.4028/www.scientific.net/AMM.644-650.3053

Paper Titles

Analysis of College Students' Online Business in China
p.3036

Thought about the Construction of Digital Employment Information Service System of Rural Migrant Workers in the West Area of Jilin
p.3040

Design and Implementation on Sina Micro-Blog Client Based on the Android System
p.3045

On SOA Community Informationization Foundation Database Generic Interface Design
p.3049

Research on Storage Strategy of Unstructured Small Files in HDFS
p.3053

The Building of the Database of Art Resources Research for Academy of Fine Arts
p.3057

Research and Implementation of Auxiliary Teaching System Based on C/S Model
p.3061

The Improvement of the Public Service System Based on Web Technology
p.3065

Design of Extended Event Service Model Based on CORBA
p.3069

HomeApplied Mechanics and MaterialsApplied Mechanics and Materials Vols. 644-650Research on Storage Strategy of Unstructured Small...

Research on Storage Strategy of Unstructured Small Files in HDFS

Abstract:

With the wide use of HDFS and increasing scale of small files, problems of HDFS in small files storage gradually exposed. Thus the article put forward a storage strategy of unstructured small file based on the type of file, and optimized the architecture of cluster to save memory and improve the efficiency of file access. Through experiment, the strategy is proved to be effective and reliable.

You might also be interested in these eBooks

View Preview

Info:

Periodical:

Applied Mechanics and Materials (Volumes 644-650)

Pages:

3053-3056

DOI:

https://doi.org/10.4028/www.scientific.net/AMM.644-650.3053

Citation:

Cite this paper

Online since:

September 2014

Authors:

Long Tao Wu*, Tie Ning Wang, Hai Rong Hu

Keywords:

Hadoop, HDFS, Storage Strategy, Unstructured Small File

Export:

RIS, BibTeX

Price:

Permissions CCC:

Request Permissions

Permissions PLS:

Request Permissions

Сopyright:

Citation:

* - Corresponding Author

References

[1] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler. The hadoop distributed file system. Sunnyvale, California USA, IEEE 2010: 1-10.

DOI: 10.1109/msst.2010.5496972

Google Scholar

[2] GAO Ji-chao. Study and optimization of Hadoop storage strategy. Traffic university of Beijing, (2012).

Google Scholar

[3] WANG Hong-yu. The application of Hadoop in clouding computing. Software, 2011, 32(4): 36-39.

Google Scholar

[4] HONG Xu-sheng, LIN Shi-ping. MapFile-based HDFS small file storage efficiency, Computer system application, 2012, 21(11): 179-182.

Google Scholar

[5] LIU Xu-hui, Han Ji-zhong and ZHONG Yun-qin. Implementing Web GIS on Hadoop: A Case Study of Improving Small File I/O Performance on HDFS. Proceedings of 2009 IEEE Conference on Cluster Computing.

DOI: 10.1109/clustr.2009.5289196

Google Scholar

[6] MACKEY G, SEHRI S, WANG Jun. Improving metadata management for small files in HDFS. Proceedings of 2009 IEEE International Conference on Cluster Computing and Workshops.

DOI: 10.1109/clustr.2009.5289133

Google Scholar

[7] Information on http: /Hadoop. apache. org.

Google Scholar

[8] Information on http: /wiki. apache. org/Hadoop/SequenceFile.

Google Scholar

[9] LIU Xiao-jun, XU Zheng-quan. A small file storage strategy combined RDBMS and Hadoop. Journal of wuhan university, 2013, 38(1).

Google Scholar

[10] ZHANG Chun-ming, HE Ting-ting. A small file storage and reading method in Hadoop. Computer application and software, 2012, 29(11): 95-100.

Google Scholar