A Metadata Management Mechanism Based on HDFS

Abstract:

A metadata management mechanism on the basis of HDFS was proposed in this paper, which is applied for cloud storage. The basic unit of consistent hashing is DHU in the scheme. Metadata is distributed in multiple servers in this solution, and the position of metadata is recorded in a mapping table. In addition, prefetching strategy and local cache technology are used to improve the performance of the system. Operation logs and Multi-Paxos algorithm are being used to ensure system’s reliability and consistency. Improved system with a lower latency in reading and writing than the original system was seen in the experiments. Good performance for cloud storage system can be shown by this system.

• Periodical: Applied Mechanics and Materials (Volume 577)
• Pages: 1026-1029
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.577.1026
• Online since: July 2014
• Authors: Xiao Feng Chen*, Yuan Sheng Lou, Dong Mei Hu
• Keywords: Cloud Storage, Consistent Hashing, HDFS, Metadata

References

[1] Sinnamohideen S, Sambasivan R R, Hendricks J, et al. A transparently-scalable metadata service for the Ursa Minor storage system[C]/Proceedings of the USENIX Annual Technical Conference (USENIX ATC'10), Boston MA. (2010).

[2] Feng D, Wang J, Wang F, et al. DOIDFH: an effective distributed metadata management scheme[C]/Computational Science and its Applications, 2007. ICCSA 2007. International Conference on. IEEE, 2007: 245-252.

DOI: 10.1109/iccsa.2007.22

[3] Xiong J, Hu Y, Li G, et al. Metadata distribution and consistency techniques for large-scale cluster file systems[J]. Parallel and Distributed Systems, IEEE Transactions on, 2011, 22(5): 803-816.

DOI: 10.1109/tpds.2010.154

[4] Hua Y, Jiang H, Zhu Y, et al. Semantic-aware metadata organization paradigm in next-generation file systems[J]. Parallel and Distributed Systems, IEEE Transactions on, 2012, 23(2): 337-344.

DOI: 10.1109/tpds.2011.169