The Design and Implementation of Astronomical Multi-Catalog Storage and Cross-Match Based on Hadoop

Abstract:

With the development of astronomical observation techniques, telescopes around the world continue to release new spectral data. Due to the difference of geographical location and performance of variable telescopes, the catalog data contain different location information and different band information. In order to provide astronomers with more comprehensive astronomical information, celestial information of multi-band needs to be cross-matched. It’s necessary to use distributed and parallel computing techniques to process massive astronomical data. In this paper, we propose a solution to store and cross-match multi-catalog astronomical data.

• Periodical: Advanced Materials Research (Volume 1042)
• Pages: 121-125
• DOI: https://doi.org/10.4028/www.scientific.net/AMR.1042.121
• Online since: October 2014
• Authors: Xia Xu Zhang, Jing Chang Pan*
• Keywords: Cross-Match, Hadoop, HBase, Mass of Astronomical Catalog Data

References

[1] Du P, Ren J J, Pan J C, et al. New cross-matching algorithm in large-scale catalogs with ThreadPool technique [J]. Science China Physics, Mechanics and Astronomy, 2014: 1-7.

DOI: 10.1007/s11433-013-5161-y

[2] Song X, Han J Z, Wang K, et al. MapReduce for astronomical cross-matching [J]. Application Research of Computers, 2010, 10: 038.

[3] Gao D, Zhang Y X, Zhao Y H. The Realization of Cross-identification Based on Huge Multi-wavelength Catalog Data [J]. Publications of the National Astronomical Observatories of China, 2005, 2: 186-193.

[4] Du Peng. The Cross-match and Aggregation of Large-scal Astronomical Catalogs Data. MS thesis. Shandong University, (2013).

[5] Gorski K M, Hivon E, Banday A J, et al. HEALPix: A Framework for High-resolution distance and Fast Analysis of Data distributed on The Sphere [J]. The Astrophysical Journal, 2005, 622(2): 759.

DOI: 10.1086/427976

[6] Zhao Qing. Research on High-Efficient Massive Data Oriented Astronomical Cross-Match [D]. Tianjin University, (2010).

[7] Gao Dan. Development of Masive Astronomy Data Federation System and Research of Data Mining Algorithms [D]. Beijing: National Astronomical Observatories, Chinese Academy of Sciences, (2008).

[8] Dean J, Ghemawat S. MapReduce: Simplified Data Processing on Large Clusters [J]. Communications of the ACM, 2008, 51(1): 107-113.

DOI: 10.1145/1327452.1327492

[9] Zhao Q, Sun J, Yu C, et al. A Paralleled Large-scale Astronomical Cross-matching Function [M] Algorithms and Architectures for Parallel Processing. Springer Berlin Heidelberg, 2009: 604-614.

DOI: 10.1007/978-3-642-03095-6_57