Massive Data Analysis Based on Hadoop Distributed Parallel Computing Framework

Ling Leng; Lin Wang; Ben Lai  Liang; Jiang De  Yu

doi:10.4028/www.scientific.net/AMM.380-384.2765

Paper Titles

A Study on Distributed License Authorization System Based on Threshold Secret Sharing
p.2749

Fuzzy Amendments of Trust Model with Intrusion Tolerance
p.2753

Test Paper Generating Algorithm Based on Potential Weaknesses of Knowledge Points Mined by Association Rule
p.2757

Optimization of Object-Based Knowledge Mesh Structure Based on Time Performance
p.2761

Massive Data Analysis Based on Hadoop Distributed Parallel Computing Framework
p.2765

A Complete Performance Evaluation Indexs of Survivable System
p.2770

Online Self-Test System Grouping Function of Genetic Algorithm-Based Optimization
p.2776

Construction of Knowledge Model in Collaborative Learning System
p.2781

Based on the Characteris of Learners Personalized-Learning System Design
p.2786

HomeApplied Mechanics and MaterialsApplied Mechanics and Materials Vols. 380-384Massive Data Analysis Based on Hadoop Distributed...

Massive Data Analysis Based on Hadoop Distributed Parallel Computing Framework

Abstract:

With the development of Internet, Mobile Internet and The Internet of things, no one can deny that the era of massive data has come. Hadoop distributed system provides reliable parallel computing service by cheap PC clustering. It has irreplaceable advantages in scalability, robustness, calculated performance and cost. In fact, it has been a mainstream analysis platform of big data. In this paper, the classification characteristics of big data analysis were enumerated, the Pig platform system architecture was explained mainly. The functions of Pig platform engine front and after were introduced exactly. When Pig platform processes Pig Latin script, logic plan, physical plan and MapReduce plan will be produced. The concrete process was illustrated combined with examples in paper.

You might also be interested in these eBooks

View Preview

Info:

Periodical:

Applied Mechanics and Materials (Volumes 380-384)

Pages:

2765-2769

DOI:

https://doi.org/10.4028/www.scientific.net/AMM.380-384.2765

Citation:

Cite this paper

Online since:

August 2013

Authors:

Ling Leng, Lin Wang, Ben Lai Liang, Jiang De Yu

Keywords:

Hadoop Distributed System, Massive Data Analysis, Pig Platform

Export:

RIS, BibTeX

Price:

Permissions CCC:

Request Permissions

Permissions PLS:

Request Permissions

Сopyright:

Citation:

References

[1] A. Gates,O. Natkovich etc. Building a high-level dataflow system on top of Map-Reduce: The Pig experience. In Proc. of VLDB, 2009: 1414-1425).

DOI: 10.14778/1687553.1687568

Google Scholar

[2] C. Olston, B. Reed, U. Srivastava,R. Kumar and A. Tomkins. Pig Latin: A not-so-foreign language for dataprocessing. In Proc. of ACM SIGMOD, (2008).

DOI: 10.1145/1376616.1376726

Google Scholar

[3] Li Boduo, Mazur Edward, Diao Yanlei , McGr egor Andrew , Shenoy Prashant J. A platform for scalable one-pass analytics using MapReduce/Proceedings of the ACM SIGMOD International Conferen ce on Management of Dat a( SIGM OD' 11) . Athens , Greece, 2011: 985-996.

DOI: 10.1145/1989323.1989426

Google Scholar

[4] Nykiel T , Potamias M , Mishra C , Kollios G, Koudas N. MRShar e: Sharing across multiple queries in MapRedu ce. PVLDB, 2010, 3( 1) : 494-505.

DOI: 10.14778/1920841.1920906

Google Scholar

[5] http: / / www . asterdata. com / product / mapreduce. php.

Google Scholar

[6] http: / / www . greenplum. com/ technology/ mapreduce.

Google Scholar

[7] http: / / hive. apache. org.

Google Scholar