Research and Design of Data Processing Based on ETL Framework

Xiao Li Guo; Bo Chen

doi:10.4028/www.scientific.net/AMR.1049-1050.1966

Paper Titles

Projection Transformation on 3D Object’s Three Views
p.1950

Based on Simulink the MSK Communication System Design and Research
p.1954

An Online Balance Weigh Service Based on Web Service
p.1958

Build a Fully Distributed Hadoop Cluster Based on VM Scene
p.1962

Research and Design of Data Processing Based on ETL Framework
p.1966

Research on Software Testing to Ensure Web Application Usability, Reliability and Security
p.1972

Design and Implementation of Question and Answer System Based on Mobile Phone Platform
p.1977

Design and Implementation of Content Management System Based on J2EE
p.1981

Application of Computer Technology in Maize Production Implementation Precision Agriculture
p.1985

HomeAdvanced Materials ResearchAdvanced Materials Research Vols. 1049-1050Research and Design of Data Processing Based on...

Research and Design of Data Processing Based on ETL Framework

Abstract:

ETL is a key link in the construction of data warehouse. On the base of analyzing the mainstream ETL tool Datastage, the data extraction, transformation and loading, proposes a ETL framework based on data processing, and the realization method and steps are discussed in detail. The framework uses HIVE as a data processing station, improve the operating efficiency of the file; data task according to the E, T and L three parts and hierarchical partitioning, conversion of data users to better grasp the process; development data using the configuration file of the task, the development personnel free out from the heavy code, will to shift the focus of the work to the data logical task, which has greatly improved the efficiency of development personnel data processing.

You might also be interested in these eBooks

Modern Technologies in Materials, Mechanics and Intelligent Systems

View Preview

Info:

Periodical:

Advanced Materials Research (Volumes 1049-1050)

Pages:

1966-1971

DOI:

https://doi.org/10.4028/www.scientific.net/AMR.1049-1050.1966

Citation:

Cite this paper

Online since:

October 2014

Authors:

Xiao Li Guo*, Bo Chen

Keywords:

Data Conversion, Data Extraction, Data Loading, Data Processing, ETL Framework

Export:

RIS, BibTeX

Price:

Permissions CCC:

Request Permissions

Permissions PLS:

Request Permissions

Сopyright:

Citation:

* - Corresponding Author

References

[1] PVassiliadis, ASimitsis, PGeorgantas, MTerrovitis, SSkiadopoulos. Ageneric and customizable framework for the design of ETL scenarios[J]. Information Systems Journa, l2005, 30(7): 492-525.

Google Scholar

[2] PVassiliadis, eta. l ARKTOS: to wards the modeling, design, control and execution of ETL processes[J]. Information Systems, 2001, 26: 537-561.

DOI: 10.1016/s0306-4379(01)00039-4

Google Scholar

[3] Chen Xian, Chen Song Qiao. According to the design and realization of the in general use ETL tool of data warehouse[J]. Calculator application study, 2004, (8) : 214-216.

Google Scholar

[4] You Yu Lin, Zhang Xian Min. The ETL strategy and the structure design in a kind of dependable data warehouse[J]. Calculator engineering and application, 2006, 3: 172-175.

Google Scholar

[5] week luxuriant Wei, Deng Su, Huang Hong Bin. According to the ETL tool design and realization of dollar data[J]. Science technique and engineering, 2006, 6(21): 3503-3507.

Google Scholar

[6] Hong source, the week is good. According to the design and realization of CWM standard ETL[J]. University college journal(information science version) in Jilin, 2006, 1(24): 50-55.

Google Scholar

[7] JoseZubcof, fJuanTrujillo. AUML2. 0 profile to design Association Rulemining models in the multidimensional conceptual modeling of data warehouses[J]. Data&Knowledge Engineering? October2007, 63(1): 44-62.

DOI: 10.1016/j.datak.2006.10.007

Google Scholar