There are many etl tools in market. I did some research on Pentaho Data Integraion Tool. In which kettle is for ETL.
How can we compare kettle with hadoop ?
Does kettle(any ETL tool) replaces hadoop ?
When do we need ETL tool on the top of hadoop ?
Please someone help me to clear my doubts.
Below I have listed a few differences between Hadoop Kettle.
Kettle (K.E.T.T.L.E - Kettle ETTL Environment) has been as of late aquired by the Pentaho gathering and renamed to Pentaho Data Integration. Kettle is a main open source ETL application available. It is delegated an ETL instrument, anyway the idea of exemplary ETL process (extricate, change, load) has been somewhat adjusted in Kettle as it is made out of four components, ETTL, which represents:
Data extraction from source databases
Transport of the data
Stacking of data into a data warehouse
Kettle is a lot of devices and applications which permits data controls over various sources.
The fundamental parts of Pentaho Data Integration are:
Spoon - a graphical device which make the plan of an ETTL procedure transformations simple to make. It plays out the common data stream capacities like perusing, approving, refining, changing, composing data to a wide range of data sources and goals. Tranformations structured in Spoon can be run with Kettle Pan and Kitchen.
Pan - is an application devoted to run data transformations planned in Spoon.
Chef - an instrument to make employments which computerize the database update process in a mind boggling way
Kitchen - it's an application which executes the occupations in a clump mode, for the most part utilizing a calendar which makes it simple to begin and control the ETL handling
Carte - a web worker which permits far off checking of the running Pentaho Data Integration ETL forms through an internet browser.
Hadoop is an Apache Project to give a structure to handling disseminated data utilizing a capacity reflection ( HDFS) and a preparing deliberation ( Map-Reduce).
ETL then again is a Data ingestion/Data development idea that began alongside a need of various organizations to fabricate business knowledge/DW backends to quantify/settle on choice on different parts of business forms.
On a superficial level Hadoop is totally unique in relation to doing ETL yet three things occurred:
After the underlying bedlam of too many open source instruments that professed to cooperate with Hadoop and vowed to process data quicker than everything else, some blurred into insensibility ( those old Pig contents … ) and some rose as true norm for data preparing ( Spark, Kafka, Cassandra … most likely hardly any more)
Web/Mobile data and next period of website detonated. ETL was being utilized to peruse/coordinate data sources whose endpoint was regularly static and all the transformation/change and DW load were clumped( for example expected to execute at certain recurrence) . The data produced by a 24x7 cooperation of clients and business through sites/application made it vital for certain utilization cases (, for example, : identifying peculiarity in bank logging endeavor, route, rich media utilization conduct and so forth) to be estimated continuously.
The operational cost/thought of improvement and sending of ETL and DW got simpler with foundation going to cloud and organization escaping a devoted framework group into designer's hand by ideals of agreeable intends to containerize any ETL application and by being able to arrangement a completely practical bunch/worker/run time env with scarcely any lines of code.
So at long last, not Hadoop however the various instruments based on worldview of Hadoop( and cloud contributions), sort of made conventional ETL old.
Apparently, there are organizations despite everything stayed with on premise data focus and with outsider ETL device created codebase. However, that number is in decrease and I don't perceive any practical explanation behind an opposite in this pattern.