@sushil nagur Some options: - Hive can be used and is a common pattern. Land the data in HDFS, and use HiveQL to cleanse, transform, into a Hive Table (e.g. ORC format). HBase can also be a target (or indeed Solr). - SparkSQL is often used to ingest data. Again, land in HDFS, and use SparkSQL to process and add to Hive/HBase tables. - HDF (Ni-Fi) is more of a stealth ETL Tool or simple event processing, but can perform a number of transforms (also includes an expression builder/language, and many out of the box processors for different sources/targets. - Pig can be used to build data pipelines. Sqoop can be used to extract data, but only performs basic transforms. - Hortonworks has an eco-system of Partners with ETL solutions (e.g. Syncsort, etc.). - Storm and Spark Streaming are options for streaming operations, can be use Kafka as a buffer. In terms of commercial ETL vs Open Source, it comes down to many points - requirements, budget, time, skills, strategy, etc. The commercial ETL tools are mature, and some have sophisticated functionality. transformations, and connectivity. Hortonworks partners with commercial ETL vendors when the scenario fits. In other scenarios, native HDP tooling (as listed above) is sufficient. HTH, Graham
... View more
