For a comparison between compression formats take a look at this link: http://comphadoop.weebly.com/ ... View more

Please share a link to the tutorial you're using ... View more

@Varun R Take a look at the below articles that cover Tez performance tuning as well as an overview of how it works. https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html ... View more

@subash sharma Which version of HDP are you using? Even though Column Level Lineage is advertised as available with Atlas 0.8 in the Apache page, it has only been made GA with HDP 2.6.1 rather than HDP 2.6.0. The delay was to ensure it works with the other appropriate HiveQL commands beyond CTAS. ... View more

Adding to Sonu's response: Moving to Hadoop from BI/EDW background is certainly a very common path. Those coming from that background usually find themselves more comfortable with Hive as an entry point. Hive provides an abstraction layer on top of MapReduce/Tez, and is based on SQL-like syntax that is ANSI compliant. It also has the advantage of providing a JDBC/ODBC connector, so most of the industry BI tools such as Tableau, Qlik, Microstrategy, etc.. can integrate and interact with Hive. This means that business analysts may continue to use the tools they are already familiar with while leveraging the power of Hadoop in the background I would recommend you start by looking at Hive. Once comfortable with it, you can start to explore Hive data modelling and optimization, and then branch out to the other areas that Sonu recommended. I've also seen people in the field focus their entire career/job around just Hive. Take a look at the link below for an introduction to Hive. There's plenty of internet resources and books that you can leverage to advance your knowledge. Hortonworks also provides Developer Training that covers an introduction to Hive as well as other engines/tools. https://hortonworks.com/tutorial/how-to-process-data-with-apache-hive/ ... View more

@Smart Data I am assuming that you are trying to create entities and lineage for HDFS files. If so, then yes, you would need to use the REST API to create the lineage. You can use the API to create the entities themselves rather than going through Kafka. If you're using HDP 2.6.1, you can also create your entities through the Atlas UI as per the link below. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-governance/content/ch_atlas_searching_and_viewing_entities.html#atlas_manually_creating_entities Finally, below is a step-by-step example of creating entities and lineage for an HDFS file that is picked and processed by Spark and the results written back to HDFS. It will give you a good idea of how the APIs may be leveraged. https://community.hortonworks.com/content/kbentry/91237/creating-custom-types-and-entities-in-atlas.html ... View more