In the HDP 2.5 release notes it says that Hive 2.1 is TP http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_release-notes/content/tech_previews.html Hive New in this release: Stored Procedures Client-Side LLAP Daemons (HIVE-7193) LLAP / Spark security interface Hive 2.1 In the HDP 2.5 Release email to customers it was stated that Hive 2.1 is TP but that Hive ACID is certified for production with Hive 1.2.1 Apache Hive Includes Apache Hive 1.2.1 for production and Hive 2.1 (Technical Preview) for cutting-edge performance Hive LLAP (Technical Preview): Persistent query servers and optimized in-memory caching for blazing fast SQL. Up to 25x faster for BI workloads. 100% compatible with existing Hive workloads Hive ACID and Streaming Ingest certified for production use with Hive 1.2.1 Dynamic user-based security policies for data masking and filtering HPL/SQL: Procedural programming within Hive Hive View v1.5.0, improved robustness and security Parquet format fully certified with Hive 1.2.1 / 2.1 In the Hortonworks.com Hive Overview section it states (confusingly) that ACID is GA in Hive 2.1 (though originated in 0.14) http://hortonworks.com/apache/hive/#section_3 ... View more

Syncsort DMX-h is an excellent ETL tool that is YARN-enabled http://www.syncsort.com/en/Products/BigData/DMXh ... View more

You need to take three approaches: minimize your data before join (e.g. load only columns needed for join and output, filter before join), then optimize your joins, then optimize settings (including compressing intermediate results) For 1, see: https://pig.apache.org/docs/r0.7.0/cookbook.html For 1 and 2, see: https://pig.apache.org/docs/r0.9.1/perf.html After performing these optimizations, for 3 see: http://chimera.labs.oreilly.com/books/1234000001811/ch08.html#pig_tuning Also, be sure you are running pig on Tez. ... View more