@Sami Ahmad You'll want to use beeline going forward since the Hive CLI will be deprecated. Beeline's JDBC connection provides for a higher level of security than the Hive CLI. ... View more

@Isa Mllr See this version matrix https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-component-versioning ... View more

Hi @Dagmawi Mengistu. Make sure you setup Hive proxy as described here http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-views/content/setup_HDFS_proxy_user.html ... View more

Hi @Adnan Alvee. Having already narrowed performance issues down to a particular job, the primary thing you will want to look for within each stage is skew. Skew is when a small number of tasks take significantly longer to execute than others. You'll want to look specifically at task runtime to see if something is wrong. As you drill down into slower running tasks you'll want to focus on where they are slow. Is the slowness in writing data, reading data, or computation? This can narrow things down to a particular problem with a node or maybe you don't have enough disks to handle the scratch space for shuffles. Keep in mind that Spark scales linearly. Your processing may be slow simply due to not enough hardware. You'll want to focus on how much memory and cpu you've allocated to your executers as well has how many disks you have in each node. It also looks as if your #executers is quite large. Consider having fewer executors with more resources per executor. Also the executor memory is memory per executor. The number you have is a bit large. Try playing with those numbers and see if it makes a difference. Hope this helps. ... View more

Thanks @Andrew Grande! That worked! I feel like a noob 🙂 but appreciate all the help! ... View more

Here is the repeating error in nifi-app.log.nifi-app.txt ... View more

I apologize. The actual error is "File name is too long". The name of the file is COWFILE1.DAT and COWFILE1.ARC. ... View more