@Aravindan Vijayan, Clearing the contents of /var/lib/ambari-metrics-collector/hbase-tmp/* and restarting the Ambari metrics collector totally fixed our problem on HDInsight Hadoop 3.6.1 (HDP 2.6.5). It was a little confusing because HBase is not deployed on the Azure Hadoop 3.6.1 cluster. Evidently, Ambari Metrics is running it's own internal HBase instance? Fyi...The error in /var/log/ambari-metrics-collector/ambari-metrics-collector.log was... 21:09:16,997 INFO [main-SendThread(xxxx.cx.internal.cloudapp.net:61181)] ClientCnxn:1032 - Opening socket connection to se rver xxx.cx.internal.cloudapp.net/xx.xxx.xx.xx:61181. Will not attempt to authenticate using SASL (unknown error) 21:09:16,998 WARN [main-SendThread(xxx.cx.internal.cloudapp.net:61181)] ClientCnxn:1162 - Session 0x16886b291270010 for s erver null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Thanks ... View more

This can be done without first writing to local disk... echo "`date` hi" | hdfs dfs -appendToFile - /tmp/abc.txt The hyphen "-" reads from stdin ... View more

I too had this problem. My bash script worked fine in my DEV environment as a background job and as a foreground job. However, in my TEST environment the job would only run as a foreground job. In TEST, running as a nohup job would seem to stop at the point where my Sqoop step was called. Ultimately I came across this thread which pointed me in the right direction. Essentially you can emulate nohup by "daemonizing" your script. setsid ./sqoop.sh </dev/null &>myLog.out & ... View more

If you don't already have a table with at least one row, you can accomplish the desired result without creating a dummy table. insert into test_array select 1, array('a','b') from (select '123') x; ... View more

Guilherme's solution worked for me on the HDP 2.5 sandbox VM when I used the hive-client directory instead of the hive-server2 directory. Create directory /usr/hdp/current/hive-client/auxlib/ Move jar files to that directory Restart Hive ... View more

For those interested in viewing column level stats try this... analyze table orderdetails compute statistics for columns; describe formatted orderdetails.unitprice; col_name data_type min max num_nulls distinct_count avg_col_len max_col_len num_trues num_falses comment unitprice double 2.0 26.3 0 127 foo ... View more