at the end it turned out not to be a enctype issue 😉   The problem was that the hostname of the solr server in the curl URL did not match the hostname part of solr's kerberos principal. After putting the FQDN in the call: curl --negotiate -u : 'http://mgr-node1.test.demo:8983.........   the error disappeard and everything works nice. ... View more

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded   Increase the container and heap size.  I am not sure whether it is a mapper or reducer that is failing but here are the settings to look into.   set hive.exec.reducers.bytes.per.reducer= set mapreduce.map.memory.mb= set mapreduce.reduce.memory.mb= set mapreduce.map.java.opts=<roughly 80% of container size> set mapreduce.reduce.java.opts=<roughly 80% of container size> ... View more

man took a bit of trial and error. The issue with the first run is that it is returning an empty line. I tried a few awk specific was to get around it but they didn't work. So here is a hack. And using the variable withing awk as well.   DC=PN hdfs dfs -ls /lib/ | grep "drwx" | awk '{system("hdfs dfs -count " $8) }' | awk '{ gsub(/\/lib\//,"'$DC'"".hadoop.hdfs.",$4); print $4 ".folderscount",$1"\n"$4 ".filescount",$2"\n"$4 ".size",$3;}' PN.hadoop.hdfs.archive.folderscount 9 PN.hadoop.hdfs.archive.filescount 103 PN.hadoop.hdfs.archive.size 928524788 PN.hadoop.hdfs.dae.folderscount 1 PN.hadoop.hdfs.dae.filescount 13 PN.hadoop.hdfs.dae.size 192504874 PN.hadoop.hdfs.schema.folderscount 1 PN.hadoop.hdfs.schema.filescount 14 PN.hadoop.hdfs.schema.size 45964 DC=VA hdfs dfs -ls /lib/ | grep "drwx" | awk '{system("hdfs dfs -count " $8) }' | awk '{ gsub(/\/lib\//,"'$DC'"".hadoop.hdfs.",$4); print $4 ".folderscount",$1"\n"$4 ".filescount",$2"\n"$4 ".size",$3;}' VA.hadoop.hdfs.archive.folderscount 9 VA.hadoop.hdfs.archive.filescount 103 VA.hadoop.hdfs.archive.size 928524788 VA.hadoop.hdfs.dae.folderscount 1 VA.hadoop.hdfs.dae.filescount 13 VA.hadoop.hdfs.dae.size 192504874 VA.hadoop.hdfs.schema.folderscount 1 VA.hadoop.hdfs.schema.filescount 14 VA.hadoop.hdfs.schema.size 45964 ... View more