yarn-client is not supported, only yarn-cluster is supported ... View more

@elkan li which user are you logged in as when launching? Can you try an su hive then launch the hive cli? It seems the issue is with the launching user ... View more

@sagar pavan the Diagnostic message indicates the Users AM resource limit is exceeded. Please review the capacity scheduler's AM resource limit and raise it from the default 20%, this should allow the AM container to be launched ... View more

This does not look like an issue with the jar being included but rather an issue with the import statement. I breifly looked on google and see similar descriptions stating to try org.mongodb. I would focus on the import statement more than the inclusion of the jar for livy. ... View more

@Mickaël GERVAIS check to make sure livy interpreter is listed in the interpreter bindings for the notebook. Also, set DEBUG on the livy server and check in the livy out file produced on the server. Finally, make sure you have restarted livy and zeppelin to pick up the changes. I tested and it did work for me. ... View more

The jars should be able to be added by using the parameter key livy.spark.jars and pointing to an hdfs location in the livy interpreter settings. This does not seem to work. I had to place the needed jar in the following directory on the livy server: /usr/hdp/2.5.3.0-37/livy/repl-jars ... View more

That is not a good solution as you are saying yarn does not need any healthy disks to function. You essentially disable the health check with this. There is an underlying problem with the disks and that is the reason for it being marked unhealthy. ... View more

@Mourad Chahri You can go to the ResourceManager UI. From there you should see a nodes link on the left side of the screen. If you click on that, you should see all of your NodeManagers and the reason for it being listed as unhealthy may be shown here. It is most likely due to yarn local dirs or log dirs. You may be hitting the disk threshold for this. There are a couple of parameters you can check for this. yarn.nodemanager.disk-health-checker.min-healthy-disks yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb Finally, if that does not reveal the issue, you should look in /var/log/hadoop-yarn/yarn. Your previous comment shows you were looking in /var/log/hadoop/yarn which is not where the NodeManager log is located. I hope this helps. ... View more

@rama These directories are used by yarn for job logs. There are similar directories used for localization, the yarn-local dirs. They are not distributed so much as used when containers are allocated on that node. They should get cleaned up when jobs complete but can leave orphaned files in the event of a Resource Manager restart or a Node Manager restart. The directories are configured via yarn and it is a comma separated list of locations so you can add additional mounts/directories but they will apply to all node managers managed by Yarn. Hope this helps. ... View more

oops, your input is slashes, not dashes. Can you try with the needed format? ... View more

@Mayank Pandey what does the following produce? select log_source_time, from_unixtime(unix_timestamp(substr(log_source_time,0,11),'dd-MMM-yyyy')) as todateformat from table1 limit 2; ... View more

@Mayank Pandey there are some ways of converting the date. For example something like select inp_dt, from_unixtime(unix_timestamp(substr(inp_dt,0,11),'dd-MMM-yyyy')) as todateformat from table; There are several ways to attempt this if you do a google search on your needs. ... View more

@mike harding to add to this, Tez by default first initializes an AM whereas MapReduce does so at submission only. This is the reason you see the behavior you describe. The tez container has a timeout setting as you stated and that will determine how long lived that initial AM is ... View more