I was getting the same error. I solved it by creating the directory on all the machines. (In my case a 3 nodes cluster). So, please make sure that you have "/var/log/spark2/lineage" directory present on all the machines. Also, the ownership should be as follows: chown spark:spark /var/log/spark2/lineage ... View more

In your cluster, check on exactly which node you have your hive service running. On that node, you will find log files at "/var/log/hive" location. ... View more