@Saurabh Kumar Then I can only think of increasing the yarn.nodemanager.log-dirs size by adding multiple mount points. But still i'm suspecting that something else is also occupying the space. ... View more

@Artem Ervits how this can be an answer? its doesn't provide any info on spark sql. ... View more

@Saurabh Kumar Can you please share the below parameters values? yarn.nodemanager.local-dirs hadoop.tmp.dir ... View more

Ok, So if I roughly calculate the the size vs no. of jobs i.e if each job generate 100MB of logs on each node then it means you can have up to 1000 jobs running at the same time. Is that the case?. 1. Either some other log occupied the space in same partition. 2. Or the yarn job logs are not getting cleaned up fast. 3. Or you have a big cluster where you are running hundreds of jobs with some extra debugging. If this is the case then you need to reorganized the logging confs and consider increasing/adding space in yarn.nodemanager.log-dirs partition. Can you share the disk usage of parent directories from that 100GB partition? @Saurabh Kumar ... View more

Can you please set below property in your workflow and see if that works? @Mark Thorson oozie.use.system.libpath=true ... View more

@Mark Thorson Hue and oozie server logs please? ... View more

@Mark Thorson Please follow below command and see if this resolve your issue. Reference doc HERE /usr/hdp/current/oozie/bin/oozie-setup.sh sharelib create -fs hdfs://<namenode>:8020 To verify that the sharelibs extracted correctly, run the following command: oozie admin -oozie http://<oozie server host address>:11000/oozie -shareliblist ... View more

@Saurabh Kumar ohh so you mean you have 100GB delicately on each NM node for yarn log and YARN log agg is also enabled and still you are facing this issue with local log dir? I think we need to check why that location is not getting cleared after job completion, may be something else occupying the space? ... View more