Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

monitoring job failed from time to time on unable to create userchache dir for the application

Highlighted

monitoring job failed from time to time on unable to create userchache dir for the application

Super Collaborator

Hi,

 

I have a monitoring job that from time to time it failed on unable to create userchache dir for the application:

 

example of the erros:

 

Diagnostics: File file:/liveperson/data/server_hdfs/data/disk1/yarn/nm/usercache/cloudera-scm/appcache/application_1491680609319_95697 does not exist
Failing this attempt. Failing the application.

 

Diagnostics: mkdir of /liveperson/data/server_hdfs/data/disk7/yarn/nm/usercache/cloudera-scm/appcache/application_1491680609319_93673/container_e14_1491680609319_93673_02_000001/tmp failed
Failing this attempt. Failing the application.

 

 

I checked the server and don't see any file system issue, and once i tried to banigate the logs i got that no logs available for teh container.

 

Any help is much apprecoiated.

 

 

1 REPLY 1

Re: monitoring job failed from time to time on unable to create userchache dir for the application

Super Collaborator

The root cause was inode issue on specific disks which its defintion was small than other disks on the same server. worked with the system team and fixed the inodes definiton on the specific disks.