Reply
Expert Contributor
Posts: 244
Registered: ‎01-25-2017

monitoring job failed from time to time on unable to create userchache dir for the application

Hi,

 

I have a monitoring job that from time to time it failed on unable to create userchache dir for the application:

 

example of the erros:

 

Diagnostics: File file:/liveperson/data/server_hdfs/data/disk1/yarn/nm/usercache/cloudera-scm/appcache/application_1491680609319_95697 does not exist
Failing this attempt. Failing the application.

 

Diagnostics: mkdir of /liveperson/data/server_hdfs/data/disk7/yarn/nm/usercache/cloudera-scm/appcache/application_1491680609319_93673/container_e14_1491680609319_93673_02_000001/tmp failed
Failing this attempt. Failing the application.

 

 

I checked the server and don't see any file system issue, and once i tried to banigate the logs i got that no logs available for teh container.

 

Any help is much apprecoiated.

 

 

Highlighted
Expert Contributor
Posts: 244
Registered: ‎01-25-2017

Re: monitoring job failed from time to time on unable to create userchache dir for the application

The root cause was inode issue on specific disks which its defintion was small than other disks on the same server. worked with the system team and fixed the inodes definiton on the specific disks.

Announcements