Support Questions
Find answers, ask questions, and share your expertise

The directory item limit of /tmp/logs/<<appln>>/logs is exceeded: limit=1048576 items=1048576

Explorer

We are using CDH 5.4.5 with parcels...

 

We get the following error in hdfs logs Pls suggest

 

 

2016-01-25 05:34:31,265 INFO org.apache.hadoop.ipc.Server: IPC Server handler 76 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from 10.80.34.22:42733 Call#3937671 Retry#0: org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException: The directory item limit of /tmp/logs/Appln/logs is exceeded: limit=1048576 items=1048576

2016-01-25 05:34:45,873 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<<appln>>@PROD.HADOOP.COM (auth:TOKEN) cause:org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException: The directory item limit of /tmp/logs/<<appln>>/logs is exceeded: limit=1048576 items=1048576

2016-01-25 05:34:45,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from 10.80.34.8:48244 Call#3987372 Retry#0: org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException: The directory item limit of /tmp/logs/<<appln>>/logs is exceeded: limit=1048576 items=1048576

2016-01-25 05:34:46,231 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<<appln>>@PROD.HADOOP.COM (auth:TOKEN) cause:org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException: The directory item limit of /tmp/logs/<<appln>>/logs is exceeded: limit=1048576 items=1048576

2016-01-25 05:34:46,231 INFO org.apache.hadoop.ipc.Server: IPC Server handler 119 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from 10.80.34.23:51099 Call#4006916 Retry#0: org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException: The directory item limit of /tmp/logs/<<appln>>/logs is exceeded: limit=1048576 items=1048576



[root@PROD tmp]# hadoop fs -copyFromLocal test /tmp/logs/<<appln>>/logs
copyFromLocal: The directory item limit of /tmp/logs/<<appln>>/logs is exceeded: limit=1048576 items=1048576
[root@PROD tmp]#

1 REPLY 1

Re: The directory item limit of /tmp/logs/<<appln>>/logs is exceeded: limit=1048576 items=1048576

Contributor

The error means maximum number of folders or file in one directory has reached its limit and thats when we see the error. 

 

"The directory item limit of /tmp/logs/Appln/logs is exceeded: limit=1048576 items=1048576"

 

There's a parameters which we can invoke in NameNode safety valves in order to increase this limit

 

-> dfs.namenode.fs-limits.max-directory-items determines the maximum number of folders or files (not recursive) in one directory. The value range of this parameter is 1 to 6400000, and the default value is 1048576.

-> After setting the above parameter, restart the NameNode to make changes come into effect and verify with  hdfs-site.xml file whether changes are imbibed.