ozone FS datanodes disk space full

[#1]

ozone FS datanodes disk space is getting occupied immediately even after extend the disk space without any operations performed.

[#2]

even container balancer is not helping to distributed the data in nodes

[root@invcdp91dei01 ~]# ozone admin containerbalancer start
Container Balancer started successfully.

[#3]

Hi @satvaddi 

Thanks for connecting with cloudera,

could you please let me know the current CDP version and whats the size of disk you provide for ozone, open your Recon webui and check if container has been created and it's the one storing the disk size i suspect this should be the case

[#4]

cdp version used - cdh7.1.9.p1000 

tmp volume is showing as disk space occupied. 

[root@invcdp91dei01 ~]# hadoop fs -du -h ofs://ozoneid/tmp
0 0 ofs://ozoneid/tmp/.cloudera_health_monitoring_canary_files
0 0 ofs://ozoneid/tmp/hive

[#5]

ozone admin namespace du ofs://<ozone nameservice>  please use this command

[#6]

/tmp is occupied more space, please share command to cleanup /tmp

[root@invcdp91dei01 ~]# ozone admin namespace du ofs://ozoneid/tmp
Connecting to Recon: http://invcdp91dei04.informatica.com:9888/api/v1/namespace/du?path=/tmp ...

Path : /tmp
Total Size : 562 GB

DU
Size Path Name
405 GB /tmp/24c9e15e52afc47c225b757e7bee1f9d/
134 GB /tmp/5af6b2617513b81608501adea4166625/
459 MB /tmp/8a4ac216fb230da3834de641b3e5d0f7/
0 bytes /tmp/adpqa/
0 bytes /tmp/bb3462b62cd8db3f9ba007d86f8d1c6d/
21 GB /tmp/bc3dcd485adfa1c339eab38f1516c6c5/
0 bytes /tmp/d58da82289939d8c4ec4f40689c2847e/
0 bytes /tmp/tmp/

[#7]

COuld you please confirm what is below in tmp dir looks like container to me, you can confirm the same form recon ui and find out the path it belongs to

405 GB /tmp/24c9e15e52afc47c225b757e7bee1f9d/
134 GB /tmp/5af6b2617513b81608501adea4166625/

21 GB /tmp/bc3dcd485adfa1c339eab38f1516c6c5/

[#8]

do not find such folders in /tmp

405 GB /tmp/24c9e15e52afc47c225b757e7bee1f9d/
134 GB /tmp/5af6b2617513b81608501adea4166625/

21 GB /tmp/bc3dcd485adfa1c339eab38f1516c6c5/

[#9]

bc3dcd485adfa1c339eab38f1516c6c5 >> These alpha numeric related to tablet from kudu, region from habse or container for ozone, did you get a chance to check recon ui