Reply
Expert Contributor
Posts: 160
Registered: ‎09-29-2014

CDH 5.10.2 Impala bug

CDH: 5.10.2 i have found a bug of imapla, the total story is below: we all know we need to refresh metadata by invalidate metadata by impala, so impala can visit the newest table or data. two days ago, i have written a job using impala-shell to refresh the metadata , and deploy the job in oozie. but it cause catalogd process host's /tmp directory to full. the reason is because everytime to refresh the metadata, it will create 6 files by catalogd process, this file is temp file and shows to me it's deleted(lsof | grep "catalogd process " | grep deleted ), but actully, catalogd didn't release the file, and cause the /tmp directory to full. in order to confirm this is 5.10.2 impala bug, i am going to test in 5.14.2 , no any problem. 1. [root@xxxxxx~]# lsof | grep 8669 | grep deleted |wc -l 4819 2. [root@xxxxxx~]# lsof | grep 8669 | grep deleted | more catalogd 8669 impala 251r REG 253,0 9521 7077906 /tmp/c0153b57-c4fa-4699-b9ab-9ba913c1d8a3.jar (deleted) catalogd 8669 impala 253r REG 253,0 9521 7077912 /tmp/ce4084c7-cb39-45f2-8889-671526121083.jar (deleted) catalogd 8669 impala 254r REG 253,0 9521 7077916 /tmp/2d04c43b-eef6-4560-8ebb-e7bdb9db1a63.jar (deleted) catalogd 8669 impala 256r REG 253,0 9521 7077914 /tmp/ea84b7b7-ea85-4ada-8257-d8e0779d9c92.jar (deleted) catalogd 8669 impala 257r REG 253,0 9521 7077915 /tmp/a5c51c99-6399-46e0-931b-e2a116b7841f.jar (deleted) catalogd 8669 impala 267r REG 253,0 9521 7077917 /tmp/c0501b1a-567d-4e9a-a4b6-2db9aa4f3f2a.jar (deleted) catalogd 8669 impala 268r REG 253,0 9521 7077922 /tmp/0661490e-6820-4132-96b9-736c75818860.jar (deleted) catalogd 8669 impala 269r REG 253,0 9521 7077918 /tmp/d446013e-d23d-4cea-b3e3-3a1952a46da2.jar (deleted) catalogd 8669 impala 270r REG 253,0 9521 7077920 /tmp/081b6379-e23a-431e-8da4-8babc9189afc.jar (deleted) catalogd 8669 impala 271r REG 253,0 9521 7077923 /tmp/bb306bfc-0c25-43eb-a43f-9d9b3de07494.jar (deleted) catalogd 8669 impala 272r REG 253,0 9521 7077926 /tmp/f078ee37-cbfd-466b-be81-981be9c354c1.jar (deleted) catalogd 8669 impala 273r REG 253,0 9521 7077924 /tmp/cf500aae-ffca-4ad0-8ccb-35088b14dae0.jar (deleted) catalogd 8669 impala 274r REG 253,0 9521 7077925 /tmp/c5e66f4a-95c7-48ea-b1c2-7c81aac1ff5a.jar (deleted) who can tell me to fix this bug ? thanks.
Cloudera Employee
Posts: 28
Registered: ‎11-17-2017

Re: CDH 5.10.2 Impala bug

[ Edited ]

Thank you for the report.

 

This resembles IMPALA-3983, however it has been fixed in CDH 5.10.2 and the temporary jars are being removed at the end of the extractFunctions method. The output of lsof shows that the files were deleted but the space might not be freed from disk as catalogd is keeping the file open.

 

A possible root cause could be is that another process is removing the files before the CatalogD could remove it.

Highlighted
Expert Contributor
Posts: 160
Registered: ‎09-29-2014

Re: CDH 5.10.2 Impala bug

it's really happend in CDH 5.10.2, i think there is no another process to delete the file before catalogd .
Announcements