Impala cannot write to HDFS

[#1]

Hi,

 one of our INSERT query failed in Impala, one of the fragment could not write the data into HDFS. The message is quite descriptive, but I am not able to find out what is the root cause of this failure - the HDFS did not reported any issue at that time, neither Impala daemons. 

Query Status: Failed to write data (length: 38425) to Hdfs file: 
hdfs://hanameservice/data/target_table/data/_impala_insert_staging/e64692c97276103f_d0ba1f0500000000/.e64692c97276103f-d0ba1f0500000001_1863061457_dir/e64692c97276103f-d0ba1f0500000001_2018403269_data.0.parq 
Error(255): Unknown error 255 Root cause: IllegalMonitorStateException:

The Impala daemon log:

I0516 06:43:06.444519 32188 krpc-data-stream-recvr.cc:557] cancelled stream: fragment_instance_id=44488c54916f20c9:261655f90000000e node_id=5
I0516 06:43:06.444725 32188 query-state.cc:412] Instance completed. instance_id=44488c54916f20c9:261655f90000000e #in-flight=3 status=OK
I0516 06:43:06.444736 32188 query-exec-mgr.cc:155] ReleaseQueryState(): query_id=44488c54916f20c9:261655f900000000 refcnt=2
I0516 06:43:06.793349 32186 query-state.cc:412] Instance completed. instance_id=44488c54916f20c9:261655f900000015 #in-flight=2 status=OK
I0516 06:43:06.793372 32186 query-exec-mgr.cc:155] ReleaseQueryState(): query_id=44488c54916f20c9:261655f900000000 refcnt=1
I0516 06:43:06.899813 10865 status.cc:125] Failed to write data (length: 38425) to Hdfs file: hdfs://hanameservice/target_table/data/_impala_insert_staging/e64692c97276103f_d0ba1f0500000000/.e64692c97276103f-d0ba1f0500000001_1863061457_dir/e64692c97276103f-d0ba1f0500000001_2018403269_data.0.parq
Error(255): Unknown error 255
Root cause: IllegalMonitorStateException:
    @           0x966e3a
    @          0x107e9fb
    @           0xe1aea3
    @           0xe1b127
    @           0xe1c54d
    @           0xdecd8c
    @           0xdedbc3
    @           0xdef090
    @           0xbadc17
    @           0xbb06af
    @           0xb9e74a
    @           0xd607ef
    @           0xd60fea
    @          0x12d8b5a
    @     0x7fa01674fdd5
    @     0x7fa016478ead
I0516 06:43:06.944746 10865 runtime-state.cc:170] Error from query e64692c97276103f:d0ba1f0500000000: Failed to close HDFS file: hdfs://hanameservice/target_table/data/_impala_insert_staging/e64692c97276103f_d0ba1f0500000000/.e64692c97276103f-d0ba1f0500000001_1863061457_dir/e64692c97276103f-d0ba1f0500000001_2018403269_data.0.parq
Error(255): Unknown error 255
Root cause: IllegalMonitorStateException:
I0516 06:43:06.966231 10865 query-state.cc:412] Instance completed. instance_id=e64692c97276103f:d0ba1f0500000001 #in-flight=1 status=GENERAL: Failed to write data (length: 38425) to Hdfs file: hdfs://hanameservice/target_table/data/_impala_insert_staging/e64692c97276103f_d0ba1f0500000000/.e64692c97276103f-d0ba1f0500000001_1863061457_dir/e64692c97276103f-d0ba1f0500000001_2018403269_data.0.parq
Error(255): Unknown error 255
Root cause: IllegalMonitorStateException:
I0516 06:43:06.966250 10865 query-state.cc:425] Cancel: query_id=e64692c97276103f:d0ba1f0500000000

Any hints what can be the root cause of this issue?

Thanks

[#2]

Hi Tomas,

Couple questions:

1. is it happening on all tables with INSERT operation or just certain tables? Any pattern with the type of tables that are failing?

2. what version of CDH are you using?

3. can you please locate the host of the failed fragment and find the full stacktrace for this exception?

This might be helpful to see what next step we can do.

Cheers
Eric

[#3]

No it was just one insert and after the repeat it succeeded, so I am not able to reproduce, and thus no patterns. CDH 5.15
Can you give me a detailed hint how to get the full stacktrace (from the Impala daemon?) of the failed fragment? I dont have the query profile (already deleted) but as I can remember one of the fragment (out of 10) was waiting for almost 2h to HDFS sink, others finished within a minute. Maybe it is a hdfs issue?