[Hive] IOException: Fail to get checksum, since file /warehouse/tablespace/managed/**** is under construction.

[#1]

Dear team,

We are getting the below error in the CDP - 7.1.6 HiveServer logs. Can you please share what's the cause of this issue and any possible solution?

2021-09-03 13:12:25,571 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checks
um, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.
2021-09-03 13:12:25,571 INFO  org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: Completed executing command(queryId=hive_20210903131225_70117bf2-c60f-4564-83e9-8a60be421f63); Time taken: 0.12 seconds
2021-09-03 13:12:25,572 INFO  org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: OK
2021-09-03 13:12:25,572 INFO  org.apache.hadoop.hive.ql.lockmgr.DbTxnManager: [HiveServer2-Background-Pool: Thread-16886]: Stopped heartbeat for query: hive_20210903131225_70117bf2-c60f-4564-83e9-8a60be421f63
2021-09-03 13:12:25,578 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-16886]: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.
        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:362) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:241) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322) [hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_242]
        at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) [hadoop-common-3.1.1.7.1.6.0-297.jar:?]
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340) [hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Fail to get checksum, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.
        at org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3509) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2245) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:515) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:432) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:742) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:497) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        ... 13 more

[#2 · Accepted solution]

Is the query failing or you are just noticing the errors in the logs?

It seems to be ACID managed table.

Please provide the hiveserver2 logs and the exact query fired.

[#3]

@Sayed016, If you are still experiencing the issue, can you provide the information @asish has requested? If the reply has resolved your issue, kindly mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  

[#4]

@asish Sorry for the delayed reply. We only saw this error in the log, however, I looked into the logs again later today and couldn't find this - "under construction" in the logs anymore. Is there any workaround for this so that in the future this doesn't happen?

[#5]

@Sayed016 This error occurs if the HDFS files are under replicated.

Did you see HDFS is under replicated when the issue occurred? Wait for the HDFS to gets replicated and you can query