Support Questions
Find answers, ask questions, and share your expertise

[Hive] IOException: Fail to get checksum, since file /warehouse/tablespace/managed/**** is under construction.

Explorer

Dear team,

 

We are getting the below error in the CDP - 7.1.6 HiveServer logs. Can you please share what's the cause of this issue and any possible solution?

 

 

2021-09-03 13:12:25,571 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checks
um, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.
2021-09-03 13:12:25,571 INFO  org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: Completed executing command(queryId=hive_20210903131225_70117bf2-c60f-4564-83e9-8a60be421f63); Time taken: 0.12 seconds
2021-09-03 13:12:25,572 INFO  org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: OK
2021-09-03 13:12:25,572 INFO  org.apache.hadoop.hive.ql.lockmgr.DbTxnManager: [HiveServer2-Background-Pool: Thread-16886]: Stopped heartbeat for query: hive_20210903131225_70117bf2-c60f-4564-83e9-8a60be421f63
2021-09-03 13:12:25,578 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-16886]: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.
        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:362) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:241) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322) [hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_242]
        at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) [hadoop-common-3.1.1.7.1.6.0-297.jar:?]
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340) [hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Fail to get checksum, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.
        at org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3509) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2245) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:515) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:432) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:742) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:497) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
        ... 13 more

 

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Expert Contributor

Is the query failing or you are just noticing the errors in the logs?

 

It seems to be ACID managed table.

 

Please provide the hiveserver2 logs and the exact query fired.

View solution in original post

4 REPLIES 4

Expert Contributor

Is the query failing or you are just noticing the errors in the logs?

 

It seems to be ACID managed table.

 

Please provide the hiveserver2 logs and the exact query fired.

View solution in original post

Community Manager

@Sayed016, If you are still experiencing the issue, can you provide the information @asish has requested? If the reply has resolved your issue, kindly mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  


Regards,

Vidya Sargur,
Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Explorer

@asish Sorry for the delayed reply. We only saw this error in the log, however, I looked into the logs again later today and couldn't find this - "under construction" in the logs anymore. Is there any workaround for this so that in the future this doesn't happen?

 

Expert Contributor

@Sayed016 This error occurs if the HDFS files are under replicated.

 

Did you see HDFS is under replicated when the issue occurred? Wait for the HDFS to gets replicated and you can query