Support Questions
Find answers, ask questions, and share your expertise

Hive compactions ACID tables - Missing files in select FileNotFoundException

New Contributor

Hello,

 

Me and my team are using Hive version 3.1.0 as part of HDInisght 4.0 in Azure.

We have a dimensional model which stores data in Hive managed tables  (ACID as default) with underlying  file system ADLS Gen2 and using Azure SQL DB as a metastore.

 

To populate the data in those tables, after some conforming done with Spark, we use MERGE Statements .

The Merges perform Upsert.

We are currently faced with one, very important problem, which goes like this:

- Sometimes when a long running query is executed and it uses a table which just had a merge executed against it before, that query fails, because, as far as I can see, the compactor runs in the same time  on the underlying data and it deletes some of the files (merges them) used by that specific table, making the query running on top of it fail.

The error is this one (I replaced the ADLS path with something dummy for anonymity):

 

ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 27, vertexId=vertex_1584051627112_5934_1_12, diagnostics=[Task failed, taskId=task_1584051627112_5934_1_12_000565, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1584051627112_5934_1_12_000565_0:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.io.FileNotFoundException: Operation failed: "The specified path does not exist.", 404, GET, https:/myADLSUri.dfs.core.windows.net/myADLSContainer/hive/warehouse/managed/new_repartition.db/call_fact/country_code%3DBR/year%3D2020/month%3D2/delete_..., PathNotFound, "The specified path does not exist. RequestId:bde0f6fa-201f-0095-043a-008c49000000 Time:2020-03-22T11:08:04.4571403Z"

        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)

        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)

        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)

        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)

        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)

        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)

        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)

        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)

        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)

        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)

        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.RuntimeException: java.io.IOException: java.io.FileNotFoundException: Operation failed: "The specified path does not exist.", 404, GET, https://myADLSUri.dfs.core.windows.net/myADLSContainer/hive/warehouse/managed/new_repartition.db/cal..., PathNotFound, "The specified path does not exist. RequestId:bde0f6fa-201f-0095-043a-008c49000000 Time:2020-03-22T11:08:04.4571403Z"

        at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)

        at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152)

        at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)

        at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)

        at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)

        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)

        ... 16 more

 

 

Do you guys have any ideas on what might be causing this - the query and the compactor conflict?

Also, any suggestions on how to fix it? (I know that we can disable compaction and run them manually, but we have all sorts of dynamic partitions and we are talking about 200 tables with multiple ETL flows running in parallel - we will see a great performance decrease and it will be a mess handling it).

 

Thanks,

Cristian.

 

 

0 REPLIES 0