Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive on Spark Exception

avatar
New Contributor

I have a Hive view with Union All operator. The View executes when I execute wth Map Reduce engine. When I use spark as my execution engine, I am getting the following error "Error: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent (state=42000,code=40000). Any advice.

5 REPLIES 5

avatar
Champion
Get into the job and container logs via the Spark History UI. Try to get more context or a full stack trace.

avatar
Super Guru
You might also want to share the full query that failed with such error, together with the full stack trace.

avatar
New Contributor

The SQL is very simple. 

Select col1, col2,col3,.............,coln from Table1

UNION ALL Select col1, col2,col3.............,coln from Table2

UNION ALL Select col1, col2,col3.............,coln from Table3

UNION ALL .......... UNION ALL Select col1, col2,col3.............,coln from Table13

 

When I break up the view into smaller chunks, they work.  Here is the sample of combinations that work and do not work.

 

Tables   1,2,3,4 Working

                                1,2,3,4,5 Not Working

                                1,2,3,5 Not Working

                                1,2,4,5 Not Working

                                1,3,4,5 Not Working

                                1,2,3,4,6 Not Working

                                1,2,3,4,7 Not Working

                                1,2,3,4,8 Not Working

                                1,5 Working

                                5,6 Working

                                5,6,7 Not Working

                                6,7 Working

                                6,7,8 Working

                                6,7,8,9 Working

                                6,7,8,9,10 Working

                                6,7,8,9,10,11 Working

                                6,7,8,9,10,11,12 Not Working

                                12,13 Working

 

These is a partial list of combinations, I have tried.

 

When it is not working, I get this stack trace.

 

Getting log thread is interrupted, since query is done!

Error: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent (state=42000,code=40000)

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:239)

        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:225)

        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:244)

        at org.apache.hive.beeline.Commands.executeInternal(Commands.java:904)

        at org.apache.hive.beeline.Commands.execute(Commands.java:1091)

        at org.apache.hive.beeline.Commands.sql(Commands.java:987)

        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1082)

        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)

        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)

        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)

        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:385)

        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:136)

        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)

       at org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)

        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:424)

        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)

        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:260)

        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)

        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)

        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)

        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)

        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:711)

        at org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.removeUnionOperators(GenSparkUtils.java:284)

        at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.generateTaskTree(SparkCompiler.java:229)

        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:204)

        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10109)

        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9902)

        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

        at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)

        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488)

        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274)

        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1261)

        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)

        ... 15 more

 

avatar
Champion

@Venkatk99

 

Since the error shows HiveSQLException, keep the Spark aside and execute your different table combinations directly in hive and make sure there is no issue

 

You can check something like

1. make sure the datatype of col1, col2, col3.. coln from table 5 is exactly matches/compatable with 6 & 7

2. Also focus on  exception from log "Exception when trying to remove partition predicates: fail to find child from parent"

etc

 

You can try to execute it from spark once all combinations of quires are working fine from hive

 

 

avatar
New Contributor

The query executes with map reduce engine and I get the desired result.  The error happens when I switch to spark engine.