Reply
New Contributor
Posts: 3
Registered: ‎07-11-2017

Hive on Spark Exception

I have a Hive view with Union All operator. The View executes when I execute wth Map Reduce engine. When I use spark as my execution engine, I am getting the following error "Error: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent (state=42000,code=40000). Any advice.

Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Hive on Spark Exception

Get into the job and container logs via the Spark History UI. Try to get more context or a full stack trace.
Cloudera Employee
Posts: 211
Registered: ‎03-23-2015

Re: Hive on Spark Exception

You might also want to share the full query that failed with such error, together with the full stack trace.
New Contributor
Posts: 3
Registered: ‎07-11-2017

Re: Hive on Spark Exception

The SQL is very simple. 

Select col1, col2,col3,.............,coln from Table1

UNION ALL Select col1, col2,col3.............,coln from Table2

UNION ALL Select col1, col2,col3.............,coln from Table3

UNION ALL .......... UNION ALL Select col1, col2,col3.............,coln from Table13

 

When I break up the view into smaller chunks, they work.  Here is the sample of combinations that work and do not work.

 

Tables   1,2,3,4 Working

                                1,2,3,4,5 Not Working

                                1,2,3,5 Not Working

                                1,2,4,5 Not Working

                                1,3,4,5 Not Working

                                1,2,3,4,6 Not Working

                                1,2,3,4,7 Not Working

                                1,2,3,4,8 Not Working

                                1,5 Working

                                5,6 Working

                                5,6,7 Not Working

                                6,7 Working

                                6,7,8 Working

                                6,7,8,9 Working

                                6,7,8,9,10 Working

                                6,7,8,9,10,11 Working

                                6,7,8,9,10,11,12 Not Working

                                12,13 Working

 

These is a partial list of combinations, I have tried.

 

When it is not working, I get this stack trace.

 

Getting log thread is interrupted, since query is done!

Error: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent (state=42000,code=40000)

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:239)

        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:225)

        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:244)

        at org.apache.hive.beeline.Commands.executeInternal(Commands.java:904)

        at org.apache.hive.beeline.Commands.execute(Commands.java:1091)

        at org.apache.hive.beeline.Commands.sql(Commands.java:987)

        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1082)

        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)

        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)

        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)

        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:385)

        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:136)

        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)

       at org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)

        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:424)

        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)

        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:260)

        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)

        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)

        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)

        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)

        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:711)

        at org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.removeUnionOperators(GenSparkUtils.java:284)

        at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.generateTaskTree(SparkCompiler.java:229)

        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:204)

        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10109)

        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9902)

        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

        at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)

        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488)

        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274)

        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1261)

        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)

        ... 15 more

 

Posts: 388
Topics: 11
Kudos: 60
Solutions: 34
Registered: ‎09-02-2016

Re: Hive on Spark Exception

@Venkatk99

 

Since the error shows HiveSQLException, keep the Spark aside and execute your different table combinations directly in hive and make sure there is no issue

 

You can check something like

1. make sure the datatype of col1, col2, col3.. coln from table 5 is exactly matches/compatable with 6 & 7

2. Also focus on  exception from log "Exception when trying to remove partition predicates: fail to find child from parent"

etc

 

You can try to execute it from spark once all combinations of quires are working fine from hive

 

 

New Contributor
Posts: 3
Registered: ‎07-11-2017

Re: Hive on Spark Exception

The query executes with map reduce engine and I get the desired result.  The error happens when I switch to spark engine.

Announcements