Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive on Spark Exception

Hive on Spark Exception

New Contributor

I have a Hive view with Union All operator. The View executes when I execute wth Map Reduce engine. When I use spark as my execution engine, I am getting the following error "Error: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent (state=42000,code=40000). Any advice.

5 REPLIES 5

Re: Hive on Spark Exception

Champion
Get into the job and container logs via the Spark History UI. Try to get more context or a full stack trace.

Re: Hive on Spark Exception

Guru
You might also want to share the full query that failed with such error, together with the full stack trace.

Re: Hive on Spark Exception

New Contributor

The SQL is very simple. 

Select col1, col2,col3,.............,coln from Table1

UNION ALL Select col1, col2,col3.............,coln from Table2

UNION ALL Select col1, col2,col3.............,coln from Table3

UNION ALL .......... UNION ALL Select col1, col2,col3.............,coln from Table13

 

When I break up the view into smaller chunks, they work.  Here is the sample of combinations that work and do not work.

 

Tables   1,2,3,4 Working

                                1,2,3,4,5 Not Working

                                1,2,3,5 Not Working

                                1,2,4,5 Not Working

                                1,3,4,5 Not Working

                                1,2,3,4,6 Not Working

                                1,2,3,4,7 Not Working

                                1,2,3,4,8 Not Working

                                1,5 Working

                                5,6 Working

                                5,6,7 Not Working

                                6,7 Working

                                6,7,8 Working

                                6,7,8,9 Working

                                6,7,8,9,10 Working

                                6,7,8,9,10,11 Working

                                6,7,8,9,10,11,12 Not Working

                                12,13 Working

 

These is a partial list of combinations, I have tried.

 

When it is not working, I get this stack trace.

 

Getting log thread is interrupted, since query is done!

Error: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent (state=42000,code=40000)

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:239)

        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:225)

        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:244)

        at org.apache.hive.beeline.Commands.executeInternal(Commands.java:904)

        at org.apache.hive.beeline.Commands.execute(Commands.java:1091)

        at org.apache.hive.beeline.Commands.sql(Commands.java:987)

        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1082)

        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)

        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)

        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)

        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:385)

        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:136)

        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)

       at org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)

        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:424)

        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)

        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:260)

        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)

        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)

        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)

        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)

        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to remove partition predicates: fail to find child from parent

        at org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:711)

        at org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.removeUnionOperators(GenSparkUtils.java:284)

        at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.generateTaskTree(SparkCompiler.java:229)

        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:204)

        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10109)

        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9902)

        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

        at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)

        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488)

        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274)

        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1261)

        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)

        ... 15 more

 

Re: Hive on Spark Exception

Champion

@Venkatk99

 

Since the error shows HiveSQLException, keep the Spark aside and execute your different table combinations directly in hive and make sure there is no issue

 

You can check something like

1. make sure the datatype of col1, col2, col3.. coln from table 5 is exactly matches/compatable with 6 & 7

2. Also focus on  exception from log "Exception when trying to remove partition predicates: fail to find child from parent"

etc

 

You can try to execute it from spark once all combinations of quires are working fine from hive

 

 

Highlighted

Re: Hive on Spark Exception

New Contributor

The query executes with map reduce engine and I get the desired result.  The error happens when I switch to spark engine.