Support Questions

Find answers, ask questions, and share your expertise

How can I enable pig -useHCatalog via Ambari. I seem to get stuck everytime I am try to execute scripts via Pig View.

avatar
Guru

This is what Ambari says:

WARNING: Use "yarn jar" to launch YARN applications. 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0.2.4.0.0-169 (rexported) compiled Feb 10 2016, 07:50:04 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:44,356 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found 2016-06-02 21:12:44,477 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020 2016-06-02 21:12:45,021 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-b1da2d85-2845-4a3f-b59b-b2e367f74c1c 2016-06-02 21:12:45,420 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-06-02 21:12:45,646 [main] INFO org.apache.pig.backend.hadoop.ATSService - Created ATS Hook 2016-06-02 21:12:46,138 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Failed to parse: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1776) at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1484) at org.apache.pig.PigServer.parseAndBuild(PigServer.java:428) at org.apache.pig.PigServer.executeBatch(PigServer.java:453) at org.apache.pig.PigServer.executeBatch(PigServer.java:439) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:502) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 17 more Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885) ... 23 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException at org.apache.hive.hcatalog.pig.HCatLoader.<init>(HCatLoader.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:742) ... 24 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 31 more 2016-06-02 21:12:46,148 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Details at logfile: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:46,167 [main] INFO org.apache.pig.Main - Pig script completed in 2 seconds and 722 milliseconds (2722 ms)

1 ACCEPTED SOLUTION

avatar
Super Guru

Have you tried adding "-useHCatalog" as a pig argument? On the pig script, there is an option in the lower right to add an argument. This worked for me. Without this option, I was unable to run an pig scripts in Ambari that required HCat based jars.

4878-screen-shot-2016-06-08-at-34626-pm.png

View solution in original post

12 REPLIES 12

avatar
Super Guru
@srai

Are you able to run same pig script through command line?

Also looks like below jar either missing or have multiple versions while runing pig.

/usr/hdp/current/hive-client/lib/hive-metastore-<version>.jar

avatar
Guru

I tried a couple of things. Ran the job via command line using this syntax "pig -x tez -useHCatalog -f script.pig" ...this works fine. The problem is, I need to find its equivalent to make it run via Pig View in Ambari.

avatar
Guru

I am able to reproduce this via CLI as well. For a refresher, working on a Sandbox downloaded from Hortonworks site, also, when I manually specify the HADOOP_CLASSPATH and HADOOP_CLIENT_OPTS via command line, I get it to work. I got the classpath and hadoop client opts using "pig -printCmdDebug -x tez -useHCatalog" there are indeed some additional jar's that are included in the hadoop class path and client opts.

avatar
Master Guru

same here with the latest sandbox, won't work from Ambari

avatar
Super Guru

Have you tried adding "-useHCatalog" as a pig argument? On the pig script, there is an option in the lower right to add an argument. This worked for me. Without this option, I was unable to run an pig scripts in Ambari that required HCat based jars.

4878-screen-shot-2016-06-08-at-34626-pm.png

avatar
Super Guru

avatar

m also facing same issue can someone help on this @Srai.

M not able to run pig queries using hcatalog

avatar
Guru

hey Abhijeet, not sure if you have tried this.

"pig -printCmdDebug -x tez -useHcatalog"

Let me know if this does not work for you.

avatar

Hey Srai thanks for replying m newbie to this hadoop world.

Can you please tell me where i need to execute this command.

I tried in both directly from shell :

[root@sandbox ~]# pig -printCmdDebug -x tez -useHcatalog;

and in pig using mapreduce mode

grunt> pig -printCmdDebug -x tez -useHcatalog;

in both case its not working @srai.