How can I enable pig -useHCatalog via Ambari. I seem to get stuck everytime I am try to execute scripts via Pig View.

[#1]

This is what Ambari says:

WARNING: Use "yarn jar" to launch YARN applications. 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0.2.4.0.0-169 (rexported) compiled Feb 10 2016, 07:50:04 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:44,356 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found 2016-06-02 21:12:44,477 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020 2016-06-02 21:12:45,021 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-b1da2d85-2845-4a3f-b59b-b2e367f74c1c 2016-06-02 21:12:45,420 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-06-02 21:12:45,646 [main] INFO org.apache.pig.backend.hadoop.ATSService - Created ATS Hook 2016-06-02 21:12:46,138 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Failed to parse: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1776) at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1484) at org.apache.pig.PigServer.parseAndBuild(PigServer.java:428) at org.apache.pig.PigServer.executeBatch(PigServer.java:453) at org.apache.pig.PigServer.executeBatch(PigServer.java:439) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:502) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 17 more Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885) ... 23 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException at org.apache.hive.hcatalog.pig.HCatLoader.<init>(HCatLoader.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:742) ... 24 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 31 more 2016-06-02 21:12:46,148 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Details at logfile: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:46,167 [main] INFO org.apache.pig.Main - Pig script completed in 2 seconds and 722 milliseconds (2722 ms)

[#2]

@srai

Are you able to run same pig script through command line?

Also looks like below jar either missing or have multiple versions while runing pig.

/usr/hdp/current/hive-client/lib/hive-metastore-<version>.jar

[#3]

I tried a couple of things. Ran the job via command line using this syntax "pig -x tez -useHCatalog -f script.pig" ...this works fine. The problem is, I need to find its equivalent to make it run via Pig View in Ambari.

[#4]

I am able to reproduce this via CLI as well. For a refresher, working on a Sandbox downloaded from Hortonworks site, also, when I manually specify the HADOOP_CLASSPATH and HADOOP_CLIENT_OPTS via command line, I get it to work. I got the classpath and hadoop client opts using "pig -printCmdDebug -x tez -useHCatalog" there are indeed some additional jar's that are included in the hadoop class path and client opts.

[#5]

same here with the latest sandbox, won't work from Ambari

[#6 · Accepted solution]

Have you tried adding "-useHCatalog" as a pig argument? On the pig script, there is an option in the lower right to add an argument. This worked for me. Without this option, I was unable to run an pig scripts in Ambari that required HCat based jars.

[#7]

Note this tutorial covers it in step 3.3: http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#...

[#8]

m also facing same issue can someone help on this @Srai.

M not able to run pig queries using hcatalog

[#9]

hey Abhijeet, not sure if you have tried this.

"pig -printCmdDebug -x tez -useHcatalog"

Let me know if this does not work for you.

[#10]

Hey Srai thanks for replying m newbie to this hadoop world.

Can you please tell me where i need to execute this command.

I tried in both directly from shell :

[root@sandbox ~]# pig -printCmdDebug -x tez -useHcatalog;

and in pig using mapreduce mode

grunt> pig -printCmdDebug -x tez -useHcatalog;

in both case its not working @srai.