hcat.bin is not defined. Define it to be your hcat script

[#1]

getting error when executing a pig script through Oozie. Script has pig and hcat commands.

[#2]

I see it's thrown from this class https://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/tools/grunt/GruntParser.java

[#3]

do I need to set $HCAT_HOME/bin/hcat in pig.properties?

[#4]

That should work I think.

[#5 · Accepted solution]

SOLUTION

Pig introduced an option to run hcatalog commands in grunt and in scripts. There's more info in pig.properties file in /etc/pig/conf/pig.properties

# In addition to the fs-style commands (rm, ls, etc) Pig can now execute

# SQL-style DDL commands, eg "sql create table pig_test(name string, age int)".

# The only implemented backend is hcat, and luckily that's also the default.

#

# pig.sql.type=hcat # Path to the hcat executable, for use with pig.sql.type=hcat (default: null)

#

hcat.bin=/usr/local/hcat/bin/hcat

this is on sandbox 2.3, HDP 2.3 and HDP 2.3.2 by default

so running any pig script with hcat commands without -useHCatalog will fail with the following, usually that will happen through Oozie

Pig Stack Trace --------------- ERROR 2997: Encountered IOException. /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. java.io.IOException: /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1286) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:631) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

so to fix it change it globally by editing pig.properties in Ambari to point to the following:

hcat.bin=/usr/bin/hcat

or

copy pig.properties to your own location, override the pig.properties with right path (i.e. hcat.bin=/usr/bin/hcat) and execute script like so

pig -P pig.properties test.pig

or

override the property on the fly

pig -Dhcat.bin=/usr/bin/hcat test.pig

Or even lesser intrusive way:

In your pig script put this in the beginning

set hcat.bin /usr/bin/hcat;