Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hcat.bin is not defined. Define it to be your hcat script

avatar
Master Mentor

getting error when executing a pig script through Oozie. Script has pig and hcat commands.

1 ACCEPTED SOLUTION

avatar
Master Mentor

SOLUTION

Pig introduced an option to run hcatalog commands in grunt and in scripts. There's more info in pig.properties file in /etc/pig/conf/pig.properties

# In addition to the fs-style commands (rm, ls, etc) Pig can now execute

# SQL-style DDL commands, eg "sql create table pig_test(name string, age int)".

# The only implemented backend is hcat, and luckily that's also the default.

#

# pig.sql.type=hcat # Path to the hcat executable, for use with pig.sql.type=hcat (default: null)

#

hcat.bin=/usr/local/hcat/bin/hcat

this is on sandbox 2.3, HDP 2.3 and HDP 2.3.2 by default

so running any pig script with hcat commands without -useHCatalog will fail with the following, usually that will happen through Oozie

Pig Stack Trace --------------- ERROR 2997: Encountered IOException. /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. java.io.IOException: /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1286) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:631) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

so to fix it change it globally by editing pig.properties in Ambari to point to the following:

hcat.bin=/usr/bin/hcat

or

copy pig.properties to your own location, override the pig.properties with right path (i.e. hcat.bin=/usr/bin/hcat) and execute script like so

pig -P pig.properties test.pig

or

override the property on the fly

pig -Dhcat.bin=/usr/bin/hcat test.pig

Or even lesser intrusive way:

In your pig script put this in the beginning

set hcat.bin /usr/bin/hcat;

View solution in original post

4 REPLIES 4

avatar
Master Mentor

avatar
Master Mentor

do I need to set $HCAT_HOME/bin/hcat in pig.properties?

avatar

That should work I think.

avatar
Master Mentor

SOLUTION

Pig introduced an option to run hcatalog commands in grunt and in scripts. There's more info in pig.properties file in /etc/pig/conf/pig.properties

# In addition to the fs-style commands (rm, ls, etc) Pig can now execute

# SQL-style DDL commands, eg "sql create table pig_test(name string, age int)".

# The only implemented backend is hcat, and luckily that's also the default.

#

# pig.sql.type=hcat # Path to the hcat executable, for use with pig.sql.type=hcat (default: null)

#

hcat.bin=/usr/local/hcat/bin/hcat

this is on sandbox 2.3, HDP 2.3 and HDP 2.3.2 by default

so running any pig script with hcat commands without -useHCatalog will fail with the following, usually that will happen through Oozie

Pig Stack Trace --------------- ERROR 2997: Encountered IOException. /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. java.io.IOException: /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1286) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:631) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

so to fix it change it globally by editing pig.properties in Ambari to point to the following:

hcat.bin=/usr/bin/hcat

or

copy pig.properties to your own location, override the pig.properties with right path (i.e. hcat.bin=/usr/bin/hcat) and execute script like so

pig -P pig.properties test.pig

or

override the property on the fly

pig -Dhcat.bin=/usr/bin/hcat test.pig

Or even lesser intrusive way:

In your pig script put this in the beginning

set hcat.bin /usr/bin/hcat;