@bsaini@hortonworks.com Node 1 - no storage - I guess you wont be using this for data node but it can play role of worker node (Nodemanager if you like and other components) You will be creating a config group for node1 Please see this screenshot. In my case, node4 is datanode and I want to customize the data directories Created new config for node4 select the config and change parameters that you want to change. There is Doc example ... View more

The answer is the logic to group puts by regionserver is now built-in with HBase API 1.0+. It is no longer necessary to leverage any other code to achieve it. ... View more

SOLUTION Pig introduced an option to run hcatalog commands in grunt and in scripts. There's more info in pig.properties file in /etc/pig/conf/pig.properties # In addition to the fs-style commands (rm, ls, etc) Pig can now execute # SQL-style DDL commands, eg "sql create table pig_test(name string, age int)". # The only implemented backend is hcat, and luckily that's also the default. # # pig.sql.type=hcat # Path to the hcat executable, for use with pig.sql.type=hcat (default: null) # hcat.bin=/usr/local/hcat/bin/hcat this is on sandbox 2.3, HDP 2.3 and HDP 2.3.2 by default so running any pig script with hcat commands without -useHCatalog will fail with the following, usually that will happen through Oozie Pig Stack Trace --------------- ERROR 2997: Encountered IOException. /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. java.io.IOException: /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1286) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:631) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) so to fix it change it globally by editing pig.properties in Ambari to point to the following: hcat.bin=/usr/bin/hcat or copy pig.properties to your own location, override the pig.properties with right path (i.e. hcat.bin=/usr/bin/hcat) and execute script like so pig -P pig.properties test.pig or override the property on the fly pig -Dhcat.bin=/usr/bin/hcat test.pig Or even lesser intrusive way: In your pig script put this in the beginning set hcat.bin /usr/bin/hcat; ... View more