In HDFS, the permissions model for owner and group follow the BSD rule. The owner is set to the authenticated user, but the group is inherited from the parent directory. This is documented in the Permissions Guide: http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Overview """ When a file or directory is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule). """ The Group Mapping is purely used at the authorisation side, not at the creation side as you are expecting it to be. Since your /user/username directory's group is by default the username itself, that's the value you will naturally see for all groups. If you'd like that changed, you will need to chgrp the /user/username directory to be username:user-group instead of username:username. Subsequent files will now be created with username:user-group under it. ... View more

One part of HBase Replication is that of turning on configs on the Server end to enable the feature. This can be done via the API exactly like Marcell described. The other part of replication configs is the peer configuration, but this is not doable via CM API as it is not a service-passed configuration, but more of a runtime one. You will need to use HBase's Java API (ReplicationAdmin class) directly for this: http://archive.cloudera.com/cdh5/cdh/5/hbase/apidocs/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.html. To do this in Python, I'd guess you will need to use Jython or such. ... View more

I would use the spark action as much OOTB as possible leverage sharelib for since handles a number of things for you. You can use multiple versions of sharelib as described here check for overriding the sharelib.   Wilfred   ... View more

Lets say you want to execute "script.sh" 1. If you have script.sh inside your WF/lib/ path on HDFS, you just need <exec>script.sh</exec> 2. If you have script.sh on an arbitrary path on HDFS, you need: <exec>script.sh</exec> <file>/path/to/script.sh#script.sh</file> 3. Use of the below form with (1) is redundant, but the subsequent form is when you want to invoke it as a different name: <exec>script.sh</exec> <file>script.sh#script.sh</file> <exec>linked-script-name.sh</exec> <file>original-script-name.sh#linked-script-name.sh</file> ... View more

Note: CDH3 is long past its supported lifetime. Netezza JDBC should be worth trying on the CDH3 Sqoop version. I don't recall if it worked without a specialised connector, but the generic SQL connector should likely make it go through. ... View more