The point about having the job.properties on the local filesystem is just within the context of using Oozie CLI - it is explicitly mentioned to avoid confusion in the process of writing a workflow XML (the XML needs to be on HDFS, but the properties are a local file read by the Oozie CLI when invoked - people often get confused by this relationship). When you use Hue, which in turn simply uses the Oozie REST API, you can use whatever mechanism Hue offers you to manage your job.properties (as a HDFS file, defined within-workflow, etc.). The properties are something used to resolve the workflow's variables and supply it some necessary parameters. Once submitted, the properties file or list of properties do not matter anymore to Oozie. ... View more

If you've previously set manual overrides to MaxPermSize options in your configurations, you can remove them away safely having switched to using JDK8. If you still have parts that use JDK7, leave them be and ignore the warning. The warnings themselves do not pose a problem, as JDK8 will simply note that it won't be using them anymore and continue to start up normally. However, a future Java version (JDK9 or 10) may choose to interpret it as an invalid command and fail. Read more on this JDK level change at https://dzone.com/articles/java-8-permgen-metaspace ... View more

In HDFS, the permissions model for owner and group follow the BSD rule. The owner is set to the authenticated user, but the group is inherited from the parent directory. This is documented in the Permissions Guide: http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Overview """ When a file or directory is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule). """ The Group Mapping is purely used at the authorisation side, not at the creation side as you are expecting it to be. Since your /user/username directory's group is by default the username itself, that's the value you will naturally see for all groups. If you'd like that changed, you will need to chgrp the /user/username directory to be username:user-group instead of username:username. Subsequent files will now be created with username:user-group under it. ... View more

One part of HBase Replication is that of turning on configs on the Server end to enable the feature. This can be done via the API exactly like Marcell described. The other part of replication configs is the peer configuration, but this is not doable via CM API as it is not a service-passed configuration, but more of a runtime one. You will need to use HBase's Java API (ReplicationAdmin class) directly for this: http://archive.cloudera.com/cdh5/cdh/5/hbase/apidocs/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.html. To do this in Python, I'd guess you will need to use Jython or such. ... View more

Thanks for reporting this. Upon investigating, there's a small bug with how we load/respect lib_dir in the agent. There's two ways to supply it, one via the CMF_AGENT_ARGS (--lib_dir=/path) inside /etc/default/cloudera-scm-agent and another via the line inside /etc/cloudera-scm-agent/config.ini. The small bug is that for the UUID file we seem to look at the default path before we load up the config. I'd have expected some form of error in your case vs. a blatant ignore, especially if /var/lib/cloudera-scm-agent no longer exists, so I'd first make sure you have lib_dir configured without the prefix # comment-marking keyword in your config.ini, and if that is so, apply the workaround to the bug (specify both in CMF_AGENT_ARGS and in config.ini). The internal bug report is OPSAPS-32501 P.s. Make sure to copy over the uuid file before you erase /var/lib/cloudera-scm-agent. If you've lost it, no worries, you can create the uuid file under your lib_dir with the content matching the one shown on the CM -> Hosts -> (the specific host's page) -> "Host ID" field on top. Failure in keeping the same UUID will result in duplicated hosts (though that can be fixed trivially by fixing the uuid file and restarting agent yet again, then deleting the role-less duped host). ... View more