Hi guys, Using Maven is now allowed by the Legal team here, we can only use Gradle, and only what Gradle packages out of the box, no add-ons. Ergo Gradle does not have the ability to re-locate package names to prevent collisions like we can do with Maven Shade. Yes @schintalapani we are using Storm .10. This Jira looks EXACTLY what we need. https://issues.apache.org/jira/browse/STORM-129 ... View more

Just a note that YARN may need to execute things that are placed into its local cache on the NMs, its not purly a data storage. This is why you cant have directories that are YARN related mounted as NOEXEC in /etc/fstab... ... View more

I think we are going to try to go the route of having 2 HS2, one with Ranger Integration, one without. No matter what we do we will end up with 1000's of policies as there are potentially 1000's of different data sources being onboarded to the system. Most of these don't require anything more then HDFS Security, so having to enter them into Hive for access is a management nightmare. By having a 'Ranger HS2 with doAs=False' and a 'Non-Ranger doAs=true' HS2 we can at least decide which tables require Ranger benefits and then force ownership of that data to Hive and make the policcies as a one off, rather then polices for every new data source that get on-boarded. Today after a simple migration from ACL's we had 17+ pages of policies in Ranger, which we all thought was crazy when most of them don't need Ranger benefits. Just to note this is also using wildcards for resources and multiple users and groups in a policy. ... View more

Can't we just overload the HDFS Policies? So for example at a client we are using doAs false so we can use column security via Hive, but then for the 'application' that loads the data also has an HDFS Policy so it can directly run MR jobs and the like to get the data loaded for end Hive users. ... View more

So what we have done for example is made some assumptions about who will access data and how. We break this down into 2 groups of users: Analysts, and Power Users. Analysts (90% of users) ONLY access the data via Hive, they never go from HDFS or use any other tools. Analysts also need to have column level security in place to ensure they only access data related to thier clearance - ie public, pii, spii... Power Users (10 % of users) can access the datasets with any tool from Hive or HDFS and have no restrictions on the columns they can see. The service application also counts as a poweruser as it deals with the ingest and preping of the data. To facilitate this we did a few things Hive owns all the data on HDFS We no longer have to manage policies per user at an HDFS Level (due to the below) Hive doAs=false (no impersonation) This enables Ranger to decide if users can access tables or not Jobs now run as Hive, with Hive owning the data there is no no issue PowerUsers get Ranger HDFS Policies (think overloaded) This lets them access anything from HDFS We also Give Power Users Select * policies in Hive so they can query any column Hope this helps ... View more

There is also the list of all the contrib views on github. https://github.com/apache/ambari/tree/trunk/contrib/views ... View more