Hi All, In the secured cluster, we have a HiveServer2 started as "hive". When we connect to Hive via Beeline, Ranger policies (defined via Hive Ranger plugin) are respected as the control flow goes through HiveServer2. It verifies if the user has access to table x,y,z or not,etc. Since we have set impersonation set to false, the query gets executed as hive user, and hive user has more permission on HDFS and things work well in the case of Beeline-Hive scenario. Now, with Spark, when the end user needs to connect to Hive using Spark-Shell or Python-shell for example, we see that the connections directly go to HiveMetaStore and not HS2, so Ranger does not play its part. Also all the queries are executed as the end user, obviously, the end user does not have permission to access the file directly on HDFS and the Spark-Hive fails. I know its a pending issue, but for now we need a work around so that we can give end user permissions to access the data, without creating (or with minimally changing ) existing hdfs access permissions. (This is on HDP 2.5.x) Many thanks, Arpan ... View more

Hi All,   In our CDH production cluster, we have setup spark, we plan to give access to all the end users/developers/BI users,etc. But we learnt any valid user after getting their own user kerb TGT, can get hold of sqlContext (in program or in shell) and can run any query against any secure databases.   This puts us in a critical condition as we do not want to give blanket permission to everyone. We are looking forward to a solution or a work around, by which we can give secure access only to the selected users to sensetive tables/database.   Failing to do so, we would like to remove/disable the SparkSQL context/feature for everyone.     Any pointers in this direction will be very valueable.   Thank you, Arpan   ... View more

Guys,   I am t rying to run the Spark PI application in YARN cluster mode, here are the steps briefly. (This works for service users like spark, hdfs, hive,etc) - try doing kinit using spark headless keytab - run sparkPI example, check application ID - check yarn logs using yarn logs -applicationId <ID> I see the logs / PI value,etc.  Now follow these steps: - kdestory - kinit adm_user (user in active directory) - check klist and user has valid TGT. - run same spark PI application in YARN Cluster mode - get the application id from history server or from console. - Execute : yarn logs -applicationId <appID> from above step. I see following : 16/06/09 12:11:53 INFO impl.TimelineClientImpl: Timeline service address: http://host:8188/ws/v1/timeline/ 16/06/09 12:11:54 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 /app-logs/<USER_WHO_WAS_USED_TO_KINIT>/logs/application_1465312809105_0024 does not have any log files. I checked /logs/<KINIT_USER>/logs/applicationID, indeed it does not have any logs. Could you please suggest what is missing?   Thanks in advance, Arpan   ... View more