Atlas Tags not getting synced to Ranger

[#1]

We are trying to setup Ranger for Atlas and not able to get Ranger Tagsync working.We have created a classification tag in atlas and added it to a hive_column.

Here is the snippet from ranger tagsync logs:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}



27 Sep 2018 15:37:38 DEBUG AtlasResourceMapperUtil [Thread-6] - 71 <== getRangerServiceResource(65ff70ed-09ab-4470-b407-81f5c40d4931): resource=RangerServiceResource={ id={null} guid={65ff70ed-09ab-4470-b407-81f5c40d4931} isEnabled={true} createdBy={null} updatedBy={null} createTime={null} updateTime={null} version={null} guid={65ff70ed-09ab-4470-b407-81f5c40d4931} serviceName={atlastest_hive} resourceElements={database={RangerPolicyResource={values={xxxxxx } isExcludes={false} isRecursive={false} }} column={RangerPolicyResource={values={customer } isExcludes={false} isRecursive={false} }} table={RangerPolicyResource={values={xxxxxx } isExcludes={false} isRecursive={false} }} } resourceSignature={null}  }

27 Sep 2018 15:37:39 DEBUG AbstractTagSource [Thread-6] - 64 Uploading serviceTags={"op":"add_or_update","serviceName”:”atlastest_hive","tagVersion":0,"tagDefinitions":{"0":{"name":"mytestpii","source":"Atlas","attributeDefs":[],"id":0,"isEnabled":true}},"tags":{"0":{"type":"mytestpii","owner":0,"attributes":{},"options":{},"validityPeriods":[],"id":0,"isEnabled":true}},"serviceResources":[{"serviceName”:”atlastest_hive","resourceElements":{"database":{"values”:[“xxxxxx”],”isExcludes":false,"isRecursive":false},"column":{"values":["customer"],"isExcludes":false,"isRecursive":false},"table":{"values”:["xxxxx"],"isExcludes":false,"isRecursive":false}},"id":0,"guid":"65ff70ed-09ab-4470-b407-81f5c40d4931","isEnabled":true}],"resourceToTagIds":{"0":[0]}}

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-6] - 129 ==> upload()

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 190 ==> doUpload()

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 222 ==> uploadServiceTagCache()

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 239 ==> tryWithCred

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 250 <== tryWithCred

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 232 <== uploadServiceTagCache()

27 Sep 2018 15:37:39 ERROR TagAdminRESTSink [Thread-5] - 204 Upload of service-tags failed with message HTTP 400

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 214 <== doUpload()

27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-6] - 140 <== upload()

We have an external hive metastore(which is not managed by Ranger) and we run a hive hook there which pushes the updates to our Atlas Instance(via Kafka).

Is it mandatory to run the Ranger plugin on this external hive instance to get the Tagsync working?

Also, we came across couple of posts where it is advised to add this property as the clustername that was used to get data to Atlas might be different from Ranger's constructed service name - TAGSYNC_ATLAS_TO_RANGER_SERVICE_MAPPING.

Can you please provide an example for what values can TAGSYNC_ATLAS_TO_RANGER_SERVICE_MAPPING have(in the format 'clusterName,componentName,serviceName')

Thanks

[#2]

@Megha U

Is it mandatory to run the Ranger plugin on this external hive instance to get the Tagsync working? => No it is not mandatory to run ranger plugin on your external hive as long as you have the service with name = 'atlastest_hive' defined in ranger admin.

1. Can you confirm if you have the service with name defined 'atlastest_hive' in Ranger ?

2. If you have the service 'atlastest_hive' defined already, can you provide us with ranger admin logs to help debug further.

[#3]

@Sarath Subramanian :

1. Can you confirm if you have the service with name defined 'atlastest_hive' in Ranger ? - No, we don't have any such service running in Ranger. We only have Atlas plugin enabled in ranger. So, in Ranger UI > Audit > Plugins - we only see one service(atlas service) being listed

Just to give a brief of what we have done. We have an external hive metastore on which hive hook is running and this pushes data to Atlas kafka. The cluster name used in this hive hook is "atlastest". For any columns coming from this hive store, the qualified name is xxxxxx.xxxxxxx.xxxxxx@atlastest.

We have added a few tags in Atlas and tagged such hive columns with those tags. When we go to Ranger UI > Access Manager > Tag Based Policies > Add new Policy : We don’t see any of the Atlas Tags here.

And we are seeing the error that was posted earlier.

Will this information help ?

What should I be looking for in Ranger Admin logs ?

Thanks !

[#4]

@Megha U

The reason for the error reported on tagsync log (and returned by ranger-admin) is directly because there is no service "atlastest_hive" in the ranger-admin. As part of processing of kafka notification for tag creation, tagsync builds a ranger-admin request from the data in the notification. This request contains the service-name to which tagged entity belongs, and sends it over to ranger-admin for persisting in the ranger-admin database. When ranger-admin validates it, it does find the corresponding service and hence the error.

What do you want to do with tags in your installation? If you don't plan to use ranger-authorizer for hive-server2, then the tagged entities are not expected to be authorized by ranger and tags (which are basically classifications for hive entities) are not very useful to you.

Thanks,