Member since
09-18-2018
4
Posts
0
Kudos Received
0
Solutions
11-06-2018
03:04 PM
Hi, Use case is to search for a hive_table or a hive_column where name contains "customers" or description contains "customers" or comment contains "customers" Using the search api : /v2/search/basic , we can achieve this for one particular type(hive_table) by using the following json. How can I query for both hive_table and hive_column in the same query ? Is it possible ? One option is to use a full text search, But that will return me all the types (But, I am only interested in a subset of types) p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545}
span.Apple-tab-span {white-space:pre} { "typeName": "hive_table”, "excludeDeletedEntities": true, "offset": 0, "limit":25, "entityFilters": { "condition": "OR", "criterion": [ { "attributeName":"name", "operator": "contains", "attributeValue": "customer" }, { "attributeName":"comment", "operator": "contains", "attributeValue": "provider" }, { "attributeName":"description", "operator": "contains", "attributeValue": "provider" } ] }, "attributes": [ "db", "qualifiedName" ] } Thanks
... View more
Labels:
- Labels:
-
Apache Atlas
09-29-2018
04:28 AM
@Sarath Subramanian : 1. Can you confirm if you have the service with name defined 'atlastest_hive' in Ranger ? - No, we don't have any such service running in Ranger. We only have Atlas plugin enabled in ranger. So, in Ranger UI > Audit > Plugins - we only see one service(atlas service) being listed Just to give a brief of what we have done. We have an external hive metastore on which hive hook is running and this pushes data to Atlas kafka. The cluster name used in this hive hook is "atlastest". For any columns coming from this hive store, the qualified name is xxxxxx.xxxxxxx.xxxxxx@atlastest. We have added a few tags in Atlas and tagged such hive columns with those tags. When we go to Ranger UI > Access Manager > Tag Based Policies > Add new Policy : We don’t see any of the Atlas Tags here. And we are seeing the error that was posted earlier. Will this information help ? What should I be looking for in Ranger Admin logs ? Thanks !
... View more
09-28-2018
03:19 AM
We are trying to setup Ranger for Atlas and not able to get Ranger Tagsync working.We have created a classification tag in atlas and added it to a hive_column. Here is the snippet from ranger tagsync logs: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
27 Sep 2018 15:37:38 DEBUG AtlasResourceMapperUtil [Thread-6] - 71 <== getRangerServiceResource(65ff70ed-09ab-4470-b407-81f5c40d4931): resource=RangerServiceResource={ id={null} guid={65ff70ed-09ab-4470-b407-81f5c40d4931} isEnabled={true} createdBy={null} updatedBy={null} createTime={null} updateTime={null} version={null} guid={65ff70ed-09ab-4470-b407-81f5c40d4931} serviceName={atlastest_hive} resourceElements={database={RangerPolicyResource={values={xxxxxx } isExcludes={false} isRecursive={false} }} column={RangerPolicyResource={values={customer } isExcludes={false} isRecursive={false} }} table={RangerPolicyResource={values={xxxxxx } isExcludes={false} isRecursive={false} }} } resourceSignature={null} }
27 Sep 2018 15:37:39 DEBUG AbstractTagSource [Thread-6] - 64 Uploading serviceTags={"op":"add_or_update","serviceName”:”atlastest_hive","tagVersion":0,"tagDefinitions":{"0":{"name":"mytestpii","source":"Atlas","attributeDefs":[],"id":0,"isEnabled":true}},"tags":{"0":{"type":"mytestpii","owner":0,"attributes":{},"options":{},"validityPeriods":[],"id":0,"isEnabled":true}},"serviceResources":[{"serviceName”:”atlastest_hive","resourceElements":{"database":{"values”:[“xxxxxx”],”isExcludes":false,"isRecursive":false},"column":{"values":["customer"],"isExcludes":false,"isRecursive":false},"table":{"values”:["xxxxx"],"isExcludes":false,"isRecursive":false}},"id":0,"guid":"65ff70ed-09ab-4470-b407-81f5c40d4931","isEnabled":true}],"resourceToTagIds":{"0":[0]}}
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-6] - 129 ==> upload()
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 190 ==> doUpload()
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 222 ==> uploadServiceTagCache()
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 239 ==> tryWithCred
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 250 <== tryWithCred
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 232 <== uploadServiceTagCache()
27 Sep 2018 15:37:39 ERROR TagAdminRESTSink [Thread-5] - 204 Upload of service-tags failed with message HTTP 400
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-5] - 214 <== doUpload()
27 Sep 2018 15:37:39 DEBUG TagAdminRESTSink [Thread-6] - 140 <== upload()
We have an external hive metastore(which is not managed by Ranger) and we run a hive hook there which pushes the updates to our Atlas Instance(via Kafka). Is it mandatory to run the Ranger plugin on this external hive instance to get the Tagsync working? Also, we came across couple of posts where it is advised to add this property as the clustername that was used to get data to Atlas might be different from Ranger's constructed service name - TAGSYNC_ATLAS_TO_RANGER_SERVICE_MAPPING. Can you please provide an example for what values can TAGSYNC_ATLAS_TO_RANGER_SERVICE_MAPPING have(in the format 'clusterName,componentName,serviceName') Thanks
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Ranger