Member since
04-20-2017
39
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1184 | 07-28-2017 05:41 AM |
08-07-2017
07:18 AM
Hi @Greg Keys Thks so much for your comment. I will try your advices and let you know.
... View more
08-02-2017
01:37 PM
Hello, When we use a Filter with Ranger on Hive tables: is it possible to define a dynamic value ? My use case : I have different groups on my application. Each group has at least one user. Data could be accessed by only an appropriate group. My SQL request : select * from table t1, table t2 where t1.id=t2.id and t2.group="group_admin". Instead of "group_admin" I would like to put a dynamic value. Is it possible or any suggestions ? Thks for your helps
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Ranger
08-01-2017
09:44 AM
You could use Messaging API with Kafka to access to entities. It is very impressive.
... View more
08-01-2017
09:42 AM
@Graham Martin Thks for your expanation. I am going to install & to use Kerberos.
... View more
08-01-2017
08:09 AM
@Graham Martin Thks for your quick reply. Is there an alternative to Kerberos ? May be Apache Knox + LDAP ? I went to the link: indeed, it explains only the use of Kerberos.
... View more
08-01-2017
07:08 AM
Hello, I could access all topics on kafka without authentification. My question : how I could secure access on kafka topics ? Thks.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Kafka
07-28-2017
05:41 AM
Please look at: https://community.hortonworks.com/questions/115749/messaging-api-with-kafka-lineage.html
... View more
07-28-2017
05:37 AM
@Sarath Subramanian Excellent! It works. Your are the best 🙂 Thks also to @Ashutosh Mestry for helping.
... View more
07-27-2017
07:35 AM
@Ashutosh Mestry With REST APIs: it works very well. For my use case I need to use Kafka messaging.
... View more
07-26-2017
07:13 AM
Hi @Ashutosh Mestry 2 entities exist : 513cd2cd-3139-4baa-a017-16f83ca2b383: {
"requestId": "pool-2-thread-9 - 8629abee-a74c-43a4-a969-758fd20704e4",
"definition": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "513cd2cd-3139-4baa-a017-16f83ca2b383",
"version": 0,
"typeName": "hdfs_path",
"state": "ACTIVE"
},
"typeName": "hdfs_path",
"values": {
"name": "Data_Kafka2",
"path": "/user/data/dept.csv",
"posixPermissions": null,
"createTime": "1970-01-01T00:00:00.000Z",
"description": "Test with Kafka",
"isSymlink": false,
"extendedAttributes": null,
"numberOfReplicas": 0,
"qualifiedName": "deptData2",
"isFile": false,
"fileSize": 0,
"owner": "admin",
"modifiedTime": "1970-01-01T00:00:00.000Z",
"clusterName": null,
"group": null
},
"traitNames": [
"Catalog.Technical.IT",
"Retainable"
],
"traits": {
"Catalog.Technical.IT": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
"typeName": "Catalog.Technical.IT",
"values": {
"name": "Catalog.Technical.IT",
"acceptable_use": null,
"description": "technical test",
"available_as_tag": false,
"taxonomy.namespace": "atlas.taxonomy"
}
},
"Retainable": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
"typeName": "Retainable",
"values": {
"retentionPeriod": 100
}
}
},
"systemAttributes": {
"createdBy": "admin",
"modifiedBy": "admin",
"createdTime": "2017-06-28T07:32:37.288Z",
"modifiedTime": "2017-06-29T12:34:30.304Z"
}
}
}
d5c27069-d7b6-4b58-8e61-adbdc4b8d208 :
{
"requestId": "pool-2-thread-6 - e16f006a-962d-45b5-a2b6-6914ec6ea0ab",
"definition": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "d5c27069-d7b6-4b58-8e61-adbdc4b8d208",
"version": 0,
"typeName": "hdfs_path",
"state": "ACTIVE"
},
"typeName": "hdfs_path",
"values": {
"name": "Data_Kafka1",
"path": "/user/data/dept.csv",
"posixPermissions": null,
"createTime": "1970-01-01T00:00:00.000Z",
"description": "Test with Kafka",
"isSymlink": false,
"extendedAttributes": null,
"numberOfReplicas": 0,
"qualifiedName": "deptData",
"isFile": false,
"fileSize": 0,
"owner": "admin",
"modifiedTime": "1970-01-01T00:00:00.000Z",
"clusterName": null,
"group": null
},
"traitNames": [ ],
"traits": { },
"systemAttributes": {
"createdBy": "admin",
"modifiedBy": "admin",
"createdTime": "2017-07-21T07:40:37.357Z",
"modifiedTime": "2017-07-21T07:52:59.405Z"
}
}
}
... View more
07-26-2017
06:41 AM
Hi @Sarath Subramanian This is my message: {
"message": {
"entities": [
{
"id": {
"id": "-1467290565135246000",
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"state": "ACTIVE",
"typeName": "hdfs_path",
"version": 0
},
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"traitNames": [],
"traits": {},
"typeName": "hdfs_path",
"values": {
"createTime": "1970-01-01T00:00:00.000Z",
"description": "Test with Kafka",
"extendedAttributes": null,
"isSymlink": false,
"name": "Data_Kafka",
"numberOfReplicas": 0,
"owner": "admin",
"path": "/user/data/dept.csv",
"posixPermissions": null,
"qualifiedName": "deptData"
}
}
],
"type": "ENTITY_CREATE",
"user": "admin"
},
"version": {
"version": "1.0.0"
}
}
... View more
07-25-2017
01:47 PM
Hello, I try to create a Process Entity with Kafka : {
"message": {
"entities": [
{
"id": {
"id": "-1467290565135246000",
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"state": "ACTIVE",
"typeName": "Process",
"version": 0
},
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"traitNames": [],
"traits": {},
"typeName": "Process",
"values": {
"qualifiedName": "qn_process_cp",
"processor_name": "file_copy_",
"name": "n_process_cp",
"inputs": [{"guid": "d5c27069-d7b6-4b58-8e61-adbdc4b8d208","typeName": "hdfs_path"}],
"outputs": [{"guid": "513cd2cd-3139-4baa-a017-16f83ca2b383","typeName": "hdfs_path"}]
}
}
],
"type": "ENTITY_CREATE",
"user": "admin"
},
"version": {
"version": "1.0.0"
}
} I have this error message (on Atlas log) : 2017-07-25 13:24:58,645 DEBUG - [NotificationHookConsumer thread-0:] ~ Set property __modifiedBy = "admin" to vertex[id=163872776 type=Process guid=null] (GraphHelper:413)
2017-07-25 13:24:58,645 DEBUG - [NotificationHookConsumer thread-0:] ~ <== createStructVertex(Process) (EntityGraphMapper:188)
2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> addProperty(vertex[id=163872776 type=Process guid=null], __superTypeNames, Referenceable) (AtlasGraphUtilsV1:132)
2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> addProperty(vertex[id=163872776 type=Process guid=null], __superTypeNames, Asset) (AtlasGraphUtilsV1:132)
2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> setProperty(vertex[id=163872776 type=Process guid=null], __guid, f789fb6e-ad0e-4702-b787-e2f5f6a61bdf) (AtlasGraphUtilsV1:141)
2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ Setting property __guid in vertex[id=163872776 type=Process guid=null] (AtlasGraphUtilsV1:159)
2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> setProperty(vertex[id=163872776 type=Process guid=f789fb6e-ad0e-4702-b787-e2f5f6a61bdf], __version, 0) (AtlasGraphUtilsV1:141)
2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ Setting property __version in vertex[id=163872776 type=Process guid=f789fb6e-ad0e-4702-b787-e2f5f6a61bdf] (AtlasGraphUtilsV1:159)
2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> mapAttributes(CREATE, Process) (EntityGraphMapper:211)
2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> AtlasTypeRegistry.getType(Process) (AtlasStructType:82)
2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ <== AtlasTypeRegistry.getType(Process): org.apache.atlas.type.AtlasEntityType@75b239ad (AtlasStructType:108)
2017-07-25 13:24:58,649 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> mapArrayValue(org.apache.atlas.repository.store.graph.v1.AttributeMutationContext@fc69927c) (EntityGraphMapper:550)
2017-07-25 13:24:58,649 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> mapObjectIdValue(org.apache.atlas.repository.store.graph.v1.AttributeMutationContext@fc5b7afb) (EntityGraphMapper:437)
2017-07-25 13:24:58,649 DEBUG - [NotificationHookConsumer thread-0:] ~ <== AtlasErrorCode.getMessage([null]) (AtlasErrorCode:125)
2017-07-25 13:24:58,650 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> AtlasErrorCode.getMessage([null]): ObjectId is not valid null (AtlasErrorCode:132)
2017-07-25 13:24:58,650 WARN - [NotificationHookConsumer thread-0:] ~ Error handling message (NotificationHookConsumer:325)
org.apache.atlas.exception.AtlasBaseException: ObjectId is not valid null
at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapObjectIdValue(EntityGraphMapper.java:455)
at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapCollectionElementsToVertex(EntityGraphMapper.java:644)
at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapArrayValue(EntityGraphMapper.java:568)
at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapToVertexByTypeCategory(EntityGraphMapper.java:318)
at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapAttribute(EntityGraphMapper.java:260)
at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapAttributes(EntityGraphMapper.java:221)
at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapAttributesAndClassifications(EntityGraphMapper.java:127)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:230)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:246)
at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:266)
at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:235)
Atlas version : 0.8 Kafka version : 0.10.1 Any suggestions ? Thks
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Kafka
06-28-2017
01:21 PM
I never did it but I could suggest : Create a new simple type :
curl -u admin:admin -ik -H "Content-Type: application/json" -X POST -d '{"enumDefs" : [], "structDefs" : [], "classificationDefs": [], "entityDefs" : [{"name": "nifi_process_test", "superTypes": ["DataSet"], "typeVersion": "1.0", "attributeDefs": [{"name": "processor_name", "typeName": "string", "cardinality": "SINGLE", "isIndexable": false, "isOptional": false, "isUnique": false } ] }] }' http://YOUR_HOST:21000/api/atlas/v2/types/typedefs I know that Nifi could connect to KAFKA via a processor. So, you just have to publish into Kafka Topic like this : 1- Open Producer console:
$KAFKA_HOME/kafka-console-producer.sh --broker-list YOUR_HOST:6667 --topic ATLAS_HOOK
2- Send ENTITY_CREATE message :
{"version": {"version": "1.0.0"}, "message": {"entities": [{"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id": {"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id": "-1467290565135246000", "version": 0, "typeName": "nifi_process_test", "state": "ACTIVE"}, "typeName": "nifi_process_test", "values": {"qualifiedName": "testNifi", "owner": "admin", "description": "Test NIFI", "name": "Test Nifi"}, "traitNames": [], "traits": {} }], "type": "ENTITY_CREATE", "user": "admin"} }
I hope it could help you.
... View more
06-28-2017
09:57 AM
Hello, You could create a simpe hdfs_path entity : curl -u admin:admin -ik -H "Content-Type: application/json" -X POST -d '{"entity": {"typeName" : "hdfs_path", "attributes" : {"name" : "hdfs_file_test", "qualifiedName" : "testfile@hdpcluster", "path" : "/user/data/test.csv", "clusterName":"hdpcluster"}}}' http://YOUR_HOST:21000/api/atlas/v2/entity
... View more
06-28-2017
08:40 AM
Hello, I used the Kafka topic (ATLAS_HOOK) to create a new entity (for hdfs_path).
It seems there is no operations to create a lineage between entities and processes (At the current version).
Should I use Kafka to create only entities & REST API to create lineage ? Any suggestion ? Thanks.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Kafka
06-26-2017
07:40 AM
Thank you su much @Sharmadha Sainath! Your comment is very useful !!! I tested these commands : $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list MY_HOSTNAME:6667 --topic ATLAS_HOOK
Then in Kafka Procuder console: {"version": {"version": "1.0.0"}, "message": {"entities": [{"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id": {"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id": "-1467290565135246000", "version": 0, "typeName": "hdfs_path", "state": "ACTIVE"}, "typeName": "hdfs_path", "values": {"qualifiedName": "TestKafka", "owner": "admin", "description": "Test kafka", "path":"/user/data/testkafka.csv", posixPermissions": null, "createTime": "1970-01-01T00:00:00.000Z", "isSymlink": false, "extendedAttributes": null, "numberOfReplicas": 0, "name": "DataKFK"}, "traitNames": [], "traits": {} }], "type": "ENTITY_CREATE", "user": "admin"} }
So, Atlas notifies into ATLAS_ENTITIES topic this new entity. I could see it on Atlas UI ! Have a good day !
... View more
06-23-2017
03:07 PM
1 Kudo
Thanks for your comment @Sharmadha Sainath I use HDP SandBox, so I use the default configuration. Mu cluster is not kerberized. First, I tried to use kafka-console-producer.sh to create an entity on ATLAS_HOOK. I had the same error message. Command : lsof -i -P -n | grep kafka
Result : java x kafka x IPv6 x 0t0 TCP *:60141 (LISTEN)
java x kafka x IPv6 x 0t0 TCP 192.x.x.x:49648->192.x.x.x:2181 (ESTABLISHED)
java x kafka x IPv6 x 0t0 TCP 192.x.x.x:6667 (LISTEN)
java x kafka x IPv6 x 0t0 TCP 1192.x.x.x:6667->192.x.x.x:32837 (ESTABLISHED)
java x kafka x IPv6 x 0t0 TCP 192.x.x.x:47712->192.x.x.x:6667 (CLOSE_WAIT)
java x kafka x IPv6 x 0t0 TCP 192.x.x.x:45976->192.x.x.x:6188 (ESTABLISHED)
java x kafka x IPv6 x 0t0 TCP 192.x.x.x:6667->192.x.x.x:47717 (ESTABLISHED)
Command : $KAFKA_BROKER_HOME/bin/kafka-topics.sh --describe --topic ATLAS_HOOK --zookeeper localhost:2181
Result : Topic:ATLAS_HOOK PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ATLAS_HOOK Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
... View more
06-23-2017
12:37 PM
Hi, I tried to use this command line : > $KAFKA_BORKER_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
> $KAFKA_BORKER_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
When I typed a test message like: test message 1 I had this error message : [2017-06-23 10:17:43,467] ERROR Error when sending message to topic test with key: null, value: 14 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
Do you have a suggestion ? A this time, I am investigated around Ranger configuration. Thanks for your help.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Kafka
06-23-2017
12:24 PM
@Nixon RodriguesThank you so much for your help. It will try it; it should work. May we shoud notify a JIRA ? Or Pull Request on github for this change. It should better if we could put this on atlas-application.properties. Best regards
... View more
06-08-2017
07:42 AM
Hello @Nixon Rodrigues My snippet code : <html>
<head>
</head>
<body>
<h1>TEST</h1>
<script type="text/javascript">
var myTestRequest = function() {
var myUrl = "http://SERVER_IP:21000/api/atlas/types/";
//Instantiate HTTP Request
var request = ((window.XMLHttpRequest) ? new XMLHttpRequest() : new ActiveXObject("Microsoft.XMLHTTP"));
request.open("GET", myUrl, true);
request.setRequestHeader("Content-Type","application/json");
//request.channel.QueryInterface(Components.interfaces.nsIHttpChannel).redirectionLimit = 0;
request.setRequestHeader("Authorization", "Basic " + btoa("admin:admin"));
//request.withCredentials = true;
request.send(null);
}
</script>
<div id="myTestRequest" style="display: block">
<button onclick="myTestRequest()">Login</button>
</div>
</body>
</html>
Error (in attached file):
XMLHttpRequest cannot load http://SERVER_IP:21000/api/atlas/types/. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://SERVER_IP:8000' is therefore not allowed access. The response had HTTP status code 400. Thank you for your help.
... View more
06-07-2017
03:53 PM
Hello, I would like to access to Apache Atlas via a XmlHtttRequest object. It doesn't work: Access-Control-Allow-Origin is missing. Thks for your help.
... View more
Labels:
- Labels:
-
Apache Atlas
05-17-2017
09:37 AM
Hi, The easiest way to find your entity: 1/- On Basic/advanced mode: - select hive_db on Search by Type 2/- Or on advanced mode (DSL), you could filter on Search by Query & just put that: hive_db. To filter by owner, for instance: hive_db where owner="public"
... View more
05-15-2017
01:31 PM
Thank you for your reply. I will try LDAP mode...
... View more
05-15-2017
08:53 AM
Thank you for your reply. Indeed, the value is NONE.
... View more
05-04-2017
02:47 PM
Hi all,
I installed Hive on HDP 2.5. I configured Ranger to allow access to the HDFS path for only hive user.
In Audit Ranger, there is an Anonymous user who wants to use /tmp/hive on HDFS.
What is the purpose of this Anonymous user ?
Thank you for your advice.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Ranger
05-02-2017
06:45 AM
@Ashutosh Mestry Thank you for your reply. Probably it would be interesting to develop a specific bridge for HDFS?
... View more
04-27-2017
08:30 AM
Hello, I use HDP 2.6. I would like to create data lineage for Hdfs files with Apache Atlas. The ideal solution uses Apache Atlas to make this data lineage probably via an Hdfs Hook. But this current version seems there is no Hdfs Hook. So I am asking you some advices : - have I create a HDFS entities to manage this lineage ? - Have I create a Hive external table to realize this lineage ? Than you for your help.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hadoop
04-26-2017
08:15 AM
I will recognize that Hortonworks could be a greatest company in Open Source Community mainly in Big Data technologies if Hortonworks could share more details :)
In any case, thank you for having taken the time to reply.
have a good day,
... View more
04-24-2017
03:59 PM
Thks. Please could you tell me more about this new solution ? This new development will be included in Apache Atlas ?
... View more