Cloudera Community
smartdatabundle
user rank
Contributor
My Accepted Solutions
Show All

Messaging API with Kafka : lineage

by Contributor in Support Questions
‎07-25-2017 01:47 PM
‎07-25-2017 01:47 PM
Hello, I try to create a Process Entity with Kafka : { "message": { "entities": [ { "id": { "id": "-1467290565135246000", "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "state": "ACTIVE", "typeName": "Process", "version": 0 }, "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "traitNames": [], "traits": {}, "typeName": "Process", "values": { "qualifiedName": "qn_process_cp", "processor_name": "file_copy_", "name": "n_process_cp", "inputs": [{"guid": "d5c27069-d7b6-4b58-8e61-adbdc4b8d208","typeName": "hdfs_path"}], "outputs": [{"guid": "513cd2cd-3139-4baa-a017-16f83ca2b383","typeName": "hdfs_path"}] } } ], "type": "ENTITY_CREATE", "user": "admin" }, "version": { "version": "1.0.0" } } I have this error message (on Atlas log) : 2017-07-25 13:24:58,645 DEBUG - [NotificationHookConsumer thread-0:] ~ Set property __modifiedBy = "admin" to vertex[id=163872776 type=Process guid=null] (GraphHelper:413) 2017-07-25 13:24:58,645 DEBUG - [NotificationHookConsumer thread-0:] ~ <== createStructVertex(Process) (EntityGraphMapper:188) 2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> addProperty(vertex[id=163872776 type=Process guid=null], __superTypeNames, Referenceable) (AtlasGraphUtilsV1:132) 2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> addProperty(vertex[id=163872776 type=Process guid=null], __superTypeNames, Asset) (AtlasGraphUtilsV1:132) 2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> setProperty(vertex[id=163872776 type=Process guid=null], __guid, f789fb6e-ad0e-4702-b787-e2f5f6a61bdf) (AtlasGraphUtilsV1:141) 2017-07-25 13:24:58,646 DEBUG - [NotificationHookConsumer thread-0:] ~ Setting property __guid in vertex[id=163872776 type=Process guid=null] (AtlasGraphUtilsV1:159) 2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> setProperty(vertex[id=163872776 type=Process guid=f789fb6e-ad0e-4702-b787-e2f5f6a61bdf], __version, 0) (AtlasGraphUtilsV1:141) 2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ Setting property __version in vertex[id=163872776 type=Process guid=f789fb6e-ad0e-4702-b787-e2f5f6a61bdf] (AtlasGraphUtilsV1:159) 2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> mapAttributes(CREATE, Process) (EntityGraphMapper:211) 2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> AtlasTypeRegistry.getType(Process) (AtlasStructType:82) 2017-07-25 13:24:58,648 DEBUG - [NotificationHookConsumer thread-0:] ~ <== AtlasTypeRegistry.getType(Process): org.apache.atlas.type.AtlasEntityType@75b239ad (AtlasStructType:108) 2017-07-25 13:24:58,649 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> mapArrayValue(org.apache.atlas.repository.store.graph.v1.AttributeMutationContext@fc69927c) (EntityGraphMapper:550) 2017-07-25 13:24:58,649 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> mapObjectIdValue(org.apache.atlas.repository.store.graph.v1.AttributeMutationContext@fc5b7afb) (EntityGraphMapper:437) 2017-07-25 13:24:58,649 DEBUG - [NotificationHookConsumer thread-0:] ~ <== AtlasErrorCode.getMessage([null]) (AtlasErrorCode:125) 2017-07-25 13:24:58,650 DEBUG - [NotificationHookConsumer thread-0:] ~ ==> AtlasErrorCode.getMessage([null]): ObjectId is not valid null (AtlasErrorCode:132) 2017-07-25 13:24:58,650 WARN - [NotificationHookConsumer thread-0:] ~ Error handling message (NotificationHookConsumer:325) org.apache.atlas.exception.AtlasBaseException: ObjectId is not valid null at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapObjectIdValue(EntityGraphMapper.java:455) at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapCollectionElementsToVertex(EntityGraphMapper.java:644) at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapArrayValue(EntityGraphMapper.java:568) at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapToVertexByTypeCategory(EntityGraphMapper.java:318) at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapAttribute(EntityGraphMapper.java:260) at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapAttributes(EntityGraphMapper.java:221) at org.apache.atlas.repository.store.graph.v1.EntityGraphMapper.mapAttributesAndClassifications(EntityGraphMapper.java:127) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:230) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:246) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:266) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:235) Atlas version : 0.8 Kafka version : 0.10.1 Any suggestions ? Thks ... View more
Labels:

Re: How to create a New Entity in Apache Atlas 0.7...

by Contributor in Support Questions
‎06-28-2017 01:21 PM
‎06-28-2017 01:21 PM
I never did it but I could suggest : Create a new simple type : curl -u admin:admin -ik -H "Content-Type: application/json" -X POST -d '{"enumDefs" : [], "structDefs" : [], "classificationDefs": [], "entityDefs" : [{"name": "nifi_process_test", "superTypes": ["DataSet"], "typeVersion": "1.0", "attributeDefs": [{"name": "processor_name", "typeName": "string", "cardinality": "SINGLE", "isIndexable": false, "isOptional": false, "isUnique": false } ] }] }' http://YOUR_HOST:21000/api/atlas/v2/types/typedefs I know that Nifi could connect to KAFKA via a processor. So, you just have to publish into Kafka Topic like this : 1- Open Producer console: $KAFKA_HOME/kafka-console-producer.sh --broker-list YOUR_HOST:6667 --topic ATLAS_HOOK 2- Send ENTITY_CREATE message : {"version": {"version": "1.0.0"}, "message": {"entities": [{"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id": {"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id": "-1467290565135246000", "version": 0, "typeName": "nifi_process_test", "state": "ACTIVE"}, "typeName": "nifi_process_test", "values": {"qualifiedName": "testNifi", "owner": "admin", "description": "Test NIFI", "name": "Test Nifi"}, "traitNames": [], "traits": {} }], "type": "ENTITY_CREATE", "user": "admin"} } I hope it could help you. ... View more

Re: How to create a New Entity in Apache Atlas 0.7...

by Contributor in Support Questions
‎06-28-2017 09:57 AM
‎06-28-2017 09:57 AM
Hello, You could create a simpe hdfs_path entity : curl -u admin:admin -ik -H "Content-Type: application/json" -X POST -d '{"entity": {"typeName" : "hdfs_path", "attributes" : {"name" : "hdfs_file_test", "qualifiedName" : "testfile@hdpcluster", "path" : "/user/data/test.csv", "clusterName":"hdpcluster"}}}' http://YOUR_HOST:21000/api/atlas/v2/entity ... View more

Atlas : Lineage only with REST API not with Kafka...

by Contributor in Support Questions
‎06-28-2017 08:40 AM
‎06-28-2017 08:40 AM
Hello, I used the Kafka topic (ATLAS_HOOK) to create a new entity (for hdfs_path). It seems there is no operations to create a lineage between entities and processes (At the current version). Should I use Kafka to create only entities & REST API to create lineage ? Any suggestion ? Thanks. ... View more
Labels:

Re: Kafka Producer on Apache Atlas via command lin...

by Contributor in Support Questions
‎06-26-2017 07:40 AM
‎06-26-2017 07:40 AM
Thank you su much @Sharmadha Sainath! Your comment is very useful !!! I tested these commands : $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list MY_HOSTNAME:6667 --topic ATLAS_HOOK Then in Kafka Procuder console: {"version": {"version": "1.0.0"}, "message": {"entities": [{"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id": {"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id": "-1467290565135246000", "version": 0, "typeName": "hdfs_path", "state": "ACTIVE"}, "typeName": "hdfs_path", "values": {"qualifiedName": "TestKafka", "owner": "admin", "description": "Test kafka", "path":"/user/data/testkafka.csv", posixPermissions": null, "createTime": "1970-01-01T00:00:00.000Z", "isSymlink": false, "extendedAttributes": null, "numberOfReplicas": 0, "name": "DataKFK"}, "traitNames": [], "traits": {} }], "type": "ENTITY_CREATE", "user": "admin"} } So, Atlas notifies into ATLAS_ENTITIES topic this new entity. I could see it on Atlas UI ! Have a good day ! ... View more

Re: Kafka Producer on Apache Atlas via command lin...

by Contributor in Support Questions
‎06-23-2017 03:07 PM
1 Kudo
‎06-23-2017 03:07 PM
1 Kudo
Thanks for your comment @Sharmadha Sainath I use HDP SandBox, so I use the default configuration. Mu cluster is not kerberized. First, I tried to use kafka-console-producer.sh to create an entity on ATLAS_HOOK. I had the same error message. Command : lsof -i -P -n | grep kafka Result : java x kafka x IPv6 x 0t0 TCP *:60141 (LISTEN) java x kafka x IPv6 x 0t0 TCP 192.x.x.x:49648->192.x.x.x:2181 (ESTABLISHED) java x kafka x IPv6 x 0t0 TCP 192.x.x.x:6667 (LISTEN) java x kafka x IPv6 x 0t0 TCP 1192.x.x.x:6667->192.x.x.x:32837 (ESTABLISHED) java x kafka x IPv6 x 0t0 TCP 192.x.x.x:47712->192.x.x.x:6667 (CLOSE_WAIT) java x kafka x IPv6 x 0t0 TCP 192.x.x.x:45976->192.x.x.x:6188 (ESTABLISHED) java x kafka x IPv6 x 0t0 TCP 192.x.x.x:6667->192.x.x.x:47717 (ESTABLISHED) Command : $KAFKA_BROKER_HOME/bin/kafka-topics.sh --describe --topic ATLAS_HOOK --zookeeper localhost:2181 Result : Topic:ATLAS_HOOK PartitionCount:1 ReplicationFactor:1 Configs: Topic: ATLAS_HOOK Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 ... View more

Kafka Producer on Apache Atlas via command line

by Contributor in Support Questions
‎06-23-2017 12:37 PM
‎06-23-2017 12:37 PM
Hi, I tried to use this command line : > $KAFKA_BORKER_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test > $KAFKA_BORKER_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test When I typed a test message like: test message 1 I had this error message : [2017-06-23 10:17:43,467] ERROR Error when sending message to topic test with key: null, value: 14 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. Do you have a suggestion ? A this time, I am investigated around Ranger configuration. Thanks for your help. ... View more
Labels:

Re: Atlas Cross Origin CORS

by Contributor in Support Questions
‎06-23-2017 12:24 PM
‎06-23-2017 12:24 PM
@Nixon RodriguesThank you so much for your help. It will try it; it should work. May we shoud notify a JIRA ? Or Pull Request on github for this change. It should better if we could put this on atlas-application.properties. Best regards ... View more

Re: Atlas Cross Origin CORS

by Contributor in Support Questions
‎06-08-2017 07:42 AM
‎06-08-2017 07:42 AM
Hello @Nixon Rodrigues My snippet code : <html> <head> </head> <body> <h1>TEST</h1> <script type="text/javascript"> var myTestRequest = function() { var myUrl = "http://SERVER_IP:21000/api/atlas/types/"; //Instantiate HTTP Request var request = ((window.XMLHttpRequest) ? new XMLHttpRequest() : new ActiveXObject("Microsoft.XMLHTTP")); request.open("GET", myUrl, true); request.setRequestHeader("Content-Type","application/json"); //request.channel.QueryInterface(Components.interfaces.nsIHttpChannel).redirectionLimit = 0; request.setRequestHeader("Authorization", "Basic " + btoa("admin:admin")); //request.withCredentials = true; request.send(null); } </script> <div id="myTestRequest" style="display: block"> <button onclick="myTestRequest()">Login</button> </div> </body> </html> Error (in attached file): XMLHttpRequest cannot load http://SERVER_IP:21000/api/atlas/types/. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://SERVER_IP:8000' is therefore not allowed access. The response had HTTP status code 400. Thank you for your help. ... View more

Atlas Cross Origin CORS

by Contributor in Support Questions
‎06-07-2017 03:53 PM
‎06-07-2017 03:53 PM
Hello, I would like to access to Apache Atlas via a XmlHtttRequest object. It doesn't work: Access-Control-Allow-Origin is missing. Thks for your help. ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎08-07-2017 07:18 AM
smartdatabundle's Photos
Community Statistics
Member Since ‎04-20-2017 02:01 PM
Last Visited ‎08-07-2017 07:18 AM
Posts 39
Kudos received 2