Unable to Create entity in Atlas

[#1]

Hi Team,

We are using HDP-2.6.5.0 and Ambari-2.6.2.0. While creating new entity in Atlas we are getting following error in application.log. Currently we are not using Ranger. we trying to create new entity without ranger.

Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
        at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:220)
        at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:182)
        at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:117)
        at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:208)
        at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:102)

        at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityC

hangeNotifier.java:176)
        ... 100 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
        at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:207)
        ... 105 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ ==> releaseLockedObjects(): lockedGuids.size: 0 (GraphTransactionInterceptor:187)
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ <== releaseLockedObjects(): lockedGuids.size: 0 (GraphTransactionInterceptor:197)
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ PERF|EntityREST.createOrUpdate()|30228 (EntityREST:77)
2018-11-28 01:35:02,638 ERROR - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ Error handling a request: 168f59a001543090 (ExceptionMapperUtil:32)
org.apache.atlas.exception.AtlasBaseException: Failed to notify NotificationEntityChangeListener for change UPDATE
        at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:183)


 

I can describe topic for ATLAS_ENTITIES

/usr/hdp/2.6.5.0-292/kafka/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper hadmgrndcc03-1.lifeway.org:2181,hadmgrndcc03-2.lifeway.org:2181,hadmgrndcc03-3.lifeway.org:2181
Topic:ATLAS_ENTITIES    PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ATLAS_ENTITIES   Partition: 0    Leader: 1003    Replicas: 1003  Isr: 1003
ACL Permission:-
 /usr/hdp/2.6.5.0-292/kafka/bin/kafka-acls.sh --add --allow-principal user:* --producer --topic ATLAS_ENTITIES --authorizer-properties zookeeper.connect=hadmgrndcc03-1.lifeway.org:2181,hadmgrndcc03-2.lifeway.org:2181,hadmgrndcc03-3.lifeway.org:2181
Adding ACLs for resource `Topic:ATLAS_ENTITIES`:
        user:* has Allow permission for operations: Write from hosts: *
        user:* has Allow permission for operations: Describe from hosts: *
Adding ACLs for resource `Cluster:kafka-cluster`:
        user:* has Allow permission for operations: Create from hosts: *
Current ACLs for resource `Topic:ATLAS_ENTITIES`:
        User: has Allow permission for operations: Read from hosts: *
        User: has Allow permission for operations: Describe from hosts: *
        user:* has Allow permission for operations: Write from hosts: *
        user:* has Allow permission for operations: Describe from hosts: *
        User:* has Allow permission for operations: Write from hosts: *
        user:* has Allow permission for operations: Write from hosts: application.log
        User:* has Allow permission for operations: Describe from hosts: *
        User:atlas has Allow permission for operations: Write from hosts: *
        user:* has Allow permission for operations: Read from hosts: application.log
        User:* has Allow permission for operations: Read from hosts: *
        User:atlas has Allow permission for operations: Describe from hosts: *

Could you please help in this issue and provide a solution for it.

Thanks in advance.

Regards,

Owez Mujawar

[#2]

@Owez Mujawar

You see the NotificationException because Atlas is unable to send a message to kafka topic ATLAS_ENTITIES. Can you check if Kafka service is healthy? Another possible reason is when a new kafka broker was added and assigned leader and atlas doesn't know about it. To isolate this issue, can you try publishing a sample message to ATLAS_ENTITIES topic as atlas user and see if you see any permission issues.

[#3]

@Sarath Subramanian

Hi,

As we are trying to execute the query but we are failing. below error for the same.

could you please help me to resolved this issue.

atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/current/atlas-server/bin/quick_start.py
Enter username for atlas :- admin
Enter password for atlas :-
Creating sample types:
Created type [DB]
Created type [Table]
Created type [StorageDesc]
Created type [Column]
Created type [LoadProcess]
Created type [View]
Created type [JdbcAccess]
Created type [ETL]
Created type [Metric]
Created type [PII]
Created type [Fact]
Created type [Dimension]
Created type [Log Data]
Creating sample entities:
Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClientV2$API_V2@514646ef failed with status 500 (Internal Server Error) Response Body ({"errorCode":"ATLAS-500-00-007","errorMessage":"Failed to notify NotificationEntityChangeListener for change CREATE"})
        at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:387)
        at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:320)
        at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:208)
        at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:281)
        at org.apache.atlas.examples.QuickStartV2.createInstance(QuickStartV2.java:339)
        at org.apache.atlas.examples.QuickStartV2.createDatabase(QuickStartV2.java:362)
        at org.apache.atlas.examples.QuickStartV2.createEntities(QuickStartV2.java:268)
        at org.apache.atlas.examples.QuickStartV2.runQuickstart(QuickStartV2.java:150)
        at org.apache.atlas.examples.QuickStartV2.main(QuickStartV2.java:132)
No sample data added to Apache Atlas Server.

Thanks in advance.

Regards,

Owez Mujawar

[#4]

@Sarath Subramanian

Please find below issue while publishing message to ATLAS_ENTITIES.

atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-producer.sh --broker-list hadwkrndcc03-3.lifeway.org:6667 --topic ATLAS_ENTITIES
>olla
>[2018-11-29 05:50:15,694] ERROR Error when sending message to topic ATLAS_ENTITIES with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 1526 ms has passed since batch creation plus linger time

Could you please help in this issue and provide a solution for it.

Thanks in advance.

Regards,

Owez Mujawar

[#5]

@Owez Mujawar

So you see the same exception when you try to publish a sample message to ATLAS_ENTITIES topic as atlas user. Some kafka config has been messed up - kafka brokers have been added or removed.

Please follow https://community.hortonworks.com/content/supportkb/150063/how-to-add-new-kafka-brokers-to-a-cluster...

to reassign ATLAS_ENTITIES/ATLAS_HOOK topic with new broker information.

[#6]

@Owez Mujawar

So you see the same exception when you try to publish a sample message to ATLAS_ENTITIES topic as atlas user. Some kafka config has been messed up - kafka brokers have been added or removed.

Please follow https://community.hortonworks.com/content/supportkb/150063/how-to-add-new-kafka-brokers-to-a-cluster...

to reassign ATLAS_ENTITIES/ATLAS_HOOK topic with new broker information.

[#7 · Accepted solution]

@Sarath Subramanian

now i am able publish a sample message to ATLAS_ENTITIES.

Resolution:

Deleted the older Atlas_entities and created a new one.

Thanks for the help.

Regards,

Owez Mujawar