Created 07-26-2018 02:15 PM
Hello everyone, I tried to export metadata and this is my logs from atlas
graph rollback due to exception (GraphTransactionInterceptor:71) org.apache.atlas.exception.AtlasBaseException: Failed to notify NotificationEntityChangeListener for change UPDATE at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:175) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.onEntitiesMutated(AtlasEntityChangeNotifier.java:84) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:165) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:173) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$FastClassBySpringCGLIB$80c00649.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$EnhancerBySpringCGLIB$fea0e85f.createOrUpdate(<generated>) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:423) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.doWork(NotificationHookConsumer.java:310) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:239) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:212) at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:116) at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:191) at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:89) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:168) ... 19 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:230) ... 24 more Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time
thank you
Any suggesion please
Created 07-26-2018 08:29 PM
Hi,
Could you please verify the Kafka broker and port that you have configured in atlas? And since you have tagged kerberos I assume this is kerberized env, in that case could you please also check the security.protocol in atlas properties to connect to secure Kafka.
Thank you!
Created 07-27-2018 12:54 PM
hello @dbains
how could i check the security.protocol in atlas properties to connect to secure Kafka
thank you
Created 07-27-2018 01:05 PM
hello @dbains
atlas.kafka.sasl.kerberos.service.name : kafka
atlas.kafka.security.protocol : PLAINTEXTSASL
Created 07-27-2018 01:05 PM
hello @dbains
atlas.kafka.sasl.kerberos.service.name : kafka
atlas.kafka.security.protocol : PLAINTEXTSASL
Created 07-27-2018 02:22 PM
hello @dbains
The kafka broker atlas entities failed
Created 07-27-2018 07:53 PM
1. May I know if you have Ranger enabled or not? and whether the user 'atlas' has proper permissions? If Ranger is enabled, please verify that policy is enabled for both topics: ATLAS_HOOK and ATLAS_ENTITIES. If Ranger is not enabled then atlas user should be provided ACLs for read/write operations.
If you are using Ranger, kindly refer the section 'Configure Atlas Security' of the following doc for detailed explanation:
2. Kindly run the following command and provide the output:
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper <zk_host>:2181
Also please verify the broker name in atlas properties is correct ( as configured in listeners property in kafka).
Thank you!
Created 07-30-2018 10:00 AM
hello @dbains
i checked the ATLAS_ENTITIES and ATLAS_HOOK and i found this :
Exception in thread "main" org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context 'Client'
i folowed this policies to add ranger kafka atlas plocies :
thank you
Created 07-30-2018 06:47 PM
Yes, as I suspected in the first point that it could be authorization issue. Thank you for sharing.
Created 07-31-2018 08:58 AM
hello @dbains
what should i do ? please
thanks you