Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Export metadata to Atlas proplem with kafka

Export metadata to Atlas proplem with kafka

New Contributor

Hello everyone, I tried to export metadata and this is my logs from atlas

graph rollback due to exception (GraphTransactionInterceptor:71) org.apache.atlas.exception.AtlasBaseException: Failed to notify NotificationEntityChangeListener for change UPDATE at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:175) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.onEntitiesMutated(AtlasEntityChangeNotifier.java:84) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:165) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:173) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$FastClassBySpringCGLIB$80c00649.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$EnhancerBySpringCGLIB$fea0e85f.createOrUpdate(<generated>) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:423) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.doWork(NotificationHookConsumer.java:310) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:239) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:212) at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:116) at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:191) at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:89) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:168) ... 19 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:230) ... 24 more Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time

thank you

Any suggesion please

12 REPLIES 12

Re: Export metadata to Atlas proplem with kafka

Rising Star

@nisrine elloumi

Hi,

Could you please verify the Kafka broker and port that you have configured in atlas? And since you have tagged kerberos I assume this is kerberized env, in that case could you please also check the security.protocol in atlas properties to connect to secure Kafka.

Thank you!

Highlighted

Re: Export metadata to Atlas proplem with kafka

New Contributor

hello @dbains

how could i check the security.protocol in atlas properties to connect to secure Kafka

thank you

Re: Export metadata to Atlas proplem with kafka

New Contributor

hello @dbains

atlas.kafka.sasl.kerberos.service.name : kafka

atlas.kafka.security.protocol : PLAINTEXTSASL

Re: Export metadata to Atlas proplem with kafka

New Contributor

hello @dbains

atlas.kafka.sasl.kerberos.service.name : kafka

atlas.kafka.security.protocol : PLAINTEXTSASL

Re: Export metadata to Atlas proplem with kafka

New Contributor

hello @dbains

The kafka broker atlas entities failed

Re: Export metadata to Atlas proplem with kafka

Rising Star

@nisrine elloumi

1. May I know if you have Ranger enabled or not? and whether the user 'atlas' has proper permissions? If Ranger is enabled, please verify that policy is enabled for both topics: ATLAS_HOOK and ATLAS_ENTITIES. If Ranger is not enabled then atlas user should be provided ACLs for read/write operations.

If you are using Ranger, kindly refer the section 'Configure Atlas Security' of the following doc for detailed explanation:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-governance/content/ch_hdp_data_gove...

2. Kindly run the following command and provide the output:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper <zk_host>:2181

Also please verify the broker name in atlas properties is correct ( as configured in listeners property in kafka).

Thank you!

Re: Export metadata to Atlas proplem with kafka

New Contributor

hello @dbains

i checked the ATLAS_ENTITIES and ATLAS_HOOK and i found this :

Exception in thread "main" org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context 'Client'

i folowed this policies to add ranger kafka atlas plocies :

https://community.hortonworks.com/articles/90962/atlas-install-validation-and-troubleshooting-tips.h...

thank you

Re: Export metadata to Atlas proplem with kafka

Rising Star

Hi @nisrine elloumi

Yes, as I suspected in the first point that it could be authorization issue. Thank you for sharing.

Re: Export metadata to Atlas proplem with kafka

New Contributor

hello @dbains

what should i do ? please

thanks you