Export metadata to Atlas proplem with kafka

[#1]

Hello everyone, I tried to export metadata and this is my logs from atlas

graph rollback due to exception (GraphTransactionInterceptor:71) org.apache.atlas.exception.AtlasBaseException: Failed to notify NotificationEntityChangeListener for change UPDATE at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:175) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.onEntitiesMutated(AtlasEntityChangeNotifier.java:84) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:165) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:173) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$FastClassBySpringCGLIB$80c00649.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$EnhancerBySpringCGLIB$fea0e85f.createOrUpdate(<generated>) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:423) at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.doWork(NotificationHookConsumer.java:310) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:239) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:212) at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:116) at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:191) at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:89) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:168) ... 19 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:230) ... 24 more Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for ATLAS_ENTITIES-0 due to 30090 ms has passed since batch creation plus linger time

thank you

Any suggesion please

[#2]

@nisrine elloumi

Hi,

Could you please verify the Kafka broker and port that you have configured in atlas? And since you have tagged kerberos I assume this is kerberized env, in that case could you please also check the security.protocol in atlas properties to connect to secure Kafka.

Thank you!

[#3]

hello @dbains

how could i check the security.protocol in atlas properties to connect to secure Kafka

thank you

[#4]

hello @dbains

atlas.kafka.sasl.kerberos.service.name : kafka

atlas.kafka.security.protocol : PLAINTEXTSASL

[#5]

hello @dbains

atlas.kafka.sasl.kerberos.service.name : kafka

atlas.kafka.security.protocol : PLAINTEXTSASL

[#6]

hello @dbains

The kafka broker atlas entities failed

[#7]

@nisrine elloumi

1. May I know if you have Ranger enabled or not? and whether the user 'atlas' has proper permissions? If Ranger is enabled, please verify that policy is enabled for both topics: ATLAS_HOOK and ATLAS_ENTITIES. If Ranger is not enabled then atlas user should be provided ACLs for read/write operations.

If you are using Ranger, kindly refer the section 'Configure Atlas Security' of the following doc for detailed explanation:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-governance/content/ch_hdp_data_gove...

2. Kindly run the following command and provide the output:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper <zk_host>:2181

Also please verify the broker name in atlas properties is correct ( as configured in listeners property in kafka).

Thank you!

[#8]

hello @dbains

i checked the ATLAS_ENTITIES and ATLAS_HOOK and i found this :

Exception in thread "main" org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context 'Client'

i folowed this policies to add ranger kafka atlas plocies :

https://community.hortonworks.com/articles/90962/atlas-install-validation-and-troubleshooting-tips.h...

thank you

[#9]

Hi @nisrine elloumi

Yes, as I suspected in the first point that it could be authorization issue. Thank you for sharing.

[#10]

hello @dbains

what should i do ? please

thanks you

[#11]

Hey @nisrine elloumi
Are you still facing the error of org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context 'Client'? Is security.inter.broker.protocol also set up? If it's set to PLAINTEXTSASL, can you try modifying it to SASL_PLAINTEXT
I think PLAINTEXTSASL might be the root of your issues here, as this is what it used to be called before you upgraded your ambari (I recall from another thread that this is when your issues started).
Let's try to update to SASL_PLAINTEXT, on security.inter.broker.protocol & atlas.kafka.security.protocol...

Edit; PS security.inter.broker.protocol is set in 'custom kafka-broker'

[#12]

hi @Jonathan Sneep

i found this security.inter.broker.protocol=PLAINTEXTSASL in the 'advanced kafka-broker' and i changed to SASL_PLAINTEXT so i get an error in all my kafka brokers

thank you in advance

[#13]

hi @Jonathan Sneep

i found this security.inter.broker.protocol=PLAINTEXTSASL in the 'advanced kafka-broker' and i changed to SASL_PLAINTEXT so i get an error in all my kafka brokers

thank you in advance