Sqoop kafka atlas JAAS configuration error

[#1]

I try to import a sql query from RDBMS to hive table. I used these query hundreds times but today I got strange error related Kafka and Atlas.

18/07/06 17:17:49 ERROR security.InMemoryJAASConfiguration: Unable to add JAAS configuration for client [KafkaClient] as it is missing param [atlas.jaas.KafkaClient.loginModuleName]. Skipping JAAS config for [KafkaClient]


org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_HOOK-0 due to 30078 ms has passed since batch creation plus linger time
        at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:239)
        at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:212)
        at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:114)
        at org.apache.atlas.hook.AtlasHook.notifyEntitiesInternal(AtlasHook.java:143)
        at org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:128)
        at org.apache.atlas.sqoop.hook.SqoopHook.publish(SqoopHook.java:190)
        at org.apache.atlas.sqoop.hook.SqoopHook.publish(SqoopHook.java:51)
        at org.apache.sqoop.mapreduce.PublishJobData.publishJobData(PublishJobData.java:52)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:284)
        at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:748)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:509)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_HOOK-0 due to 30078 ms has passed since batch creation plus linger time
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:230)
        ... 17 more

Despite this error sometimes sqoop works, sometimes doesn't. When doesn't, it creates table but empty.

[#2]

Hi @Erkan ŞİRİN!
I'm not used to Atlas, but does your keytab has an expiry date? Check if you can use the keytabs for Atlas/Kafka.
Also, check this link
http://www.hadoopadmin.co.in/tag/error-security-inmemoryjaasconfiguration-unable-to-add-jaas-configu...

Hope this helps!

[#3]

Thank you @Vinicius Higa Murakami . I don't use kerberos I think the issue doesn't related with keytab. As for the link I applied

atlas.jaas.KafkaClient.option.renewTicket=false

atlas.jaas.KafkaClient.option.useTicketCache=false

but didn't work.

I couldn't apply Option 2: Login to ambari server and remove both parameters by running the below commands

because it was sayin don't use configs.sh instead use configs.py

But I couldn't either apply configs.py

[#4]

Hi @Erkan ŞİRİN!
Hm gotcha! Just asking, but do you also have the property atlas.authentication.method.kerberos=False on atlas-application.properties?
Do you have anything else on the logs (sqoop,kafka/atlas)?
And when sqoop works do you still see the same ERROR from InMemoryJAASConfiguration?

PS: I was taking a look at the code, and you're hitting this msg https://github.com/apache/atlas/blob/master/intg/src/main/java/org/apache/atlas/security/InMemoryJAA...
There's a lot of DEBUG trace there, if we don't get any progress with the other logs, you can try to raise the level of the logs, to get a better trace of what's happening 🙂
Hope this helps!

[#5]

Thanks @Vinicius Higa Murakami . I will check it when I get back work on Monday.

[#6]

what are the steps to reproduce this error ?

[#7]

@Erkan ŞİRİN

If you are not using kerberos and seeing this error. Do check if you have below properties in sqoop-atlas-application.properties

atlas.jaas.KafkaClient.option.renewTicket

atlas.jaas.KafkaClient.option.useTicketCache

If they are present, delete them using below commands on ambari-server machine.

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p <password> delete localhost <cluster> sqoop-atlas-application.properties atlas.jaas.KafkaClient.option.renewTicket 

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p <password> delete localhost <cluster> sqoop-atlas-application.properties atlas.jaas.KafkaClient.option.useTicketCache

P.S if you are using new version of ambari use configs.py instead of configs.sh

Let me know if this helps.

[#8]

after reading above comments, realised that you already tried running configs.py. What is the error you got when using configs.py

[#9]

Hi @Sandeep Nemuri I can't determine the parameters to use configs.py

[#10]

@Erkan ŞİRİN

Here you go.

/var/lib/ambari-server/resources/scripts/configs.py --action=delete --host=<ambarihost> --cluster=<clustername> --config-type=sqoop-atlas-application.properties --user admin --password admin --key=atlas.jaas.KafkaClient.option.renewTicket

/var/lib/ambari-server/resources/scripts/configs.py --action=delete --host=<ambarihost> --cluster=<clustername> --config-type=sqoop-atlas-application.properties --user admin --password admin --key=atlas.jaas.KafkaClient.option.useTicketCache