Atlas hook is filling atlas_hook_failed_messages.log file and it is increasing till 10GB per day

[#1]

Hello All,

I see that Atlas hook is filling atlas_hook_failed_messages.log file and it is increasing till 10GB per day and its very tough to maintain it. Surprisingly I do not see any error only info msgs are there.

So can someone please help me to reduce this size and how to get error if any ?

[#2]

@Saurabh

Failed messages are accumulated if Atlas is not able to consume messages from Kafka.

These messages are generated by plugins like Hive who generate notification messages to be about changes taken place within it. These are placed on Kafka topic, that are eventually consumed by Atlas.

To get to the root of the problem, it would help if you could tell us more about:

• Scenario that you are trying out?
• What functionality is impacted adversely because of this?
• Environment details.

Would it be possible to attach portion of the logs?