Support Questions
Find answers, ask questions, and share your expertise

Druid batch ingestion (best pratice) -- Kafka topic from beginning


I was used druid since 2 months for real time ingestion from a kafka topic.
I did lot of test :

  • I consume my topic from begin with spark streaming and use tranquility for real time ingestion. My topic contain lot of message and when i consume from begin i have more running task real time in druid and it's can't be done except i configure my window period and i loss more message
  • I try with a kafka connect druid but i have same problem
    I read lot of article about druid and i understand that for historical data the best way is to do batch ingstion. I start with index batch ingestion and then with hadoop batch ingestion. Any of this method work with me unless i am not well using batch ingestion configuration.

Please for people that work long time with druid how can i resolve batch ingestion problem?

Thanks in advance