Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Setting Kafka as Message Bus - setting foundations

Highlighted

Setting Kafka as Message Bus - setting foundations

New Contributor

All,

We are working for setting up Kafka cluster for internal message and event communications - I am very novice with Kafka, and have some architectural questions -

1) When you create Topic and publish information - Can we/Should we keep all the information ( 100+ fields ) as message

2) Message length and retention days, are there any standards we can follow?

3) Recommandation on Format of the message ( Text / JSON / Avro ?? ) - We might have hortonworks Kafka Setup, so not sure Schema registry is supposed to be there

4) I know Partitions should give good throughput - but at what message volume and sizes we should consider partitioning -=

5) How can Kafka be possible using for Batched Datasets? - e.g. nightly data snapshot?

I stumbled on web for above, but no direct answer to most of them - Please advise.