Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this solution

avatar
Master Collaborator

So am I understanding correctly?

 

Data generating server -> HTTP Post -> Flume HTTP Source -> Flume sink -> etc.

 

and you want to make two Flume HTTP Source machines that can be written to and be able to recieve that data, in case one of them went down?  You also don't want to have to manage something like a load balancer/proxy in between the Data generating server and the Flume HTTP Source box?

 

If you can handle de-duplication on your back-end, then I think you could do this by sending the same data to two different Flume HTTP Source servers at the same time, possibly tagging the data in your sink to help you de-duplicate later.

 

 

Data generating server -> HTTP Post -> Flume HTTP Source - Flume sink with tag --> to de-duplication

               |

               -------------------->  HTTP Post -> Flume HTTP Source - Flume sink with tag --> to de-duplication

View solution in original post

Who agreed with this solution