Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Have anyone done Avro file replication between two datacenters using Falcon?

avatar

The requirement is Avro file needs to be replicated between two datacenter (primary and dr). Does Falcon supports it out of the box? If no is what is the best way to go about it?

Thanks for taking a look.

Raj

1 ACCEPTED SOLUTION

avatar
Rising Star

Yes. In HDP 2.3.0 and higher, Falcon supports replication of Avro files.

View solution in original post

5 REPLIES 5

avatar
Rising Star

Yes. In HDP 2.3.0 and higher, Falcon supports replication of Avro files.

avatar

@cnormile Do you know about any potential issues with this (for ex performance)? Also do we need any special configuration for this to work? Which pair of versions you tried (In terms of HDP and Falcon?)

avatar
Rising Star

Performance is going to be largely predicated by the speed of the network / distance between the data centers. Falcon uses distcp under the covers.

avatar

@cnormile For sure. Can you share with me the HDP and Falcon version where you tried the Avro replication? Also is this documented?

avatar
Rising Star

I'm working with Falcon 0.8, but Avro replication was fixed in the version shipped with 2.3.0. The documentation is quite lacking at this point, an issue we are working to resolve now. My apologies. Are you encountering an issue?