Have anyone done Avro file replication between two datacenters using Falcon?

[#1]

The requirement is Avro file needs to be replicated between two datacenter (primary and dr). Does Falcon supports it out of the box? If no is what is the best way to go about it?

Thanks for taking a look.

Raj

[#2 · Accepted solution]

Yes. In HDP 2.3.0 and higher, Falcon supports replication of Avro files.

[#3]

@cnormile Do you know about any potential issues with this (for ex performance)? Also do we need any special configuration for this to work? Which pair of versions you tried (In terms of HDP and Falcon?)

[#4]

Performance is going to be largely predicated by the speed of the network / distance between the data centers. Falcon uses distcp under the covers.

[#5]

@cnormile For sure. Can you share with me the HDP and Falcon version where you tried the Avro replication? Also is this documented?

[#6]

I'm working with Falcon 0.8, but Avro replication was fixed in the version shipped with 2.3.0. The documentation is quite lacking at this point, an issue we are working to resolve now. My apologies. Are you encountering an issue?