question Re: Have anyone done Avro file replication between two datacenters using Falcon? in Archives of Support Questions (Read Only)

Have anyone done Avro file replication between two datacenters using Falcon?

rbiswas1 — Tue, 21 Jun 2016 22:18:07 GMT

The requirement is Avro file needs to be replicated between two datacenter (primary and dr). Does Falcon supports it out of the box? If no is what is the best way to go about it?

Thanks for taking a look.

Raj

Re: Have anyone done Avro file replication between two datacenters using Falcon?

cnormile — Tue, 21 Jun 2016 22:29:49 GMT

Yes. In HDP 2.3.0 and higher, Falcon supports replication of Avro files.

Re: Have anyone done Avro file replication between two datacenters using Falcon?

rbiswas1 — Tue, 21 Jun 2016 22:47:11 GMT

@cnormile Do you know about any potential issues with this (for ex performance)? Also do we need any special configuration for this to work? Which pair of versions you tried (In terms of HDP and Falcon?)

Re: Have anyone done Avro file replication between two datacenters using Falcon?

cnormile — Tue, 21 Jun 2016 22:56:12 GMT

Performance is going to be largely predicated by the speed of the network / distance between the data centers. Falcon uses distcp under the covers.

Re: Have anyone done Avro file replication between two datacenters using Falcon?

rbiswas1 — Tue, 21 Jun 2016 22:58:41 GMT

@cnormile For sure. Can you share with me the HDP and Falcon version where you tried the Avro replication? Also is this documented?

Re: Have anyone done Avro file replication between two datacenters using Falcon?

cnormile — Tue, 21 Jun 2016 23:18:04 GMT

I'm working with Falcon 0.8, but Avro replication was fixed in the version shipped with 2.3.0. The documentation is quite lacking at this point, an issue we are working to resolve now. My apologies. Are you encountering an issue?