Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Have anyone done Avro file replication between two datacenters using Falcon?

Solved Go to solution
Highlighted

Have anyone done Avro file replication between two datacenters using Falcon?

The requirement is Avro file needs to be replicated between two datacenter (primary and dr). Does Falcon supports it out of the box? If no is what is the best way to go about it?

Thanks for taking a look.

Raj

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Have anyone done Avro file replication between two datacenters using Falcon?

Contributor

Yes. In HDP 2.3.0 and higher, Falcon supports replication of Avro files.

5 REPLIES 5

Re: Have anyone done Avro file replication between two datacenters using Falcon?

Contributor

Yes. In HDP 2.3.0 and higher, Falcon supports replication of Avro files.

Re: Have anyone done Avro file replication between two datacenters using Falcon?

@cnormile Do you know about any potential issues with this (for ex performance)? Also do we need any special configuration for this to work? Which pair of versions you tried (In terms of HDP and Falcon?)

Re: Have anyone done Avro file replication between two datacenters using Falcon?

Contributor

Performance is going to be largely predicated by the speed of the network / distance between the data centers. Falcon uses distcp under the covers.

Re: Have anyone done Avro file replication between two datacenters using Falcon?

@cnormile For sure. Can you share with me the HDP and Falcon version where you tried the Avro replication? Also is this documented?

Re: Have anyone done Avro file replication between two datacenters using Falcon?

Contributor

I'm working with Falcon 0.8, but Avro replication was fixed in the version shipped with 2.3.0. The documentation is quite lacking at this point, an issue we are working to resolve now. My apologies. Are you encountering an issue?

Don't have an account?
Coming from Hortonworks? Activate your account here