Mapreduce jobs running through Edge node

[#1]

Hi Folks,

Hope all are doing well.

I've dev setup of 11 node(2x NN, 8x DN, 1x edge node). Basically All nodes are connected by Cluster private(only 11 machine can access to each other) networking. We are using floating IP(public in only organization) on data node as well as on Edge node for access the data from DB.

If we keep floating ip only on Edge node and run MapReduce job from edge node which is importing data from DB to HDFS, i'm getting error that DataNode IP is not able to access to source DB.

Can someone suggest does we require Floating IP on all DataNode machine? Or we can use any other solution?

Will be very thankful to you.

Regards,

Vinay K

[#2 · Accepted solution]

@Vinay K
I Had this problem before, doing a research I found something related to "When you use MapReduce Process, all of your machines needs to establish a communication" even this DB that you mentioned. I´m not sure if you can do this without public IP.

I Suggest you, use a machine (that can communicate with your network and have a floating IP) as a "stage area" and then proceed using mapreduce process, or if possible you can check if you are able to create an internal network on this DB machine to create this communication.

I hope I could help you.

[#3]

@Eric Leme

According to you, we have to open connection from our Datanode to DB server. Edge is not single pain of glass for communication. Need more research..

Well Thanks 🙂

Regards,

Vinay

[#4]

It's only solution.

Thanks.