question Re: Import Data from MongoDb to HBase using Spark in Archives of Support Questions (Read Only)

Import Data from MongoDb to HBase using Spark

vijaykumar243 — Tue, 13 Sep 2016 12:55:13 GMT

Hi All,

Can someone please kind to tell me how to import the data from MongoDB to HBase using Spark or Without Using Spark.

If not any other Way.

Regards,

Vijay

Re: Import Data from MongoDb to HBase using Spark

rajkumar_singh — Tue, 13 Sep 2016 13:02:09 GMT

did you try this?

https://github.com/colinmarc/zerowing

Re: Import Data from MongoDb to HBase using Spark

vijaykumar243 — Tue, 13 Sep 2016 14:04:15 GMT

@Rajkumar Singh

I did tried this example. But my Mongo Json data is Very complex

Re: Import Data from MongoDb to HBase using Spark

ldaluz — Tue, 13 Sep 2016 14:37:03 GMT

@Vijay Kumar J

Have you considered using Apache NiFi for this?

NiFi has inbuilt processors to work with data in both MongoDB and HBase.

You could use NiFi's GetMongo processor followed by the PutHbaseJSON processor to move the data from MongoDB to HBase.

Check out the following article for more info on using NiFi to interact with MongoDB:

https://community.hortonworks.com/articles/53554/using-apache-nifi-100-with-mongodb.html

Re: Import Data from MongoDb to HBase using Spark

vijaykumar243 — Tue, 13 Sep 2016 17:47:16 GMT

@Laurence Da Luz

Thanks for the response.

What if the process fails while fetching (or) storing the data.

Is it feasible for production use.

How to execute this NIFI on Hourly basis.

Re: Import Data from MongoDb to HBase using Spark

ldaluz — Tue, 13 Sep 2016 19:46:03 GMT

@Vijay Kumar J

NiFi is definitely feasible for production use, and it is perfectly suited for your MongoDB to HBase data movement use case. NiFi is a tool used for managing dataflow and integration between systems in an automated and configurable way. It allows you to stream, transform, and sort data and uses a drag-and-drop UI.

Dealing with failures - NiFi is configurable - when you build your flow within NiFi you can determine how you want to handle failures. In your case, you could build a flow in NiFi that retries on failure, and sends out an email on failure (this is an example, how you want to handle failures for fetch and storing data can be configured however you need)

Execute NiFi on an Hourly Basis - NiFi isn't like traditional data movement schedulers, and flows built using NiFi are treated as 'always-on' where data can be constantly streamed as it is received. That being said, NiFi provides the ability to schedule each processor if needed, so in your case you could have your GetMongo processor set to run once every hour, and your PutHBaseJSON processor to push data to HBase as soon as it is received from the GetMongo processor.

Check out this tutorial for getting started and building your first flow: http://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/