Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is there any way to keep the data in DB2 and using i.a. Apache Storm?

avatar
New Contributor

Assuming that I need to use Apache Storm, would it be possible doing so keeping the data in DB2? Or do I need to move the data from DB2 to HDFS (using for instance Sqoop or IBM CDC), in order to be able to use Apache Storm?

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
New Contributor

Thanks a lot for your help!

avatar
New Contributor

Thanks for your reply. This means that, at the end of the day, a copy of the DB2-Tables will exist in HDFS, and the Storm process will read the data from HDFS? What I try to avoid is in fact to have a copy of the DB2-Database data in HDFS.

avatar
Master Guru

No if you would implement a JDBCSpout there would be nothing in HDFS at all. Storm by itself has nothing to do with HDFS. It is however often used together with HDFS for storing realtime results. Using the HDFSBolt. I have also seen implementations reading from HDFS as well but its not a requirement.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_storm-user-guide/content/writing-data-wi...

By default Storm has no dependencies on HDFS. It is not that common to use HDFS as a source anyway, since it normally works on realtime data. ( Kafka, MQ, http calls, TCP input, reading from a spooling directory whatever ).

So if you would implement a JDBCSpout using the DB2 JDBC library it would not store anything in HDFS unless you use an HDFSBolt

avatar
Guru

Just put your JDBC driver in the classpath and then write the connection to the DB just liek you would from any Java program. Storm is not dependent on HDFS. In fact, you don't need a Hadoop cluster to run Storm. You can read and write on every event that comes through Storm into the DB2 database....