https://developer.yahoo.com/hadoop/tutorial/module4.html Map -> Combiner -> Partitioner -> Sort -> Shuffle -> Sort -> Reduce https://farm3.static.flickr.com/2374/3529959828_0b689d1d5c_o.png https://farm3.static.flickr.com/2275/3529146683_c8247ff6db_o.png ... View more

Unfortunately there is no way to enable HDFS HA without restarting the Namenode. So unless you can change the process to use a buffer in between. ( Kafka would be a very popular tool combining MQ like use with almost unlimited scalability and easy buffering of dozens to hundreds of Terabyte of data ) . I am not sure what you could do. So if you really really absolutely cannot lose a tuple or you want to have a safer architecture anyway: A) Develop a process that reads the events and puts them into kafka. You would also need a process that reads them from kafka again and puts them in hbase. B) Switch over the process from hbase to kafka C) Upgrade your cluster D) Switch on the Kafka->Hbase process. That would not be time critical since even a 3 node Kafka cluster can easily store 10-20TB of data in a replicated fashion. ... View more

Sqoop might work or if you want to be closer to realtime, IBM CDC. Change Data Capture from IBM also has a Hadoop connector I would hope they can talk to each other. ( Mainframe versions are sometimes very different but I would assume you can forward the changes to a normal CDC instance which then should have the BigData connector.) http://www-03.ibm.com/software/products/en/ibminfochandatacaptforzos Depending on how you want the data in HDFS there is for example a webhdfs connector in IBM CDC ( now InfoSphere Data replication? ) https://www.ibm.com/developerworks/community/files/app?lang=en#/file/c04518fb-8ff3-4b2a-9fb9-38733478bb9b ... View more

Yes you can use oozie. Let's concentrate on this because to run queries in parallel efficiently you most likely will need an oozie workflow anyway. ( Falcon can kick off oozie workflows ) Regardless if you use falcon or a classic oozie coordinator to schedule them. What oozie workflows provide is the ability to create an execution graph where each action can continue on to any other node. ( cycles are forbidden ) . To allow parallel action execution you have forks and joins. A fork starts two actions in parallel and a join waits for all actions it waits on to finish. So you can pretty much create any structure you want. The example below is very simple but you could also have fork in a fork etc. pp. There are surely other ways as well but Oozie most likely will be the canonical way of doing it. For example: <start to="check-files"/> <fork name="parallel-load"> <path start="load1"/> <path start="load2"/> </fork> <action name="load1"> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <jdbc-url>jdbc:hive2://hiveserver:10000/default</jdbc-url> <password>${hivepassword}</password> <script>/data/sql/load1.sql</script> </hive2> <ok to="join-node"/> <error to="kill"/> </action> <action name="load2"> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <jdbc-url>jdbc:hive2://hiveserver:10000/default</jdbc-url> <password>${hivepassword}</password> <script>/data/sql/load2.sql</script> </hive2> <ok to="join-node"/> <error to="kill"/> </action> <join name="join-node" to="end"/> ... View more