This configuration is applicable for Spark 2.2.x and above ... View more

@Daniel Kozlowski :- The kill solution will work in "client" mode. In cluster mode, driver would be any node of the cluster. Assuming, we dont have ssh access to that node, how can one kill the driver? ... View more

I am also facing the same issue. ... View more

@BIswajit Kundu : Were you able to resolve this issue? ... View more

Hi Shane, Thanks for the example. This is very useful. I have question - How will this work with multiple column families of HBase table? Do you have any tips on it? Thanks & Regards, Dhaval ... View more

@Andi Chirita : We evaluated SpringXD vs Nifi few months back, at that time both were on GA license but not enterprise ready. SpringXD: SpringXD comes with multiple connector esp. in the field of IoT (MQTT protocol) and REST APIs, also it has module to perform transformations on the fly, but the short coming is it has many issue while running on YARN. UI was not mature enough and still beta phase. Nifi+Falcon: As @Constantin Stanca mentioned, Apache Nifi + Apache Falcon is another good option, we tried it with HDP 2.4.x version. It was not production ready. Also Falcon has multiple issue and limitation with lower versions. I hope this might help you in deciding the tool for ETL pipeline. ... View more

@Sowmya Ramesh: Does this prerequisites apply to both the clusters? Currently, we are running on the mirroring job on cluster that follow the prerequisite. ... View more