ETL Tool using Yarn Distributed Shell jar

[#1]

Can I execute scripts involving 3rd part ETL Tool using Yarn Distributed Shell?

Currently I am able to execute DTL scripts inside shell script (.sh file). But whenever I am trying execute same using yarn distributed shell, it is throwing error as "command not found". Any suggestions to resolve this!

[#2 · Accepted solution]

Syncsort DMX-h is an excellent ETL tool that is YARN-enabled

http://www.syncsort.com/en/Products/BigData/DMXh

[#3]

Thanks @gkeys, I am using DMX ETL tool only :). But I am getting " dmexpress: command not found " error while running DMX script using yarn distributed shell. Below observations, I have noted

1. I have DMX installed in all the nodes of the my cluster and able to run scripts in all nodes locally.

2. Although, Same DMX Script is running fine inside shell script, It is throwing above error while running using yarn distributed shell.

Kindly suggest which all areas I need to check to resolve this.

[#4]

@Gaurab D Just to be sure ... are you using DMX-H or simply DMX. The H in the first on means Hadoop. This is the one the integrates with Hadoop and Yarn.

[#5]

I am using DMX-H only. But, I am sorting a local file for testing and not connecting to hadoop.

[#6]

Syncsort is a 3rd party partner of ours. We work very closely with them but we do not support them. I suggest you head over to their support site http://bigdatakb.syncsort.com/Default.htm#KB/FAQ/297_Creating_DMX-h_Job_Tutorial.htm

[#7]

Doing a search you can find a number of certified on YARN ETL Tools including informatica, Microstrategy, and talend.

http://hortonworks.com/partners/certified/yarn-ready/

[#8]

Thanks @Timothy Spann