question Re: ETL Tool using Yarn Distributed Shell jar in Support Questions

ETL Tool using Yarn Distributed Shell jar

gaurab_dawn — Tue, 20 Sep 2016 16:27:50 GMT

Can I execute scripts involving 3rd part ETL Tool using Yarn Distributed Shell?

Currently I am able to execute DTL scripts inside shell script (.sh file). But whenever I am trying execute same using yarn distributed shell, it is throwing error as "command not found". Any suggestions to resolve this!

Re: ETL Tool using Yarn Distributed Shell jar

gkeys — Tue, 20 Sep 2016 20:36:27 GMT

Syncsort DMX-h is an excellent ETL tool that is YARN-enabled

http://www.syncsort.com/en/Products/BigData/DMXh

Re: ETL Tool using Yarn Distributed Shell jar

TimothySpann — Tue, 20 Sep 2016 20:47:20 GMT

Doing a search you can find a number of certified on YARN ETL Tools including informatica, Microstrategy, and talend.

http://hortonworks.com/partners/certified/yarn-ready/

Re: ETL Tool using Yarn Distributed Shell jar

gaurab_dawn — Wed, 21 Sep 2016 12:20:48 GMT

Thanks @gkeys, I am using DMX ETL tool only :). But I am getting " dmexpress: command not found " error while running DMX script using yarn distributed shell. Below observations, I have noted

1. I have DMX installed in all the nodes of the my cluster and able to run scripts in all nodes locally.

2. Although, Same DMX Script is running fine inside shell script, It is throwing above error while running using yarn distributed shell.

Kindly suggest which all areas I need to check to resolve this.

Re: ETL Tool using Yarn Distributed Shell jar

gaurab_dawn — Wed, 21 Sep 2016 12:21:19 GMT

Thanks @Timothy Spann

Re: ETL Tool using Yarn Distributed Shell jar

gkeys — Wed, 21 Sep 2016 17:48:16 GMT

@Gaurab D Just to be sure ... are you using DMX-H or simply DMX. The H in the first on means Hadoop. This is the one the integrates with Hadoop and Yarn.

Re: ETL Tool using Yarn Distributed Shell jar

gaurab_dawn — Wed, 21 Sep 2016 17:52:15 GMT

I am using DMX-H only. But, I am sorting a local file for testing and not connecting to hadoop.

Re: ETL Tool using Yarn Distributed Shell jar

gkeys — Thu, 22 Sep 2016 01:37:31 GMT

Syncsort is a 3rd party partner of ours. We work very closely with them but we do not support them. I suggest you head over to their support site http://bigdatakb.syncsort.com/Default.htm#KB/FAQ/297_Creating_DMX-h_Job_Tutorial.htm