Support Questions
Find answers, ask questions, and share your expertise

ETL Tool using Yarn Distributed Shell jar

Solved Go to solution

ETL Tool using Yarn Distributed Shell jar

Contributor

Can I execute scripts involving 3rd part ETL Tool using Yarn Distributed Shell?

Currently I am able to execute DTL scripts inside shell script (.sh file). But whenever I am trying execute same using yarn distributed shell, it is throwing error as "command not found". Any suggestions to resolve this!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: ETL Tool using Yarn Distributed Shell jar

Guru

Syncsort DMX-h is an excellent ETL tool that is YARN-enabled

http://www.syncsort.com/en/Products/BigData/DMXh

View solution in original post

7 REPLIES 7

Re: ETL Tool using Yarn Distributed Shell jar

Guru

Syncsort DMX-h is an excellent ETL tool that is YARN-enabled

http://www.syncsort.com/en/Products/BigData/DMXh

View solution in original post

Re: ETL Tool using Yarn Distributed Shell jar

Contributor

Thanks @gkeys, I am using DMX ETL tool only :). But I am getting " dmexpress: command not found " error while running DMX script using yarn distributed shell. Below observations, I have noted

1. I have DMX installed in all the nodes of the my cluster and able to run scripts in all nodes locally.

2. Although, Same DMX Script is running fine inside shell script, It is throwing above error while running using yarn distributed shell.

Kindly suggest which all areas I need to check to resolve this.

Re: ETL Tool using Yarn Distributed Shell jar

Guru

@Gaurab D Just to be sure ... are you using DMX-H or simply DMX. The H in the first on means Hadoop. This is the one the integrates with Hadoop and Yarn.

Re: ETL Tool using Yarn Distributed Shell jar

Contributor

I am using DMX-H only. But, I am sorting a local file for testing and not connecting to hadoop.

Re: ETL Tool using Yarn Distributed Shell jar

Guru

Syncsort is a 3rd party partner of ours. We work very closely with them but we do not support them. I suggest you head over to their support site http://bigdatakb.syncsort.com/Default.htm#KB/FAQ/297_Creating_DMX-h_Job_Tutorial.htm

Re: ETL Tool using Yarn Distributed Shell jar

Super Guru

Doing a search you can find a number of certified on YARN ETL Tools including informatica, Microstrategy, and talend.

http://hortonworks.com/partners/certified/yarn-ready/

Re: ETL Tool using Yarn Distributed Shell jar

Contributor