I am using DMX-H only. But, I am sorting a local file for testing and not connecting to hadoop. ... View more

Thanks @Timothy Spann ... View more

Thanks @gkeys, I am using DMX ETL tool only :). But I am getting " dmexpress: command not found " error while running DMX script using yarn distributed shell. Below observations, I have noted 1. I have DMX installed in all the nodes of the my cluster and able to run scripts in all nodes locally. 2. Although, Same DMX Script is running fine inside shell script, It is throwing above error while running using yarn distributed shell. Kindly suggest which all areas I need to check to resolve this. ... View more

Thanks @mqureshi , It was indeed a queue issue. I have earlier configured capacity scheduler giving where both the datanodes are associated with one queue which I wasn't using while submitting this job. Once I removed that configuration, job is running just fine. Many thanks for your advice. ... View more

I can see below state of the job from Yarn UI YarnApplicationState: ACCEPTED: waiting for AM container to be allocated, launched and register with RM. FinalStatus Reported by AM: Application has not completed yet. ... View more

My guess this is due to job is only in not in SCHEDULED state but RUNNING state. ... View more

Whenever I am trying to check logs using the above command. It is showing below message. "/var/log/yarn/apps/hadoop/logs/application_1473064502809_0029 does not exist Log aggregation has not completed or is not enabled." ... View more

Below is the log from /var/log/yarn folder immediately after submitting the job. 2016-09-14 05:41:51,852 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService (IPC Server handler 30 on 8032): Allocated new applicationId: 34 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService (IPC Server handler 30 on 8032): Application with id 34 submitted by user hadoop 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): Storing application with id application_1473064502809_0034 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger (IPC Server handler 30 on 8032): USER=hadoop IP=********* OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1473064502809_0034 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): application_1473064502809_0034 State change from NEW to NEW_SAVING 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore (AsyncDispatcher event handler): Storing info for app: application_1473064502809_0034 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): application_1473064502809_0034 State change from NEW_SAVING to SUBMITTED 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue (ResourceManager Event Processor): Application added - appId: application_1473064502809_0034 user: hadoop leaf-queue of parent: root #applications: 10 2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Accepted application application_1473064502809_0034 from user: hadoop, in queue: default 2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1473064502809_0034_000001 amEmrLabels: CORE 2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): application_1473064502809_0034 State change from SUBMITTED to ACCEPTED 2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService (AsyncDispatcher event handler): Registering app attempt : appattempt_1473064502809_0034_000001 2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1473064502809_0034_000001 State change from NEW to SUBMITTED 2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Application application_1473064502809_0034 from user: hadoop activated in queue: default 2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Application added - appId: application_1473064502809_0034 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@48d9e852, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 10 #queue-pending-applications: 0 #queue-active-applications: 10 2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Added Application Attempt appattempt_1473064502809_0034_000001 to scheduler from user hadoop in queue default 2016-09-14 05:41:53,828 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1473064502809_0034_000001 State change from SUBMITTED to SCHEDULED ... View more