Member since
07-26-2016
36
Posts
8
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4064 | 09-13-2016 10:07 AM | |
3080 | 09-07-2016 10:57 AM |
09-21-2016
10:52 AM
I am using DMX-H only. But, I am sorting a local file for testing and not connecting to hadoop.
... View more
09-21-2016
05:21 AM
Thanks @Timothy Spann
... View more
09-21-2016
05:20 AM
Thanks @gkeys, I am using DMX ETL tool only :). But I am getting " dmexpress: command not found " error while running DMX script using yarn distributed shell. Below observations, I have noted 1. I have DMX installed in all the nodes of the my cluster and able to run scripts in all nodes locally. 2. Although, Same DMX Script is running fine inside shell script, It is throwing above error while running using yarn distributed shell. Kindly suggest which all areas I need to check to resolve this.
... View more
09-20-2016
09:27 AM
2 Kudos
Can I execute scripts involving 3rd part ETL Tool using Yarn Distributed Shell? Currently I am able to execute DTL scripts inside shell script (.sh file). But whenever I am trying execute same using yarn distributed shell, it is throwing error as "command not found". Any suggestions to resolve this!
... View more
Labels:
- Labels:
-
Apache YARN
09-14-2016
09:13 AM
Thanks @mqureshi , It was indeed a queue issue. I have earlier configured capacity scheduler giving where both the datanodes are associated with one queue which I wasn't using while submitting this job. Once I removed that configuration, job is running just fine. Many thanks for your advice.
... View more
09-14-2016
06:54 AM
I can see below state of the job from Yarn UI YarnApplicationState: ACCEPTED: waiting for AM container to be allocated, launched and register with RM. FinalStatus Reported by AM: Application has not completed yet.
... View more
09-14-2016
06:16 AM
My guess this is due to job is only in not in SCHEDULED state but RUNNING state.
... View more
09-14-2016
06:13 AM
Whenever I am trying to check logs using the above command. It is showing below message. "/var/log/yarn/apps/hadoop/logs/application_1473064502809_0029 does not exist Log aggregation has not completed or is not enabled."
... View more
09-14-2016
05:44 AM
Below is the log from /var/log/yarn folder immediately after submitting the job. 2016-09-14 05:41:51,852 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService (IPC Server handler 30 on 8032): Allocated new applicationId: 34
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService (IPC Server handler 30 on 8032): Application with id 34 submitted by user hadoop
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): Storing application with id application_1473064502809_0034
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger (IPC Server handler 30 on 8032): USER=hadoop IP=********* OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1473064502809_0034
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): application_1473064502809_0034 State change from NEW to NEW_SAVING
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore (AsyncDispatcher event handler): Storing info for app: application_1473064502809_0034
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): application_1473064502809_0034 State change from NEW_SAVING to SUBMITTED
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue (ResourceManager Event Processor): Application added - appId: application_1473064502809_0034 user: hadoop leaf-queue of parent: root #applications: 10
2016-09-14 05:41:53,826 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Accepted application application_1473064502809_0034 from user: hadoop, in queue: default
2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1473064502809_0034_000001 amEmrLabels: CORE
2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): application_1473064502809_0034 State change from SUBMITTED to ACCEPTED
2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService (AsyncDispatcher event handler): Registering app attempt : appattempt_1473064502809_0034_000001
2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1473064502809_0034_000001 State change from NEW to SUBMITTED
2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Application application_1473064502809_0034 from user: hadoop activated in queue: default
2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Application added - appId: application_1473064502809_0034 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@48d9e852, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 10 #queue-pending-applications: 0 #queue-active-applications: 10
2016-09-14 05:41:53,827 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler (ResourceManager Event Processor): Added Application Attempt appattempt_1473064502809_0034_000001 to scheduler from user hadoop in queue default
2016-09-14 05:41:53,828 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1473064502809_0034_000001 State change from SUBMITTED to SCHEDULED
... View more
09-14-2016
05:31 AM
1 Kudo
I am trying to run sqoop job but it is getting stuck without throwing an error. I am unable to see any yarn logs from this sqoop job. What can I do to identify the issue here. Last part of the Log lookslike below: 16/09/13 05:05:35 INFO db.DBInputFormat: Using read commited transaction isolation
16/09/13 05:05:35 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '1=1' and upper bound '1=1'
16/09/13 05:05:35 INFO mapreduce.JobSubmitter: number of splits:1
16/09/13 05:05:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1473064502809_0029
16/09/13 05:05:36 INFO impl.YarnClientImpl: Submitted application application_1473064502809_0029
16/09/13 05:05:36 INFO mapreduce.Job: The url to track the job: http://**********/proxy/application_1473064502809_0029/
16/09/13 05:05:36 INFO mapreduce.Job: Running job: job_1473064502809_0029
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Sqoop