ah dumb me I was not using the right host address . [hdfs@ip-172-31-8-79 ~]$ kafka-topics.sh --create --zookeeper MasterNode --replication-factor 1 --partitions 4 --topic mytopic Created topic "mytopic". [hdfs@ip-172-31-8-79 ~]$ ... View more

actually zookeeper server is running on three different servers ,namely Slave3, Master, SecondaryMaster ... View more

this step also did not help : put maxClientCnxns=30 in file /etc/zookeeper/conf/zoo.cfg retart zookeeper : ... View more

I see there is no zookeeper server running on this node but only zookeeper client . this node is called Slave1 , all three zookeeper server processes are running on MasterNode. and telnet from Slave1 to MasterNode on port 2181 works fine. so why I am getting connection refused ? I am running the kafka command on Slave1. [hdfs@ip-172-31-8-79 ~]$ telnet MasterNode 2181 Trying 172.31.8.77... Connected to MasterNode. Escape character is '^]'. ... View more

Telnet showing connection refused to port 2181 eventhough zookeeper is running : [hdfs@ip-172-31-8-79 ~]$ telnet ip-172-31-8-79.ec2.internal 2181 Trying 172.31.8.79... telnet: connect to address 172.31.8.79: Connection refused [hdfs@ip-172-31-8-79 ~]$ ps -ef | grep "zookeper.properties" hdfs 22894 22772 0 12:52 pts/1 00:00:00 grep zookeper.properties [hdfs@ip-172-31-8-79 ~]$ ... View more

ok I could grab the RM log from the server and its not able to start the job , due to error shown below, how can I fix? 2016-05-12 09:56:51,114 INFO resourcemanager.ClientRMService (ClientRMService.java:getNewApplicationId(290)) - Allocated new applicationId: 11 2016-05-12 09:56:51,225 INFO resourcemanager.ClientRMService (ClientRMService.java:submitApplication(585)) - Application with id 11 submitted by user hadoop 2016-05-12 09:56:51,226 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(170)) - USER=hadoop IP=172.31.8.77 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1463059172548_0011 CALLERCONTEXT=HIVE_QUERY_ID:hive_20160512095650_21a047fd-e7c6-427e-9b8d-5242621e0b60 2016-05-12 09:56:51,226 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1058)) - Storing application with id application_1463059172548_0011 2016-05-12 09:56:51,226 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1463059172548_0011 State change from NEW to NEW_SAVING 2016-05-12 09:56:51,227 INFO recovery.RMStateStore (RMStateStore.java:transition(199)) - Storing info for app: application_1463059172548_0011 2016-05-12 09:56:51,234 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1463059172548_0011 State change from NEW_SAVING to SUBMITTED 2016-05-12 09:56:51,234 INFO capacity.ParentQueue (ParentQueue.java:addApplication(347)) - Application added - appId: application_1463059172548_0011 user: hadoop leaf-queue of parent: root #applications: 17 2016-05-12 09:56:51,234 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplication(828)) - Accepted application application_1463059172548_0011 from user: hadoop, in queue: default 2016-05-12 09:56:51,234 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1463059172548_0011 State change from SUBMITTED to ACCEPTED 2016-05-12 09:56:51,234 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:registerAppAttempt(678)) - Registering app attempt : appattempt_1463059172548_0011_000001 2016-05-12 09:56:51,235 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(807)) - appattempt_1463059172548_0011_000001 State change from NEW to SUBMITTED 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:addApplicationAttempt(678)) - Application added - appId: application_1463059172548_0011 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@67b7a456, leaf-queue: default #user-pending-applications: 10 #user-active-applications: 2 #queue-pending-applications: 15 #queue-active-applications: 2 2016-05-12 09:56:51,237 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationAttempt(857)) - Added Application Attempt appattempt_1463059172548_0011_000001 to scheduler from user hadoop in queue default 2016-05-12 09:56:51,238 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(807)) - appattempt_1463059172548_0011_000001 State change from SUBMITTED to SCHEDULED ... View more

also found the RM UI but if I try to start it from Ambari as YARN-QUICKLINKS-Resourcemanager UI , I get the following error: Network Error (tcp_error) A communication error occurred: "Operation timed out" The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time. ... View more

hiveserver-logs.zip hive server logs ... View more