Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error starting Resource Manager - http server

avatar
New Contributor

Hi,

 

when i try to start the Resource Manager i'm getting a error that i pasted at the end of the message.

Using Debian 8 in VirtualBox.

 

I was following this:

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_cdh5_install.htm...

 

Any suggestions what i'm missing here? Have no idea what connection does the http server have with resource manager.

 

This all the data i could collect about this:

 

root@debian8Node2:~# service hadoop-yarn-resourcemanager restart
Job for hadoop-yarn-resourcemanager.service failed. See 'systemctl status hadoop-yarn-resourcemanager.service' and 'journalctl -xn' for details.
root@debian8Node2:~# systemctl status hadoop-yarn-resourcemanager.service
● hadoop-yarn-resourcemanager.service - LSB: Hadoop resourcemanager
   Loaded: loaded (/etc/init.d/hadoop-yarn-resourcemanager)
   Active: failed (Result: exit-code) since Thu 2015-07-16 12:14:41 BST; 13s ago
  Process: 1094 ExecStop=/etc/init.d/hadoop-yarn-resourcemanager stop (code=exited, status=0/SUCCESS)
  Process: 1144 ExecStart=/etc/init.d/hadoop-yarn-resourcemanager start (code=exited, status=1/FAILURE)

Jul 16 12:14:35 debian8Node2 su[1176]: Successful su for yarn by root
Jul 16 12:14:35 debian8Node2 su[1176]: + ??? root:yarn
Jul 16 12:14:35 debian8Node2 su[1176]: pam_unix(su:session): session opened for user yarn by (uid=0)
Jul 16 12:14:35 debian8Node2 hadoop-yarn-resourcemanager[1144]: starting resourcemanager, logging to /var/log/hadoop-yarn/yarn-yarn-resourcemanager-debian8Node2.out
Jul 16 12:14:41 debian8Node2 hadoop-yarn-resourcemanager[1144]: Failed to start Hadoop resourcemanager. Return value: 1 ... failed!
Jul 16 12:14:41 debian8Node2 systemd[1]: hadoop-yarn-resourcemanager.service: control process exited, code=exited status=1
Jul 16 12:14:41 debian8Node2 systemd[1]: Failed to start LSB: Hadoop resourcemanager.
Jul 16 12:14:41 debian8Node2 systemd[1]: Unit hadoop-yarn-resourcemanager.service entered failed state.
root@debian8Node2:~# journalctl -xn
-- Logs begin at Thu 2015-07-16 07:45:17 BST, end at Thu 2015-07-16 12:17:01 BST. --
Jul 16 12:14:35 debian8Node2 su[1176]: pam_unix(su:session): session opened for user yarn by (uid=0)
Jul 16 12:14:35 debian8Node2 hadoop-yarn-resourcemanager[1144]: starting resourcemanager, logging to /var/log/hadoop-yarn/yarn-yarn-resourcemanager-debian8Node2.out
Jul 16 12:14:36 debian8Node2 su[1176]: pam_unix(su:session): session closed for user yarn
Jul 16 12:14:41 debian8Node2 hadoop-yarn-resourcemanager[1144]: Failed to start Hadoop resourcemanager. Return value: 1 ... failed!
Jul 16 12:14:41 debian8Node2 systemd[1]: hadoop-yarn-resourcemanager.service: control process exited, code=exited status=1
Jul 16 12:14:41 debian8Node2 systemd[1]: Failed to start LSB: Hadoop resourcemanager.
-- Subject: Unit hadoop-yarn-resourcemanager.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit hadoop-yarn-resourcemanager.service has failed.
--
-- The result is failed.
Jul 16 12:14:41 debian8Node2 systemd[1]: Unit hadoop-yarn-resourcemanager.service entered failed state.
Jul 16 12:17:01 debian8Node2 CRON[1426]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 16 12:17:01 debian8Node2 CRON[1427]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 16 12:17:01 debian8Node2 CRON[1426]: pam_unix(cron:session): session closed for user root

 

 

Ths from the /var/log/hadoop-yarn/yarn-yarn-resourcemanager-debian8Node2.log, pasted only the exception part

2015-07-16 12:14:36,927 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2015-07-16 12:14:36,933 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8088
2015-07-16 12:14:37,287 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state STARTED; cause: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:278)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:983)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1083)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1215)
Caused by: java.io.IOException: Problem in starting http server. Server handlers failed
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:841)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:273)
        ... 4 more

1 ACCEPTED SOLUTION

avatar
Mentor
Thank you - it appears the logs are masking the thread failures at the INFO level. Could you change your logging config into DEBUG for the RM and retry to see if you notice any exceptions pertaining to the handler threads starting up?

You can switch it in the log4j.properties file under /etc/hadoop/conf/ on the RM host.

I see another email in the cdh-user@ group where something similar was observed for a user on Ubuntu, but that happened to them cause they had a mix-up with the zookeeper package provided by Ubuntu's own repository vs. CDH ones being installed. I am not sure if your Debian repository carries a zookeeper package as well but you should probably check via dpkg/etc. to ensure its a CDH version package and not a Debian-default one.

View solution in original post

9 REPLIES 9

avatar
Mentor
Could you attach your full RM log here, via pastebin.com or such service, and also the .out file you will find alongside the .log file?

The abort's happening cause we're unable to run some vital threads within the RM. From the short log excerpt though, the reason is not very clear.

avatar
New Contributor

This are todays logs when i started it up. Hope it helps.

 

The .log file:

RM Log file

 

.out file:

.out file

avatar
Mentor
Thank you - it appears the logs are masking the thread failures at the INFO level. Could you change your logging config into DEBUG for the RM and retry to see if you notice any exceptions pertaining to the handler threads starting up?

You can switch it in the log4j.properties file under /etc/hadoop/conf/ on the RM host.

I see another email in the cdh-user@ group where something similar was observed for a user on Ubuntu, but that happened to them cause they had a mix-up with the zookeeper package provided by Ubuntu's own repository vs. CDH ones being installed. I am not sure if your Debian repository carries a zookeeper package as well but you should probably check via dpkg/etc. to ensure its a CDH version package and not a Debian-default one.

avatar
New Contributor

Changed the log level, but the log contained nothing more then the log i linked before.

 

But the resource manager started when i purged the packages and made sure they are pulled from cdh5 repository.

 

Thanks for the suggestion.

avatar
New Contributor
Hi,

Can you explain how did you purge the packages ?

Thanks

avatar
New Contributor

I noticed that I was using the jdk 1.8, so I reinstalled the 1.7, but still I have an error at the start of the resource manager :

  

service hadoop-yarn-resourcemanager start

 

Job for hadoop-yarn-resourcemanager.service failed. See 'systemctl status hadoop-yarn-resourcemanager.service' and 'journalctl -xn' for details.

 

 

And in the logfile I cannot see why it fails :

 

2015-11-19 17:52:34,827 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8088
2015-11-19 17:52:35,480 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state STARTED; cause: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:278)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:985)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1085)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1217)
Caused by: java.io.IOException: Problem in starting http server. Server handlers failed
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:893)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:273)
        ... 4 more

 

avatar
New Contributor

Hi
Can you explain how you purge them in thread, please
Thanks

avatar
New Contributor

Reading through this blog helped me a lot: http://blog.cloudera.com/blog/2014/11/guidelines-for-installing-cdh-packages-on-unsupported-operatin...

 

Specially understanding that this can be a package mix/match issue because of trying to run YARN on a unsupported version (Ubuntu 16.04).

 

I removed package and reinstalled like this:

 

$> sudo apt-get remove zookeeper 

 

this will remove a bunch of stuff along with zookeeper

 

$> sudo apt-get install hadoop-yarn-resourcemanager

 

this will reinstall resource manager for you.

 

Hope this helps!

avatar
Explorer

Hi,

 

   Could you please explain how to purge the packages and start the RM again.

 

Thanks