Skip to main content
matagyula
matagyula
Gold ⭐️
May 5, 2020
Solved

CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

Greetings,

After enabling "Kerberos Authentication for HTTP Web-Consoles" for YARN the Resource Manager WebUI and the HistoryServer Web UI become inaccessible with a valid Kerberos ticket (without a ticket the UI correctly gives the "Authentication required" HTTP 401 error message).

Navigating to either of the interfaces returns the following error:

 

 

 

HTTP ERROR 500

Problem accessing /jobhistory. Reason:

    Server Error

Caused by:

java.lang.IllegalArgumentException: Empty key
	at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
	at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
	at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1553)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

 

 

 

 Meanwhile, in Cloudera Manager the YARN health checks report bad status for every component.  In the YARN logs (hadoop-cmf-yarn-RESOURCEMANAGER) the following WARN messages appear:

 

 

 

2020-05-05 11:56:42,835 WARN org.eclipse.jetty.servlet.ServletHandler: /jmx
java.lang.IllegalArgumentException: Empty key
...
2020-05-05 11:57:56,461 WARN org.eclipse.jetty.servlet.ServletHandler: /ws/v1/cluster/info
java.lang.IllegalArgumentException: Empty key
        at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
        at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
        at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
        at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
        at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1553)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
        at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:536)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

 

 

 

The cluster is Kerberized, TLS/SSL is enabled. As a side note, SPNEGO is enabled for the HBase WebUI and that works without issues.

Looking through the documentation and various online forums I only found hints that suggested adding the Service Monitor Kerberos Principal to hdfs-site.xml, but obviously my issue is with Yarn, not HDFS.

Thank you for your help in advance!

Kind regards,

Julius

Bender
BenderAnswer
Gold ⭐️⭐️⭐️
May 6, 2020

Hello @matagyula ,

 

thank you for sharing with us the exceptions you are getting after enabling for "Kerberos Authentication for HTTP Web-Consoles" for YARN. You will need to configure SPNEGO [1] and enable authentication for HDFS too [2] to overcome the issues described.

 

Please let us know if the proposed changes resolved your issue!

 

Thank you:
Ferenc

 

[1] https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_browser_access_kerberos_protected_url.html

 

[2] CM -> HDFS service -> search for and enable "Enable Kerberos Authentication for HTTP Web-Consoles", deploy client configuration, restart HDFS and YARN services

Ferenc Erdelyi, Technical Solutions Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button. Learn more about the Cloudera Community: - Community Guidelines (https://community.cloudera.com/t5/custom/page/page-id/Community_Guidelines) - How to use the forum (/t5/help/faqpage)
View in thread

3 replies

Bender
Gold ⭐️⭐️⭐️
BenderAnswer
Gold ⭐️⭐️⭐️
May 6, 2020

Hello @matagyula ,

 

thank you for sharing with us the exceptions you are getting after enabling for "Kerberos Authentication for HTTP Web-Consoles" for YARN. You will need to configure SPNEGO [1] and enable authentication for HDFS too [2] to overcome the issues described.

 

Please let us know if the proposed changes resolved your issue!

 

Thank you:
Ferenc

 

[1] https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_browser_access_kerberos_protected_url.html

 

[2] CM -> HDFS service -> search for and enable "Enable Kerberos Authentication for HTTP Web-Consoles", deploy client configuration, restart HDFS and YARN services

Translate
Ferenc Erdelyi, Technical Solutions Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button. Learn more about the Cloudera Community: - Community Guidelines (https://community.cloudera.com/t5/custom/page/page-id/Community_Guidelines) - How to use the forum (/t5/help/faqpage)
    matagyula
    matagyulaAuthor
    Gold ⭐️
    May 6, 2020

    Dear @Bender ,

    Thank you very much for your prompt response. Enabling SPNEGO for HDFS did indeed solve our issue with YARN. The UIs are now accessible again (with a valid Kerberos ticket).

    Üdvözlettel 🙂

    Gyuszi

    Translate
    Bender
    Gold ⭐️⭐️⭐️
    Bender
    Gold ⭐️⭐️⭐️
    May 6, 2020

    Hello @matagyula ,

     

    thank you for your feedback on the proposed actions and for accepting the reply as the solution! It will help Community Members facing with similar issues to find the answer faster.

     

    Üdvözlettel:

    Ferenc

    Translate
    Ferenc Erdelyi, Technical Solutions Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button. Learn more about the Cloudera Community: - Community Guidelines (https://community.cloudera.com/t5/custom/page/page-id/Community_Guidelines) - How to use the forum (/t5/help/faqpage)
      Terms & ConditionsAccessibility statement