Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

avatar
Rising Star

Greetings,

After enabling "Kerberos Authentication for HTTP Web-Consoles" for YARN the Resource Manager WebUI and the HistoryServer Web UI become inaccessible with a valid Kerberos ticket (without a ticket the UI correctly gives the "Authentication required" HTTP 401 error message).

Navigating to either of the interfaces returns the following error:

 

 

 

HTTP ERROR 500

Problem accessing /jobhistory. Reason:

    Server Error

Caused by:

java.lang.IllegalArgumentException: Empty key
	at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
	at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
	at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1553)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

 

 

 

 Meanwhile, in Cloudera Manager the YARN health checks report bad status for every component.  In the YARN logs (hadoop-cmf-yarn-RESOURCEMANAGER) the following WARN messages appear:

 

 

 

2020-05-05 11:56:42,835 WARN org.eclipse.jetty.servlet.ServletHandler: /jmx
java.lang.IllegalArgumentException: Empty key
...
2020-05-05 11:57:56,461 WARN org.eclipse.jetty.servlet.ServletHandler: /ws/v1/cluster/info
java.lang.IllegalArgumentException: Empty key
        at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
        at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
        at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
        at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
        at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1553)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
        at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:536)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

 

 

 

The cluster is Kerberized, TLS/SSL is enabled. As a side note, SPNEGO is enabled for the HBase WebUI and that works without issues.

Looking through the documentation and various online forums I only found hints that suggested adding the Service Monitor Kerberos Principal to hdfs-site.xml, but obviously my issue is with Yarn, not HDFS.

Thank you for your help in advance!

Kind regards,

Julius

1 ACCEPTED SOLUTION

avatar
Moderator

Hello @matagyula ,

 

thank you for sharing with us the exceptions you are getting after enabling for "Kerberos Authentication for HTTP Web-Consoles" for YARN. You will need to configure SPNEGO [1] and enable authentication for HDFS too [2] to overcome the issues described.

 

Please let us know if the proposed changes resolved your issue!

 

Thank you:
Ferenc

 

[1] https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_browser_access_kerberos_prot...

 

[2] CM -> HDFS service -> search for and enable "Enable Kerberos Authentication for HTTP Web-Consoles", deploy client configuration, restart HDFS and YARN services


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

View solution in original post

3 REPLIES 3

avatar
Moderator

Hello @matagyula ,

 

thank you for sharing with us the exceptions you are getting after enabling for "Kerberos Authentication for HTTP Web-Consoles" for YARN. You will need to configure SPNEGO [1] and enable authentication for HDFS too [2] to overcome the issues described.

 

Please let us know if the proposed changes resolved your issue!

 

Thank you:
Ferenc

 

[1] https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_browser_access_kerberos_prot...

 

[2] CM -> HDFS service -> search for and enable "Enable Kerberos Authentication for HTTP Web-Consoles", deploy client configuration, restart HDFS and YARN services


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

avatar
Rising Star

Dear @Bender ,

Thank you very much for your prompt response. Enabling SPNEGO for HDFS did indeed solve our issue with YARN. The UIs are now accessible again (with a valid Kerberos ticket).

Üdvözlettel 🙂

Gyuszi

avatar
Moderator

Hello @matagyula ,

 

thank you for your feedback on the proposed actions and for accepting the reply as the solution! It will help Community Members facing with similar issues to find the answer faster.

 

Üdvözlettel:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community: