CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

matagyula — Tue, 05 May 2020 12:45:09 GMT

Greetings,

After enabling "Kerberos Authentication for HTTP Web-Consoles" for YARN the Resource Manager WebUI and the HistoryServer Web UI become inaccessible with a valid Kerberos ticket (without a ticket the UI correctly gives the "Authentication required" HTTP 401 error message).

Navigating to either of the interfaces returns the following error:

HTTP ERROR 500 Problem accessing /jobhistory. Reason: Server Error Caused by: java.lang.IllegalArgumentException: Empty key at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96) at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93) at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1553) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748)

Meanwhile, in Cloudera Manager the YARN health checks report bad status for every component. In the YARN logs (hadoop-cmf-yarn-RESOURCEMANAGER) the following WARN messages appear:

2020-05-05 11:56:42,835 WARN org.eclipse.jetty.servlet.ServletHandler: /jmx java.lang.IllegalArgumentException: Empty key ... 2020-05-05 11:57:56,461 WARN org.eclipse.jetty.servlet.ServletHandler: /ws/v1/cluster/info java.lang.IllegalArgumentException: Empty key at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96) at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93) at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587) at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1553) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:536) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748)

The cluster is Kerberized, TLS/SSL is enabled. As a side note, SPNEGO is enabled for the HBase WebUI and that works without issues.

Looking through the documentation and various online forums I only found hints that suggested adding the Service Monitor Kerberos Principal to hdfs-site.xml, but obviously my issue is with Yarn, not HDFS.

Thank you for your help in advance!

Kind regards,

Julius

Re: CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

Bender — Wed, 06 May 2020 09:02:48 GMT

Hello @matagyula ,

thank you for sharing with us the exceptions you are getting after enabling for "Kerberos Authentication for HTTP Web-Consoles" for YARN. You will need to configure SPNEGO [1] and enable authentication for HDFS too [2] to overcome the issues described.

Please let us know if the proposed changes resolved your issue!

Thank you:
Ferenc

[1] https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_browser_access_kerberos_protected_url.html

[2] CM -> HDFS service -> search for and enable "Enable Kerberos Authentication for HTTP Web-Consoles", deploy client configuration, restart HDFS and YARN services

Re: CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

matagyula — Wed, 06 May 2020 14:38:52 GMT

Dear @Bender ,

Thank you very much for your prompt response. Enabling SPNEGO for HDFS did indeed solve our issue with YARN. The UIs are now accessible again (with a valid Kerberos ticket).

Üdvözlettel 🙂

Gyuszi

Re: CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

Bender — Wed, 06 May 2020 14:57:25 GMT

Hello @matagyula ,

thank you for your feedback on the proposed actions and for accepting the reply as the solution! It will help Community Members facing with similar issues to find the answer faster.

Üdvözlettel:

Ferenc

question Re: CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500 in Support Questions

CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

Re: CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

Re: CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500

Re: CDH 6.3 YARN - Enabling SPNEGO causes HTTP ERROR 500