Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

LIVY2 interpreter error - AuthenticationFilter: AuthenticationToken ignored: Invalid signature

avatar
author-rank Koffi
Contributor

Created on ‎01-13-2021 08:03 AM - last edited on ‎01-14-2021 07:51 AM by Community Manager Community Manager

 

 

 

 

HDP-3.0.1.0
Spark2 2.3.0
Zeppelin Notebook 0.8.0
Kerberos	1.10.3-30
Apache Hadoop Multi-Node Kerberized Cluster using a HA proxy.
with the following lines in haproxy.cfg:
listen spark_livy
    bind        <HA_PROXY_IP>:8999
    mode        tcp
    option      tcplog
    server      spark_livy_1 <HOST1_URL>:8999 check
    server      spark_livy_2 <HOST2_URL>:8999 check

 

 

 

HI,

When we try to run a query in zeppelin using LIVY2 interpretor, we have the following message that pop up:

 

 

 

 

org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond

 

 

 

 

Sans titre7.png

 

On the LIVY2 server logs (livy-livy-server.out) we have the following line that appears:

 

 

 

 

21/01/12 18:53:22 INFO InteractiveSession: Stopping InteractiveSession 768...
21/01/12 18:53:23 WARN RpcDispatcher: [ClientProtocol] Closing RPC channel with 1 outstanding RPCs.
21/01/12 18:53:23 INFO InteractiveSession: Stopped InteractiveSession 768.
21/01/13 10:28:47 WARN AuthenticationFilter: AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature

 

 

 

 

Sans titre8.png

 

On the logs (zeppelin-interpreter-livy2-zeppelin-<host>.log) in the zeppelin host the following line appears:

 

 

 

 

ERROR [2021-01-13 12:01:48,906] ({pool-2-thread-8} LivySharedInterpreter.java[interpret]:83) - Fail to interpret:print('hello mister ')
org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
        at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:733)
        at org.apache.zeppelin.livy.BaseLivyInterpreter.executeStatement(BaseLivyInterpreter.java:581)
        at org.apache.zeppelin.livy.BaseLivyInterpreter.interpret(BaseLivyInterpreter.java:393)
        at org.apache.zeppelin.livy.LivySharedInterpreter.interpret(LivySharedInterpreter.java:81)
        at org.apache.zeppelin.livy.BaseLivyInterpreter.interpret(BaseLivyInterpreter.java:251)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:103)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:633)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
        at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
        at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
        at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:703)
        ... 15 more
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:633)
        at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202)
        at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67)
        at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187)
        ... 18 more
Caused by: org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:91)
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:619)
        ... 24 more

 

 

 

 

Any idea on how to solve this issue?

 

Thank you in advance for your help.

 

note exemple: 

<HA_PROXY_IP>   --> xx.x.xx.xx

<HA_PROXY_URL> --> myhaproxy.test.com

<HOST1_URL> --> hdp-dev-ms01.test.com 

<HOST1> --> hdp-dev-ms01 

<HOST2_URL> --> hdp-dev-ms02.test.com 

<HOST2> --> hdp-dev-ms02 

 

1,574 Views
0 Kudos
1 REPLY 1

avatar
author-rank Koffi
Contributor

Created on ‎01-13-2021 12:56 PM - edited ‎01-13-2021 02:45 PM

Also on the Ambari UI, we have the following configuration for the service spark(Livy2)
Sans titre10.png

And in the Kerberos we have the following config:
Sans titre11.png 

My understanding is that the value in  the service spark(Livy2) conf

 

 

livy.server.launch.kerberos.principal   livy/_HOST@<REALM>

 

is replaced by the value set in the Kerberos page

 

livy.server.launch.kerberos.principal  ${livy2-env/livy2_user}/_HOST@${realm}

 

What is the value of livy2-env? Where can i get this info?

 

For livy2_user the value is livy according to the param set in ambari:
Sans titre12.png

 

also please find below the list of principals that are in /etc/security/keytabs/livy.service.keytab and /etc/security/keytabs/spnego.service.keytab on the Livy2 server <HOST1> (or <HOST2>) host:

Sans titre13.png

 

note exemple:

<HA_HOST_URL> --> myhaproxy.test.com

<HOST1_URL> --> hdp-dev-ms01.test.com 

<HOST1> --> hdp-dev-ms01 

<REALM>  --> CORP.MYREALM.COM

1,553 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements