HDP-3.0.1.0
Spark2 2.3.0
Zeppelin Notebook 0.8.0
Kerberos 1.10.3-30
Apache Hadoop Multi-Node Kerberized Cluster using a HA proxy.
with the following lines in haproxy.cfg:
listen spark_livy
bind <HA_PROXY_IP>:8999
mode tcp
option tcplog
server spark_livy_1 <HOST1_URL>:8999 check
server spark_livy_2 <HOST2_URL>:8999 check
HI,
When we try to run a query in zeppelin using LIVY2 interpretor, we have the following message that pop up:
org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
On the LIVY2 server logs (livy-livy-server.out) we have the following line that appears:
21/01/12 18:53:22 INFO InteractiveSession: Stopping InteractiveSession 768...
21/01/12 18:53:23 WARN RpcDispatcher: [ClientProtocol] Closing RPC channel with 1 outstanding RPCs.
21/01/12 18:53:23 INFO InteractiveSession: Stopped InteractiveSession 768.
21/01/13 10:28:47 WARN AuthenticationFilter: AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
On the logs (zeppelin-interpreter-livy2-zeppelin-<host>.log) in the zeppelin host the following line appears:
ERROR [2021-01-13 12:01:48,906] ({pool-2-thread-8} LivySharedInterpreter.java[interpret]:83) - Fail to interpret:print('hello mister ')
org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:733)
at org.apache.zeppelin.livy.BaseLivyInterpreter.executeStatement(BaseLivyInterpreter.java:581)
at org.apache.zeppelin.livy.BaseLivyInterpreter.interpret(BaseLivyInterpreter.java:393)
at org.apache.zeppelin.livy.LivySharedInterpreter.interpret(LivySharedInterpreter.java:81)
at org.apache.zeppelin.livy.BaseLivyInterpreter.interpret(BaseLivyInterpreter.java:251)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:103)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:633)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:703)
... 15 more
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://<HA_PROXY_URL>:8999/sessions/768/statements": <HA_PROXY_URL>:8999 failed to respond; nested exception is org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:633)
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202)
at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67)
at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187)
... 18 more
Caused by: org.apache.http.NoHttpResponseException: <HA_PROXY_URL>:8999 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:91)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:619)
... 24 more
Any idea on how to solve this issue?
Thank you in advance for your help.
note exemple:
<HA_PROXY_IP> --> xx.x.xx.xx
<HA_PROXY_URL> --> myhaproxy.test.com
<HOST1_URL> --> hdp-dev-ms01.test.com
<HOST1> --> hdp-dev-ms01
<HOST2_URL> --> hdp-dev-ms02.test.com
<HOST2> --> hdp-dev-ms02
Also on the Ambari UI, we have the following configuration for the service spark(Livy2)
And in the Kerberos we have the following config:
My understanding is that the value in the service spark(Livy2) conf
livy.server.launch.kerberos.principal livy/_HOST@<REALM>
is replaced by the value set in the Kerberos page
livy.server.launch.kerberos.principal ${livy2-env/livy2_user}/_HOST@${realm}
What is the value of livy2-env? Where can i get this info?
For livy2_user the value is livy according to the param set in ambari:
also please find below the list of principals that are in /etc/security/keytabs/livy.service.keytab and /etc/security/keytabs/spnego.service.keytab on the Livy2 server <HOST1> (or <HOST2>) host:
note exemple:
<HA_HOST_URL> --> myhaproxy.test.com
<HOST1_URL> --> hdp-dev-ms01.test.com
<HOST1> --> hdp-dev-ms01
<REALM> --> CORP.MYREALM.COM
Announcements
Product Announcements
What's New @ Cloudera
Community Announcements
Community Announcements
Product Announcements
