ZEPPELIN: org.apache.thrift.transport.TTransportException

[#1]

We are on HDP 2.6.2 and Zeppelin 0.7.2, while running the zeppelin Notebook with Spark2 Interpreter --> Per User -> Isolated mode it is keep failing with the below error:

ERROR [2017-10-20 12:28:46,619] ({pool-2-thread-5} RemoteScheduler.java[getStatus]:256) - Can't get status information
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
        at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
        at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
        at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
        at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:92)
        at org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:254)
        at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:342)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
        at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
        ... 15 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
        ... 16 moreERROR [2017-10-20 12:28:46,619] ({pool-2-thread-5} NotebookServer.java[afterStatusChange]:2050) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:379)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:101)
        at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:410)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
        at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
        at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
        at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
        at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
        at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:92)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:377)
        ... 11 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
        at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
        ... 18 more

We have seen this but this didn't help us (we are yet to try re-installing), and also found the JIRA BUGs related to this:

https://issues.apache.org/jira/browse/ZEPPELIN-1700

https://issues.apache.org/jira/browse/ZEPPELIN-1984

https://issues.apache.org/jira/browse/ZEPPELIN-2547

we wanted to know is there any workaround for this or any patch available for this.

Thanks

Venkat

[#2]

Can someone please help?

[#3]

@Venkata Sudheer Kumar M

Same Error fixed by setting below property in spark interpreter configs.

it is a know issue -- ZEPPELIN-1263

Description of Problem: spark.driver.memory will not take effect, the driver memory is always 1G. Workaround: To change the driver memory, specify it in the SPARK_DRIVER_MEMORY property on the interpreter setting page for your spark interpreter.

Refer - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_release-notes/content/known_issues.html

BUG 78035

Thanks,

Sajid.

[#4]

@sajid mohammed

this issue is not related to ZEPPELIN-1263 , as this is related to User Impersonation with Zeppelin Spark interpreter.

You can mode more details in relation to this under: https://issues.apache.org/jira/browse/ZEPPELIN-3016 and the corresponding community article

Please note zeppelin gives the error:

ERROR [2017-10-2012:28:46,619]({pool-2-thread-5}RemoteScheduler.java[getStatus]:256)-Can't get status information
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)

even for the following scenarios:

1) Log Directory not having permissions

2) User doesn't have folder/file level permissions

3) Jar files missing in the path

4) ENV variables missing

These are some of the scenarios i have seen this error with zeppelin Spark interpreter.

Thanks

Venkat

[#5]

Hi Sir, 

Did you solve it? i have same problem when setting perNote and scoped mode.

[#6]

Hi @ayayay2333, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.