question Re: Is sharing spark RDD or context a supported in HDP 2.5? in Archives of Support Questions (Read Only)

Is sharing spark RDD or context a supported in HDP 2.5?

sunile_manjee — Thu, 22 Sep 2016 23:12:15 GMT

Is sharing spark RDD or context a supported in HDP 2.5 via livy server? Everything I see is via zeppelin interrupter to livy. I want to know if strictly using spark is sharing spark (not zeppelin) RDD or context supported.

Re: Is sharing spark RDD or context a supported in HDP 2.5?

azeltov — Thu, 22 Sep 2016 23:24:59 GMT

@Sunile Manjee it is not supported by HDP 2.5 , confirmed that yesterday with @vshukla

Re: Is sharing spark RDD or context a supported in HDP 2.5?

rgelhausen — Fri, 23 Sep 2016 01:27:01 GMT

@Sunile Manjee

Consider how Spark applications run: a driver runs either on the client, or in a YARN container. If multiple users will ask the same Spark application instance to do multiple things, they need an interface to communicate that to the Driver.

Livy is the out of the box REST interface that shares a single Spark application by presenting the control interface to external users.

If you do not want to use Livy, but still want to share a Spark context, you need to build an external means of communicating with the shared Driver. One solution might be to have the driver periodically pull new queries from a database or from files on disk. This functionality is not builtin to Spark, but could be implemented with a while loop and a sleep statement.

Re: Is sharing spark RDD or context a supported in HDP 2.5?

rgelhausen — Fri, 23 Sep 2016 02:33:14 GMT

*Edit* Realistically, questions about shared SparkContexts are often about

1. Making shared use of cached DataFrames/DataSets

Livy and the Spark Thrift JDBC/ODBC server are decent initial solutions. Keep an eye on Spark-LLAP integration which will be better all around (security, efficiency, etc.)

2. Problems with Spark applications consuming all of a cluster's resources.

Spark's ability to spin up and spin down executor instances dynamically based on utilization is probably a better solution to this problem than sharing a single spark context.

Re: Is sharing spark RDD or context a supported in HDP 2.5?

sunile_manjee — Fri, 23 Sep 2016 08:25:52 GMT

@azeltov does that include with livy server?

Re: Is sharing spark RDD or context a supported in HDP 2.5?

sunile_manjee — Fri, 23 Sep 2016 08:27:02 GMT

@Randy Gelhausen Is spark RDD & context sharing supported in 2.5 via livy server?

Re: Is sharing spark RDD or context a supported in HDP 2.5?

cstanca — Sat, 24 Sep 2016 00:05:37 GMT

@Sunile Manjee

No without Livy. Yes with Livy (@vshukla). However, it is exposed only to Zeppelin, for now.

Code examples: https://github.com/romainr/hadoop-tutorials-examples/tree/master/notebook/shared_rdd

Re: Is sharing spark RDD or context a supported in HDP 2.5?

rgelhausen — Sat, 24 Sep 2016 03:34:35 GMT

Livy is not "hidden". If you have started the Livy server, you can interact with its REST API from any application.

Re: Is sharing spark RDD or context a supported in HDP 2.5?

cstanca — Sat, 24 Sep 2016 08:08:40 GMT

@Randy Gelhausen

Yes. With HDP 2.5. I missed that memo. Before that was "Installed as Spark Ambari Service, not yet exposed outside of Zeppelin". The following was needed: https://github.com/hortonworks-gallery/ambari-zeppelin-service

Re: Is sharing spark RDD or context a supported in HDP 2.5?

azeltov — Mon, 26 Sep 2016 22:49:23 GMT

Correct Livy server is only supported as zeppelin integration, not direct REST api call to Livy.

Re: Is sharing spark RDD or context a supported in HDP 2.5?

rgelhausen — Sat, 01 Oct 2016 02:23:25 GMT

yes

Re: Is sharing spark RDD or context a supported in HDP 2.5?

yjiang — Mon, 31 Oct 2016 13:02:20 GMT

Does ThriftServer in HDP support sharing RDD today?

Re: Is sharing spark RDD or context a supported in HDP 2.5?

sunile_manjee — Mon, 31 Oct 2016 22:56:13 GMT

Just to close this out as i was able to get info required. Sharing RDD and context outside of zeppelin is not supported.

Re: Is sharing spark RDD or context a supported in HDP 2.5?

cstanca — Mon, 31 Oct 2016 23:09:29 GMT

@Sunile Manjee

As I mentioned in my original response: Not without Livy. Yes with Livy (@vshukla). However, it is exposed only to Zeppelin, for now.