- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Why do we need to setup Spark Thrift Server?
Created ‎05-17-2016 12:47 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Guys,
Sorry to sound dumb, but what is the use of Spark Thirft Server? We have Kerberized HDP 2.4.0 cluster. Recently installed Spark component on the HDP. Now when I am seeing the setup document, I see the option that talks about adding Spark Thrift Server component.
I googled a bit, it talks about JDBC along with thrift spark server. Not very clearly understood though.
I would like to understand more before making any changes to our Kerberized HDP 2.4 .
Many thanks.
Created ‎05-17-2016 01:02 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Spark thrift server is pretty similar to hiveserver2 thrift, rather submitting the sql queries as hive mr job it will use spark SQL engine which underline uses full spark capabilities. As an use case tools like Tableau can easily connect to spark thrift server through ODBC driver just like hiveserver2 and access the hive or spark temp tables to run the sql queries on spark framework.
Created ‎05-17-2016 01:02 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Spark thrift server is pretty similar to hiveserver2 thrift, rather submitting the sql queries as hive mr job it will use spark SQL engine which underline uses full spark capabilities. As an use case tools like Tableau can easily connect to spark thrift server through ODBC driver just like hiveserver2 and access the hive or spark temp tables to run the sql queries on spark framework.
Created ‎02-11-2017 12:38 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Question, Spark Thrift Server should stay on Master or Worker nodes ?
Created ‎02-02-2018 08:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
with hiveserver2 you can also submit job on spark if spark is configured as execution engine in hive, right?
Created ‎05-17-2016 03:02 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Check out this article and demo for a full explanation and working example:
https://community.hortonworks.com/repos/29883/sparksql-data-federation-demo.html
Created ‎05-17-2016 06:46 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You only need to setup Spark Thrift Server if you need to provide access to SparkSQL via JDBC or ODBC. If you want to only use SparkSQL either programmatically (submit a spark app with Spark-submit) or with Spark-Shell, you don't need Spark Thrift Server.
