Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to make a standalone Spark cluster use HDFS of CHD?

How to make a standalone Spark cluster use HDFS of CHD?

Hi All,

I have Coudera 5.10 Hadoop cluster and I have a general purpose cluster running a standalone Spark under Slurm.

How can the standalone Spark utilize HDFS?

My understanding is that I need to run some service on Hadoop gateway that would export HDFS and to use some URL pointing to that HDFS in the standalone Spark? How is it done exactly?

Thank you,

Igor

 

1 REPLY 1

Re: How to make a standalone Spark cluster use HDFS of CHD?

Contributor

Hello Igor, 

 

Thanks for your post, however running a standalone Spark cluster is deprecated as of CDH 5.5.0 :

 

https://www.cloudera.com/documentation/enterprise/release-notes/topics/rg_deprecated.html#concept_y4...

 

There is some documentation that shows how this would work on earlier version of CDH:

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_ig_spark_configure.html

 

Thanks, 

Jordan