How to make a standalone Spark cluster use HDFS of CHD?

[#1]

Hi All,

I have Coudera 5.10 Hadoop cluster and I have a general purpose cluster running a standalone Spark under Slurm.

How can the standalone Spark utilize HDFS?

My understanding is that I need to run some service on Hadoop gateway that would export HDFS and to use some URL pointing to that HDFS in the standalone Spark? How is it done exactly?

Thank you,

Igor

[#2]

Hello Igor, 

Thanks for your post, however running a standalone Spark cluster is deprecated as of CDH 5.5.0 :

https://www.cloudera.com/documentation/enterprise/release-notes/topics/rg_deprecated.html#concept_y4...

There is some documentation that shows how this would work on earlier version of CDH:

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_ig_spark_configure.html

Thanks, 

Jordan