Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What hadoop configuration setting determines number of nodes available in spark?

What hadoop configuration setting determines number of nodes available in spark?

Contributor

Not much experience with spark and trying to determine amount of available memory, number of executors, and nodes for a submitted spark job. Code just looks like...

 

 

 

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
import time

sparkSession = SparkSession.builder.appName("node_count_test").getOrCreate()
sparkSession._jsc.sc().setLogLevel("WARN")

# see <a href="<a href="<a href="https://stackoverflow.com/a/52516704/8236733" target="_blank">https://stackoverflow.com/a/52516704/8236733</a>" target="_blank"><a href="https://stackoverflow.com/a/52516704/8236733</a" target="_blank">https://stackoverflow.com/a/52516704/8236733</a</a>>" target="_blank"><a href="<a href="https://stackoverflow.com/a/52516704/8236733</a" target="_blank">https://stackoverflow.com/a/52516704/8236733</a</a>" target="_blank"><a href="https://stackoverflow.com/a/52516704/8236733</a</a" target="_blank">https://stackoverflow.com/a/52516704/8236733</a</a</a>>>
print("Giving some time to let session start in earnest...")
time.sleep(15)
print("...done")

print("\n\nYou are using %d nodes in this session\n\n" % sparkSession._jsc.sc().getExecutorMemoryStatus().keySet().size())

 

 

 

and the output is...

 

 

 

Giving some time to let session start in earnest...
...done


You are using 3 nodes in this session

 

 

 

I would think this number should be the number of data nodes in the cluster, which I can see in ambari is 4, so I would think the output above would be 4. Can anyone tell me what determines the number of available nodes in spark?

Don't have an account?
Coming from Hortonworks? Activate your account here