Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark/Hadoop Cluster and Elasticsearch Integration

Spark/Hadoop Cluster and Elasticsearch Integration

New Contributor

When trying to connect elastic search from spark i am getting the below error org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'.

Below is the command used to initiate the spark shell :

spark-shell --packages com.databricks:spark-csv_2.11:1.2.0 --jars /home/ubuntu/elasticsearch-hadoop-2.3.2/dist/elasticsearch-hadoop-2.3.2.jar --conf spark.es.nodes="XXXXX" --conf spark.es.port=443 --conf spark.es.nodes.client.only=true .

Spark version : 1.6.3

Scala version : 2.10.5

Is there anything in conf part needs to setup to communicate with es?

What kind of configuration i need to use for es-spark integration?