Support Questions
Find answers, ask questions, and share your expertise

Spark/Hadoop Cluster and Elasticsearch Integration


Spark/Hadoop Cluster and Elasticsearch Integration

New Contributor

When trying to connect elastic search from spark i am getting the below error org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'.

Below is the command used to initiate the spark shell :

spark-shell --packages com.databricks:spark-csv_2.11:1.2.0 --jars /home/ubuntu/elasticsearch-hadoop-2.3.2/dist/elasticsearch-hadoop-2.3.2.jar --conf"XXXXX" --conf --conf .

Spark version : 1.6.3

Scala version : 2.10.5

Is there anything in conf part needs to setup to communicate with es?

What kind of configuration i need to use for es-spark integration?