Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to save Spark DataFrame to Solr

Unable to save Spark DataFrame to Solr

New Contributor

My Environment:

solr-spec - 4.10.3

solr-impl - 4.10.3-cdh5.13.0

Spark 1.6

spark-solr-3.0.2

spark-solr-3.0.2-shaded

solr-solrj-6.5.1

 

While I am trying to save the Spark Dataframe 'df.write().format("solr").options(options).save()' I am getting the following error:

Exception in thread "main" java.lang.AbstractMethodError: solr.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/DataFrame;)Lorg/apache/spark/sql/sources/BaseRelation;
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
at org.apache.spark.sql.DataFrameWriter.dataSource$lzycompute$1(DataFrameWriter.scala:181)
at org.apache.spark.sql.DataFrameWriter.org$apache$spark$sql$DataFrameWriter$$dataSource$1(DataFrameWriter.scala:181)
at org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply$mcV$sp(DataFrameWriter.scala:188)
at org.apache.spark.sql.DataFrameWriter.executeAndCallQEListener(DataFrameWriter.scala:154)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:188)
at Test.main(Test.java:65)

 

2 REPLIES 2
Highlighted

Re: Unable to save Spark DataFrame to Solr

New Contributor

Unfortunately, the version of Solr in Cloudera 5.x does not support that. You can write a Spark app to index a batch of  org.apache.solr.common.SolrInputDocument objects into  org.apache.solr.client.solrj.impl.CloudSolrServer.

Re: Unable to save Spark DataFrame to Solr

New Contributor

Would it be possible in CDH6 with 7-th version of Solr?