Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎06-22-2017

Creating CloudSolrClient Fails When Using SparkSumit

I have spark streaming program written in scala that connects to a kafka topic, reads messages, parses them into Solr documents to be added or removed from an existing collection.

My JUnits that creats an instance of the CloudSolrClient to test search queries, inserts and deletes all succeed.  However, when I integrate my program with spark and run it via sparkSubmit it fails. 

It fails in cluster mode and client mode with the same error, shown below.

It is  failing before getting to the streaming part of the progam.  It seems to be failing in this code:

HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer())
_solr = new CloudSolrClient(_zkHosts)
_solr.setDefaultCollection(_solrCollection)

 

I even tried running that same code via the Spak-Shell.  In all cases I receive the same error:

Exception Details:
Location:
org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/http/impl/client/CloseableHttpClient; @57: areturn
Reason:
Type 'org/apache/http/impl/client/SystemDefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
Current Frame:
bci: @57
flags: { }
locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/SystemDefaultHttpClient' }
stack: { 'org/apache/http/impl/client/SystemDefaultHttpClient' }
Bytecode:
0000000: bb00 0359 2ab7 0004 4cb2 0005 b900 0601
0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
0000020: b600 0a2b b600 0bb6 000c b900 0d02 00b8
0000030: 000e 4d2c 2bb8 000f 2cb0
Stackmap Table:
append_frame(@47,Object[#144])

at org.nccourts.SolrPipelineSparkStreamAppConfig.init(SolrPipelineSparkStreamAppConfig.scala:64)
at org.nccourts.SolrPipelineSparkStreamAppConfig.<init>(SolrPipelineSparkStreamAppConfig.scala:35)

I am using the following dependencies in my gradle.buid file:

dependencies {
compile 'org.slf4j:slf4j-api:1.7.25'
compile 'org.slf4j:log4j-over-slf4j:1.7.25'
compile 'org.scala-lang:scala-compiler:2.11.0'
compile 'org.scala-lang:scala-library:2.11.0'
compile 'ch.qos.logback:logback-classic:1.0.13'
compile('org.apache.solr:solr-solrj:5.5.4')
compile('commons-codec:commons-codec:1.11')
compile('org.apache.spark:spark-core_2.11:2.1.0.cloudera1')
compile('org.apache.spark:spark-streaming_2.11:2.1.0.cloudera1')
compile('org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0.cloudera1')
testCompile group: 'org.scalatest', name: 'scalatest_2.11', version: '3.2.0-SNAP10'
// Use JUnit test framework
testImplementation 'junit:junit:4.12'
}
I am using CDH 5.10
Any help or pointer is appreciated.

Announcements