I made change suggested to point to /etc/hadoop/conf.cloudera.yarn as suggested. That took care of the earlier
error. When I reran the script, I got the error below.
-------------
Error: java.io.IOException: Batch Write Failure
at org.apache.solr.hadoop.BatchWriter.throwIf(BatchWriter.java:239)
at org.apache.solr.hadoop.BatchWriter.queueBatch(BatchWriter.java:181)
at org.apache.solr.hadoop.SolrRecordWriter.close(SolrRecordWriter.java:275)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:550)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:629)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.solr.common.SolrException: ERROR: [doc=1966-05-19 10:36:59.373733] unknown field 'file_length'
-----------
It seems not to like the id field which is a string representation of a timestamp.
Here is an excerpt of the schema that I am including:
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<uniqueKey>id</uniqueKey>
<field name="county" type="text_general" indexed="false" stored="true"/>
<field name="year" type="int" indexed="false" stored="true"/>
<field name="court_type" type="text_general" indexed="false" stored="true"/>
<field name="seq_num" type="int" indexed="false" stored="true"/>
<field name="role" type="text_general" indexed="false" stored="true"/>
<field name="num" type="int" indexed="false" stored="true"/>
<field name="stat" type="text_general" indexed="false" stored="true"/>
<field name="biz_name" type="text_general" indexed="true" stored="true"/>
--------------------
And here is an excerpt of my files to be indexed:
id,county,year,court_type,seq_num,party_role,party_num,party_status,biz_name,prefix,last_name,first_name,middle_name,suffix,in_regards_to,case_status,row_of_origin
1994-11-03 12:15:32.12172,180,1994,CVM,558,P,1,DISPOSED,WINDSOR ARMS HOUSING LTD PTNSHP,null,null,null,null,null,null,null,T48
1999-04-16 14:28:37.009778,000,1999,CVD,862,P,1,null,null,null,CRITZER,KAREN,YVONNE,null,null,null,T46
-----------------------
Here is the readCSV.conf
SOLR_LOCATOR : {
# Name of solr collection
collection : party_name
# ZooKeeper ensemble
zkHost : "dwh-mst-dev02.stor.nccourts.org:2181/solr"
# The maximum number of documents to send to Solr per network batch (throughput knob)
# batchSize : 100
}
morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
commands : [
{
readCSV {
separator : ","
columns : [id,county,year,court_type,seq_num,party_role,party_num,party_status,biz_name,prefix,last_name,first_name,middle_name,suffix,in_regards_to,case_status,row_of_origin]
ignoreFirstLine : true
trim : true
charset : UTF-8
}
}
{ logDebug { format : "output record: {}", args : ["@{}"] } }
# load the record into a Solr server or MapReduce Reducer.
{
loadSolr {
solrLocator : ${SOLR_LOCATOR}
}
}
]
}
]
--
Thanks
