Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Make Solr to use HDFS

Solved Go to solution

Re: Make Solr to use HDFS

Explorer

Hi Bala,

 

  I have found that very rarely is data truly unstructured.  What kind of data is it?  Typically, there is some form of structure to the data.  Can you send me a sample file kevin@cloudera.com

Re: Make Solr to use HDFS

Expert Contributor
Kevin , The data consists of Rich documents (txt , pdf , doc files) It does not hold any particular structure . Is it possible to extract the data out of this format ??
Thanks
Bala

Re: Make Solr to use HDFS

Explorer

Bala,

 

  It absolutely is.  I was just giving you a sample set of instructions so you could play with a CSV file ingest.  You will be looking to use Apache Tika.  The good news is there is a morphline to help you with that.  The bad new is you will have to write that morphline.  I would recommend starting here: https://github.com/cloudera/search#cdk-morphlines-solr-cell

Re: Make Solr to use HDFS

Expert Contributor
Kevin , in the earlier briefing you have mentioned about morphline . So should i proceed with the earlier steps you have asked me to follow . Or should i go through this first ? https://github.com/cloudera/search#cdk-morphlines-solr-cell
Thanks
Bala

Re: Make Solr to use HDFS

Explorer

You can follow the same steps I sent you, but you will need to switch to https://github.com/cloudera/search#cdk-morphlines-solr-cellcdk-morphline-solr-cell morphline instead of the CSV one in the example.

Re: Make Solr to use HDFS

Expert Contributor
Kevin, How to use CDK ?
Thanks
Bala

Re: Make Solr to use HDFS

Expert Contributor
Hello Kevin ,

I am still not able to figure out how to use the CDK u have mentioned :( .. Need help ..

Thanks
Bala
Thanks
Bala
Highlighted

Re: Make Solr to use HDFS

Expert Contributor

Kevin , I followed the steps , It working as expected in dry run. But when i run without dry--run argument . It stops at this step :( :(

 

770  [main] INFO  org.apache.solr.cloud.ZkController  – Write file /tmp/1404354031741-0/velocity/facet_fields.vm
771  [main] INFO  org.apache.solr.cloud.ZkController  – Write file /tmp/1404354031741-0/elevate.xml
773  [main] INFO  org.apache.solr.cloud.ZkController  – Write file /tmp/1404354031741-0/admin-extra.menu-bottom.html
774  [main] INFO  org.apache.solr.cloud.ZkController  – Write file /tmp/1404354031741-0/schema.xml
897  [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  – Indexing 1 files using 1 real mappers into 1 reducers

 

It stops in 897 itself . I restarted and tried , still the same .

 

Any help .

 

Thanks

Bala

Thanks
Bala

Re: Make Solr to use HDFS

New Contributor

Is incremental load to Solr is possible? Meaning that If the dataset set that is going to load in the solr has some unique keys ( with or without update in other fields of the record) that are already present in the solr collection, I want existing records get updated and new record get inserted in Solr collection. Could you please let me know if it is possible in Solr or not. If yes, please advice in achieving the same.

Don't have an account?
Coming from Hortonworks? Activate your account here