Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problems with Flume and Morphline Solr Sink

SOLVED Go to solution

Problems with Flume and Morphline Solr Sink

New Contributor

Hi guys, im having some problems with morphline solr sink. This is my flume conf file

 

tier1.sources = fuente_lt 
tier1.channels = memoryChannel_lt
tier1.sinks = solrSink_lt 

tier1.sources.fuente_lt.type = netcat
tier1.sources.fuente_lt.max-line-lenght = 512
tier1.sources.fuente_lt.bind = 0.0.0.0
tier1.sources.fuente_lt.port = 6667

tier1.channels.memoryChannel_lt.type = memory
tier1.channels.memoryChannel_lt.capacity = 100
tier1.channels.memoryChannel_lt.transactionCapacity = 100

tier1.sinks.solrSink_lt.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
tier1.sinks.solrSink_lt.morphlineFile =  /home/xxxx/morphline.conf
tier1.sinks.solrSink_lt.morphlineId = morphline1


tier1.sources.fuente_lt.channels = memoryChannel_lt
tier1.sinks.solrSink_lt.channel = memoryChannel_lt

I've created the collection in Solr, modified my schema.xml file and that i think its working fine.

This is my morphline.conf file

SOLR_LOCATOR : {
  collection : logCollection
  zkHost : "x.x.x.x:2181/solr"
}

morphlines : [
  {
    id : morphline1     
    importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
    
   commands : [    
                
      {
       readCSV {
	  separator: " "
	  columns: [ip,tipo,dispositivo,descripcion]
	  charset : UTF-8
          }
      }

      {
	generateUUID {
		field: id
	}
      }
      
      { 
        loadSolr {
          solrLocator : ${SOLR_LOCATOR}
        }
      }

    ]
  }

]

When i restart the flume service im getting this

 

....
mar. 28, 13:24:31.652	INFO	org.apache.flume.instrumentation.MonitoredCounterGroup	
Component type: SINK, name: solrSink_lt started
mar. 28, 13:24:31.660	INFO	org.apache.flume.source.NetcatSource	
Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/0.0.0.0:6667]
mar. 28, 13:24:31.693	INFO	org.mortbay.log	
Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
mar. 28, 13:24:31.746	INFO	org.mortbay.log	
jetty-6.1.26.cloudera.4
mar. 28, 13:24:31.794	INFO	org.mortbay.log	
Started SelectChannelConnector@0.0.0.0:41414
mar. 28, 13:24:32.156	INFO	org.kitesdk.morphline.api.MorphlineContext	
Importing commands

and from that point after a couple of seconds it starts again from this point

mar. 28, 13:25:12.339	INFO	org.apache.flume.node.PollingPropertiesFileConfigurationProvider	
Configuration provider starting
mar. 28, 13:25:12.355	INFO	org.apache.flume.node.PollingPropertiesFileConfigurationProvider	
Reloading configuration file:/var/run/cloudera-scm-agent/process/290-flume-AGENT/flume.conf
mar. 28, 13:25:12.359	INFO	org.apache.flume.conf.FlumeConfiguration	
Processing:solrSink_lt
mar. 28, 13:25:12.359	INFO	org.apache.flume.conf.FlumeConfiguration	
Processing:solrSink_lt
mar. 28, 13:25:12.360	INFO	org.apache.flume.conf.FlumeConfiguration	
Added sinks: solrSink_lt Agent: tier1
....
....

anybody experienced this kind of issue?

 

regards.-

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Problems with Flume and Morphline Solr Sink

New Contributor

Hi, i solved this issue changing the java memory configuration. I see this same issue in another topic. Thanks anyway.

 

Regards!

4 REPLIES 4
Highlighted

Re: Problems with Flume and Morphline Solr Sink

Rising Star

I'm a bit hazy on the topic, but your Morphline file looks a bit light to me.

 

I believe you should have some additional pieces beyond the SOLR_LOCATOR piece like follows:

 

morphlines : [
   {
      id : morphline1
      importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
   
      commands : [
         { %function here, like readCSV or something % } 
         ...
         { sanitizeUnknownSolrFields { solrLocator : ${SOLR_LOCATOR} } }
         { loadSolr {solrLocator : ${SOLR_LOCATOR} } } 
      ]
   }
]

Re: Problems with Flume and Morphline Solr Sink

Rising Star
sorry... I blame Internet Explorer for not showing me the entirety of your morphline... What you have looks okay to me. I don't know that you nee the sanitizeUnknownSolrFields like I put in either. Your SOLR collection is created AND activated on zookeeper right?

Re: Problems with Flume and Morphline Solr Sink

New Contributor

Hi, i solved this issue changing the java memory configuration. I see this same issue in another topic. Thanks anyway.

 

Regards!

Re: Problems with Flume and Morphline Solr Sink

Rising Star

Good catch!  I did notice that in an example I had done, I didn't have the memory configurations even set and things worked fine.  I do recall those memory settings being a bit guess and check for me.  Something I should probably work on. ;)