question Re: Problems with Flume and Morphline Solr Sink in Archives of Support Questions (Read Only)

Problems with Flume and Morphline Solr Sink

roditoxb — Fri, 16 Sep 2022 10:11:10 GMT

Hi guys, im having some problems with morphline solr sink. This is my flume conf file

tier1.sources = fuente_lt 
tier1.channels = memoryChannel_lt
tier1.sinks = solrSink_lt 

tier1.sources.fuente_lt.type = netcat
tier1.sources.fuente_lt.max-line-lenght = 512
tier1.sources.fuente_lt.bind = 0.0.0.0
tier1.sources.fuente_lt.port = 6667

tier1.channels.memoryChannel_lt.type = memory
tier1.channels.memoryChannel_lt.capacity = 100
tier1.channels.memoryChannel_lt.transactionCapacity = 100

tier1.sinks.solrSink_lt.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
tier1.sinks.solrSink_lt.morphlineFile =  /home/xxxx/morphline.conf
tier1.sinks.solrSink_lt.morphlineId = morphline1


tier1.sources.fuente_lt.channels = memoryChannel_lt
tier1.sinks.solrSink_lt.channel = memoryChannel_lt

I've created the collection in Solr, modified my schema.xml file and that i think its working fine.

This is my morphline.conf file

SOLR_LOCATOR : {
  collection : logCollection
  zkHost : "x.x.x.x:2181/solr"
}

morphlines : [
  {
    id : morphline1     
    importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
    
   commands : [    
                
      {
       readCSV {
	  separator: " "
	  columns: [ip,tipo,dispositivo,descripcion]
	  charset : UTF-8
          }
      }

      {
	generateUUID {
		field: id
	}
      }
      
      { 
        loadSolr {
          solrLocator : ${SOLR_LOCATOR}
        }
      }

    ]
  }

]

When i restart the flume service im getting this

....
mar. 28, 13:24:31.652	INFO	org.apache.flume.instrumentation.MonitoredCounterGroup	
Component type: SINK, name: solrSink_lt started
mar. 28, 13:24:31.660	INFO	org.apache.flume.source.NetcatSource	
Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/0.0.0.0:6667]
mar. 28, 13:24:31.693	INFO	org.mortbay.log	
Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
mar. 28, 13:24:31.746	INFO	org.mortbay.log	
jetty-6.1.26.cloudera.4
mar. 28, 13:24:31.794	INFO	org.mortbay.log	
Started SelectChannelConnector@0.0.0.0:41414
mar. 28, 13:24:32.156	INFO	org.kitesdk.morphline.api.MorphlineContext	
Importing commands

and from that point after a couple of seconds it starts again from this point

mar. 28, 13:25:12.339	INFO	org.apache.flume.node.PollingPropertiesFileConfigurationProvider	
Configuration provider starting
mar. 28, 13:25:12.355	INFO	org.apache.flume.node.PollingPropertiesFileConfigurationProvider	
Reloading configuration file:/var/run/cloudera-scm-agent/process/290-flume-AGENT/flume.conf
mar. 28, 13:25:12.359	INFO	org.apache.flume.conf.FlumeConfiguration	
Processing:solrSink_lt
mar. 28, 13:25:12.359	INFO	org.apache.flume.conf.FlumeConfiguration	
Processing:solrSink_lt
mar. 28, 13:25:12.360	INFO	org.apache.flume.conf.FlumeConfiguration	
Added sinks: solrSink_lt Agent: tier1
....
....

anybody experienced this kind of issue?

regards.-

Re: Problems with Flume and Morphline Solr Sink

tseader — Tue, 29 Mar 2016 18:56:24 GMT

I'm a bit hazy on the topic, but your Morphline file looks a bit light to me.

I believe you should have some additional pieces beyond the SOLR_LOCATOR piece like follows:

morphlines : [
   {
      id : morphline1
      importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
   
      commands : [
         { %function here, like readCSV or something % } 
         ...
         { sanitizeUnknownSolrFields { solrLocator : ${SOLR_LOCATOR} } }
         { loadSolr {solrLocator : ${SOLR_LOCATOR} } } 
      ]
   }
]

Re: Problems with Flume and Morphline Solr Sink

tseader — Tue, 29 Mar 2016 19:05:08 GMT

sorry... I blame Internet Explorer for not showing me the entirety of your morphline... What you have looks okay to me. I don't know that you nee the sanitizeUnknownSolrFields like I put in either. Your SOLR collection is created AND activated on zookeeper right?

Re: Problems with Flume and Morphline Solr Sink

roditoxb — Tue, 29 Mar 2016 19:19:00 GMT

Hi, i solved this issue changing the java memory configuration. I see this same issue in another topic. Thanks anyway.

Regards!

Re: Problems with Flume and Morphline Solr Sink

tseader — Wed, 30 Mar 2016 11:54:08 GMT

Good catch! I did notice that in an example I had done, I didn't have the memory configurations even set and things worked fine. I do recall those memory settings being a bit guess and check for me. Something I should probably work on. 😉