Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Missing parameter: morphlineFile

Missing parameter: morphlineFile

New Contributor

Hi,

I copy/paste my flume and morphline configuration file into Cloudera Manager/Flume Configuration.

 

# Flume configuration file
mongotask.sources = r1
mongotask.sinks = k1
mongotask.channels = c1

mongotask.sources.r1.type = spooldir
mongotask.sources.r1.deletePolicy = immediate
mongotask.sources.r1.includePattern = ^.+[0-9]+(\.gz)
mongotask.sources.r1.recursiveDirectorySearch = false
mongotask.sources.r1.spoolDir = /data3/flume/spooldir/mongodb
mongotask.sources.r1.pollDelay = 36000

# Use a channel which buffers events in memory
mongotask.channels.c1.type = memory
mongotask.channels.c1.capacity = 10000
mongotask.channels.c1.transactionCapacity = 1000

mongotask.sinks.k1.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
mongotask.sinks.k1.morphlineFile = $ZK_HOST
mongotask.sinks.k1.morphlineId = morphline1

# Bind the source and sink to the channel
mongotask.sources.r1.channels = c1
mongotask.sinks.k1.channel = c1
# Morphline configuration file
SOLR_LOCATOR : {
  collection : collection1

  zkHost : "$ZK_HOST"
}
morphlines : [
  {
    id : morphline1
    importCommands : ["com.cloudera.**", "org.kitesdk.**", "org.apache.solr.**"]

    commands : [
      {
        readLine {
          charset : UTF-8
        }
      }
      {
        grok {
          dictionaryFiles : [target/test-classes/grok-dictionaries]
          expressions : {
            message : """<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"""
          }
        }
      }
      {
        sanitizeUnknownSolrFields {
          # Location from which to fetch Solr schema
          solrLocator : ${SOLR_LOCATOR}
        }
      }
      { logDebug { format : "output record: {}", args : ["@{}"] } }
      {
        loadSolr {
          solrLocator : ${SOLR_LOCATOR}
        }
      }
    ]
  }
]

 


I get missing parameter 'morphlineFile' error.
[root@hdchx02:~]# tail -f /var/log/flume-ng/flume-cmf-flume-AGENT-hdchx02.log
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:45)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2019-09-09 13:25:11,528 INFO org.apache.flume.sink.solr.morphline.MorphlineSink: Starting Morphline Sink k1 (MorphlineSolrSink) ...
2019-09-09 13:25:11,528 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started
2019-09-09 13:25:11,528 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6fd6733c counterGroup:{ name:null counters:{} } } - Exception follows.
org.kitesdk.morphline.api.MorphlineCompilationException: Missing parameter: morphlineFile


What value should I have for morphlineFile in the flume configuration for solrSink?

Don't have an account?
Coming from Hortonworks? Activate your account here