How do you debug MorphlineCompilationException occurrences?

[#1]

I'm getting the following error when trying to use Flume-ng Solr Sink. There is not enough info to point me to the acctual cause of the problem 😞 If there is something wrong with the syntax of the morphline.conf then shouldn't there be some error message that points to the offending line in the config?

Thanks in advance for any guidance in debugging this.

2014-01-22 13:26:32,902 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@24988707 counterGroup:
{ name:null counters:{} } } - Exception follows.
java.lang.NoClassDefFoundError: com/cloudera/cdk/morphline/api/MorphlineCompilationException
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:93)
        at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
        at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: com.cloudera.cdk.morphline.api.MorphlineCompilationException
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 15 more

[#2 · Accepted solution]

Here is how it works with CM: Paste the contents of your morphline config file into the CM text box as described here:

https://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Manag...

Cloudera Manager automatically uploads and deploys said pasted morphline config to the Flume agent's process directory as "morphlines.conf" when it starts the Flume agent. Therefore, you can reference the morphline config file in the Flume agent's configuration file using only the (relative path) name. For example, in flume.conf you can use the name morphlines.conf to refer to the location of the morphlines configuration file.

Wolfgang.

[#3]

Your classpath is missing the CDK jar files.

Installation instructions are here: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Search/latest/Cloudera-Search-Install...

Wolfgang.

[#4]

Why would this be the case if I installed via parcels via Cloudera Manager 4.8?

[#5]

If the problem is the Exception class itself is not found, then that makes no sense to me because the folowing shows it occurs in core

find / -iname \*morphline\*.jar -ls -exec jar tvf {} com/cloudera/cdk/morphline/api/MorphlineCompilationException \;
1720710   32 -rwxr-xr-x   1 root     root        30376 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-solr-core-0.8.1.jar
1720706  312 -rwxr-xr-x   1 root     root       318835 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-core-0.8.1.jar
  2077 Tue Jan 01 00:00:00 EST 1980 com/cloudera/cdk/morphline/api/MorphlineCompilationException.class
1720655    8 -rwxr-xr-x   1 root     root         6260 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-hadoop-core-0.8.1.jar
1720646   16 -rwxr-xr-x   1 root     root        15103 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-solr-cell-0.8.1.jar
1720708   12 -rwxr-xr-x   1 root     root        10751 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-json-0.8.1.jar
1720688   12 -rwxr-xr-x   1 root     root         8997 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-hadoop-sequencefile-0.8.1.jar
1720731    8 -rwxr-xr-x   1 root     root         7812 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-twitter-0.8.1.jar
1720679   12 -rwxr-xr-x   1 root     root        11217 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-tika-decompress-0.8.1.jar
1720751   12 -rwxr-xr-x   1 root     root        10873 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-metrics-servlets-0.8.1.jar
1720677   32 -rwxr-xr-x   1 root     root        32665 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-saxon-0.8.1.jar
1720750    8 -rwxr-xr-x   1 root     root         7622 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-tika-core-0.8.1.jar
1720752   28 -rwxr-xr-x   1 root     root        26101 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/search/lib/cdk-morphlines-avro-0.8.1.jar
2624257   32 -rwxr-xr-x   1 root     root        30409 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-solr-core-0.8.1.jar
2624249  312 -rwxr-xr-x   1 root     root       319093 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-core-0.8.1.jar
  2077 Wed Oct 23 14:09:30 EDT 2013 com/cloudera/cdk/morphline/api/MorphlineCompilationException.class
2624178    8 -rwxr-xr-x   1 root     root         6270 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-hadoop-core-0.8.1.jar
2624167   16 -rwxr-xr-x   1 root     root        15108 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-solr-cell-0.8.1.jar
2624255   12 -rwxr-xr-x   1 root     root        10756 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-json-0.8.1.jar
2624226   12 -rwxr-xr-x   1 root     root         9000 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-hadoop-sequencefile-0.8.1.jar
2624292   24 -rwxr-xr-x   1 root     root        20998 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/hbase-indexer-morphlines-1.3-search-1.1.0.jar
2624287    8 -rwxr-xr-x   1 root     root         7819 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-twitter-0.8.1.jar
2624212   12 -rwxr-xr-x   1 root     root        11227 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-tika-decompress-0.8.1.jar
2624316   12 -rwxr-xr-x   1 root     root        10885 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-metrics-servlets-0.8.1.jar
2624209   32 -rwxr-xr-x   1 root     root        32699 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-saxon-0.8.1.jar
2624315    8 -rwxr-xr-x   1 root     root         7627 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-tika-core-0.8.1.jar
2624317   28 -rwxr-xr-x   1 root     root        26127 Nov 21 06:12 /opt/cloudera/parcels/SOLR-1.1.0-1.cdh4.3.0.p0.21/lib/hbase-solr/lib/cdk-morphlines-avro-0.8.1.jar
2621816   28 -rwxr-xr-x   1 root     root        26195 Nov 20 21:07 /opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/flume-ng/lib/flume-ng-morphline-solr-sink-1.4.0-cdh4.5.0.jar

[#6]

Ah, this is a known issue with parcels (but not with packages) - CDH-16144.

The work-around is to add the setting SEARCH_HOME=/opt/cloudera/parcels/SOLR/lib/search to the CM configuration > Flume > Service Wide > Flume Service Environment Safety Valve.

(The issue is already fixed for the next upcoming release)

Wolfgang.

[#7]

OK, thanks I will try this and let you know...

[#8]

OK, that fixes that error, but now I get this...

com.cloudera.cdk.morphline.api.MorphlineCompilationException: Cannot parse morphline file: /etc/flume-ng/conf/morphline.conf

My Flume config in CM has the following as per http://archive.cloudera.com/cdh4/cdh/4/flume-ng/FlumeUserGuide.html#morphlinesolrsink

tier1.sinks.mysink.morphlineFile = /etc/flume-ng/conf/morphline.conf

However, no such file exists...

ls /etc/flume-ng/conf/morphline.conf
ls: cannot access /etc/flume-ng/conf/morphline.conf: No such file or directory

but I noticed that there are a whole bunch of morphlines.conf files managed by the agent (see below), so is there some env var to substitute for morphlineFile assignment that will always point to most recent conf managed y CM?

find / -name morphlines.conf -ls
2625258    8 -rw-r-----   1 flume    flume        6661 Jan 18 14:09 /var/run/cloudera-scm-agent/process/336-flume-AGENT/morphlines.conf
2625806    4 -rw-r-----   1 flume    flume         957 Jan 22 13:26 /var/run/cloudera-scm-agent/process/401-flume-AGENT/morphlines.conf
2625310    8 -rw-r-----   1 flume    flume        6661 Jan 18 14:44 /var/run/cloudera-scm-agent/process/342-flume-AGENT/morphlines.conf
2625624    8 -rw-r-----   1 flume    flume        6661 Jan 20 19:22 /var/run/cloudera-scm-agent/process/378-flume-AGENT/morphlines.conf
2625206    8 -rw-r-----   1 flume    flume        6661 Jan 17 23:40 /var/run/cloudera-scm-agent/process/328-flume-AGENT/morphlines.conf
2625572    8 -rw-r-----   1 flume    flume        6661 Jan 20 18:04 /var/run/cloudera-scm-agent/process/372-flume-AGENT/morphlines.conf
2625780    4 -rw-r-----   1 flume    flume        1011 Jan 22 11:10 /var/run/cloudera-scm-agent/process/396-flume-AGENT/morphlines.conf
2625676    8 -rw-r-----   1 flume    flume        6661 Jan 20 20:37 /var/run/cloudera-scm-agent/process/384-flume-AGENT/morphlines.conf
2625754    4 -rw-r-----   1 flume    flume        1011 Jan 22 10:38 /var/run/cloudera-scm-agent/process/393-flume-AGENT/morphlines.conf
2625520    8 -rw-r-----   1 flume    flume        6661 Jan 18 23:55 /var/run/cloudera-scm-agent/process/366-flume-AGENT/morphlines.conf
2625546    8 -rw-r-----   1 flume    flume        6661 Jan 19 22:34 /var/run/cloudera-scm-agent/process/369-flume-AGENT/morphlines.conf
2625232    8 -rw-r-----   1 flume    flume        6661 Jan 18 13:51 /var/run/cloudera-scm-agent/process/333-flume-AGENT/morphlines.conf
2625832    4 -rw-r-----   1 flume    flume         957 Jan 22 14:18 /var/run/cloudera-scm-agent/process/404-flume-AGENT/morphlines.conf
2625494    8 -rw-r-----   1 flume    flume        6661 Jan 18 23:28 /var/run/cloudera-scm-agent/process/363-flume-AGENT/morphlines.conf
2625336    8 -rw-r-----   1 flume    flume        6661 Jan 18 15:10 /var/run/cloudera-scm-agent/process/345-flume-AGENT/morphlines.conf
2625388    8 -rw-r-----   1 flume    flume        6661 Jan 18 15:14 /var/run/cloudera-scm-agent/process/351-flume-AGENT/morphlines.conf
2625598    8 -rw-r-----   1 flume    flume        6661 Jan 20 19:12 /var/run/cloudera-scm-agent/process/375-flume-AGENT/morphlines.conf
2625468    8 -rw-r-----   1 flume    flume        6661 Jan 18 21:48 /var/run/cloudera-scm-agent/process/360-flume-AGENT/morphlines.conf
2625284    8 -rw-r-----   1 flume    flume        6661 Jan 18 14:34 /var/run/cloudera-scm-agent/process/339-flume-AGENT/morphlines.conf
2625414    8 -rw-r-----   1 flume    flume        6661 Jan 18 18:32 /var/run/cloudera-scm-agent/process/354-flume-AGENT/morphlines.conf
2625362    8 -rw-r-----   1 flume    flume        6661 Jan 18 15:12 /var/run/cloudera-scm-agent/process/348-flume-AGENT/morphlines.conf
2625650    8 -rw-r-----   1 flume    flume        6661 Jan 20 19:44 /var/run/cloudera-scm-agent/process/381-flume-AGENT/morphlines.conf
2625728    8 -rw-r-----   1 flume    flume        6661 Jan 20 20:52 /var/run/cloudera-scm-agent/process/390-flume-AGENT/morphlines.conf
2625702    8 -rw-r-----   1 flume    flume        6661 Jan 20 20:39 /var/run/cloudera-scm-agent/process/387-flume-AGENT/morphlines.conf
2625858    4 -rw-r-----   1 flume    flume         957 Jan 22 14:21 /var/run/cloudera-scm-agent/process/407-flume-AGENT/morphlines.conf
2625177    8 -rw-r-----   1 flume    flume        6661 Jan 17 10:06 /var/run/cloudera-scm-agent/process/325-flume-AGENT/morphlines.conf
2625440    8 -rw-r-----   1 flume    flume        6661 Jan 18 18:44 /var/run/cloudera-scm-agent/process/357-flume-AGENT/morphlines.conf

[#9 · Accepted solution]

Here is how it works with CM: Paste the contents of your morphline config file into the CM text box as described here:

https://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Manag...

Cloudera Manager automatically uploads and deploys said pasted morphline config to the Flume agent's process directory as "morphlines.conf" when it starts the Flume agent. Therefore, you can reference the morphline config file in the Flume agent's configuration file using only the (relative path) name. For example, in flume.conf you can use the name morphlines.conf to refer to the location of the morphlines configuration file.

Wolfgang.

[#10 · Accepted solution]

Ok, setting morphlineFile = morphlines.conf works

Thanks