The javax/jms/JMSException class is found in the geronimo-jms jar file, which isn't included by default in the flume classpath, as it is a third party jar. To include that, you would need to use the flume plugin architecture [1] to add any of the jars needed by activemq -pd [1] http://flume.apache.org/FlumeUserGuide.html#installing-third-party-plugins ... View more

Is this the pattern that you are trying to match on: ^\s*\#+|\#+$   Try it without quoting or the forward slashes, like:   agent.sources.localsource.interceptors.search-replace.searchPattern = ^\s*\#+|\#+$ agent.sources.localsource.interceptors.search-replace.replaceString = |   -pd ... View more

The exec source is generally not recommended for production environments, as it doesn't handle things well if the process that is getting spawned gets killed unexpectedly. With regards to your log files that you are transferring, are you trying to stream them, or just transport them into hdfs? You may want to consider just using an hdfs put command with a cron job, or mounting the hdfs filesystem via nfs, especially if you want to preserve the files in hdfs as-is. Flume is designed for streaming data, not as a file transport mechanism. If you do want to stream them then, the spooldir source would be used if the files are not being appended to. If they are being appended to while flume is reading them, then you would want to use the new taildir source (as of CDH5.5) [1], as it provides a more reliable handling of streaming log files. The spool dir source requires that files are not modified once they are in the spool directory, and they are removed or marked with .COMPLETED when ingestion is finished. -PD [1] http://archive.cloudera.com/cdh5/cdh/5/flume-ng/FlumeUserGuide.html#taildir-source ... View more

You are using a 'tail -f' command on your (I assume) idempotent csv file, which is tailing the last 10 lines (by default) and would continue to tail if you are writing more data to that CSV file. If this file is in fact no longer being modified, and you want to index the whole file, then I would recommend using the spooldir source instead: http://archive.cloudera.com/cdh5/cdh/5/flume-ng/FlumeUserGuide.html#spooling-directory-source -PD ... View more

The default maxLineLength for the LINE deserializer is 2048: http://archive.cloudera.com/cdh5/cdh/5/flume-ng/FlumeUserGuide.html#line You can set the following to accomodate your large events: agent.sources.axon_source.deserializer.maxLineLength=10000 ... View more

The Exec source is called with the ProcessBuilder: https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html It inherits the environment of the current running flume process ... View more

Are you using Cloudera Manager to start the flume agent?  If so, you'll want to configure the heap size through Cloudera Manager.     If you are not using Cloudera Manager, you will want to specify the following on the command line, not as a -Dproperty value: flume-ng agent -n a1 -f flume_config.conf -Xms1024m -Xmx=204800m   We recommend to set the -Xms and -Xmx values to the same amount so the jvm does not have to resize the heap which can cause performance issues. ... View more