Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

morphline interceptor not work

morphline interceptor not work

New Contributor

Hello guys!!

 

I want to set a morphline interceptor in an agent wich send the dato to collector agent allocate in cloudera server.

But i have a the following problem:

 

 

tParser.hasAtLeastOneAttachment:164) - Command failed because of missing attachment for record: {hostname=[magnifix], plataforma=[linux]}
01 feb 2018 15:23:34,420 DEBUG [PollableSourceRunner-TaildirSource-linuxLog] (org.kitesdk.morphline.base.AbstractCommand.process:158) - Command failed!
01 feb 2018 15:23:34,420 DEBUG [PollableSourceRunner-TaildirSource-linuxLog] (org.kitesdk.morphline.base.AbstractCommand.process:158) - Command failed!
01 feb 2018 15:23:34,421 WARN [PollableSourceRunner-TaildirSource-linuxLog] (org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.process:132) - Morphline /home/flume/pruebaMorphline.conf@preMorphlineLinuxLog failed to process record: {hostname=[magnifix], plataforma=[linux]}

 

 

My agent configuration is:

 

############################################################################################
### LINUX-LOG ####
############################################################################################


# ------------------------- Declaracion sources, channels y sinks ------------------------ #

sourceAgent.sources = linuxLog
sourceAgent.channels = fileChannelLinux
sourceAgent.sinks = avroSinkLinux1
sourceAgent.sinkgroups = group


# --------------------------- Configuracion source linux-log ----------------------------- #

sourceAgent.sources.linuxLog.type = TAILDIR
sourceAgent.sources.linuxLog.positionFile = /home/flume/flume-metadata/flumedatadirLinuxLog/position_log_linux-events.json
sourceAgent.sources.linuxLog.filegroups = linuxLogs
sourceAgent.sources.linuxLog.filegroups.linuxLogs = /var/log/pruebas
sourceAgent.sources.linuxLog.filegroups.linuxLogs.headerkey=linuxLog


# ----------------------------- Configuracion de interceptores ---------------------------- #

#Definicion interceptores para adjuntar cabezeras al mensaje
sourceAgent.sources.linuxLog.interceptors = regex hostInterceptor plataforma morphlineinterceptor

sourceAgent.sources.linuxLog.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
sourceAgent.sources.linuxLog.interceptors.morphlineinterceptor.morphlineFile = /home/flume/pruebaMorphline.conf
sourceAgent.sources.linuxLog.interceptors.morphlineinterceptor.morphlineId = preMorphlineLinuxLog

sourceAgent.sources.linuxLog.interceptors.regex.type = regex_filter
sourceAgent.sources.linuxLog.interceptors.regex.regex = .*\\.(err|error|crit|alert|emerg|panic):\\s.*
sourceAgent.sources.linuxLog.interceptors.regex.excludeEvents = false

 

#Definicion interceptor que agrega el nombre del host
sourceAgent.sources.linuxLog.interceptors.hostInterceptor.type = org.apache.flume.interceptor.HostInterceptor$Builder
sourceAgent.sources.linuxLog.interceptors.hostInterceptor.hostHeader = hostname
sourceAgent.sources.linuxLog.interceptors.hostInterceptor.preserveExisting = false
sourceAgent.sources.linuxLog.interceptors.hostInterceptor.useIP = false

 

#Definicion interceptor que agrega el nombre de la plataforma a la cabezera
sourceAgent.sources.linuxLog.interceptors.plataforma.type = static
sourceAgent.sources.linuxLog.interceptors.plataforma.preserveExisting = true
sourceAgent.sources.linuxLog.interceptors.plataforma.key = plataforma
sourceAgent.sources.linuxLog.interceptors.plataforma.value = linux


# -------------------------------- Configuracion de los sink ------------------------------ #

sourceAgent.sinks.avroSinkLinux1.type = avro
sourceAgent.sinks.avroSinkLinux1.hostname = 172.36.1.32
sourceAgent.sinks.avroSinkLinux1.port = 9997

 

# ---------------------------- Configuracion de los channels ------------------------------ #

# Configure channel Linux-Log
sourceAgent.channels.fileChannelLinux.type = file
sourceAgent.channels.fileChannelLinux.checkpointDir = /home/flume/flume-metadata/flumecheckdirLinuxLog/
sourceAgent.channels.fileChannelLinux.dataDirs = /home/flume/flume-metadata/flumedatadirLinuxLog/


# ------------------- Union de los sink con el source y el channel ------------------------ #

sourceAgent.sources.linuxLog.channels = fileChannelLinux
sourceAgent.sinks.avroSinkLinux1.channel = fileChannelLinux

 

And the morphline is:

 

 

 

# Created by jromero

morphlines : [
{
# Morphline usado para agregar el campo de detail que es usado por el
id : preMorphlineLinuxLog

# Import all morphline commands in these java packages and their subpackages.
# Other commands that may be present on the classpath are not visible to this morphline.
importCommands : ["org.kitesdk.**", "org.apache.solr.**","com.cloudera.**"]

commands : [

{
readLine {
charset : UTF-8
}
}

{ addValuesIfAbsent {
bodybak : "@{_attachment_body}"
}
}


{
grok {

dictionaryFiles : ["/home/flume/grok-patterns"]

expressions : {
message : """%{TIMESTAMP_ISO8601}-%{INT}:%{INT} %{HOSTNAME} %{PROG}.%{WORD}%{GREEDYDATA:detalle}"""
}

}
}

{
logDebug {
format : "output record: {}", args : ["@{}"]
}
}
]
}
]

 

 

I hava try several options like:

-add java code in morphiline to try to agregate the _attachment_body field .

-modify de source code of the morphline core.

-Change the library versions

 

No one of this option work!! . I need their help !! Please Guys its necesary for me. Thanks !!!

 

 

 

 

 

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here