Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Checking the return value of a morphline read loop for each record processed

Checking the return value of a morphline read loop for each record processed

Explorer

Is there any way of checking the per-record return value of a compiled morphline command that has a reader loop (e.g., readLine)?

 

For example, let's assume the following code:

 

morphlines: [
  {
    id: aMorphline

    importCommands: ["org.kitesdk.**", "org.apache.solr.**"]

    commands: [

      {
        readLine {
          charset: UTF-8
        }
      }

      {
        tryRules {
          copyRecords: false
          catchExceptions: false
          throwExceptionIfAllRulesFailed: true

          rules: [
            {
              commands: [
                {
                  grok {
                    dictionaryResources: [.../grok-dictionaries/patterns]
                    expressions: {
                      message: ${grok_expression}
                    }
                  }
                }
                {
                  ...
                }
              ]
            }
          ]
        }
      }
    ]
  }
]

If this command is compiled into a Java Command object, e.g.

 

Command com = ... // Compile conf file

then, a check like

 

if (com.process(record)) {...}

does not return until all input has been consumed.

 

Is there any way of implementing such a check where the per-record return value can be checked from within a Java program?

 

 

2 REPLIES 2
Highlighted

Re: Checking the return value of a morphline read loop for each record processed

Expert Contributor
If you want that you'd need to replace the readLine command with your own custom command.

Re: Checking the return value of a morphline read loop for each record processed

Explorer

That was my conclusion, too, but I was hoping that I was missing something. :-)

 

Thanks.

 

Don't have an account?
Coming from Hortonworks? Activate your account here