Reply
Expert Contributor
Posts: 131
Registered: ‎08-08-2013

hbase-indexer mr : missing morphline TRACE

Hello,

I am currently investigating into HBase-indexer (batch/MR version). In morphline.conf I added logTrace as follows:

 

 

morphlines : [
  {
    id : morphline1
    importCommands : ["org.kitesdk.morphline.**", "com.ngdata.**"]

    commands : [
      {
        extractHBaseCells {
          mappings : [
            {
              inputColumn : "info:firstname"
              outputField : "firstname_s"
              type : string
              source : value
            }
            {
              inputColumn : "info:cat"
              outputField : "cat"
              type : string
              source : value
            }
          ]
        }
      }
      { logTrace { format : "bliblablub output record: {}", args : ["@{}"] } }
    ]
  }
]

 

and the log4j.properties file looks like:

log4j.rootLogger=INFO,stdout,fout

log4j.logger.com.ngdata=TRACE
log4j.logger.org.apache.solr=INFO
#log4j.logger.org.apache.solr.hadoop=DEBUG
log4j.logger.org.kitesdk.morphline=TRACE
log4j.logger.org.kitesdk.examples.morphlines=TRACE
log4j.logger.org.apache.solr.update.processor.LogUpdateProcessor=WARN
log4j.logger.org.apache.solr.core.SolrCore=WARN
log4j.logger.org.apache.solr.search.SolrIndexSearcher=ERROR

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# use a simple message format
log4j.appender.stdout.layout.ConversionPattern=%m%n

log4j.appender.fout=org.apache.log4j.FileAppender
# create a log file
log4j.appender.fout.File=/tmp/hbase-solr-indexer/batchindexer.log
log4j.appender.fout.layout=org.apache.log4j.PatternLayout
# use a more detailed message pattern
log4j.appender.fout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p [%t] %C %M %c{1}:%L - %m%n

 

The whole Index-run is started via:

hadoop --config /etc/hadoop/conf jar /usr/lib/hbase-solr/tools/hbase-indexer-mr-*-job.jar --conf /etc/hbase/conf/hbase-site.xml -D 'mapred.child.java.opts=-Xmx500m' --hbase-indexer-file $HOME/hbase-solr/morphline-hbase-mapper.xml --zk-host quickstart.cloudera/solr --collection hbase-collection2 --go-live --log4j $HOME/hbase-solr/log4j.properties

 

Both of the fields are getting indexed in SOLR, but I cannot find the "logTrace" line including "bliblablub" in the created logfile "/tmp/hbase-solr-indexer/batchindexer.log".

Any hint highly appreciated.

 

Thanks in advance..

 

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: hbase-indexer mr : missing morphline TRACE

The log file will be on the remote hosts that ran the map tasks, not on the host that started the map reduce driver.

Wolfgang
Expert Contributor
Posts: 131
Registered: ‎08-08-2013

Re: hbase-indexer mr : missing morphline TRACE

Hi @whosch ,

thanks for answering quickly.

I am doing this exercise in the Cloudera sandbox, hence it is just the single node env., and the general logfile I configured in the log4j properties is being created in /tmp , but without the morphline trace output...

Any further hints where to deep-dive ?

 

Thanks, Gerd