Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hbase-indexer mr : missing morphline TRACE

hbase-indexer mr : missing morphline TRACE

Guru

Hello,

I am currently investigating into HBase-indexer (batch/MR version). In morphline.conf I added logTrace as follows:

 

 

morphlines : [
  {
    id : morphline1
    importCommands : ["org.kitesdk.morphline.**", "com.ngdata.**"]

    commands : [
      {
        extractHBaseCells {
          mappings : [
            {
              inputColumn : "info:firstname"
              outputField : "firstname_s"
              type : string
              source : value
            }
            {
              inputColumn : "info:cat"
              outputField : "cat"
              type : string
              source : value
            }
          ]
        }
      }
      { logTrace { format : "bliblablub output record: {}", args : ["@{}"] } }
    ]
  }
]

 

and the log4j.properties file looks like:

log4j.rootLogger=INFO,stdout,fout

log4j.logger.com.ngdata=TRACE
log4j.logger.org.apache.solr=INFO
#log4j.logger.org.apache.solr.hadoop=DEBUG
log4j.logger.org.kitesdk.morphline=TRACE
log4j.logger.org.kitesdk.examples.morphlines=TRACE
log4j.logger.org.apache.solr.update.processor.LogUpdateProcessor=WARN
log4j.logger.org.apache.solr.core.SolrCore=WARN
log4j.logger.org.apache.solr.search.SolrIndexSearcher=ERROR

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# use a simple message format
log4j.appender.stdout.layout.ConversionPattern=%m%n

log4j.appender.fout=org.apache.log4j.FileAppender
# create a log file
log4j.appender.fout.File=/tmp/hbase-solr-indexer/batchindexer.log
log4j.appender.fout.layout=org.apache.log4j.PatternLayout
# use a more detailed message pattern
log4j.appender.fout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p [%t] %C %M %c{1}:%L - %m%n

 

The whole Index-run is started via:

hadoop --config /etc/hadoop/conf jar /usr/lib/hbase-solr/tools/hbase-indexer-mr-*-job.jar --conf /etc/hbase/conf/hbase-site.xml -D 'mapred.child.java.opts=-Xmx500m' --hbase-indexer-file $HOME/hbase-solr/morphline-hbase-mapper.xml --zk-host quickstart.cloudera/solr --collection hbase-collection2 --go-live --log4j $HOME/hbase-solr/log4j.properties

 

Both of the fields are getting indexed in SOLR, but I cannot find the "logTrace" line including "bliblablub" in the created logfile "/tmp/hbase-solr-indexer/batchindexer.log".

Any hint highly appreciated.

 

Thanks in advance..

 

2 REPLIES 2

Re: hbase-indexer mr : missing morphline TRACE

Expert Contributor
The log file will be on the remote hosts that ran the map tasks, not on the host that started the map reduce driver.

Wolfgang

Re: hbase-indexer mr : missing morphline TRACE

Guru

Hi @whosch ,

thanks for answering quickly.

I am doing this exercise in the Cloudera sandbox, hence it is just the single node env., and the general logfile I configured in the log4j properties is being created in /tmp , but without the morphline trace output...

Any further hints where to deep-dive ?

 

Thanks, Gerd