Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop - Override log4j properties to add custom appender

Highlighted

Sqoop - Override log4j properties to add custom appender

New Contributor

HI All,

I am using Hortonoworks HDP cluster with 4 nodes.

I have a sqoop job which pulls the data from MySQL Db into HDFS.

Now, What I am trying to do is push the Sqoop Job logs into MongoDB by overriding log4j properties.

However, after trying out several options I am unable to override Log4j.

Could you please tell me how to do it?

Please note that I do not want to override the global log4j of Hadoop since it will affect other components too which I do not want.

Please let me know if you need any more details.

Regards,

Shiva

3 REPLIES 3
Highlighted

Re: Sqoop - Override log4j properties to add custom appender

Mentor

what are you using to execute sqoop jobs? If it's Oozie, the logs are easily accessible from Oozie logs dir and Oozie database. If it's shell command, you can redirect the logs to sysout and push that to Mongo. @Shiva Moorthy. No matter the way, when you do achieve what you're doing, it will be a good candidate for an article here as a representation of something cool that can be done.

Re: Sqoop - Override log4j properties to add custom appender

New Contributor

@Artem Ervits

hi i have a simple oozie shell action to echo "hell o world", I am trying to over ride the oozie shell action logging Please let me know how this can be achieved

I tried setting the map reduce properties of shell action still it does not work

I have a bash script like this

bash

--------------------------

#!/bin/bash

echo "log4j logger checking the target" # End of script tasks, returning error code

I added the following property in workflow and i tried to execute

<code><property>
      <name>mapred.child.java.opts</name>
      <value>-Dlog4j.configuration=${log4jConfig}</value>
 </property>

the log4j content is similar to this

log4j.rootLogger=${hadoop.root.logger} hadoop.root.logger=INFO,oozie log4j.appender.oozie=org.apache.log4j.ConsoleAppender log4j.appender.oozie.target=System.err log4j.appender.oozie.layout=org.apache.log4j.PatternLayout log4j.appender.oozie.layout.ConversionPattern=[%p] %d{yyyy-MM-dd'T'HH:mm:ss.SSSZ} [sds-to-hdfs][%c] %m%n

I want to change the logging target to System.out just to check if the log4j is overirdden

But still the log4j is not overriden please let me know if there is smothing wrong

Highlighted

Re: Sqoop - Override log4j properties to add custom appender

Mentor

@Shiva Moorthy has this been resolved? Can you post your solution or accept best answer?

Don't have an account?
Coming from Hortonworks? Activate your account here