I am working on a Hortonwork cluster which run Spark jobs using Livyclient as follow
LivyClient client = new LivyClientBuilder().setURI(new URI(livyUrl))
// .setConf("spark.executor.instances", "40")
String localJar = "hdfs://auper01-01-20-01-0.prod.vroc.com.au:8020/tmp/simple-project/mdm_mahdi.jar";
I then try to log inside my application using
org.apache.log4j.LogManager.getRootLogger().info("This is an information log");
I am abit confused as there is no yarn installed on the cluster and I can not see logs in Spark history job when I go to executors. Can anyone suggest me how to get my own logs inside Spark applications. when running by Livyclient.