Could you please try passing /etc/hadoop/conf( where hadoon conf files are residing into HADOOP_CONF_DIR and then submit Spark job? {{ export HADOOP_CONF_DIR=“/etc/hadoop/conf:/etc/hive/conf }} ... View more

Hello   It looks like Hadoop configuration files are not passed to Spark. So please check if hdfs-site and core-site xml files are passed to sprk-sql properly ... View more

Hello,   You try 'spark.scheduler.mode' as 'FAIR'   conf.set("spark.scheduler.mode", "FAIR")  so that multiple job will be executed in parallel.   Please refer document [1]   [1] https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application ... View more

Hi   In order to check available Resource after allocation of container to your application with below grep command on Resource Manager logs   {{ grep "Released container" RM_logs_file | grep "1577297544619_0002" | grep "> available" }} ... View more

Hi   Are you not able to see note which you created in "NoteBook List Box"? If not please try creating new Note from "Notebook Listbox on top left-side screen"-->Create New note--Provide Name and Interpreter. Then you will be able to see the note which you created ... View more

Hi    Please find inline answers,   1) How can I view the history of switching modes (standby/active) in namenode service?   --> You can check history of switching modes (standby/active) in namenode service in failover controler's log file. 2) After how long of active namenode unavailability, standby becomes active?   -->  It depends on how large your edits/fsimage files are to be synched.   We recommend you to refer documents[1][2]   [1] https://docs.cloudera.com/runtime/7.0.1/hdfs-overview/topics/hdfs-moving-highly-available-namenode-failover-controller-and-journalnode-roles-using-the-migrate-roles-wizard.html [2] https://stackoverflow.com/questions/27266267/namenode-ha-failover-time ... View more

Hi,   ``You can configure proxy user using properties hadoop.proxyuser.$superuser.hosts along with either or both of hadoop.proxyuser.$superuser.groups and hadoop.proxyuser.$superuser.users.``   Refer: [1] https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html ... View more

Hello    Could you please ensure if spark streaming connect to the right Kafka broker host; Check if 10.20.0.44:29092 is the correct IP:port.   Also please monitor Kafka broker logs to verify if Spark streaming job is connected to Kafka broker. ... View more