About mokkan

mokkan · ‎05-21-2025

Spark job is writing event log to hdfs://namenode:8021/spark-history , but our job creates many events within 10 to 12 minutes. As per the document spark.eventLog.buffer.kb =100kb Does it mean when event writes to /spark-history/application_xxxxxx_xx/xx , whenever buffer gets 100kb? It means every time it is going to call fsync? spark.eventLog.buffer.kb 100k Buffer size to use when writing to output streams, in KiB unless otherwise specified.

mokkan · ‎11-16-2019

Thank you for the info. Yes, I have created with backup with another directory and I was about to boot restart the namenode from that image.

mokkan · ‎11-16-2019

Thank you for the information. If I have multiple directories configured for dfs.namenode.name.dir. Defualt: /hadoop/hdfs/namenode I can configure /backup/namenode /goldbackup/namenode In this case I will have multiple copy of FSimage and Editlog and in case one the directory is corrupted, I can use another one. Does it make sense?

mokkan · ‎11-15-2019

Hello, I have a basic question about about namenode format. If we perform namenode format, will clean up all the metadata info in namenode like directory structure, etc. After it completed, if we restart namenode, would it rebuild the directory structure from datanode information? Or we are lost.

Online	Offline
Last Visited	‎05-21-2025 11:09 AM

Member Since	‎09-22-2017 06:34 PM
Last Visited	‎05-21-2025 11:09 AM
Posts	38

Cloudera Community

Re: namenode format question

spark history server event log question

Re: namenode format question

Re: namenode format question

namenode format question