@hayhapra  Have a look at this Cloudera Druid tableau connection    Happy hadooping ... View more

@Amn_468  The Namenode is the brain of the cluster, it has the footprint of the cluster location of the files, ACL's, stores the HDFS metadata, the directory tree of all files in the file system, and tracks the files across the cluster and does not store the actual data or the dataset. The data itself is actually stored in the Datanodes. Your error   2020-10-27 16:20:05,140 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1821ms GC pool 'ParNew' had collection(s): count=1 time=2075ms)   This indicates that the NameNode paused for longer than the expected time of 60000ms. This also explains why DataNode did not get a response from NameNode in designated 60000ms.   The warning also indicates that the pause was due to GC which calls for a memory and GC Tuning. NameNode knows the location, list of the blocks with this information NameNode knows how to construct the file from blocks. The fastest way to render this information is to store it in memory that's the reason the NN is usually on a high-end server configured with a lot of memory (RAM). because the block locations are stored in RAM   An ideal starter config in production for a datanode and Namende would be Name Node Configuration   Processors: 2 Quad Core CPUs running @ 2 GHz RAM: 128 GB Disk: 6 x 1TB SATA Network: 10 Gigabit Ethernet   Data Node Configuration   Processors: 2 Quad Core CPUs running @ 2 GHz RAM: 64 GB Disk: 12-24 x 1TB SATA Network: 10 Gigabit Ethernet   A fundamental parameter to tune for garbage collectors is the number of HDFS blocks stored in the Hadoop cluster in your case 23,326,719 files. The number of files, and associated blocks, is a fundamental parameter in the tuning process. The Namenode maintains the complete directory structure in memory. Therefore, more files mean more objects to manage. Most of the time, Hadoop clusters are configured without knowledge of the final workload in terms of the number of files that will be stored. Having in mind the strong connection between these two aspects is crucial to anticipate future turbulence in the hdfs quality of service.   You should analyze log prints produced by the garbage collector the gc.log files found in the Namenode logs directory the available memory is filling up before the garbage collector activity is able to release it. Hope that helps ... View more

@jlguti    Can you share the output of the below command?  Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container. yarn logs -applicationId application_1604418534431_0001 That could give us pointers to the potential issue either memory or some misconfiguration.   Happy Happoingg       ... View more

@Amn_468    The NameNode is solely responsible for the Cluster Metadata so please increase the NN heap size and restart the services. Please revert    ... View more

@Amn_468  Increasing the Java Heap Size for the NameNode and Secondary NameNode Services,you could be using the default 1GB setting for heap size As a general rule of thumb take a look at the configuration of your Heap Sizes for every 1 Million Blocks in your cluster should have at least 1GB of Heap Size. 2 Million Blocks 2GB heap size 3 Million Blocks 3GB heap size ..... n Million Blocks n GB heap size After increasing the Java Heap Size and restart the HDFS Services that should resolve the issue.   Please revert ... View more