Member since
11-26-2019
8
Posts
0
Kudos Received
0
Solutions
07-21-2020
03:39 AM
I have one multinode HDP3.1.5 cluster and As per given in the Apache Hadoop 3.1.1 Documentation , it is expected to show all the metrics including NumTransactionsBatchedInSync60sNumOps and NumTransactionsBatchedInSync60s50thPercentileLatencyMicros on Journal nodes REST response given on 8480 port, these metrics are missing in REST responses of Journal Node. I even tried to configure dfs.metrics.percentiles.intervals property in hdfs-site.xml, but instead of exposing all metrics on journal node REST, the metrics are exposed on Namenodes REST response given on port 50070. Could you please help me how to expose such metrics on Journal Node as well? Is there any other journal node specific property which I need to configure to expose such metrics?
... View more
Labels:
06-23-2020
01:26 AM
I need to check the status of ZKFailOverController(ZKFC) service whether it is running or not, is there any way to collect the ZKFC health/ monitoring metrics in java code easily? Also does the dfs.ha.zkfc.port property is version specific? I had one observation that this property is visible automatically on HDP 3.1 but not on HDP 2.5 although it is HA enable.
... View more
Labels:
06-18-2020
04:48 AM
I want to fetch the config parameters from the conf REST response, but not sure whether the solution of fetching config details from http://hostname:50070/conf worked or not for all types of clusters which are having the secured endpoints. Please assist here.
... View more
Labels:
06-11-2020
06:28 AM
Looking for the Journal port in any JMX response except http://hostname:50070/conf, Also searching for the List of active journal nodes Currently i'm getting the list of journal nodes from "NameJournalStatus" : "[{\"manager\":\"QJM to ["host1:8485","host2:8485","host3:8485"]" but its not clear whether this is list is of active journal nodes or not and also the port 8485 is IPC port but I want JMX port like default ports 8480 or 8481 of journal node
... View more
Labels:
03-20-2020
01:50 AM
@venkatsambath If it is the value of blocks, then how 2097152 blocks can be fit in 100gb space cluster each of 128 mb of block? Acutally i'm not getting how this block capacity value will be calculated, while calculating this metric total memory is considered or not? I have single node hadoop cluster as well as multinode test envirnonment in both the value for block capacity is same 2097152. Where should i found the block size in rest response? Is there any other URI for this metric? Initially it was assumed that the block capacity is same block size but over the time this value got changed.
... View more
03-16-2020
04:44 AM
BlockCapacity given in the REST reponse of Namenode is different than what we have set in the Ambari under HDFS dfs.blocksize
On Rest I am getting the value for BlockCapacity is 2097152 which equivalent to 2mb although the value of blocksize in Ambari is 128mb by default Fetching the blockcapacity from from following REST url https://<namenode-ip>:50470/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Does BlockCapacity from namenodes rest api and dfs.blocksize from Ambari are the same or different?
... View more
Labels:
11-26-2019
11:19 PM
Response of Jobs API : http://<history server http address:port>/ws/v1/history/mapreduce/jobs
{ "jobs": { "job": [{ "submitTime": 1574157410876, "startTime": 1574157419359, "finishTime": 1574157459602, "id": "job_1568869492319_0009", "name": "word count", "queue": "default", "user": "hdfs", "state": "FAILED", "mapsTotal": 0, "mapsCompleted": 0, "reducesTotal": 0, "reducesCompleted": 0 }] } }
Response of Job API : http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}
{ "job": { "submitTime": 1574157410876, "startTime": 1574157419359, "finishTime": 1574157459602, "id": "job_1568869492319_0009", "name": "word count", "queue": "default", "user": "hdfs", "state": "FAILED", "mapsTotal": 9, "mapsCompleted": 0, "reducesTotal": 1, "reducesCompleted": 0, "uberized": false, "diagnostics": "Task failed task_1568869492319_0009_m_000006\nJob failed as tasks failed. failedMaps:1 failedReduces:0", "avgMapTime": 0, "avgReduceTime": 0, "avgShuffleTime": 0, "avgMergeTime": 0, "failedReduceAttempts": 0, "killedReduceAttempts": 0, "successfulReduceAttempts": 0, "failedMapAttempts": 17, "killedMapAttempts": 7, "successfulMapAttempts": 0 } }
See the difference in the mapsTotal and reducesTotal values, Could anybody have idea whats happening here?
I'm running the jobs on HDP3.0 platform.
... View more