Member since
10-29-2015
121
Posts
29
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
478 | 06-27-2024 02:42 AM | |
2067 | 06-24-2022 09:06 AM | |
3386 | 01-19-2021 06:56 AM | |
54487 | 01-18-2016 06:59 PM |
08-13-2024
03:12 AM
1 Kudo
So do you mean that there is no need of migration, just bring up the new server, assign required roles and then decomm the old one?
... View more
08-13-2024
03:11 AM
1 Kudo
Thank you for the response. But its my old post and I was able to get Yarn QM working then.
... View more
08-07-2024
04:21 AM
1 Kudo
Got it @AyazHussain. I was unclear with the statement "namespace updated for NN for RM". In our cluster we already have namespaces updated and also apps reach namespace instead of to NN directly. So that is okay. Lastly, would you be able to comment on how and what precautions are needed while moving below roles from one server to another? Target is to decommission old server. Atlas Server HBase REST Server HBase Thrift Server HDFS Balancer HDFS HttpFS Hive on Tez HiveServer2 Hue Server Hue Kerberos Ticket Renewer Impala Daemon Livy Server As per my understanding, we will need to just add new hosts and assign them the relevant roles, however, few of these might also need data migration. Any comments on that? Thanks Snm1523
... View more
08-07-2024
01:31 AM
Thank you for the response @AyazHussain. Possible to please reiterate "you need to have namespace updated for NN for RM" Thanks Snm1523
... View more
08-06-2024
08:29 AM
Hello, We are in process of adding new servers to existing CDP PB cluster. These servers will have master and worker roles along with Kafka brokers distributed / assigned to them. As part of the process, we will also decommission few servers which will be replaced by these new ones. Questions are: 1. What will be the functional impact of this change on data engineers / analysts / developers of team? 2. What changes they will need to do in their codes / applications / integrations / connections, etc? I am aware of few like: - Zookeeper ensemble will need to be modified - Teams will need to be informed of new brokers - Custom Knox topologies will need to be modified What are the other changes that will need to be done by the user / engineers group? 3. I am aware that connection strings will need to be modified. Which ones, just Hive or any other as well? Services that will be impacted of this change are: HDFS YARN HIVE IMPALA KAFKA HBASE Also, we have Tableau and other applications that connect to cluster to fetch reports / data. Kindly advise. Thanks Snm1523
... View more
06-27-2024
02:42 AM
1 Kudo
For the benefit of anyone looking for this, Java Configuration Options for NodeManager Java Configuration Options for ResourceManager Java Configuration Options for <component name> are the configuration that needs to be updated with the -javaagent command. This allows to pick the Prometheus agent jar and enable JMX exporter to collect metrics.
... View more
06-26-2024
04:06 AM
Hello, We are on CDP PB 7.1.9 and goal is to monitor YARN applications and performance along with few HDFS metrics on Grafana via dashboards and ultimately trigger alerts. Metrics has to be collected via Prometheus agent and shared with Grafana. At this point, I have downloaded a Prometheus agent and created a below yml configuration. I understand this will collect all the metrics, which is intentional to start with. lowercaseOutputName: true
rules:
# All Gauge type Hadoop Metrics
- pattern: 'hadoop<name=.*><>(Count|Value)'
name: hadoop_${1}_gauge
type: GAUGE
# All Counter type Hadoop Metrics
- pattern: 'hadoop<name=.*><>(Count|Value)'
name: hadoop_${1}_counter
type: COUNTER Prometheus agent jar and this config file is stored at /var/lib/prometheus_jmx_config/. For now testing this only on resource manager instances to verify if metrics are getting collected. Following few articles and Grafana documentation, I understood that I will need to run Prometheus as a Java agent (our case) or a stand alone HTTP server. To achieve this, we need to expose / enable JMX for the components we need to monitor (Resource Manager in this case) which has to be done by adding java agent command in hadoop-env.sh. Referring to the documentation, below is the command I think should work: YARN_RESOURCEMANAGER_OPTS="-javaagent:/var/lib/prometheus_jmx_config/jmx_prometheus_javaagent-0.20.0.jar=9091:/var/lib/prometheus_jmx_config/hadoop_jmx_exporter.yml" I tried adding this command in Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh of YARN configuration and it prompted to restart impacted services as this would change yarn-conf for multiple services. However, YARN service (RM to be specific) was never restarted. So I restarted it thinking it will execute prometheus agent as Java process and ultimately enable JMX on port 9091. However, the agent never started so JMX did not got enabled. We also have few Java related properties specific to components like below: Java Configuration Options for NodeManager Java Configuration Options for ResourceManager However, as I am not confident if those are correct, would be great if someone could advise on the same or if any configuration that I have missed. Thanks Snm1523
... View more
Labels:
06-21-2024
07:27 AM
Hello @wbekker, Which one and how did you turned it off? Thanks snm1523
... View more
06-07-2024
01:19 AM
1 Kudo
UPDATE: There were configs done incorrectly in topology files due to which I was unable to access Service UIs. Fixing the topology config file helped.
... View more
06-05-2024
02:30 AM
1 Kudo
Thank you for the detailed explanation, @ShankerSharma. However, we ultimately had the engineering team along with developers who did this job. But I will keep this in my notes for reference.
... View more