Hello @StanislavJ , The Linux kernel parameter, vm.swappiness, is a value from 0-100 that controls the swapping of application data (as anonymous pages) from physical memory to virtual memory on disk. The higher the value, the more aggressively inactive processes are swapped out from physical memory. The lower the value, the less they are swapped, forcing filesystem buffers to be emptied. On most systems, vm.swappiness is set to 60 by default. This is not suitable for Hadoop clusters because processes are sometimes swapped even when enough memory is available. This can cause lengthy garbage collection pauses for important system daemons, affecting stability and performance. Cloudera recommends that you set vm.swappiness to a value between 1 and 10, preferably 1, for minimum swapping on systems where the RHEL kernel is 2.6.32-642.el6 or higher. To view your current setting for vm.swappiness, run: #cat /proc/sys/vm/swappiness To set vm.swappiness to 1, run: #sudo sysctl -w vm.swappiness=1 Also, To give an overview, swapping alerts are generated in Cloudera Manager when host swapping or role process swap usage exceeds a defined threshold. The warning threshold of "500 MiB" will mean that any swap usage beyond this on a given host will generate an alert and critical if set to any would generate an alert even if a small amount of swapping occurs. The swap memory usage threshold value can be set at the "host" level or at the "process/service" level. >> To set the threshold at the process level settings can be done as follows: From CM UI >> Clusters >> yarn >> Configuration >> search for "Process Swap Memory Thresholds" >> (For resource manager) Warning and Critical >> Select Specify >> Specify value here (You can set the values in Bytes/KB/MB/GB) >> Save changes You can increase the value and then further suggest you monitor the cluster for the swap usage and adjust the values accordingly. If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post. Thank you, Babasaheb Jagtap ... View more

Hello @Timo & @George-Megre    If you are facing an issue after enabling Kerberos and are unable to produce/consume then we would suggest you to please follow the below steps and let us know how it goes:    Make sure that all partitions are in a healthy state using the Kafka describe command or that there is no warning/alerts for Kafka in CM. If there is no alert for Kafka then, please follow the below steps to connect to the Kafka topic:    1) kinit with the keytab and make sure that the user is having required permissions enabled in Ranger  2) Create jaas.conf file with the contents:    vi /tmp/jaas.conf   KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="zookeeper"; };   3) Run the following command export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/jaas.conf   Note: Make sure and replace jaas.conf complete path.    4)  Create the client.properties file containing the following properties. vi /tmp/client.properties   security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name=kafka   5) Start console producer kafka-console-producer --broker-list <broker1.test.com:6667,broker2.test.com:6667> --topic <topic-name> --producer.config /tmp/client.properties   6) Start console consumer kafka-console-consumer --bootstrap-server <broker1.test.com:6667,broker2.test.com:6667> --topic <topic-name> --consumer.config /tmp/client.properties --from-beginning   Note: Use the complete hostname of the broker,  Also, replace the topic, client.properties name in the above commands.    Please check and f you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.   Thank you.  ... View more

Hello @AndreyKravtsov    Please refer to the article below to Integrating Apache NiFi and Apache Kafka [1]   [1]  https://community.cloudera.com/t5/Community-Articles/Integrating-Apache-NiFi-and-Apache-Kafka/ta-p/247433   This example is with the PLAINTEXT Kafka protocol. It looks like you are using SSL/TLLS for kafka. You should check whether you are using SSL or SASL_SSL protocol for kafka from CM >> Kafka conf and created a StandardSSLContextService controller setting and update with Kafka Keystore and Truststore details. You can refer the below article for details [2]:   [2] https://community.cloudera.com/t5/Support-Questions/Need-help-with-SSL-config-in-Nifi-ConsumeKafka/td-p/320594   Additionally, refer the following articles for more details:  [1] - https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.17.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html [2] - https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.20.0/org.apache.nifi.ssl.StandardSSLContextService/index.html   If this information helped you, then it will be appreciated if you will take a moment to click on KUDOS 🙂 ... View more

Hello @Yam    Below are my inline answers to your queries:    #1 How to find In which server the Kafka cluster and topic are created and  >> You can check hostnames and broker IDs on which Kafka broker roles are installed from CM >> Kafka >> Instances. Here you will get the hostnames and broker IDs.   Now to check on which brokers topics are created either you can check it from SMM which very useful monitoring and administrative tool Cloudera is providing or you simply describe all topics using the command:    kafka-topics --zookeeper ZK_HOST:2181 --describe   OR kafka-topics --bootstrap-server <broker-hostname>:9092 --describe     Note: If the cluster is kerberized, then you should use keytab, jaas.conf, client.properties and the correct port which you are using to describe topics   #2 Can we find any IP addresses anywhere in the configuration/properties files >> You can check hostnames, IPs and roles installed on those nodes from CM >> Hosts >> All Hosts.   If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post. ... View more

Hello @Amn_468    1) The quick fix might be to restart both RM and YQM and see if that clears the issue.   If you still face the same issue, then   2) Check if you are using TLSL for CM and YARN. If yes, then configure TLS for Yarn Queue Manager as well [1].   [1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/yarn-security/topics/yarn-configure-tls-ssl-yarn.html   3) If Kerberos is enabled for YARN, then enable it for the YARN queue manager. Could you please verify if Kerberos security is configured correctly using the below doc [1]: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-kerberos-authentication/topics/cm-security-kerberos-enabling-step8-verify-kerberos.html   If you still face the same issue, then please upload the below queue manager log files from the Queue manager host for the issue time stamp:- *queuemanager-webapp.log *config-service.log *config-service-requestlog.log location : /var/log/yarn/queuemanager/   Please upload ResourceManager log files for the issue time-stamp - location : /var/log/hadoop-yarn/   If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.   Thank you. ... View more

 @DataMike    Yes, you can use the CC APIs as per your requirements [1] [2]   [1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cctrl-managing/topics/cctrl-using-rest-api.html [2] https://community.cloudera.com/t5/Customer/Frequently-Used-CRUISE-CONTROL-API-and-important-DOCs/ta-p/324729   From the above articles you can use the following API to rebalance topics/partitions:    curl -k --negotiate -u: -X POST "https://<CC FQDN>:8899/kafkacruisecontrol/rebalance?dryrun=false&rebalance_disk=true"   To avoid high CPU, memory and disk read write you can initiate rebalance process in batches. It will automatically create a batch and rebalance topics.   curl -X POST  "http://$HOSTNAME:8899/kafkacruisecontrol/rebalance?dryrun=true&concurrent_partition_movements_per_broker=10&concurrent_leader_movements=500"   If it will help you then please click on ”Accept as Solution" below this post.   Thank you. ... View more