Resources

SYMPTOM: Ambari is showing Alert about a connection failed to the journal node service. Below is the alert -2016-06-30 18:50:39,865 [CRITICAL] [HDFS] [journalnode_process] (JournalNode Process) Connection failed to http://jn1.example.com:8480 (Execution of 'curl -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/f8ed47d4-f63e-482c-be70-36755387ca4b -c /var/lib/ambari-agent/tmp/cookies/f8ed47d4-f63e-482c-be70-36755387ca4b -w '%{http_code}' http://jn.example.com:8480 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpE9v3mg 2>/tmp/tmpKOSncN' returned 28. % Total % Received % Xferd Average Speed Time Time Time Current ERROR: Below are the journal logs2016-07-01 10:21:29,390WARN namenode.FSImage (EditLogFileInputStream.java:scanEditLog(350)) - Caughtexception after scanning through 0 ops from/hadoop/hdfs/journal/phadcluster01/current/edits_inprogress_0000000002510372012while determining its valid length. Position was 712704 java.io.IOException: Can't scan a pre-transaction

OverviewData Governance is unique for each organization and every organization needs to track a different set of properties for their data assets. Fortunately, Atlas provides the flexibility to add new data asset properties to support your organization’s data governance requirements. The objective for this article is to describe the steps utilizing the Atlas REST API to add new Atlas properties to your Atlas Types. Add a new Property for an existing Atlas TypeTo simplify this article, we will focus in on the 3 steps required to add and enable for display a custom property to the standard Atlas property ‘hive_table’. Following these steps, you should be able to modify the ‘hive_table’ Atlas Type and add custom properties which are available to enter values, view in the Atlas UI and search.To make the article easier to read the JSON file is shown in small chunks. To view the full JSON file as well as other files used to research for this article, check out this repo.Step 1: Define th

SYMPTOM: The Standby NameNode process running on our 2nd of four management node servers isn't running. Interrogating the log files, I've found an exception relating to an Oozie jobERROR: Below was the error logs -2016-12-20 09:20:17,286 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'http://node1:8480/getJournal?jid=namenodeha&segmentTxId=16740759&storageInfo=-63%3A1400038789%3A0%3ACID-031f35b2-59c9-42f9-8942-550aee3d39e6, http://node1:8480/getJournal?jid=namenodeha&segmentTxId=16740759&storageInfo=-63%3A1400038789%3A0%3ACID-031f35b2-59c9-42f9-8942-550aee3d39e6' to transaction ID 167130782016-12-20 09:20:17,287 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'http://node1:8480/getJournal?jid=namenodeha&segmentTxId=16740759&storageInfo=-63%3A1400038789%3A0%3ACID-031f35b2-59c9-42f9-8942-550aee3d39e6' to transaction ID 167130782016-12-20 09:20:18,28

SYMPTOMS: No visible errors in Resource manager / Node Manager logs for any resource bottleneck.Logs from container/task which is not progressing are as follows:-Error: java.io.IOException: com.sap.db.jdbc.exceptions.jdbc40.SQLNonTransientConnectionException: Connection to database server lost; check server and network status [System error: Socket closed] at org.apache.sqoop.mapreduce.db.DBRecordReader.close(DBRecordReader.java:173) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:523) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:791) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: com.sap.db.j

SYMPTOM: All the services in the cluster are down and restarting the services fails with the following error:2016-11-17 21:42:18,235 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.io.IOException: Login failure for nn/lnx21131.examplet.ex.com@EXAMPLE.AD.EX.COM from keytab /etc/security/keytabs/nn.service.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) ...Caused by: KrbException: Client not found in Kerberos database (6) ...Caused by: KrbException: Identifier doesn't match expected value (906)Regeneration of Keytabs using Ambari too failed as follows:17 Nov 2016 23:58:59,136 WARN [Server Action Executor Worker 12702] CreatePrincipalsServerAction:233 - Principal, HTTP/lnx21142.examplet.ex.com@EXAMPLE.AD.EX.COM, does not exist, creating new principal 17 Nov 2016 23:58:59,151 ERROR [Server Action Executor Worker 12702] CreatePrincipalsServerAction:284 - Failed to create or update principal, HTTP/lnx21142.exampl

often we have need to read the parquet file, parquet-meta data or parquet-footer, parquet tools is shipped with parquet-hadoop library which can help us to read parquet. these are simple steps to build parquet-tools and demonstrate use of it.prerequisites: maven 3,git, jdk-7/8 // Building a parquet tools git clone https://github.com/Parquet/parquet-mr.git cd parquet-mr/parquet-tools/ mvn clean package -Plocal // know the schema of the parquet file java -jar parquet-tools-1.6.0.jar schema sample.parquet // Read parquet file java -jar parquet-tools-1.6.0.jar cat sample.parquet // Read few lines in parquet file java -jar parquet-tools-1.6.0.jar head -n5 sample.parquet // know the meta information of the parquet file java -jar parquet-tools-1.6.0.jar meta sample.parquet

SYMPTOM: After upgrading ambari from 2.1.1 to 2.2.2.2 tried restarting oozie service which failed with error - " su: cannot set user id: Resource temporarily unavailable"ERROR: Below are the error logs-Execution, [[0000002-160227115902137-oozie-oozi-C@4]::CoordActionInputCheck:: Ignoring action. Coordinator job is not in RUNNING/RUNNINGWITHERROR/PAUSED/PAUSEDWITHERROR state, but state=SUSPENDED], Error Code: E1100 2016-07-02 13:04:42,457 WARN CoordActionInputCheckXCommand:523 - SERVER[hdmlup000a.machine.group] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-160227115902137-oozie-oozi-C] ACTION[0000002-160227115902137-oozie-oozi-C@5] E1100: Command precondition does not hold before execution, [[0000002-160227115902137-oozie-oozi-C@5]::CoordActionInputCheck:: Ignoring action. Coordinator job is not in RUNNING/RUNNINGWITHERROR/PAUSED/PAUSEDWITHERROR state, but state=SUSPENDED], Error Code: E1100 2016-07-02 13:04:42,459 WARN CoordActionInputCheckXCommand:523 - SERVER[hdmlup000a.machine.group]

SYMPTOM:user complains that he is seeing some special characters (^@^@^@^@^@^@^) in GC logs similar to thisconcurrent-mark-sweep perm gen total 29888K, used 29792K [0x00000007e0000000, 0x00000007e1d30000, 0x0000000800000000)^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@2016-12-10T00:09:31.648-0500: 912.497: [GC2016-12-10T00:09:31.648-0500: 912.497: [ParNew: 574321K->15632K(629120K), 0.0088960 secs] 602402K->43713K(2027264K), 0.0090030 secs]ROOT CAUSE:there was some query which was running on hiveserver2 in mr mode which spins Map Join act

SYMPTOM: HDFS service is not able to start and throwing below error in logs -python error " File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 124, in create_ranger_repository" ERROR: Ambari operation logs shows below message -stderr: /var/lib/ambari-agent/data/errors-22280.txtTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 433, in <module> NameNode().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 524, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 102, in start namenode(action="start", hdfs_binary=hdfs_binary, upgrade_type=upgrade_type, env=en

SYMPTOM: Ambari smoke test fails for hbase service. Below is the current scenario - Ranger is installed in the cluster HBase policy has been enabledambari-qa user has the privileges correctly defined in the HBase policyERROR: ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=ambari-qa, scope=default, params=[namespace=default,table=default:ambarismoketest,family=family],action=CREATE)2015-10-27 09:52:03,342 ERROR [main] client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.TableNotFoundException: Table 'ambarismoketest' was not found, got: XXXXX01. ROOT CAUSE: If the Ranger co-processor is not correctly defined in the HBase configuration, the smoke test from Ambari would fail. Any table creation as non-hbase user could also fail.RESOLUTION:Verify the Ranger configuration for HBase. Ensure that the following properties are set correctly and that the co-processors include Ranger classeshbase.coprocessor.master.classeshbase

SYMPTOM: Ranger plugin is enabled for HIVE. Restarting HIVE service its not able to start and stucking on below error ERROR: 2015-10-15 13:02:51,683 - u"File['/var/lib/ambari-agent/data/tmp/ojdbc6.jar']" {'content': DownloadSource('http://sjcservicenode04-prod.xxxinternal.com:8080/resources//oracle-jdbc-driver.jar')} 2015-10-15 13:02:51,796 - Not downloading the file from http://sjcservicenode04-prod.xxxinternal.com:8080/resources//oracle-jdbc-driver.jar, because /var/lib/ambari-agent/data/tmp/oracle-jdbc-driver.jar already exists 2015-10-15 13:02:51,996 - call['hdp-select status hadoop-client'] {'timeout': 20} ROOT CAUSE: Ranger Hive policies http url calls were taking forever to return results Ranger makes a lot of calls to urllib2.urlopen(request) that don't have a timeout in Ambari 2.0 Opened Ambari BUG to put time=5 in the ranger_function.py file. https://hortonworks.jira.com/browse/BUG-46275   RESOLUTION: 1) Edit /usr/lib/python2.6/site-packages/resource-management/librar

Scenario: Lets say you have 2 ranger admin instances configured in your cluster. Now if you want to enable Ranger HA, you need to delete one of the ranger admin instance[since Ranger HA admin wizard will create 2nd instance of ranger for you] In such case you need to remove one instance of Ranger admin which is already installed. The following steps will guide you how to remove Ranger Admin instance using Ambari API Backup Ambari Server database [https://ambari.apache.org/current/installing-hadoop-using-ambari/content/ambari-chap11-1.html] Stop the Ranger service using Ambari. In case the Ranger Admin fails to stop, try stopping the Ranger Service as follows using Ambari API: curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' \http://xxx.hostname:8080/api/v1/clusters/TEST/services/RANGERcurl -u admin:admin -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop Service"},

SYMPTOM: User was not able to browse ambari UI after ambari server restart. Ambari version : 2.1.2Below was the error seen in logsERROR: 06 Jul 2016 09:40:26,505 ERROR [Stack Version Loading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.1 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out) 06 Jul 2016 09:40:26,506 INFO [Stack Version Loading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.2 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:28,508 ERROR [Stack VersionLoading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.2 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out)06 Jul 2016 09:40:28,509 INFO [Stack Version Loading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.3 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:30,511 ERROR [Stack Version Loading Thread] Lat

IntroductionThe producer sends data directly to the broker that is the leader for the partition without any intervening routing tier.Optimization ApproachBatching is one of the big drivers of efficiency, and to enable batching the Kafka producer will attempt to accumulate data in memory and to send out larger batches in a single request. The batching can be configured to accumulate no more than a fixed number of messages and to wait no longer than some fixed latency bound (say 64k or 10 ms). This allows the accumulation of more bytes to send, and few larger I/O operations on the servers. This buffering is configurable and gives a mechanism to trade off a small amount of additional latency for better throughput.In order to find the optimal batch size and latency, iterative test supported by producer statistics monitoring is needed.Enable MonitoringStart the producer with the JMX parameters enabled:JMX_PORT=10102 bin/kafka-console-producer.sh --broker-list localhost:9092--topic testtopic

While accessing the Zeppelin View is shows the "zeppelin service is not running". But when we check at the back end then we find that the Zeppelin server was running. PID file has the correct PID information mentioned in it.- As we see that Zepplin Notebok is running via ambari as well. But still when we try to access the Zeppelin View it shows failure saying "Zeppelin service is not running".We will need to check the Zeppelin log to see if it shows any error? We see the following kind of error:INFO [2016-12-01 09:42:08,926] ({main} ZeppelinServer.java[setupWebAppContext]:266) - ZeppelinServer Webapp path: /usr/hdp/current/zeppelin-server/webapps INFO [2016-12-01 09:42:09,331] ({main} ZeppelinServer.java[main]:114) - Starting zeppelin server INFO [2016-12-01 09:42:09,333] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210 WARN [2016-12-01 09:42:09,367] ({main} WebAppContext.java[doStart]:514) - Failed startup of context o.e.j.w.WebAppContext@69b794e2{/,null,null}{/usr/hdp/curr

Ambari allows its users to have different configurations for different hosts for different components via configuration groups. Ambari initially assigns all hosts in your cluster to one, default configuration group for each service you install. When using Configuration groups, it enforces configuration properties that allow override, based on installed components for the selected service and group.For more information on this please refer to: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/using_host_config_groups.htmlHere we will try to make a very small change in "flume-conf" content and will apply it to specific host ("erie3.example.com") where as all the other hosts will be using the default "flume-conf" configuration. We can simply use the following ambari API to list all the config_groups.http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/config_groupsExample:http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups.Step-1).

SYMPTOMS: Sometimes the Hive CLI hangs and the gives no response. At the same time, the /var/log/hive/hivemetastore.log reports timeoutsERROR [<hostname>-47]: txn.TxnHandler (TxnHandler.java:getDbConn(984)) - There is a problem with aconnection from the pool, retrying(rc=9): Timed out waiting for a free available connection.(SQLState=08001,ErrorCode=0) java.sql.SQLException: Timed out waiting for a free available connection.ROOT CAUSE: The Hive Metastore service is waiting for the database connections to get free. At the time of error, not enough concurrent connections were available. RESOLUTION: (Note: This is for mysql DB only).To fix this problem, the number of max database connection should be increased.1. Open the /etc/my.cnf in the text editor.vi /etc/my.cnf2. Under [mysqld] section, add:max_connections = 2503. Save the file and restart the mysqld serviceservice mysqld restart

HDP Stack Version: 2.4.0SYMPTOMS: WARN Error while fetching metadata [{TopicMetadata for topic <topic-name> -> No partition metadatafor topic <topic-name> due to kafka.common.TopicAuthorizationException}] for topic <topic-name>: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) .... ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: <topic-name> (kafka.producer.async.DefaultEventHandler) ROOT CAUSE: At the moment we can’t use user/group based access to authorize Kafka access over a non-secure channel. This is because it is not possible to assert client’s identity over the non-secure channel. It is ip based in a non-secure environment. Reference: https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin See: Authorizing Kafka access over non-authenticated channel via Ranger RESOLUTION: The policy doesn't work if, IP address is not configured. This address will

Run hive shell debug so it prints detailed error: hive -hiveconf hive.log.file=hivecli_tez.log -hiveconf hive.log.dir=/tmp/hivecli -hiveconf hive.execution.engine=tez -hiveconf hive.root.logger=DEBUG,DRFA SYMTOMS: DEBUG [main]: amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1152)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: null; Status Code: 403; Error Code: 403 Forbidden; Request ID: 85BA6566D33A519B), S3 Extended Request ID: 228pqAjcCjTHo+ExpZ+86INHAhkIeE+DQoicPLkan8GDaraxsklIuHwK3f+QmjtIBzw/z5OSWaM= WARN [main]: avro.AvroSerDe (AvroSerDe.java:determineSchemaOrReturnErrorSchema(169)) - Encountered AvroSerdeException determining schema. Returning signal schema to indicate problem org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read schema from given path: s3a://<file-path> WORKAROUND: 01. mkdir joda-backup 02. cd /tmp 03. wget http://central.maven.org/maven2/joda-time/joda-time/2.8.1/joda-time-

SYMTOMPS:Ambari server log shows:WARN [qtp-client-41858] ObjectGraphWalker:209 - The configured limit of 1,000 object references was reached while attempting to calculate the size of the object graph. Severe performance degradation could occur if the sizing operation continues. This can be avoided by setting the CacheManger or Cache <sizeOfPolicy> elements maxDepthExceededBehavior to "abort" or adding stop points with @IgnoreSizeOf annotations. If performance degradation is NOT an issue at the configured limit, raisethe limit value using the CacheManager or Cache <sizeOfPolicy> elements maxDepth attribute. For more information, see the Ehcache configuration documentation. WORK-AROUND Disable cache by setting below property in in /etc/ambari-server/conf/ambari.propertiesserver.timeline.metrics.cache.disabled = true ROOT CAUSE See: AMBARI-13517 Disabling cache is not going have adverse effect. It was introduced in Ambari 2.1.2 to create a Caching layer that provides sli

Setting up a Websocket Client and Server with Apache NiFi 1.1.I wanted to test out the new WebSocket Listener in Apache NiFi 1.1, but I needed a server to serve up my HTML client. So I ran that web server with NiFi as well. So my full solution is hosted and runs through Apache NiFi.This simple WebSockets Server and Client does the hello world of web sockets, Echo! Whatever the client sends, we send it back.My Suggested Use Cases for WebSocketsWebSocket Client to Slack InterfaceWebSocket Client to Email WebSocket Chat stored to Apache PhoenixWebSocket To Communicate From Mobile Web AppsWebSocket To Send Detailed Log Details From Enterprise Web Applications Directly To Log Ingest Platform, bypassing local filesystem.Step 1: HandleHTTPRequest This accepts the HTTP calls from browsersStep 2: ExecuteStreamCommand Returns HTML page (could do getfile or any number of other ways of getting the HTML as a flowfile)Step 3: HandleHttpResponse this serves up our web page to browser

Steps:Oozie server timezone For Ambari users, login and navigate to Custom Oozie-site.Ambari > Oozie > configs > custom oozie-site Add Property "oozie.processing.timezone=GMT-0500"oozie.processing.timezone Default value=UTCOozie server timezone. Valid values are UTC and GMT(+/-)####, for example 'GMT+0530' would be India timezone. All dates parsed and generated dates by Oozie Coordinator/Bundle will be done in the specified timezone. The default value of 'UTC' should not be changed under normal circumstances. If for any reason is changed, note that GMT(+/-)#### timezones do not observe DST changes. Save and Restart Oozie ServiceOozie Web ConsoleTo View the Job in EST Timezone on Oozie Web Console, follow below steps. Open Oozie Web Console on

PROBLEM: Unable to start Resource Manager which fails with below errors:-STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9e75108092247d96ce7d70839b6945e9eba2a0b7; compiled by 'jenkins' on 2014-11-04T04:31ZSTARTUP_MSG: java = 1.7.0_67************************************************************/2014-11-04 08:41:08,705 INFO resourcemanager.ResourceManager (SignalLogger.java:register(91)) - registered UNIX signal handlers for [TERM, HUP, INT]2014-11-04 08:41:10,636 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to loginorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to loginat org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main

SYMPTOM: RM is down due to below error. Earlier we were suspicion the ulimit could be culprit though we have increased it to 128K. But still no luck.ERROR: 2016-07-25 12:19:47,125 WARN security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(873)) - Unable to add the application to the delegation token renewer. java.lang.OutOfMemoryError: unable to create new native thread. Below was few steps followed -1. Checked the error and saw that previously the same issue and increasing ulimit resolved the issue. 2. Checked the ulimit and lsof output - $ulimit -n 131072 $lsof |grep yarn |wc 1726 15553 242741 3. Checked the heap size for yarn process which was set to 8Gb and looks good.Below error was displayed in RM out.log fileJava HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f89641cf000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)## There is insufficient memory for the Java Runtime Environment to continue.# Native memory

SYMPTOM:The following jmx tool command is failing with "port already in user" error/usr/hdp/current/kafka-broker/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec --jmx-url service:jmx:rmi:///jndi/rmi://`hostname`:1099/jmxrmiROOT CAUSE:JMX tool invoke the kafka-run-class.sh after reading kafka-env.sh, since we have already have export $JMX_PORT=1099 there then it tried to bind the same host while invoking the jmx tool which result into error.WORKAROUND:NARESOLUTION:edit kafka-run-class.shReplace section # JMX port to use if [ $JMX_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" fi To the following: Changed security protocol to the correct ones and kafka cli script as follows. # JMX port to use if [ $ISKAFKASERVER = "true" ]; then JMX_REMOTE_PORT=$JMX_PORT else JMX_REMOTE_PORT=$CLIENT_JMX_PORT fi if [ $JMX_REMOTE_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management