About RajeshBodolla

RajeshBodolla · ‎01-14-2020

Hi, I have a cluster with datanodes across two racks, i'm removing one complete rack for OS upgrade. When i tried to decommission one complete rack, the process was stuck. However when i decommissioned all but one datanode in that rack(for testing), the process started. How will the data be replicated to the only existing rack if we take out the complete rack?

RajeshBodolla · ‎12-17-2019

@pdev have you disabled TLS in the management services as well? After you disable/enable TLS, you will have restart the management services so they communicate with CM with the changed port and protocol.

RajeshBodolla · ‎12-16-2019

Hi, We are planning to install Navigator encrypt in our cluster to encrypt kafka data and as part of this we would like to know if it is mandatory to setup TLS? We would also like to know the performance impact navigator encrypt has on kafka, does navigator encrypt have any logs to look at in case of any issue and can it be managed by cloudera manager. Thanks, Rajesh

RajeshBodolla · ‎07-28-2019

Hi, I have two clusters in different DC's with same domain name and same realm but the KDC's are different. How can i enable trust between these two KDC's ??

RajeshBodolla · ‎07-03-2019

@satz We were able to resolve this. We had the kerberos auth principles in default kafka group while all the broker were in a different config group. Adding the auth principles to the kafka config group has solved the issue.

RajeshBodolla · ‎06-09-2019

Hi, We have just built a new kafka cluster( 3.1.0-1.3.1.0.p0.35 ) and integrated it with kerberos. Kerberos is integrated with AD. We are able to produce and consume with kafka principle which is local but with AD users, it fails with below error on console. 19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to CLIENT_COMPLETE 19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to COMPLETE 19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Initiating API versions fetch from node -1. 19/06/09 08:14:22 DEBUG network.Selector: [Producer clientId=console-producer] Connection with server2.kafka4.corp/180.20.92.23 disconnected java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196) at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:545) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483) at org.apache.kafka.common.network.Selector.poll(Selector.java:412) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:481) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at java.lang.Thread.run(Thread.java:748) 19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Node -1 disconnected. 19/06/09 08:14:22 INFO clients.NetworkClient: [Producer clientId=console-producer] API versions request failed via disconnect. Defaulting legacy API versions 19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Give up sending metadata request since no node is available 19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Give up sending metadata request since no node is available ^C19/06/09 08:14:22 INFO producer.KafkaProducer: [Producer clientId=console-producer] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 19/06/09 08:14:22 DEBUG internals.Sender: [Producer clientId=console-producer] Beginning shutdown of Kafka producer I/O thread, sending remaining records. ========================================== The broker logs only have errors about the shortname for users. Caused by: org.apache.kafka.common.security.kerberos.NoMatchingRule: No rules apply to RAJESH@KAFKA4.CORP, rules [DEFAULT]

RajeshBodolla · ‎05-29-2019

If you install cloudera manager packages and start the agent, it will not distribute the parcels. The parcels will be distributed only when you add the host to the cluster.

RajeshBodolla · ‎05-29-2019

Clouder amanager would automatically distribute the parcels that the cluster requires.

RajeshBodolla · ‎05-29-2019

@Fawze You can mention multiple parcel links in cloudera manager so it should not be an issue.

RajeshBodolla · ‎04-05-2019

@Harsh J We did try NFS didn't get the write performance we were looking at and more over that required an NFS server as well.

RajeshBodolla · ‎04-02-2019

Hi , I'm using CDH 5.14 and have FUSE integrated. I'm having issue with the mounted HDFS sometimes( disconnecting and connecting ) and since FUSE doesn't have any log, its difficult to identify what is happening. Is it possible to start FUSE with logging? I do not see any logs in /var/log/messages regarding this.

RajeshBodolla · ‎04-01-2019

If we run a simple select on an hbase table limiting the result to 1 row using phoenix, how many read request does the jmx metrics display and how is it calculated. And what is the difference between scan size and scan number?

RajeshBodolla · ‎11-28-2018

Hi @manuroman I have read the page but couldn't validate myself on how many nodes are actually under billing. The License page in CM shows below and makes sense as we have 10 gateways and they are excluded in the total below. Is the billing calculated based on how many nodes have the core product or based on how many nodes are under navogator? Total Nodes : 101

RajeshBodolla · ‎11-28-2018

Hi, How do we know how many nodes in the cluster are used for licensing? Like edge nodes are not used for billing etc.,. We have clusters in CDH 5.5.4 and CDH 5.14.2.

RajeshBodolla · ‎11-27-2018

CFile is an on-disk columnar storage format which holds data and associated B-Tree indexes. https://github.com/cloudera/kudu/blob/master/docs/design-docs/cfile.md

RajeshBodolla · ‎11-27-2018

Hi , I it possible to set idle_query_timeout in impala shell in CDH 5.14.2? The documentation in below link tells that "You can override the --idle_session_timeout value with the IDLE_SESSION_TIMEOUT Query Option (CDH 5.15 / Impala 2.12 or higher only) at the session level". Do we have any other option in CDH 5.14.2? https://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html

RajeshBodolla · ‎10-31-2018

This is because navigator upgrade generally takes lot of time based on the number of objects and relations you have. Increasing the navigator heap size can help. Calculation of the required heap is available in cloudera site.

RajeshBodolla · ‎10-27-2018

Hi, We often see NOT_LEADER_FOR_PARTITION in kafka. On what basis does kafka leader election happen ? What are all the reasons that cause kafka partition leader election to trigger?

RajeshBodolla · ‎09-24-2018

Hi, We are using CDK 3.1.0 (1.3.1.0) and have recently observed that the kafka active controller is changing very frequently. Any specific reasons that can be triggered? Kafka clients are receiving the old active controller for some time ~30min even after the new active controller is elected. There are not specific erors in zookeeper.

RajeshBodolla · ‎09-12-2018

@Harsh J Thanks for the detailed answer. The cluster we have is having very less data and all the datanodes haven't utilized atleast 10% of storage yet but is expected to use atleast 60% in a week. Would running HDFS balancer and compaction make any difference in this case!

RajeshBodolla · ‎09-11-2018

Hi, What is the procedure to follow when addin additional region server to an existing cluster? We use CDH 5.14

RajeshBodolla · ‎08-31-2018

Hi, We have recently upgraded from 5.11.2 to 5.14.2. The upgrade was untill we identified that many databases were not visible/loaded in navigator. The logs have below. Any related pointers are welcome. 2018-08-31 10:12:17,344 ERROR com.cloudera.nav.hive.extractor.AbstractHiveExtractor [CDHExecutor-0-CDHUrlClassLoader@7fbf5fdd]: Failed to extract database dummy_database with error: java.net.SocketException: Broken pipe (Write failed) org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed) at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147) at org.apache.thrift.transport.TTransport.write(TTransport.java:107) at org.apache.thrift.transport.TSaslTransport.writeLength(TSaslTransport.java:391) at org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:499) at org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.TFilterTransport.flush(TFilterTransport.java:77) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_database(ThriftHiveMetastore.java:664) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:656) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) at com.cloudera.nav.hive.extractor.AbstractHiveExtractor.extractDatabase(AbstractHiveExtractor.java:147) at com.cloudera.nav.hive.extractor.AbstractHiveExtractor.extractDatabases(AbstractHiveExtractor.java:133) at com.cloudera.nav.hive.extractor.HiveExtractor.run(HiveExtractor.java:63) at com.cloudera.nav.hive.extractor.AbstractHiveExtractor.run(AbstractHiveExtractor.java:118) at com.cloudera.nav.hive.extractor.HiveExtractorShim.run(HiveExtractorShim.java:35) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:221) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:211) at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:236) at com.cloudera.cmf.cdhclient.CdhExecutor$SecurityWrapper$1.run(CdhExecutor.java:189) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:44) at com.cloudera.cmf.cdhclient.CdhExecutor$SecurityWrapper.doWork(CdhExecutor.java:186) at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) Thanks

RajeshBodolla · ‎08-30-2018

Hi, Try only below in your consumer properties file. security.protocol=SASL_PLAINTEXT sasl.mechanism=GSSAPI sasl.kerberos.service.name=kafka group.id=testgroup

RajeshBodolla · ‎07-04-2018

Hi @bgooley We have users in multiple domins that want access to navigator UI but the configurations allow only one AD domain and there is no option in Navigator to add individual users as well if we have to add local users instead of AD users. The option in cloudera manager provides navigator admin access to local users while we require read-only in navigator.

RajeshBodolla · ‎07-03-2018

Hi, Yes it was infact the java version issue and compatability between CM5.5 and Centos 7.4. Had to tweek client_configs.py to make it work.

RajeshBodolla · ‎07-03-2018

Hi, We are using CM 5.14 and CDH 5.14. We have configured Hue to use multiple AD domains but we do not have any info on how it can be done on Cloudera navigator. Any pointers are appreciated. Thanks

RajeshBodolla · ‎06-28-2018

Hi, Our cluster is in RHEL 6 and i'm trying to add a new gateway that is on CentOS7. I have manually inatalled the agents but starting the agent shows below error. And adding the host to cluster gets stuck in activating the parcel. [28/Jun/2018 18:49:04 +0000] 25958 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache [28/Jun/2018 18:49:04 +0000] 25958 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache [28/Jun/2018 18:49:04 +0000] 25958 MainThread agent ERROR Caught unexpected exception in main loop. Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 578, in start self._init_after_first_heartbeat_response(heartbeat_response["data"]) File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 700, in _init_after_first_heartbeat_response self.client_configs.load() File "/usr/lib64/cmf/agent/src/cmf/client_configs.py", line 682, in load new_deployed.update(self._lookup_alternatives(fname)) File "/usr/lib64/cmf/agent/src/cmf/client_configs.py", line 432, in _lookup_alternatives return self._parse_alternatives(alt_name, out) File "/usr/lib64/cmf/agent/src/cmf/client_configs.py", line 444, in _parse_alternatives path, _, _, priority_str = line.rstrip().split(" ") ValueError: too many values to unpack ^C

RajeshBodolla · ‎06-25-2018

Hi, What is the use of .sanpshot files in the data.log directory of a kafka topic and how do I make use of it? Is this present in apache kafka as well or available only on cloudera?

RajeshBodolla · ‎06-25-2018

Hi, Do we have csd for spark 1.6 supporting on CDH5.5.X? Any pointer are appreciated.

RajeshBodolla · ‎06-21-2018

Hi, We have a CDH cluster in 5.14.2. Is it compatable with KTS 3.8.x ? Due to some limitations, we are unable to use KTS 5.14.0 in this cluster. The cloudera documentation shows KTS compatability with KMS and CM but nor CDH.

Online	Offline
Last Visited	‎01-30-2020 08:50 AM

Date Registered	‎10-04-2017 08:26 AM
Last Visited	‎01-30-2020 08:50 AM
Total Messages Posted	108
Total Kudos Received	9

Re: Kafka console producer/consumer failing with A...

Re: Navigator not loading all hive databases after...

Re: Zookeeper issue with java.net.SocketTimeoutExc...

Re: Adding a master server in kudu

Re: What should we backup for kudu??

Block distribution during decommissioning

Re: After disabling TLS/SSL ,cloudera management s...

Navigator encrypt for Kafka

How to enable trust between two KDC's with same re...

Re: Kafka console producer/consumer failing with A...

Kafka console producer/consumer failing with AD us...

Re: Nodes with different OS Versions

Re: Nodes with different OS Versions

Re: Nodes with different OS Versions

Re: Logs for hadoop-fuse-dfs

Logs for hadoop-fuse-dfs

How is hbase read request count calculated

Re: Number of licensed/billable nodes?

Number of licensed nodes?

Re: Data replication in Kudu

idle_query_timeout in impala shell CDH 5.14.2

Re: Navigator not loading all hive databases after...

Kafka partition leader election

Kafka ACTIVE CONTROLLER changing frequently

Re: HBase compaction while adding region servers

HBase compaction while adding region servers

Navigator not loading all hive databases after upg...

Re: Console producer/consumer not working in kafka...

Re: How to configure cloudera navigator to access ...

Re: Adding new host to cluster fails

How to configure cloudera navigator to access mult...

Adding new host to cluster fails

.sanpshot file in kafka log directory

spark 1.6 csd for CDH5.5

KTS and CDH Compatability