Q&A

Hello,We are getting alerts for Block Count on one of our data nodes as it has crossed the threshold of 10000. Since HDFS balancer did not fix the issue, the next thing I turned my focus to see if we are hitting small files issue. I was trying to put up a report via terminal script ( hdfs dfs -ls -R /tmp |grep ^- |awk '{if ($5 < 134217728) print $5, $8;}'| head -5 | column –t) but when I compare the result from the script output vs HDFS Report from Cloudera Manager I see a difference in the size of the same file. Could anyone provide any guidance / assistance on this, or am I doing something wrong. ThanksAmn

Hello, I am seeing this value for numBlocksFailedToUncache property is going up. Should I be concerned? From apache documentation I can see it means https://blog.woopi.org/wordpress/files/hadoop-2.6.0-javadoc/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.html"Number of uncache commands that could not be completed successfully" Not sure how to interpret this.. ThanksM

Hi,  We  recently updated  value of hive.notification.sequence.lock.max.retries from 20 to 40 and started getting multiple SocketTimeOut Failures Caused by: java.io.IOException: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.2.1.11:49971 remote=node011.mapreduce.net/10.2.8.4:45454]at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:685).  .  .Is this related? 

Hi experts,When I start hive LLAP, it seems to be working as I am able to access databases and tables via LLAP.However, the hive LLAP UI "state = completed". Also shows the wrong information such as the App id or does not even show any llap daemons running. I can clearly see that the AM and LLAP daemon is running. {"amInfo" : {"appName" : "llap_service","appType" : "org-apache-slider","appId" : "application_1616963398366_0079","containerId" : "container_e59_1616963398366_0079_01_000001","hostname" : "test04.domain.com","amWebUrl" : "http://test04.domain.com:44972/"},"state" : "RUNNING_ALL","originalConfigurationPath" : "hdfs://test/user/hive/.slider/cluster/llap_service/snapshot","generatedConfigurationPath" : "hdfs://test/user/hive/.slider/cluster/llap_service/generated","desiredInstances" : 1,"liveInstances" : 1,"launchingInstances" : 0,"appStartTime" : 1617064445107,"runningThresholdAchieved" : true,"runningInstances" : [ {"hostname" : "test03.domain.com","containerId" : "conta

Hi All, I have two Cloudera clusters and each cluster has one HBase table .I first write to cluster1 and then get my data replicated to cluster2 with WAL replication.Cluster1(has 3 Region servers) and cluster2(has 80 Region servers).I am using org.apache.hadoop.hbase.spark.HBaseContext.bulkPut to write data to Hbase.My table has two column families and writes/flushes are successful for one column family in source cluster(cluster1) and the other column family is not logging any errors/warnings to Flush/write failures in region server logs.One weird issue here is, my writes are reaching WAL successfully(both Column Family data) and they are replicating successfully in destination cluster(cluster2) and I can see data in both Column families in destinationwhereas in source, I believe the writes are reaching WAL only, somehow it is not being picked up by memstore and hence not being flushed to one of the column families.(It works for one column family and not working for the other

Hi experts,I recently installed hive2.3.8 and trying to enable LLAP.I am getting the below error.beeline> !connect jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2Connecting to jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2Enter username for jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/:Enter password for jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/:21/03/24 20:34:01 [main]: INFO jdbc.HiveConnection: Connected to test02.domain.com:1050021/03/24 20:34:01 [main]: WARN jdbc.HiveConnection: Failed to connect to test02.domain.com:1050021/03/24 20:34:01 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeperError: Could not open client transport for any of the Server URI's in ZooKeeper: Failed

hello i'm using CDH 6.2.0. I have a problem with the Cloudera management service not running.When I turn on the cloudera-scm-agent and cloudera-scm-server, it looks like the picture below.And when I run a service, I get a message saying "Role not started due to unhealthy host". this is my cloudera-scm-agent log.      [23/Mar/2021 17:37:21 +0000] 3013 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-6bd66bc3db6890d3e7e602f52c725199Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 121, in _send self._port) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__ self.conn.connect() File "/usr/lib64/python2.7/httplib.py", line 824, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise errerror: [Errno 111]

hello guys hope you doing well I want to put data into hdfs using putHDFS processor of apache nifi and keep getting me invalid processor here's my configuration conf and here's my pipeline pipeline Hope there's a good tutorial that can help me ? Thank you in advance !!

Hello Everyone, I've recently enabled Kerberos in my cluster and since then one of my datanodes is not able to connect to the namenode: I see these entries in the namenode logs:  2021-02-03 12:06:15,699 INFO ipc.Server (Server.java:saslProcess(1573)) - Auth successful for $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)2021-02-03 12:06:15,700 INFO ipc.Server (Server.java:authorizeConnection(2235)) - Connection from <datanode-ip>:42328 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/<datanode-hostname>@<Realm-Name> (auth:PROXY) via $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)2021-02-03 12:06:15,700 INFO ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client <datanode-ip> threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: $4E8100-MH1MCLUV65LO@<Realm-Name> is not allowed to impersonate dn/<da

HI,  I am running a select distinct count on a column, am getting the below error. can you please help me.select distinct count (id) from hive_table_tbs; error messageError while compiling statement: FAILED: SemanticException [Error 10128]: line 1:16 Not yet supported place for UDAF 'count' Thank You

Hi,I am wondering what are the pros and cons to enabling or disabling "Run as end user instead of Hive user"?Currently we are trying to implement proper permissions for our hadoop cluster to prevent any accidents etc.My understanding is if a user logs into hiveserver2, and if they submit a query, and it runs as the hive user, then technically they should have access to all databases and tables etc, which can cause damage if not careful. Since there is this setting, I would like to better understand the use case scenarios or pros and cons etc. Thanks,

Hi,I was just wondering what is the best practice on where to store hive external tables in HDFS?Currently they are being stored in the /tmp directory in HDFS but sometimes someone accidentally deletes the files under /tmp and right now I am trying to add permissions to ranger to prevent this from happening. Thanks,

First don't laugh or tell me to upgrade, I inherited this and we are unable to upgrade it before it gets decommissioned in the next couple years. Hadoop Version: 0.20.2, r911707We just moved from one datacenter to another by creating a new hadoop cluster (from old servers) and moving data. The data is transient and only stays in hadoop for 3 months. The old cluster hovered around 500TB for a long time. The new cluster has grown up to now 940TB. I have double checked the data removal script and it is working. The data is no longer available and the logs "say" the data gets deleted but we are steadily growing about 2TB/day whis is about how much new data we get per day. looking at the logs on a datanode the actual OS file on the datanode that says gets deleted does in fact get deleted. looking at the OS filesystems they are almost full where on the old cluster they were about 50% used. Hopefully that covers the background. The only thing I can find is when I run "hadoop dfsadmi

Hello Everyone, We have recently deployed CDP. I need to fetch data from one of my other cluster (HDFS HA-Enabled) to this cluster using distcp. For configuring the remote nameservice in my CDP cluster, I followed the steps given in this link and added the configuration properties to "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" section of HDFS configuration in Cloudera Manager. After this, I restarted the services and tried to run the following command from one of my hosts in the cluster: hadoop fs -ls hdfs://<remote-nameservice-name>/ But it throws this error:   21/03/22 13:59:34 WARN fs.FileSystem: Failed to initialize fileystem hdfs://<remote-nameservice-name>/: java.lang.IllegalArgumentException: java.net.UnknownHostException: <remote-nameservice-name>-ls: java.net.UnknownHostException: <remote-nameservice-name>Usage: hadoop fs [generic options] [-appendToFile <localsrc> .

Hi I perform many queries with the below statement and I thought that instead of writing over and over again the same cases if it is possible to create a UDF in impala to handle this like sql server.   (select people, sum(new_money) as new_money from ( select people, case when pocket_01 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_02 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_03 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_04 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_05 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_06 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_07 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end ) as new_money from bank where date = '20201010' ) target group by people ) as target  Is this possi

Hi, Is public repo is no longer available, Is there alternative way to connect and get packages ?[root@nm-01 mchitrala]# yum repolistLoaded plugins: fastestmirror, langpacks, priorities, psDetermining fastest mirrors * base: mirror.atlanticmetro.net * centosplus: mirror.math.princeton.edu * epel: mirror.math.princeton.edu * extras: mirror.ash.fastserv.com * updates: repos-va.psychz.nethttp://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] HTTP Error 403 - ForbiddenTrying other mirror.To address this issue please refer to the below wiki article https://wiki.centos.org/yum-errors If above article doesn't help to resolve this issue please use https://bugs.centos.org/. http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] HTTP Error 403 - ForbiddenTrying other mirror.base/7/x86_64                   

 %pysparkjava.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)at java.net.Socket.connect(Socket.java:607)at java.net.Socket.connect(Socket.java:556)at sun.net.NetworkClient.doConnect(NetworkClient.java:180)at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)at sun.net.www.http.HttpClient.New(HttpClient.java:339)at sun.net.www.http.HttpClient.New(HttpClient.java:357)at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)at sun.net.www.protocol.htt

Hi experts,I am trying to upgrade my current hive version from 1.2 to 2.3.8.I am able to connect to hive via hiveserver2 but when I do "show databases", there are no databases showing up. 0: jdbc:hive2://test02.domain.com:100> show databases;+----------------+| database_name |+----------------++----------------+No rows selected (0.124 seconds)0: jdbc:hive2://test02.domain.com:100> When accessing the mysql database I can see the databases in hive.mysql> use hive;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedmysql> select * from DBS;+-------+-----------------------+-------------------------------------------+---------+------------+------------+| DB_ID | DESC | DB_LOCATION_URI | NAME | OWNER_NAME | OWNER_TYPE |+-------+-----------------------+-------------------------------------------+---------+------------+------------+| 1 | Default Hive database | hdfs://test/apps/hiv

Hello,I need to increase the query timeout (backend_client_rpc_timeout_ms) from current 5 minutes to 30 minutes. Could anyone guide me from where in Impala Configurations I can do this change. I did some checking but cannot find any related, appreciate any assistance / guidance. Thanks Amn

CDSW version 1.7I need to add a file to every image that cdsw builds.jupter_notebook_config.py will need to be placed in the in the .jupyter directory of every project.This is because our organization needs the cdsw sessions to be culled after inactivity and jupyter notebooks prevents the built in idle timeout from functioning. 

There is a Regionserver that is not in a dead state, but it doen not have any regions and has not received any Request.This situation occurred after a memstore ERROR, and then the status of RS on the HBase Master UI was always 0. Then this RS does not show any ERROR Log anymore.Also tried to delete this RS and reinstall it, but it still shows 0 on the HBase Master UI.And still No ERROR Log... memstore error: HBase Master UI: Hadoop Release: HDP 3.1.0

We have CDH 6.1.1 cluster and Kafka cluster. I want to make possible to select data from Kafka topics using Hive. Is there any way to do it? Where I can find Kafka storage handler version for Hive 2.1.1 ?

This is just a knowledge sharing article. I had faced this issue in production and took me a day to resolve it.The workaround that I'm sharing will help you in getting your table back online in "Enabled" without deleting Zookeeper Hbase table znode or any data as a matter of fact..Here are steps to resolve it.1. Run a "get" command against hbase:meta for the affected tablehbase(main):003:0> get 'hbase:meta', '<AFFECTED_TABLE_NAME>', 'table:state' COLUMN                                   CELL  table:state                             timestamp=1551456805377, value=\x08\x022. Notice the above "value". Its pointing to \x08\x02 which is wrong. The value should either be \x08\x00 (Enabled) or \x08\x01 (Disabled)3. Edit the value manually.hbase(main):003:0> put 'hbase:meta','<AFFECTED_TABLE_NAME>','table:state',"\b\0"

I heard that Apache Flume has been removed with the update from CDH to CDP. With the update to CDP, who will do Apache Flume's functions?Help me please....

For instance: http://localhost:18080/nifi-registry/#/explorer/grid-listIs there a way to remove the `#` and still have the url work?