Solved

Hello Team, This is Vaibhav Waghmode from Pune, India. it is regarding downloading the cloudera manager CHD5. also getting below mentioned page while downloading.could you please suggest me on this this? 

Hello all,I'm trying to kerberize the Ambari 2.7.3 cluster. However, during the setup, I get the following error:Caused by: org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_142308985016794830cc) while initializing kadmin interface at org.apache.ambari.server.serveraction.kerberos.MITKerberosOperationHandler.invokeKAdmin(MITKerberosOperationHandler.java:323) at org.apache.ambari.server.serveraction.kerberos.MITKerberosOperationHandler.principalExists(MITKerberosOperationHandler.java:123) at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.testAdministratorCredentials(KerberosOperationHandler.java:314) at org.apache.ambari.server.controller.KerberosHelperImpl.validateKDCCredentials(KerberosHelperImpl.java:2133)All of the authentication settings are okay, because I am able to kinit and

Hi ,I have created 2 flows -1] in this flow i have used GenerateFlowFile->UpdateAttribute->InvokeHttpGenerateFlowFile config- Dataformat = textCustomText = Science is magic$$@@ i love Data ## UpdateAttribute config -mime.type = application/x-www-form-urlencodedInvokeHttp config - HTTP method = POSTand this works super fine . But when i use this with gettwitter it fails as shown in below -  1] GetTwitter is working fine - 2] EvaluateJsonPath - 3] AttributeTOJson- 4] UpdateAttribute- 5] ReplaceText -   6] InvokeHTTP -     this is the error i am getting -  but if i set empty text at this from invokehttp processor - then this error goes away but the api responds with missing api_key and text -  Somebody pls help, this is eating my head up . thanks in advance   

Hi everyone! I have a file that contains a Json array of json objects. My first step is to split this flowfile into many flowfiles and ach one of them has a json object using SplitJson Processor. Then I'm trying to set the attributes of json to the flowfile attributes of each flowfile so I can use it after to build a query using Evaluatejsonpath processor. Everything worked fine at the beginning but then I noticed some flowfiles go to failure queue of Evaluatejsonpath.   I checked logs and the error is Invalid Json but when I went to see json files in failure queue the json has no problem and valid. Then I tried to change the processor that I'm using so I used ExecuteScript Processor. I used a script that does the some job I wanted, but still some problem some number of files go the failure queue. Help!         Evaluatejsonpath Processor:         SplitJson Processor:    

Hello,Need some assistance / guidance on how we can reduce Non-HDFS Space. We see Non-HDFS Space of around 270 used, as we are facing space crunch, we would explore possibilities for getting non-hdfs space reduced.I have cleared all Yarn logs for the applications which were killed/ failed etc (our /data mountpoint houses dfs, yarn, kudu, impala), yet this does not solve our issue. Any assistance / guidance is much appreciated. Thanks Amn

Hi all,I just discover NIFI few weeks ago and I tried to replicate a simple curl command with the processor InvokeHTTP :   I tried this curl command in the VM where my NIFI server is installed and it works well, I received all the needed information. So what I did in the invokeHTTP processor is the folowing: in properties1) HTTP Method =GET2) Remote URL = "URL used in the curl command"then I added a new property Authorisation = my_login:token But the processor stay invalid. I know that there is plenty of other properties and settings but I'am a little bit lost, if someone can help me it will be greatThanks  

Hi community, We run Spark 2.3.2 on Hadoop 3.1.1. We use external ORC tables stored on HDFS. We are encountering an issue on a job run under CRON when issuing the command `sql("msck repair table db.some_table")`. The table is partitioned and the issue is the following: 21/03/22 22:44:13 WARN HiveConf: HiveConf of name hive.heapsize does not exist21/03/22 22:44:13 WARN HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist21/03/22 22:44:13 WARN HiveConf: HiveConf of name hive.plan.serialization.format does not existHive Session ID = 2625af79-e021-4b57-9435-e0fea4f0080321/03/22 22:44:13 INFO SessionState: Hive Session ID = 2625af79-e021-4b57-9435-e0fea4f0080321/03/22 22:44:13 ERROR ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.io.IOException: Unable to create directory /tmp/hive/2625af79-e021-4b57-9435-e0fea4f00803_resources;org.apache.spark.sql.AnalysisException: java.lang.R

Hello, How do we balance data which are stored on individual disk on a particular datanode, we have 5 disks on a single node and one of the disk is 90% full, running balancer is not fixing the issue.Would like to get some suggestions/ comments to fix this issue.I was going through this article and it says it is not possible to balance disks within a single node, so what other options can we use to fix this until we upgrade to CDH 6.3(https://community.cloudera.com/t5/Community-Articles/HDFS-Balancer-Balancing-Data-Between-Disks-on-a-DataNode/ta-p/244650) CM & CDH - 5.16.3 Appreciate all inputs. ThanksWert

Hello,We are getting alerts for Block Count on one of our data nodes as it has crossed the threshold of 10000. Since HDFS balancer did not fix the issue, the next thing I turned my focus to see if we are hitting small files issue. I was trying to put up a report via terminal script ( hdfs dfs -ls -R /tmp |grep ^- |awk '{if ($5 < 134217728) print $5, $8;}'| head -5 | column –t) but when I compare the result from the script output vs HDFS Report from Cloudera Manager I see a difference in the size of the same file. Could anyone provide any guidance / assistance on this, or am I doing something wrong. ThanksAmn

Hello, I am seeing this value for numBlocksFailedToUncache property is going up. Should I be concerned? From apache documentation I can see it means https://blog.woopi.org/wordpress/files/hadoop-2.6.0-javadoc/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.html"Number of uncache commands that could not be completed successfully" Not sure how to interpret this.. ThanksM

Hi,  We  recently updated  value of hive.notification.sequence.lock.max.retries from 20 to 40 and started getting multiple SocketTimeOut Failures Caused by: java.io.IOException: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.2.1.11:49971 remote=node011.mapreduce.net/10.2.8.4:45454]at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:685).  .  .Is this related? 

Hi experts,When I start hive LLAP, it seems to be working as I am able to access databases and tables via LLAP.However, the hive LLAP UI "state = completed". Also shows the wrong information such as the App id or does not even show any llap daemons running. I can clearly see that the AM and LLAP daemon is running. {"amInfo" : {"appName" : "llap_service","appType" : "org-apache-slider","appId" : "application_1616963398366_0079","containerId" : "container_e59_1616963398366_0079_01_000001","hostname" : "test04.domain.com","amWebUrl" : "http://test04.domain.com:44972/"},"state" : "RUNNING_ALL","originalConfigurationPath" : "hdfs://test/user/hive/.slider/cluster/llap_service/snapshot","generatedConfigurationPath" : "hdfs://test/user/hive/.slider/cluster/llap_service/generated","desiredInstances" : 1,"liveInstances" : 1,"launchingInstances" : 0,"appStartTime" : 1617064445107,"runningThresholdAchieved" : true,"runningInstances" : [ {"hostname" : "test03.domain.com","containerId" : "conta

Hi All, I have two Cloudera clusters and each cluster has one HBase table .I first write to cluster1 and then get my data replicated to cluster2 with WAL replication.Cluster1(has 3 Region servers) and cluster2(has 80 Region servers).I am using org.apache.hadoop.hbase.spark.HBaseContext.bulkPut to write data to Hbase.My table has two column families and writes/flushes are successful for one column family in source cluster(cluster1) and the other column family is not logging any errors/warnings to Flush/write failures in region server logs.One weird issue here is, my writes are reaching WAL successfully(both Column Family data) and they are replicating successfully in destination cluster(cluster2) and I can see data in both Column families in destinationwhereas in source, I believe the writes are reaching WAL only, somehow it is not being picked up by memstore and hence not being flushed to one of the column families.(It works for one column family and not working for the other

Hi experts,I recently installed hive2.3.8 and trying to enable LLAP.I am getting the below error.beeline> !connect jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2Connecting to jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2Enter username for jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/:Enter password for jdbc:hive2://test01.domain.com:2181,test02.domain.com:2181,test03.domain.com:2181/:21/03/24 20:34:01 [main]: INFO jdbc.HiveConnection: Connected to test02.domain.com:1050021/03/24 20:34:01 [main]: WARN jdbc.HiveConnection: Failed to connect to test02.domain.com:1050021/03/24 20:34:01 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeperError: Could not open client transport for any of the Server URI's in ZooKeeper: Failed

hello i'm using CDH 6.2.0. I have a problem with the Cloudera management service not running.When I turn on the cloudera-scm-agent and cloudera-scm-server, it looks like the picture below.And when I run a service, I get a message saying "Role not started due to unhealthy host". this is my cloudera-scm-agent log.      [23/Mar/2021 17:37:21 +0000] 3013 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-6bd66bc3db6890d3e7e602f52c725199Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 121, in _send self._port) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__ self.conn.connect() File "/usr/lib64/python2.7/httplib.py", line 824, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise errerror: [Errno 111]

hello guys hope you doing well I want to put data into hdfs using putHDFS processor of apache nifi and keep getting me invalid processor here's my configuration conf and here's my pipeline pipeline Hope there's a good tutorial that can help me ? Thank you in advance !!

Hello Everyone, I've recently enabled Kerberos in my cluster and since then one of my datanodes is not able to connect to the namenode: I see these entries in the namenode logs:  2021-02-03 12:06:15,699 INFO ipc.Server (Server.java:saslProcess(1573)) - Auth successful for $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)2021-02-03 12:06:15,700 INFO ipc.Server (Server.java:authorizeConnection(2235)) - Connection from <datanode-ip>:42328 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/<datanode-hostname>@<Realm-Name> (auth:PROXY) via $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)2021-02-03 12:06:15,700 INFO ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client <datanode-ip> threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: $4E8100-MH1MCLUV65LO@<Realm-Name> is not allowed to impersonate dn/<da

HI,  I am running a select distinct count on a column, am getting the below error. can you please help me.select distinct count (id) from hive_table_tbs; error messageError while compiling statement: FAILED: SemanticException [Error 10128]: line 1:16 Not yet supported place for UDAF 'count' Thank You

Hi,I am wondering what are the pros and cons to enabling or disabling "Run as end user instead of Hive user"?Currently we are trying to implement proper permissions for our hadoop cluster to prevent any accidents etc.My understanding is if a user logs into hiveserver2, and if they submit a query, and it runs as the hive user, then technically they should have access to all databases and tables etc, which can cause damage if not careful. Since there is this setting, I would like to better understand the use case scenarios or pros and cons etc. Thanks,

Hi,I was just wondering what is the best practice on where to store hive external tables in HDFS?Currently they are being stored in the /tmp directory in HDFS but sometimes someone accidentally deletes the files under /tmp and right now I am trying to add permissions to ranger to prevent this from happening. Thanks,

First don't laugh or tell me to upgrade, I inherited this and we are unable to upgrade it before it gets decommissioned in the next couple years. Hadoop Version: 0.20.2, r911707We just moved from one datacenter to another by creating a new hadoop cluster (from old servers) and moving data. The data is transient and only stays in hadoop for 3 months. The old cluster hovered around 500TB for a long time. The new cluster has grown up to now 940TB. I have double checked the data removal script and it is working. The data is no longer available and the logs "say" the data gets deleted but we are steadily growing about 2TB/day whis is about how much new data we get per day. looking at the logs on a datanode the actual OS file on the datanode that says gets deleted does in fact get deleted. looking at the OS filesystems they are almost full where on the old cluster they were about 50% used. Hopefully that covers the background. The only thing I can find is when I run "hadoop dfsadmi

Hello Everyone, We have recently deployed CDP. I need to fetch data from one of my other cluster (HDFS HA-Enabled) to this cluster using distcp. For configuring the remote nameservice in my CDP cluster, I followed the steps given in this link and added the configuration properties to "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" section of HDFS configuration in Cloudera Manager. After this, I restarted the services and tried to run the following command from one of my hosts in the cluster: hadoop fs -ls hdfs://<remote-nameservice-name>/ But it throws this error:   21/03/22 13:59:34 WARN fs.FileSystem: Failed to initialize fileystem hdfs://<remote-nameservice-name>/: java.lang.IllegalArgumentException: java.net.UnknownHostException: <remote-nameservice-name>-ls: java.net.UnknownHostException: <remote-nameservice-name>Usage: hadoop fs [generic options] [-appendToFile <localsrc> .

Hi I perform many queries with the below statement and I thought that instead of writing over and over again the same cases if it is possible to create a UDF in impala to handle this like sql server.   (select people, sum(new_money) as new_money from ( select people, case when pocket_01 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_02 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_03 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_04 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_05 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_06 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end + case when pocket_07 = 'leather' then sum(nvl(money,0.0))/100 else 0.0 end ) as new_money from bank where date = '20201010' ) target group by people ) as target  Is this possi

Hi, Is public repo is no longer available, Is there alternative way to connect and get packages ?[root@nm-01 mchitrala]# yum repolistLoaded plugins: fastestmirror, langpacks, priorities, psDetermining fastest mirrors * base: mirror.atlanticmetro.net * centosplus: mirror.math.princeton.edu * epel: mirror.math.princeton.edu * extras: mirror.ash.fastserv.com * updates: repos-va.psychz.nethttp://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] HTTP Error 403 - ForbiddenTrying other mirror.To address this issue please refer to the below wiki article https://wiki.centos.org/yum-errors If above article doesn't help to resolve this issue please use https://bugs.centos.org/. http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] HTTP Error 403 - ForbiddenTrying other mirror.base/7/x86_64                   

 %pysparkjava.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)at java.net.Socket.connect(Socket.java:607)at java.net.Socket.connect(Socket.java:556)at sun.net.NetworkClient.doConnect(NetworkClient.java:180)at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)at sun.net.www.http.HttpClient.New(HttpClient.java:339)at sun.net.www.http.HttpClient.New(HttpClient.java:357)at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)at sun.net.www.protocol.htt