Ask the Community

I have run manually minor and major compaction. Table data is referring to the new delta and base folder but the old delta and base folders are not removing and it consumes the hive storage. Moreover, No running job is referring to this table.Queries:  create table compaction_check.check_file(name string, age int);insert into compaction_check.check_file values('user 1', 10),('user 2', 20),('user 3', 30);insert into compaction_check.check_file values('user 4', 10),('user 5', 30),('user 6', 15);ALTER TABLE compaction_check.check_file COMPACT 'major';  Check input files: select distinct INPUT__FILE__NAME FROM compaction_check.check_file; output:  Actual table directory : Why stale the base and delta folders are not removed?  

Hello,I am getting following error when trying to start Hive server = 2021-05-20 12:21:50,088 - Will retry 7 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)2021-05-20 12:22:00,098 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}2021-05-20 12:22:00,780 - call returned (1, '')2021-05-20 12:22:00,781 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)2021-05-20 12:22:10,791 - Process with pid 12962 is not running. Stale pid file at /var/run/hive/hive-server.pid  I referred to similar threads at this forum but didn't help.Pls suggest how to fix this issue thank you.Regards,Amey.

Hello, I would like to download the CDH sources to securize my deployed clusters (one in 5.4.3 and the other in 5.12.1) but when I try to download with wget and the detail found in pagehttps://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download_54.html#concept_j4t_2x3_1torhttps://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download_512.html#cdh_vd_cdh_download_512 I try with my account informations but when I use them directly I obtain a n error "Wrong port number" and I try to encode the password for url and this time I obtain.Does anyone could help me please --2021-06-07 16:18:12-- https://*login*:*password*@archive.cloudera.com/p/cdh5/ubuntu/trusty/amd64/cdh/Résolution de archive.cloudera.com (archive.cloudera.com)… 151.101.120.167Connexion à archive.cloudera.com (archive.cloudera.com)|151.101.120.167|:443… connecté.requête HTTP transmise, en attente de la réponse… 401 Authentication requiredAuthentification sé

I get two flow files and I need to look at the value of data in flow files based on two different conditions in RouteOnAttribute. If both the conditions match, then only the flow should be routed to executeSQL processor, else it should flow down to MonitorActivity Processor. I tried multiple ways, but this is not working. Every time, when one of the conditions match, it gets routed to executeSQL Processor.  

Hi,  I got a problem while sending a Zero Byte flow file through Nifi PutSFTP processor. Wondering if there is some configuration I have to set? thanks. Use case: get a file from a directory (GetFile Processor) send the file through SFTP (PutSFTP Processor)if success, change the filename to new filename (UpdateAttribute Processor)Remove the all of the content (ReplaceText Processor). Zero Byte flow file send the new filename through SFTP (PutSFTP Processor) I gotten this error while sending Zero Byte flow file over using PutSFTP ProcessorPutSFTP[id=1a091db4-1f9e-14cd-5206-1514485febc8] Rejecting StandardFlowFileRecord[uuid=247457a8-6608-4271-a626-3c016a1bbe7c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1618805512561-40, container=default, section=40], offset=15043, length=0],offset=0,name=AS_MPS_20200927_010010.xml.done,size=0] because it is zero bytes It will be great if someone could help me.  Thanks in adv

Hello,   I am trying to execute 1 .hql script from Edge node. [hdp-slave4.mydomain.com] hdp-slave4 machine is not running any server roles i.e. zookeeper or hive. Only having zookeeper & hive clients.HDP version: 3.1.4 Command =  hive -f ./dumping_test_script.hql I am getting following error =INFO : Moving data to local directory /home/user1/user1_Run_Jar/AB_HIVE_1805 2021/neo4j-graph-preparation/data-files/user_stats from hdfs://hdp-master.mydomain .com:8020/tmp/hive/hive/df588c46-4e05-4a5c-8fde-5b7aaf9329a6/hive_2021-06- 01_13-55-09_157_7991483949013764396-29555/-mr-10000ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.ex ec.MoveTask. Unable to move source hdfs://hdp-master.mydomain.com:8020/tmp/hive/hive/df588c46-4e05-4a5c-8fde-5b7aaf9329a6/hive_2021-06-01_13-55-09_157_79914 83949013764396-29555/-mr-10000 to destination /home/user1/user1_Run_Jar/AB_HIVE_18052021/neo4j-graph-preparation/data-files/user_statsDEBUG : Shutting down query INSERT

Hello,We have a 9 node HDP 3.1.5 cluster with erasure coding enabled. The hdfs fsck command returns under replicated blocks though the filesystem under path '/' is HEALTHY. Please advise how to reconstruct the replication for the under replicated blocks. Please find the hdfs fsck report below: [hdfs@hostname (prod) ~] $ hdfs fsck /Connecting to namenode via http://namenode:50070/fsck?ugi=hdfs&path=%2FFSCK started by hdfs (auth:SIMPLE) from /10.19.224.62 for path / at Wed May 26 22:48:22 PDT 2021/app/warehouse/db/data/eventdate=20210516/00000-21028-5a7dd08a-8c9e-4f62-b920-b6265292da8c-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850161008_890690. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00047-789-396cdf80-9b0b-4411-a6e2-a1094cfb223e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438512_87

Hello, I am taking a big data class and am new to this whole field, so I apologize for any incorrect use of terminology. I am trying to use sandbox. I was able to create a virtual machine, login to port 4200 with the username "root" and change the admin password using "ambari-admin-password-reset". The password works when I log into port 8080. But when I try to go back to port 4200 and login as "admin", instead of "root", the password doesn't work. Is there a different password? What am I missing? Any suggestions would be great. Thanks

I'm trying the processor ReplaceText to change all my dot to comma, but i can't I really need some help here, please. This is my CSV file: MONEY (header)109.29 I need this result MONEY (header)109,28

Is there a processor that can detect hidden characters in NiFi? In our test environment I am using an API invoke to deliver a JSON payload, for added security we want to filter out any hidden and potentially malicious text. I have been trying to use RouteText, RouteOnContent and ExtractText processor to only allow alphanumeric and punctuation characters through but can't seem to get the regex to work when looking for 'uncommon' text or characters? any help would be appreciated. Andy 

Hello,  I have followed this guide:https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_ag_change_hostnames.html In order to change the hostname on my Cloudera single node CMS system. However, now none of my services will start. Everything is set to localhost (because its all on this one host). I notice that the old hostname is still referenced for the server... Even though Hive, Hue, YARN, etc are all set to "localhost" for their databases my ZooKeeper server keeps thinking it's still the previous hostname.How do I get rid of the hostname that is stuck in Cloudera Manager / ZooKeeper? The only other thing I can think to do is remove the "Server Name override" for the ZooKeeper myid and then restart the service to see if it picks up the new hostname.

Hi, According to the official Nifi documentation, the state allows Nifi processors to "resume from the place where it left off after NiFi is restarted. Additionally, it allows for a Processor to store some piece of information so that the Processor can access that information from all of the different nodes in the cluster". If my understanding is good, when we configure a zookeeper Provider, the state will not be persisted locally, instead, the data will be sent to zookeeper. I've explored the zookeeper znodes and could not find any data related to the state, all I can find are the informations about the Coordinator and Primary nodes. However, the local state directory is still filled. The configuration is very simple, I've 3 external ZK nodes and 3 Nifi instances. Here is an exerpt of the nifi.properties file:nifi.cluster.is.node=truenifi.zookeeper.connect.string=zk-node1:2181,zk-node2:2181,zk-node3:2181nifi.state.management.embedded.zookeeper.start=falsenifi.

How Can I get access (login/password) to old HDP repository ? I need 3.1.5. from https://archive.cloudera.com

File 'hdfs://xxx:8020/data/hive/warehouse/xxxx.db/test_bak/8743abe2968ecbe7-3f9c13700000000_2053286657_data.0.parq' has an incompatible Parquet schema for column 'xxxx.test_bak.ordered_revenue'. Column type: STRING, Parquet schema: optional double ordered_revenue [i:20 d:1 r:0] I made this change in Hue.

Hi, One of our customer willing to migrate to CDP 7.x, however customer wants to use code base of Flume developed in earlier version of CDP.Customer is not willing for NiFi migration and want to check if we can copy the open source Flume binaries to CDP 7.x and setup the runtime of Flume. Can this be achievable? Any suggestion are appreciated. ThanksVenkata Madhu 

Hi experts,Many of the devs are getting the following error when trying to connect with squirrel:Error: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed outSQLState:  08S01ErrorCode: 0 I believe the issue may be because there are too many concurrent hive connections. Can someone help suggest how should i configure this to allow more hive connections?Also I find it weird that I can connect through beeline using the edge node of that cluster but Squirrel is giving an error. Thanks,Any help is much appreciated.

Hi Community I am trying to copy HDFS from an HDP 2.6.x cluster (kerberized) to a CDP Private Cloud Base 7.1.5 cluster (not kerberized) and using other ports as well, it gives me an error So I write the command in consolehadoop distcp hdfs://svr2.localdomain:1019/tmp/distcp_test.txt hdfs://svr1.local:9866/tmp/ what could be the origin of the fault?  Thank you ***********ERROR*************************Java config name: nullNative config name: /etc/krb5.confLoaded from native config>>>KinitOptions cache name is /tmp/krb5cc_11259>>>DEBUG <CCacheInputStream> client principal is useradm/admin@LOCALDOMAIN>>>DEBUG <CCacheInputStream> server principal is krbtgt/LOCALDOMAIN@LOCALDOMAIN>>>DEBUG <CCacheInputStream> key type: 18>>>DEBUG <CCacheInputStream> auth time: Sun May 09 18:39:04 VET 2021>>>DEBUG <CCacheInputStream> start time: Sun May 09 18:39:04 VET 2021>>>DEBUG <CC

Hello, I'd like to implement a lock mechanism in a data flow that prevents a flow file from progressing.  Similar in a way to  the wait processor, holding a flowfile in a queue while another flowfile with a corresponding signal attribute passes through a 'gate' further down the flow, similar to a notify. In my specific case I load a number of files received together in to target tables. I have a single flow which does the following: PutSql - issues a truncate table command that empties the target staging table.  The table name is taken from an attribute.PutDatabaseRecord - Inserts the records into the staging table based on the same attribute.PutSql - executes a stored procedure passing the relevant tables name as parameter from the attribute.  This SP merges data from the staging table into a 'main' table.A number of these files, share the same staging and target tables and on occasions I have a race condition where the the staging table is trun

Hi, I am new to NIFI, I wanted to Get total count of number of events per minute from consumer kafka.After searching on internet, I get 'UpdateCounter' , but information was very less. Can someone suggest me how to add counter . I added ${filename} in UpdateCounter>properities>counter name, It is giving individual file name with value 1 in hamburger menu>counter.Or is there any other processor that can give the event count from the consumer kafka.Thank you in advance.

Hello,   We are currently using CDH 6.3 with Kafka as one of the components.  We have Kafka and Zookeeper services installed and running.   Want to install the Kafka Connect service to install and run a few source and sink connectors.  Is it possible to do so with this distribution.

I am really new to Hive/Impala. I am looking to convert some SQL queries into Hive. I have  a column called date (Ex:2011-09-22 05:01:38) and I would like to convert it into the following format: 09-22-2011-05.01.38 AM Any help is appreciated.

I am looking for some configuration where I would like to set a timeout for jobs that are been running for a long time. I have looked up documentation and could find different properties such as  zeppelin.interpreter.connect.timeoutzeppelin.interpreter.lifecyclemanager.timeout.thresholdzeppelin.interpreter.lifecyclemanager.timeout.checkinterval Can someone please assist me in selecting the correct property to set time out for long running jobs and please differentiate between the above configuration parameters. Best Regards,Chandu

I authenticated with hadoop-admin user. hdfs dfsadmin -fetchImage ./output command works well. However, when I try to do this with curl command,  curl https://{namenode_address}:9470/imagetransfer?getimage=1&txid=latest <html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"/><title>Error 401 Authentication required</title></head><body><h2>HTTP ERROR 401</h2><p>Problem accessing /imagetransfer. Reason:<pre> Authentication required</pre></p></body></html> Which authentication is needed additionally?

To be honest right at the beginning of this post: I am new at Cloudera and would like to download CDH6.1.1 for a customer as a trial (no production from my perspective). However, I got stuck in the paywall, which requires credentials, which I cannot request as the request page requires the credentials I am asking for. So can someone give me credentials, a download-link to be used without credentials or a hint for any third option?Likely this was answered somewhere in the community but most threads I found did not work out for me.

Hi,I have many flow file which has multiple field like mentioned belowcustomefields_12345 : nullcustomefields_12346: { "welcome to this issue"} I have to route these flow files on the condition :if customfield is null --> navigate to other processorif customfield is notnull or having some value other than null should be navigate to other processor.Please let me know how to achieve this. early response will be appreciated. Note: customfield should be used rather than customfield_12345. thanks!