Q&A

Hello, I would like to download the CDH sources to securize my deployed clusters (one in 5.4.3 and the other in 5.12.1) but when I try to download with wget and the detail found in pagehttps://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download_54.html#concept_j4t_2x3_1torhttps://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download_512.html#cdh_vd_cdh_download_512 I try with my account informations but when I use them directly I obtain a n error "Wrong port number" and I try to encode the password for url and this time I obtain.Does anyone could help me please --2021-06-07 16:18:12-- https://*login*:*password*@archive.cloudera.com/p/cdh5/ubuntu/trusty/amd64/cdh/Résolution de archive.cloudera.com (archive.cloudera.com)… 151.101.120.167Connexion à archive.cloudera.com (archive.cloudera.com)|151.101.120.167|:443… connecté.requête HTTP transmise, en attente de la réponse… 401 Authentication requiredAuthentification sé

I get two flow files and I need to look at the value of data in flow files based on two different conditions in RouteOnAttribute. If both the conditions match, then only the flow should be routed to executeSQL processor, else it should flow down to MonitorActivity Processor. I tried multiple ways, but this is not working. Every time, when one of the conditions match, it gets routed to executeSQL Processor.  

Hi,  I got a problem while sending a Zero Byte flow file through Nifi PutSFTP processor. Wondering if there is some configuration I have to set? thanks. Use case: get a file from a directory (GetFile Processor) send the file through SFTP (PutSFTP Processor)if success, change the filename to new filename (UpdateAttribute Processor)Remove the all of the content (ReplaceText Processor). Zero Byte flow file send the new filename through SFTP (PutSFTP Processor) I gotten this error while sending Zero Byte flow file over using PutSFTP ProcessorPutSFTP[id=1a091db4-1f9e-14cd-5206-1514485febc8] Rejecting StandardFlowFileRecord[uuid=247457a8-6608-4271-a626-3c016a1bbe7c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1618805512561-40, container=default, section=40], offset=15043, length=0],offset=0,name=AS_MPS_20200927_010010.xml.done,size=0] because it is zero bytes It will be great if someone could help me.  Thanks in adv

Hello,   I am trying to execute 1 .hql script from Edge node. [hdp-slave4.mydomain.com] hdp-slave4 machine is not running any server roles i.e. zookeeper or hive. Only having zookeeper & hive clients.HDP version: 3.1.4 Command =  hive -f ./dumping_test_script.hql I am getting following error =INFO : Moving data to local directory /home/user1/user1_Run_Jar/AB_HIVE_1805 2021/neo4j-graph-preparation/data-files/user_stats from hdfs://hdp-master.mydomain .com:8020/tmp/hive/hive/df588c46-4e05-4a5c-8fde-5b7aaf9329a6/hive_2021-06- 01_13-55-09_157_7991483949013764396-29555/-mr-10000ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.ex ec.MoveTask. Unable to move source hdfs://hdp-master.mydomain.com:8020/tmp/hive/hive/df588c46-4e05-4a5c-8fde-5b7aaf9329a6/hive_2021-06-01_13-55-09_157_79914 83949013764396-29555/-mr-10000 to destination /home/user1/user1_Run_Jar/AB_HIVE_18052021/neo4j-graph-preparation/data-files/user_statsDEBUG : Shutting down query INSERT

Hello,We have a 9 node HDP 3.1.5 cluster with erasure coding enabled. The hdfs fsck command returns under replicated blocks though the filesystem under path '/' is HEALTHY. Please advise how to reconstruct the replication for the under replicated blocks. Please find the hdfs fsck report below: [hdfs@hostname (prod) ~] $ hdfs fsck /Connecting to namenode via http://namenode:50070/fsck?ugi=hdfs&path=%2FFSCK started by hdfs (auth:SIMPLE) from /10.19.224.62 for path / at Wed May 26 22:48:22 PDT 2021/app/warehouse/db/data/eventdate=20210516/00000-21028-5a7dd08a-8c9e-4f62-b920-b6265292da8c-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850161008_890690. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00047-789-396cdf80-9b0b-4411-a6e2-a1094cfb223e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438512_87

Hello, I am taking a big data class and am new to this whole field, so I apologize for any incorrect use of terminology. I am trying to use sandbox. I was able to create a virtual machine, login to port 4200 with the username "root" and change the admin password using "ambari-admin-password-reset". The password works when I log into port 8080. But when I try to go back to port 4200 and login as "admin", instead of "root", the password doesn't work. Is there a different password? What am I missing? Any suggestions would be great. Thanks

I'm trying the processor ReplaceText to change all my dot to comma, but i can't I really need some help here, please. This is my CSV file: MONEY (header)109.29 I need this result MONEY (header)109,28

Is there a processor that can detect hidden characters in NiFi? In our test environment I am using an API invoke to deliver a JSON payload, for added security we want to filter out any hidden and potentially malicious text. I have been trying to use RouteText, RouteOnContent and ExtractText processor to only allow alphanumeric and punctuation characters through but can't seem to get the regex to work when looking for 'uncommon' text or characters? any help would be appreciated. Andy 

Hello,  I have followed this guide:https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_ag_change_hostnames.html In order to change the hostname on my Cloudera single node CMS system. However, now none of my services will start. Everything is set to localhost (because its all on this one host). I notice that the old hostname is still referenced for the server... Even though Hive, Hue, YARN, etc are all set to "localhost" for their databases my ZooKeeper server keeps thinking it's still the previous hostname.How do I get rid of the hostname that is stuck in Cloudera Manager / ZooKeeper? The only other thing I can think to do is remove the "Server Name override" for the ZooKeeper myid and then restart the service to see if it picks up the new hostname.

Hi, According to the official Nifi documentation, the state allows Nifi processors to "resume from the place where it left off after NiFi is restarted. Additionally, it allows for a Processor to store some piece of information so that the Processor can access that information from all of the different nodes in the cluster". If my understanding is good, when we configure a zookeeper Provider, the state will not be persisted locally, instead, the data will be sent to zookeeper. I've explored the zookeeper znodes and could not find any data related to the state, all I can find are the informations about the Coordinator and Primary nodes. However, the local state directory is still filled. The configuration is very simple, I've 3 external ZK nodes and 3 Nifi instances. Here is an exerpt of the nifi.properties file:nifi.cluster.is.node=truenifi.zookeeper.connect.string=zk-node1:2181,zk-node2:2181,zk-node3:2181nifi.state.management.embedded.zookeeper.start=falsenifi.

How Can I get access (login/password) to old HDP repository ? I need 3.1.5. from https://archive.cloudera.com

File 'hdfs://xxx:8020/data/hive/warehouse/xxxx.db/test_bak/8743abe2968ecbe7-3f9c13700000000_2053286657_data.0.parq' has an incompatible Parquet schema for column 'xxxx.test_bak.ordered_revenue'. Column type: STRING, Parquet schema: optional double ordered_revenue [i:20 d:1 r:0] I made this change in Hue.

Hi, One of our customer willing to migrate to CDP 7.x, however customer wants to use code base of Flume developed in earlier version of CDP.Customer is not willing for NiFi migration and want to check if we can copy the open source Flume binaries to CDP 7.x and setup the runtime of Flume. Can this be achievable? Any suggestion are appreciated. ThanksVenkata Madhu 

Hi experts,Many of the devs are getting the following error when trying to connect with squirrel:Error: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed outSQLState:  08S01ErrorCode: 0 I believe the issue may be because there are too many concurrent hive connections. Can someone help suggest how should i configure this to allow more hive connections?Also I find it weird that I can connect through beeline using the edge node of that cluster but Squirrel is giving an error. Thanks,Any help is much appreciated.

Hi Community I am trying to copy HDFS from an HDP 2.6.x cluster (kerberized) to a CDP Private Cloud Base 7.1.5 cluster (not kerberized) and using other ports as well, it gives me an error So I write the command in consolehadoop distcp hdfs://svr2.localdomain:1019/tmp/distcp_test.txt hdfs://svr1.local:9866/tmp/ what could be the origin of the fault?  Thank you ***********ERROR*************************Java config name: nullNative config name: /etc/krb5.confLoaded from native config>>>KinitOptions cache name is /tmp/krb5cc_11259>>>DEBUG <CCacheInputStream> client principal is useradm/admin@LOCALDOMAIN>>>DEBUG <CCacheInputStream> server principal is krbtgt/LOCALDOMAIN@LOCALDOMAIN>>>DEBUG <CCacheInputStream> key type: 18>>>DEBUG <CCacheInputStream> auth time: Sun May 09 18:39:04 VET 2021>>>DEBUG <CCacheInputStream> start time: Sun May 09 18:39:04 VET 2021>>>DEBUG <CC

Hello, I'd like to implement a lock mechanism in a data flow that prevents a flow file from progressing.  Similar in a way to  the wait processor, holding a flowfile in a queue while another flowfile with a corresponding signal attribute passes through a 'gate' further down the flow, similar to a notify. In my specific case I load a number of files received together in to target tables. I have a single flow which does the following: PutSql - issues a truncate table command that empties the target staging table.  The table name is taken from an attribute.PutDatabaseRecord - Inserts the records into the staging table based on the same attribute.PutSql - executes a stored procedure passing the relevant tables name as parameter from the attribute.  This SP merges data from the staging table into a 'main' table.A number of these files, share the same staging and target tables and on occasions I have a race condition where the the staging table is trun

Hi, I am new to NIFI, I wanted to Get total count of number of events per minute from consumer kafka.After searching on internet, I get 'UpdateCounter' , but information was very less. Can someone suggest me how to add counter . I added ${filename} in UpdateCounter>properities>counter name, It is giving individual file name with value 1 in hamburger menu>counter.Or is there any other processor that can give the event count from the consumer kafka.Thank you in advance.

Hello,   We are currently using CDH 6.3 with Kafka as one of the components.  We have Kafka and Zookeeper services installed and running.   Want to install the Kafka Connect service to install and run a few source and sink connectors.  Is it possible to do so with this distribution.

I am really new to Hive/Impala. I am looking to convert some SQL queries into Hive. I have  a column called date (Ex:2011-09-22 05:01:38) and I would like to convert it into the following format: 09-22-2011-05.01.38 AM Any help is appreciated.

I am looking for some configuration where I would like to set a timeout for jobs that are been running for a long time. I have looked up documentation and could find different properties such as  zeppelin.interpreter.connect.timeoutzeppelin.interpreter.lifecyclemanager.timeout.thresholdzeppelin.interpreter.lifecyclemanager.timeout.checkinterval Can someone please assist me in selecting the correct property to set time out for long running jobs and please differentiate between the above configuration parameters. Best Regards,Chandu

I authenticated with hadoop-admin user. hdfs dfsadmin -fetchImage ./output command works well. However, when I try to do this with curl command,  curl https://{namenode_address}:9470/imagetransfer?getimage=1&txid=latest <html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"/><title>Error 401 Authentication required</title></head><body><h2>HTTP ERROR 401</h2><p>Problem accessing /imagetransfer. Reason:<pre> Authentication required</pre></p></body></html> Which authentication is needed additionally?

To be honest right at the beginning of this post: I am new at Cloudera and would like to download CDH6.1.1 for a customer as a trial (no production from my perspective). However, I got stuck in the paywall, which requires credentials, which I cannot request as the request page requires the credentials I am asking for. So can someone give me credentials, a download-link to be used without credentials or a hint for any third option?Likely this was answered somewhere in the community but most threads I found did not work out for me.

Hi,I have many flow file which has multiple field like mentioned belowcustomefields_12345 : nullcustomefields_12346: { "welcome to this issue"} I have to route these flow files on the condition :if customfield is null --> navigate to other processorif customfield is notnull or having some value other than null should be navigate to other processor.Please let me know how to achieve this. early response will be appreciated. Note: customfield should be used rather than customfield_12345. thanks!  

Hello experts, I have nifi registry with MySQL db as metadata store and FileSystem as Persistence provider.And this registry has 10+ flows with multiple versions. Now I am trying to integrate registry to GIT, that is trying to change persistence provider from FileSystem to GIT. I referred some docs (below urls) and steps to switch persistence provider etc, those all help for fresh GIT integration set and they wont carry/migrate already created buckets/flows/versions from FileSystem to GIT https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#switching-from-other-flow-persistence-provider https://community.cloudera.com/t5/Community-Articles/Storing-Apache-NiFi-Versioned-Flows-in-a-Git-Repository/ta-p/248713 Any help on this would be much appreciated. ThanksMahendra

Hey everyone. I'm in the process of migrating data from a CDH 6.3.2 cluster to CDP 7.1.6.   I'm using distcp to migrate data as it seems to be impossible to perform in-place migrations from CDH 6.3.2 and CDP 7.1.6 (according to cloudera documentation -- https://docs.cloudera.com/cdp-private-cloud/latest/upgrade/topics/cdpdc-upgrade-paths.html).     I migrated all the resources that I need from HDFS but now I would like to also migrate the yarn application logs (the ones that we can access in resource manager UI or using yarn logs command).  I tried to use distcp to copy the logs from the old cluster to the new, to the directory that is configurable using yarn-site.xml. However, the migrated logs don't appear in resource manager UI neither using yarn logs -applicationId <app-id>. There is any way to make the logs from the old cluster available in the new cluster and accessible via resource manager or using the yarn logs command?   Thanks in advance. C