Solved

upgradation

Hi All. I'm relatively new to NiFi and would like a little help in designing a data flow that looks like this: SELECT from a SQL data sourcesplit/convert to JSONoptionally transform JSONmerge to 500 record batchesPOST to target via REST API (requires session token from API auth POST)I know how to create this flow in NiFi. The part I'm having trouble with is including a step that gets a session token via the REST API auth endpoint. I can get the session token using InvokeHTTP no problem. I just don't know how to add the session token to the flow file(s) containing the data to POST to the API. Thanks for your help.

Hi All, I want to append cluster name in the 'Cloudera Alert mail subject'. I am not getting any configuration for same. Please suggest how to do that. Current Subject: [Cloudera Alert] 4 Alerts since 2:31:41 AM Want to change as below (assuming cluster name as ABC):[Cloudera Alert - ABC ] 4 Alerts since 2:31:41 AM Regards,Banshi.

在CDP集群hive的外部仓库目录权限默认为1777，而内部仓库目录为0770，这样会出现一个问题，hive的外部仓库目录任何用户都可以写入数据问题。是设计就是这样还是起塔问题？

Hi guys,   we added solr srever in solrCloud，when I want to achieve to HA in solrCloud. but it is failed in transfer. when one is stop , the other one is living and used by Atlas.     

Hello all,Please bear with me if I ask a simple/basic question. I am new to NiFi and appreciate your patience and support.I am setting up secure NiFi in 2 clusters. I am using NiFi 1.15.3 version in my setup. My cluster 1 has nodes: node1, node2, node3 and cluster 2 has nodes: node4, node5, node6.I have created certificates in my organization by setting node1 as primary node and other nodes as alternative nodes. I use the same certificates in both clusters since I added all the nodes to the certificate when creating it.My cluster 1 starts and I can able to login to web ui with admin user without any issues. However, my cluster 2 starts without any issues but when I try to login, it throws:Insufficient permission. Untrusted proxy CN=node1, O=xxxx, L=yyyy, ST=zzzz, C=USand in logs:o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed GET https://node4:8443/nifi-api/flow/current-user [Untrusted proxy CN=node1, O=xxxx, L=yyyy, ST=zzzz, C=US]As a part of troubleshooting, I added the node

Hello, A simple create table as select fails for some tables in my original_db and I don't know why.The query I use is below, and the error HIVE gives out is really long, but the main theme is "Vertex Failed", "Error while processing row", "No enum constant" - as you can see below. However, when ran without "STORED AS PARQUET" option, the query runs fine.Any ideas what I can try to maybe create the tables as compressed parquet? Thanks   beeline -e "SET parquet.compression=SNAPPY;SET hive.exec.compress.output=true;SET parquet.output.codec=snappy;set hive.tez.container.size = 8192;DROP table if exists DB.table;CREATE table DB.tableSTORED AS parquetAsSELECT * FROM ORIGINAL_DB.table;" ERROR : Status: FailedERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1640013186854_27946_1_00, diagnostics=[Task failed, taskId=task_1640013186854_27946_1_00_000004, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1640

How can I transform a string to an integer in NiFi? I tried JoltTransform, but the string is transformed to a negative (??) integer. In the example, below, the timestamp_start original string value 1677720088000 is transformed to -1612124736. What am I missing in my Jolt specification? Any help is appreciated. Is there an alternative to the JoltTransform - another processor? Looks like this is a known issue: https://github.com/bazaarvoice/jolt/issues/749 Original JSON Transformed JSON  Jolt Specification [ { "operation": "modify-overwrite-beta", "spec": { "timestamp_start": "=toInteger" } }, { "operation": "default", "spec": {} }]   

I have recently upgraded Private Cloud Base components (CM 7.5.4, CDH 7.1.7). I am unable to locate a Spark 3 or Spark3 LIVY service after following the CDS 3.2, installation steps. ( https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-install-spark-3-parcel.html ) I am able to download, distribute, and activate the Spark 3 parcel but there are no options to add it as a service. I did add the `csd` jars to the directories with no luck. logs indicate conflicts with the csd (I tried with both the page listed .49 and also the unlisted .1001). Removing the csd jars removes the warning but the service is still not available.  "...2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.42022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170

I started a apache phoenix query server, phonenix version is apache-phoenix-5.0.0-HBase-2.0-binAll other functions work well in the cmd line of sqlline-thin.py. But when I tried to get the primary key of a table, its just listed nothing. 0: jdbc:phoenix:thin:url=http://localhost:876> !primarykeys FN2.TESTYL2+------------+--------------+-------------+--------------+----------+----------+| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | COLUMN_NAME | KEY_SEQ | PK_NAME |+------------+--------------+-------------+--------------+----------+----------++------------+--------------+-------------+--------------+----------+----------+ when I tried in phoenix cmd line sqlline.py, it works well.do you know why the thin phoenix query server cannot find the primary key? thanksYang

I just sow a video with a rather weird claim. It stated that if nifi fell, the offset of the kafkarecordreader will be lost. This however, seems to me a bit weird, since the offset should be kept by kafka to the specific consumer group ID, AFAIK. Am I m

I have a WF which runs daily on Real and Beta.I found that there is a problem running my WF in beta today.The WF in real takes around 90-120 minutes.But in beta, the WF takes about 8 hours.I don’t understand the 8 hours run time in beta because the real data is much larger than the beta data.Can you please tell me how to reduce WF execution time in beta?

I've encountered this error log at cloudera HDFS sink connector. (I erased only hostname) org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.connect.errors.ConnectException: java.io.IOException: DestHost:destPort Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.  As this log says, the destination HDFS uses SIMPLE authentication. (hadoop.security.authentication=simple)Is there any way to connect cloudera HDFS sink connector to HDFS with simple authentication?

Hi Is there a way to get the number of the files in the input then assign the count value to an attribute and assign a number value to each file I have files to merge using MergeContent so I should assign fragment.index for each file and fragment.count as total count of files to merge.

Hi all, My company is now moving work to CDSW under Citrix Gateway.As my local computer is not connected to the cloud, there are many troubles like file transfer, copy and paste texts which make me a suffering life. I would like to ask for help for sending email with attachment in CDSW under Citrix Gateway.There is no outlook in my Citrix Gateway. I am not sure can it get done. Does anyone has the python script that can help?Many thanks in advance.

Hi All, Thanks in Adv... Wanted top some POC, for that i want install CDH or HDP hadoop cluster on AWS. Now after merging new product is : CDP Can i still install HDP 3.1  with ambari  as before without any license issue. if yes kindly help the link to downloads source. If not can i install CDP without license or trail versions. if kindly provide links to download and installation document as i never installed CDP. Thanks,Suryakant  

Hi All, Before I have worked in CDH 5.X,Could you please let me know, is there any CDP sanbox/quickstart VM like as Hortenworks sand-box, CDH quick start VMPlease provide the link do download and configure , if Cloudera is providing CDP sanbox/quickstart VM

We tried with root user and with test user also, but still getting the above error. We also did below config change for root usersudo vi /etc/ssh/sshd_configSearch and change the below lines as below,PermitRootLogin yessudo systemctl restart sshd For test user, we did below config changesudo vi /etc/sudoersadded below lineec2-user ALL=(ALL)  ALLHowever, with doing the the above changes we are not able to login with both root and test user. Can you please help us on this

Hi,   I have the following Json. I would like to create a Jolt Spec to remove all nulls from all levels. How can I do that? Any help would be appreciated. Thanks {"ID": "c22b657e-227b-4c65-e6f8-08d88a3e1118","Name": "SomeName","TaggedItemName": "SomeValue","ITRName": "SomeValue","TestReference": null,"JobCardName": null,"Comments": "test","PrimaryHandoverName": null,"SecondaryHandoverNumber": null,"CertificationGroupingName": null,"DocumentCode": "SomeValue","AssignedToName": "SomeName","DocumentReference": null,"TagITRCompletionStatusName": "Rejected","ScheduleRevision": null,"Completed": {"SignOff_AuthP": "John","SignOff_Date": null},"Accepted": {"SignOff_AuthP": "Smith","SignOff_Date": null},"Approved": {"SignOff_AuthP": "Ali","SignOff_Date": null},"ScheduleGroup": null,"DownloadUri": "SomeUrl","UpdatedDate": "2022-01-18T11:36:33.9057626Z","CreatedDate": "2020-11-30T10:57:20.8534287Z"}

Trying to determine if HDF/HDP is a valid choice going forward.Using this document (https://www.cloudera.com/downloads/paywall-expansion.html), it seems like these legacy tools require to an active subscription to download the binaries (as well as get support like bug-fixes and such). My question is if I set up a HDF/HDP cluster while I have an active subscription, then subscription lapses, what happens? Do I need to shutdown the cluster? Can I spin up local hosts and add them to a pre-existing cluster? What are the limitations? What about older clusters before this paywall was in place?EDIT: Adding this link here: https://www.cloudera.com/products/pricing/pricing-update.html"Subscription for binary accessA key part of adopting the Red Hat model was that access to enterprise software binaries requires an active subscription. Cloudera requires a subscription to any compute and storage capacity assigned to run Cloudera software. All Cloudera software version releases, maintenan

Hi, I have installed Ranger in my CDP cluster with RangerAdmin, Tagsync and Usersync instances. I have a problem when trying to access the admin web UI (6080) with the credentials admin:admin or ranger:ranger.I get "Unable to connect to DB" on the interface and  WARN com.mchange.v2.resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@2bf8ec30 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.WARN com.mchange.v2.resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@781133d5 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:org.postgresql.util.PSQLException: FATAL: password authentication failed for user "ranger" in the ran

Hi,   i am having the error message in the title, but I am stuck as I have already checked the followings:   * ranger permissions: cm_s3 / all - bucket, path -- the user in the the list for read / write / all permissions * IDBroker role: user has a cdp-datalake-admin-role, which has a cdp-datalake-admin-policy-s3access   any other idea and what to check? thanks  

For security reasons I need to encrypt a column before putting the data in the repository. How can I do this using python or some processor. I tried to use the cryptography library but without success. If you can guide me which is the best way I would be very grateful.

Hi, How do i change the email in my profile? 

I'm using ExecuteSQL,SplitAvro,ConvertAvroToJSON,EvaluateJsonPath,ReplaceText,ExecuteSQL.  I am trying to replace the content in the flowfile using replaceText processor.Now, i can replace like this. -> INSERT INTO values (${id},'${name}') . ReplaceText process like this : INSERT into x values(1,'xx')INSERT into x values(2,'yy')INSERT into x values(3,'zz')I'm sending INSERT query for each line. But i want to send executesql process like thisINSERT INTO x  values (1,'xx'),(2,'yy'),(3,'zz')