Solved

Hello,I am trying to get an key from GetMongo Processor and then I am trying to replace the value  in the format key=valueusing  ReplaceText where value here is the result which I am getting and then passing  to different POST API which supports `application/x-www-form-urlencoded`From GetMongo Processor I am getting value {match_id: 789456}and In replaceText Processor I am using in Search Value : (?s)(^.*$) and Replacement Value in match_id=${match_id:urlEncode()}But in API data is getting passed as { match_id: '' } though it should pass as  { match_id: 789456 } What I am doing wrong in ReplaceText Processor Can you please help me out with this ?  

Hello, - I've setup a Sandbox HDP 3.0.1.- Determine Network Adapter of Your VirtualBox Sandbox (Bridged Adapter, IP:192.168.1.37).- I edited the hosts file (on my computer, windows) and assigned the domain sandbox-hdp.hortonworks.com to the IP address of 192.168.1.37, but When I start all services via the Ambari UI, I receive the following error message:   safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.2 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2019-09-29 18:25:10,968 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.2 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection

I have a requirement to add custom user for allowing impersonation. I can see that CDH 6.3 documentation has some references to it. However, I could not see how I can add similar entries through Cloudera Manager in CDP 7.x. Can someone help with that information ?Essentially, I would like to add the entries like this.  <property> <name>hadoop.proxyuser.super.hosts</name> <value>10.222.0.0/16,10.113.221.221</value></property><property> <name>hadoop.proxyuser.super.users</name> <value>user1,user2</value></property>or something like this.<property> <name>hadoop.proxyuser.alice.hosts</name> <value>host_a,host_b</value></property>  Is there a way to achieve the same through Cloudera Manager in CDP 7.x ? I am not sure if I should be editing the core-site.xml on the nodes directly as those were Auto Generated by the Cloudera Manager.

Hi,System: NiFi 1.19, Java 11, Win 64bit, 610 processors, 245 runningI have a problem at my current running flows. Example: the LogMessage-Processor shows, that there are 112 flows in the last 5 minutesBut if I select "view data provenance", there are no flows displayedAs you can see, the "Oldest event available: 08/29/2023", this is one month ago.I checked the hard disk, where NiFi is running: there is 25 GB free memory, so it shouldn't be a full-disk-problemAny idea, what we have mis-configured?

output file:{"name" : "name","class" : "a","grade" : "A","address" : "state""url" :{"first url":"url","second url":"url"}}

Hi all,I've been looking for solutions to expose Kafka data in Hive. I found a few things, however HiveKafkaStorageHandler caught my attention :).I was looking for trying it out, but I don't find anywhere in the documentation how does it support a connection to Kafka through SSL (which is a requirement for my setup).Can someone tell me for sure if this storage handler supports a broker listener that uses SSL? Thanks a lot in advance !

I have a nifi flow. I observe that if input file is split into smaller files and fed into the flow one by one, then overall time taken(sum of time taken for individual files) is considerably low compared to when I feed single big file.What can be a possible cause for this performance difference?Note:Flow has many processors that use avro readers/writers.I calculate time using following in a LogMessage processor:${now():toNumber():minus(${lineageStartDate}):format("HH:mm:‌​ss", "GMT")}

Accessing zeppelin with pam credentials are failing with bad credentials

Hi, I would like to enquire about the missing access policies features as I run my nifi instance. Step that I did:1) Generate the certificate and key (keystore.jks, truststore.jks, nifi.properties) Command: .\bin\tls-toolkit.sh standalone -n "localhost" 2) Copy the above 3 cert to ".\nifi-1.23.2\conf"3) Generate the user's client certificate to authenticate to NiFiCommand: .\bin\tls-toolkit.sh standalone -C "CN=sys_admin, OU=NIFI"4) Load the client certificate into web browser (EDGE)5) Populate the Initial Admin Identity Open authorizers.xml in ".\nifi-1.23.2\conf"In UserGroupProvider section, replace <property name = "Initial User Identity 1"><property> to <property name = "Initial User Identity 1">CN=sys_admin, OU=NIFI<property>In accessPolicyProvider section, replace <property name = "Initial Admin Identity"><property> to <property name = "Initial Admin Identity">CN=sys_admin, OU=NIFI<property>6) Star

Hi, I just completed setting up CDP Private Base in a POC environment. I was in process of attempting AD (LDAP based) integration so that users get authenticated via Active Directory.I am unsure if there was a mis configuration, however, after restarting Cloudera Manager, I can't login to it via "Admin" account (local). Thought it has got AD integrated, tried multiple accounts from AD, however, none of them are working. Please help to re-enable admin (local) account. Thankssnm1523

As of HDFS CM5.12.1, the standby namenode is busy under high availability,I'm not sure if the busy standby namenode is hot standby, so please help me see how to fix it. The busy state of the standby namenode restores it to the start state.The action buttons of the standby namenode have all turned gray, making it impossible to restart, stop, and start the namenode.

Hi ,I require assistance in configuring my secure cluster to function with LDAP.Which procedure is recommended when doing that on a running cluster?What steps do I need to take first?Do I need to shut down all the nodes? Thanks Edi

Hi,I am trying to implement OIDC authentication with Azure AD in NIFI, I am able to enable it and user is able to login but now I want to use App Roles created in Azure App Registration instead of AD groups, how can I do that. 

from hdfs dfs -du -h / we see that spark history take a lot space from HDFS I want delete only some that I specified for example all that are of the year 2019 and 2019, and keep the rest. If I use the commandhdfs dfs -rm -R /spark2-history/* this delete all and I want not delete all. Thanks

Hi ladies and girls,I'm trying to merge error logs generated by ExecuteSQL and send them via email. Currently, I have the following flow:ExecuteSQL - failure - > MergeContent - merged -> PutEmail.If I disable MergeContent, the logs are being sent as they should but ( an inidividual email for each error log). If I enable MergeContent, I receive a single email however it duplicates the first error.ERROR: syntax error at or near ""1qa""Where: PL/pgSQL function "extract".truncate_tables(character varying) line 5 at EXECUTEERROR: syntax error at or near ""1qa""Where: PL/pgSQL function "extract".truncate_tables(character varying) line 5 at EXECUTEERROR: syntax error at or near ""1qa""Where: PL/pgSQL function "extract".truncate_tables(character varying) line 5 at EXECUTEERROR: syntax error at or near ""1qa""Here are the settings that I use:I also added a LogMessage processor and the flow for original is showing the correct content. The problem seems to be with the merged logs. Can someo

Hi,I'm using a lot of flows with NiFi 1.19 (Java 11, Win 64bit)I've got problems with a 3rd-party-processor (out-of-support): every few days this processor hung up, so I have to stop it, select "terminate" and start the processor new.Understanding NiFi's "Terminate" option on running components. Is there any possibilty to configure a processor, so that it is terminated and restartet automatically if it runs longer than x minutes?Frank

Hey Cloudera community, I'm an avid gamer and also have some experience working with Apache NiFi for data processing tasks. Lately, I've been wondering if there's a way to leverage NiFi to enhance my gaming experience. I know it's typically used for data integration and ETL, but I've heard it's quite versatile. So, here's my question: Are there any creative ways I can use Apache NiFi to optimize my gaming PC's  performance? Whether it's improving network latency, automating system maintenance, or something entirely different, I'm open to suggestions. I'd also appreciate any insights from fellow gamers who have experimented with NiFi in a gaming context. How has it worked for you? Any success stories, tips, or warnings to share? Thanks in advance for your input! Looking forward to some innovative ideas. Best regards, Charlie

Hi guys,So I have been struggling with a data conversion and I can't really figure out how to achieve what I am trying to achieve.I have a CSV File which comes into my flow as follows:COLUMN1 COLUMN2 COLUMN3 COLUMN4 COLUMN5 COLUMN6 MONTH0182 Tel set W27 0 2200 31-IAN-220183 Apa cai W27 0 2200 30-SEP-220185 HDM set MT8 1 2200 28-FEB-220185 Apa alo MT8 0 2200 31-OCT-220186 HDM set HG5 1 2200 30-IUN-220188 Fus alo HG5 1 2200 30-APR-22I am using afterwards an ConvertRecord to transform the CSV into an AVRO File, using the following schema:{"type": "record","name": "nifiRecord","namespace": "org.apache.nifi","fields": [{"name": "COLUMN1","type": ["string","null"]},{"name": "COLUMN2","type": ["string","null"]},{"name": "COLUMN3","type": ["string","null"]},{"name": "COLUMN4","type": ["string","null"]},{"name": "COLUMN5","type": ["string","null"]},{"name": "COLUMN6","type": ["int","null"]},{"name": "MONTH","type": ["string","null"]}]}Now, in the next step, I would like to transform the MONTH f

Hello everyone,in a cdp 7.1.7 cluster I have not found a way to implement engine spark to run hive queries on hue. I can execute hive queries with spark engine with spark-shell but if I try to execute hive queries on hue by setting set hive.execution.engine=spark I get the following error:Error while compiling statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark sessionCan anyone help me to implement hive on spark via hue?Thanks in advance

Currently I have configured apache NiFi with Ranger as per discussed in this thread. Ranger NiFi plugin is able to pull resources from ranger and auto suggest in ranger nifi service  definition UI. But I don't think the authorizations works properly. Only the /flow resource specifier has an impact. When I remove that specifier user is not able to view the screen and when I add it back the user is logging back into the Nifi UI. Rest of the resources including processors , ports etc. seems to be unclickable or accessible  no matter any of the resource specifier I provide in ranger. I even tried the wild card * but still doesn't seems to work. The UI looks as below.   The Ranger Authorizers file is shown below   <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property>

I have been trying to integrate Nifi with Apache Ranger. When I manually configure policies in Ranger for Nifi the policies are fetched by Nifi and authorization works fine.  But when I try to define the service definition and test the connection its is giving the following error.The configuration in service definition is shown below. The authentication in nifi is setup by following this article.Note: Only Nifi is setup in SSL mode. Apache Ranger doesn't have SSL mode.What could be the possible reason this is happening? 

My Production environment has existing 4 data nodes.I just added additional 4 data nodes in the cluster and would like to run the HDFS Rebalance to balance the data across the 8 data nodes. My question is, can I rebalance data but also ingesting and processing new data at the same time? Will it caused any file interrupted if do so?  Thanks.  

I had a flow which needs to be replicated for 100 jobs.one of my jolt transform looks like this. [{ "operation": "default", "spec": { "*": { "id": "#{customer_id_1}", "warehouse_loaddate" : "${now():format('yyyy-MM-dd HH:mm:ss', 'CET')}" } }}] The customer_id_1 is a parameter currently, which is different in all 10 different jobs.  like there are many different parameters in the context like customer_id_2customer_id_3 etc..With this I need to touch my code at 100 different places...But since I know the customer ID is present in my upstream processor.. so I just want to name the parameter customer_id and get the value from upstream dynamically... Is there a way that I can send a attribute to downstream processor and use it in jolt ?Without using parameters from parameter context ? We use #{<id>} and ${<func>} to access parameters and Functions.. but how can i refer an attribute from my upstream processor ? Please help me under

I have installed Zeppelin on my CDP cluster , since by default "anonymous" id is disabled what is the user ID that I can use to login to Zeppelin UI. I tried using :admin = password1user1 = password2user2 = password3But none of them seems to work

Hello,We need to partition 5 hive tables with more than 100 million records each. Tables have multiple columns to partition with integer and string data allowing less than 1000 partitions per column. Could we combine several of these fields to partition the tables? would there be a restriction on the maximum number of combined partitions? Should they have a specific order according to the number of partitions in each table? for example, first partition by year (10 partitions) , then by country (50 partitions) and then by city name (500 partitions) ? Must the combination of partitions be less than a particular number?Best Regards,H.M