Q&A

Hi,System: NiFi 1.19, Java 11, Win 64bit, 610 processors, 245 runningI have a problem at my current running flows. Example: the LogMessage-Processor shows, that there are 112 flows in the last 5 minutesBut if I select "view data provenance", there are no flows displayedAs you can see, the "Oldest event available: 08/29/2023", this is one month ago.I checked the hard disk, where NiFi is running: there is 25 GB free memory, so it shouldn't be a full-disk-problemAny idea, what we have mis-configured?

Hi Everyone, Can Anyone help us in fixing the issue:I have a flow where I am using control rate processor to control the data flow, I will submit tasks in druid using those files, for example let's say I have a total of 10 files. I will first ask my control rate processor to send 2 flow files to downstream, now I will submit those files as tasks in druid and fetch their status, if 1 task is running and another one is success I need to ask my control rate processor to send only 1 file, if 2 tasks are running dont submit tasks until they are completed, if 2 are success then send 2 files from control rate an counting the tasks which are running and adding them to an attribute,so is there a way that control rate processor takes the value of count attribute in Maximum rate property. I've checked the Rate Control Criteria property which can be set to Attribute Value but I am confused how to set the rate to attribute value.

output file:{"name" : "name","class" : "a","grade" : "A","address" : "state""url" :{"first url":"url","second url":"url"}}

Hi all,I've been looking for solutions to expose Kafka data in Hive. I found a few things, however HiveKafkaStorageHandler caught my attention :).I was looking for trying it out, but I don't find anywhere in the documentation how does it support a connection to Kafka through SSL (which is a requirement for my setup).Can someone tell me for sure if this storage handler supports a broker listener that uses SSL? Thanks a lot in advance !

I have a nifi flow. I observe that if input file is split into smaller files and fed into the flow one by one, then overall time taken(sum of time taken for individual files) is considerably low compared to when I feed single big file.What can be a possible cause for this performance difference?Note:Flow has many processors that use avro readers/writers.I calculate time using following in a LogMessage processor:${now():toNumber():minus(${lineageStartDate}):format("HH:mm:‌​ss", "GMT")}

Accessing zeppelin with pam credentials are failing with bad credentials

Hi, Currently i am having a setup where i am already having partitioned parquet data in my s3 bucket, which i want to dynamically bind with hive table. I am able to achieve it for single partition, but i need help loading data from all the partitions in the table from existing partitioned parquet data from s3 bucket. Thank you.

Hi, I would like to enquire about the missing access policies features as I run my nifi instance. Step that I did:1) Generate the certificate and key (keystore.jks, truststore.jks, nifi.properties) Command: .\bin\tls-toolkit.sh standalone -n "localhost" 2) Copy the above 3 cert to ".\nifi-1.23.2\conf"3) Generate the user's client certificate to authenticate to NiFiCommand: .\bin\tls-toolkit.sh standalone -C "CN=sys_admin, OU=NIFI"4) Load the client certificate into web browser (EDGE)5) Populate the Initial Admin Identity Open authorizers.xml in ".\nifi-1.23.2\conf"In UserGroupProvider section, replace <property name = "Initial User Identity 1"><property> to <property name = "Initial User Identity 1">CN=sys_admin, OU=NIFI<property>In accessPolicyProvider section, replace <property name = "Initial Admin Identity"><property> to <property name = "Initial Admin Identity">CN=sys_admin, OU=NIFI<property>6) Star

Hi, I just completed setting up CDP Private Base in a POC environment. I was in process of attempting AD (LDAP based) integration so that users get authenticated via Active Directory.I am unsure if there was a mis configuration, however, after restarting Cloudera Manager, I can't login to it via "Admin" account (local). Thought it has got AD integrated, tried multiple accounts from AD, however, none of them are working. Please help to re-enable admin (local) account. Thankssnm1523

CDH version - 6.3.2The package 'cloudera-manager-agent-6.3.0-1281944.el7' is removed from the data-cmhue instance and we are not able to install it (via yum/rpm command).Following packages are already installed -cloudera-manager-server-6.3.0-1281944.el7.x86_64cloudera-manager-daemons-6.3.0-1281944.el7.x86_64Can you assist on this?

As of HDFS CM5.12.1, the standby namenode is busy under high availability,I'm not sure if the busy standby namenode is hot standby, so please help me see how to fix it. The busy state of the standby namenode restores it to the start state.The action buttons of the standby namenode have all turned gray, making it impossible to restart, stop, and start the namenode.

Hi ,I require assistance in configuring my secure cluster to function with LDAP.Which procedure is recommended when doing that on a running cluster?What steps do I need to take first?Do I need to shut down all the nodes? Thanks Edi

Hi,I am trying to implement OIDC authentication with Azure AD in NIFI, I am able to enable it and user is able to login but now I want to use App Roles created in Azure App Registration instead of AD groups, how can I do that. 

from hdfs dfs -du -h / we see that spark history take a lot space from HDFS I want delete only some that I specified for example all that are of the year 2019 and 2019, and keep the rest. If I use the commandhdfs dfs -rm -R /spark2-history/* this delete all and I want not delete all. Thanks

Hi ladies and girls,I'm trying to merge error logs generated by ExecuteSQL and send them via email. Currently, I have the following flow:ExecuteSQL - failure - > MergeContent - merged -> PutEmail.If I disable MergeContent, the logs are being sent as they should but ( an inidividual email for each error log). If I enable MergeContent, I receive a single email however it duplicates the first error.ERROR: syntax error at or near ""1qa""Where: PL/pgSQL function "extract".truncate_tables(character varying) line 5 at EXECUTEERROR: syntax error at or near ""1qa""Where: PL/pgSQL function "extract".truncate_tables(character varying) line 5 at EXECUTEERROR: syntax error at or near ""1qa""Where: PL/pgSQL function "extract".truncate_tables(character varying) line 5 at EXECUTEERROR: syntax error at or near ""1qa""Here are the settings that I use:I also added a LogMessage processor and the flow for original is showing the correct content. The problem seems to be with the merged logs. Can someo

Hi,I'm using a lot of flows with NiFi 1.19 (Java 11, Win 64bit)I've got problems with a 3rd-party-processor (out-of-support): every few days this processor hung up, so I have to stop it, select "terminate" and start the processor new.Understanding NiFi's "Terminate" option on running components. Is there any possibilty to configure a processor, so that it is terminated and restartet automatically if it runs longer than x minutes?Frank

Hey Cloudera community, I'm an avid gamer and also have some experience working with Apache NiFi for data processing tasks. Lately, I've been wondering if there's a way to leverage NiFi to enhance my gaming experience. I know it's typically used for data integration and ETL, but I've heard it's quite versatile. So, here's my question: Are there any creative ways I can use Apache NiFi to optimize my gaming PC's  performance? Whether it's improving network latency, automating system maintenance, or something entirely different, I'm open to suggestions. I'd also appreciate any insights from fellow gamers who have experimented with NiFi in a gaming context. How has it worked for you? Any success stories, tips, or warnings to share? Thanks in advance for your input! Looking forward to some innovative ideas. Best regards, Charlie

Hi guys,So I have been struggling with a data conversion and I can't really figure out how to achieve what I am trying to achieve.I have a CSV File which comes into my flow as follows:COLUMN1 COLUMN2 COLUMN3 COLUMN4 COLUMN5 COLUMN6 MONTH0182 Tel set W27 0 2200 31-IAN-220183 Apa cai W27 0 2200 30-SEP-220185 HDM set MT8 1 2200 28-FEB-220185 Apa alo MT8 0 2200 31-OCT-220186 HDM set HG5 1 2200 30-IUN-220188 Fus alo HG5 1 2200 30-APR-22I am using afterwards an ConvertRecord to transform the CSV into an AVRO File, using the following schema:{"type": "record","name": "nifiRecord","namespace": "org.apache.nifi","fields": [{"name": "COLUMN1","type": ["string","null"]},{"name": "COLUMN2","type": ["string","null"]},{"name": "COLUMN3","type": ["string","null"]},{"name": "COLUMN4","type": ["string","null"]},{"name": "COLUMN5","type": ["string","null"]},{"name": "COLUMN6","type": ["int","null"]},{"name": "MONTH","type": ["string","null"]}]}Now, in the next step, I would like to transform the MONTH f

HIiI have created an Apache Ni-Fi data pipeline that fetches data from a MySQL table and after some data transformation loads the data into a postgreSQL table. The first processor is a GenerateTableFetch which is followed by an ExecuteSQL processor.The pipeline works perfectly with source tables containing over 14 million entries (around 2700 MB). However, with tables containing over 420 million entries (around 69000 MB), the GenerateTableFetch processor does not provide any output.I have used a partition size of 100000 rows and I am using as Column for Value Partitioning the column id which is the primary key in the source table. The same column is also used as Maximimum-value Columns. The Max Wait Time is set to zero, allowing also long SQL queries.The attached figure shows the other properties of the processor.In the ExecuteSQL processor I have configured the Max Rows Per Flow File to the same value as the partitioning size: 100000.I am running Ni-Fi 1.18.0 (java 11.0.20.1) on Linux

I'm new in hive so it can be very stupid question) In our project we have very difficult model that contains a lot of arrays and nested fields. This model describes interaction with other systems so its filling can be very different. It is something like logs. We save this data in Hadoop using external table and orc files. It is a very big struct field. Apache ORC Tools is used by us for creating orc files and making orc schema. We don't have problem with writing, but when we read data we get a lot of "null" values. I understand that reasons of it are difficult model and different filled fields. But output in beeline or Hue is unreadable:...{""id"":""mqqqnc"",""value"":[{""Ipv4Value"":null,""StringValue"":{""value"":""99""}}]},{""id"":""eee"",""value"":[{""Ipv4Value"":null,""StringValue"":{""value"":""254103""}}]},{""id"":""uli"",""value"":[{""Ipv4Value"":null,""StringValue"":{""value"":""asfsfsd""}} ...This example is a little part of our struct field but we have three null field.Is i

Hello everyone,in a cdp 7.1.7 cluster I have not found a way to implement engine spark to run hive queries on hue. I can execute hive queries with spark engine with spark-shell but if I try to execute hive queries on hue by setting set hive.execution.engine=spark I get the following error:Error while compiling statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark sessionCan anyone help me to implement hive on spark via hue?Thanks in advance

I have read about support of redis sentinel and standalone redis in Apache nifi but is there any way I can integrate clustered nifi for storing and fetching the value ?

Currently I have configured apache NiFi with Ranger as per discussed in this thread. Ranger NiFi plugin is able to pull resources from ranger and auto suggest in ranger nifi service  definition UI. But I don't think the authorizations works properly. Only the /flow resource specifier has an impact. When I remove that specifier user is not able to view the screen and when I add it back the user is logging back into the Nifi UI. Rest of the resources including processors , ports etc. seems to be unclickable or accessible  no matter any of the resource specifier I provide in ranger. I even tried the wild card * but still doesn't seems to work. The UI looks as below.   The Ranger Authorizers file is shown below   <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property>

I have been trying to integrate Nifi with Apache Ranger. When I manually configure policies in Ranger for Nifi the policies are fetched by Nifi and authorization works fine.  But when I try to define the service definition and test the connection its is giving the following error.The configuration in service definition is shown below. The authentication in nifi is setup by following this article.Note: Only Nifi is setup in SSL mode. Apache Ranger doesn't have SSL mode.What could be the possible reason this is happening? 

I'm hoping someone might have a suggestion here.  A PutDatabaseRecord processor which was working properly under 1.15 started generating java.sql.BatchUpdateException: ORA-00936: missing expression on a 1.18 cluster.I'm using the Oracle12+ database type and the incoming flowfile format is Avro - so there should be no Schema inference happening.It's strange, since that error usually indicates a structural problem with the SQL itself.  So, I'm concerned that there is a defect in the 1.18 version of PutDatabaseRecord.  But, maybe there is something I can change about the source data or in the properties of the processor which will workaround this issue?If anyone has seen something like this or has some suggestions I could try, I would appreciate it.  Thanks.