Ask the Community

hi everyone ! hoping everybody has an awsome day.I am trying to create a process Group with a flow definition json i downloaded.when i upload a file.json manually and checked the nifi-request.log and the only i see is this pettiotion " http://localhost:8081/process-groups/{id}/process-groups/upload" but i don't see the body in sent request, i want to use the api rest to upload this file but documentation is not clear in this aspect pettition #1:curl --location "http://localhost:8082/nifi-api/process-groups/{id}/process-groups/upload" \--form "file=@/path/to/my/flowDefinition.json"Response: Process Group name is required pettition #2:curl --location "http://localhost:8082/nifi-api/process-groups/{id}/process-groups/upload" \--header 'Content-Type: multipart/form-data' \-F 'id=root' \-F 'name=QA_FLOW' \-F 'position[x]=577.0' \-F 'position[y]=11.0' \-F 'clientId=27bf6df2-018c-1000-362e-61166913eee4' \-F 'disconnectNode=true'-F "file=@/path/to/my/flowDefinition.jso

Hey there,we're running CDP Private Cloud Base in 7.1.9 in 4 different Stages (Lab, DEV, QA and Prod). Is it possible to add a banner in different colours to the different WebUIs so that we can always distinguish the stages well? Like changing the colours in Cloudera Manager?Thanks and Regards,Timo

Anyone have guide how to upgrade default Zeppelin in Ambari 2.7.4, i cant find it anywhereThanks in advance.

Hi , AllAfter we upgarding NiFi 1.9.2 to 1.22.0 (using cluster) , we found that ListFile will lost tracking. Just like @TimothySpann  say.see: https://community.cloudera.com/t5/Support-Questions/NiFi-upgrade-1-9-2-to-1-23-0/m-p/376353#M242873Here is our ListFile setting :Listing Strategy Tracking EntitiesRecurse Subdirectories trueRecord Writer No value setInput Directory Location RemoteFile Filter [^\.].*Path Filter No value setInclude File Attributes trueMinimum File Age 1 minMaximum File Age No value setMinimum File Size 0 BMaximum File Size No value setIgnore Hidden Files trueTarget System Timestamp Precision Auto DetectEntity Tracking State Cache RedisDistributedMapCacheClientServiceEntity Tracking Time Window 365 daysEntity Tracking Initial Listing Target All AvailableEntity Tracking Node Identifier ${hostname()}Track Performance falseMaximum Number of Files to Track 1000

Hello, I have NiFi flow with EvaluateJsonPath that read a content of FlowFile with json to Data attribute. But numeric values of json are changed, i.e.:GenerateFlowFile{"Amount1": 999999999999999999.99,"Amount2": 999999999.99}->EvaluateJsonPathData $->Data{"Amount1":1.0E18,"Amount2":9.9999999999E8}How do I stop NiFi from rounding off and changing json numbers? Tested on NiFi 1.15/1.19.Thanks.

Hi ,In Apache NIFI  I am  consuming data from one Kafka topic and publishing that data to customer Kafka topic . For this iam using ConsumeKafka and PublishKafka processor respectively.If outgoing dataflow is idle for more than 5 minutes, the Kafka connection with customer should be terminated and  reconnected when there is  any new  messages consumed .how to do this with publish kafka processor ??

I have a remote NIFI with single user authentication with Secured SSL certificate, I want to enable the site to site with remote NIFI from local NIFI,I am getting error like unauthorized in the remote process group of the local NIFI If I remove the authentication from the  Remote NIFI, site to site is working properly and flow file is getting transferred to local NIFIcan someone help to make the site to site with user authentication

hi there,So I need your help with something. So I am extracting 4 columns out of a database. Out of these 4 columns, two columns are DATE in Oracle and I want to convert them into DATETIME (as they are going to be inserted into BigQuery). In addition to this, I am using an UpdateRecord to generate a new column and for that, I am using the following AVRO Schema:{"namespace": "example.avro","type": "record","name": "my_relevant_table","fields": [{"name": "ID_REQUEST","type": ["string","null"]},{"name": "ID_PROCESS","type": ["string","null"]},{"name": "ARCHIVE_DATE","type": ["null",{"type": "long","logicalType": "local-timestamp-millis"}]},{"name": "EXEC_DATE","type": ["null",{"type": "long","logicalType": "local-timestamp-millis"}]},{"name": "LOAD_DATE","type": ["null",{"type": "int","logicalType": "date"}]},]}Now, as you can imagine, I will encounter some null value within the ARCHIVE_DATE column, because this is a normal behavior. If I let the flow execute as it is, the AVRO File get's

Hi Team,I want to change the log level of the Hadoop security logger from INFO to ERROR. I've exported the following environment variable for thatexport HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=ERROR,RFA ${HADOOP_NAMENODE_OPTS}"I can see the new parameter has been reflected in the command linehdfs 13048 2.1 0.7 3899916 509000 ? Sl 01:22 0:29 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=ERROR,RFA -Dyarn.log.dir=/var/log/hadoop-hdfs -Dyarn.log.file=hadoop-hdfs-namenode-hadoop-master-1.log -Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=/usr/lib/hadoop/lib/native -Xmx2048m -Dhadoop.log.dir=/var/log/hadoop-hdfs -Dhadoop.log.file=hadoop-hdfs-namenode-hadoop-master-1.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=ERROR,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameN

How to change Nifi logo in Nifi binary packages installed

I'm encountering occasional Exception when using PUTSQL processor in apache Nifi.I have already Increased innodb_lock_wait_timeout But I'm still encountering the error occasionally. How can I fix this error? And what are the optimal configurations when using putsql processor to insert large volume of records into mariadb?Variable_name;Valueinnodb_lock_wait_timeout;300Error Snippet:2023-11-24 18:04:45,837 ERROR [Timer-Driven Process Thread-51] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c7255d17-018b-1000-143e-fc839881330c] Failed to update database due to a failed batch update, java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transaction. There were a total of 1 FlowFiles that failed, 14 that succeeded, and 0 that were not execute and will be routed to retry;java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transactionat com.mysql.cj.jdbc.exceptions.SQLError.createBatchUpdateException(SQLError.java:224)

I'm using NiFi to process .tar.gz files, extract the csv files, and upload them into S3. NiFi is currently running on a t2.large. I tested the same workflow on a t2.2xlarge, expecting an increase in performance. Yet, it didn't happen. In particular, it seems like the CompressContent and UnpackContent processors take the same amount of time, regardless of the instance type.Any idea of what I'm doing wrong or how to optimize performance?For the t2.large, I have concurrent tasks set to 2, while for the t2.2xlarge I have it set to 8. Execution on all nodes for both.Thanks

I want to set up a Nifi flow that gets data from a public RSS feed and loads it into a data lake. This RSS feed updates irregularly and when it does update it overwrites previous content. What processor(s) should I use to get data from the RSS feed (close to) when it has updated? Is it as simple as using InvokeHTTP repeatedly, checking for a change in output, then loading into data lake if the content differs from the previous invocation? Is there another way if I don't want to make the HTTP request so frequently?

Hello all, i am new to Nifi , i have Processor to InvokeHTTP and get json file ,the json format example:"{"alertDate":"2023-12-03 11:01:17","title":"XXXX","data":"XXXX, XXXX, XXXX","category":1}" "i am using the Processor ConvertJSONToSQL  to send the data to PutSQL to insert to my sql data base ,my issue is :PutSQL Failed to update database for FlowFile due to The value of the sql.args.1.value is '2023-12-02 10:54:38', which cannot be converted to a timestamp" know i need to use the JoltTransformJSON but not sure what is the correct Jolt Specification i need to put,my DB columns  are like this can you please help? - remember i am new to Nifi 🙂thanks a lot! 

I tell you that we have updated Apache Nifi from version 1.16 to version 1.22 and we are currently having a problem connecting to the Mongo DB because the server version is very low.Is there an option that does not involve having to update the database server to be able to leave this connection functional?  

Hey everyone, I want to fetch all databases and its records from CouchDB (NON-SQL) and then view all records in my Apache NiFi, after confirmation I want to send all records to my Server database.?I have not found processors of GetCouchDB. Share your thoughts that how we can do such process by using other processors ? Is there anyone to know about this process.   #How to read all databases and its records from CouchDB and then by using Apache NiFi send all those records to my Server?

Hi all,The HDFS alerted that "Stale Configuration: Refresh needed" . I followed the suggestion, but i  hesitated when the following image came out.  I want to confirm that when i click "Restart Stale Configuration" button, the entire cluster will restart? Or only serveral servers will be restarted?

I am new to Jolt, I am trying to transform JSON to a particular format using nifi, where i am moving some objects into another object, I think i am closeThis is my JSON{ "relatedParty": [ { "id": "B2BSmallandMedium" } ], "serviceOrderItem": [ { "id": "e26205f4-e144-4ded-9e64-82ede0b26f4", "action": "add", "service": { "name": "cust", "state": "active", "supportingService": [ { "id": "e26205f4-e144-4ded-9e64-82ede0b26e99", "href": "serviceInventory/v4/service" }, { "id": "7313c885-a075-4188-a6bc-c3fbfb4c7422", "href": "activationAndConfiguration/v4/service" } ] } } ]}I am new to Jolt, I am trying to transform JSON to a particular format, where i am moving some objects into another object, I think i am close.This is my JSON{ "relatedParty": [ { "id": "B2BSmallandMedium" } ], "serviceOrderItem": [ { "id": "e26205f4-e144-4ded-9e64-82ede0b26f4", "acti

I am following https://medium.com/egen/how-to-configure-oauth2-authentication-for-apache-kafka-cluster-using-okta-8c60d4a85b43 for setting up OAuth2.0But when I try to produce data I get Error while fetching metadata with correlation id 43 : {myTopic=TOPIC_AUTHORIZATION_FAILED}.Can anyone please help me outI tried adding invalid details in my sasl-oauth2-consumerapp-config.properties file to check if it is picking the right file , it seems like right file is picked.

Hi, I'm trying to get this JOLT working, but in some cases, in the output is printing null values. The qty of itens inside RUBRICS array is dynamic. What am I doing wrong ? Payload [ { "ENTITY_DOCUMENT": "11", "ENTITY_ID": "", "ENTITY_TYPE": "TEST_CARD", "CLOSED": true, "CALCULATION_DATE": "2023-11-01", "RUBRICS": [ { "name": "provisao_TEST_card", "classification": "", "code": "", "amount": 110.08, "type": "", "business_unit": "TEST", "aggregation_key": "" } ] }, { "ENTITY_DOCUMENT": "12", "ENTITY_ID": "", "ENTITY_TYPE": "TEST_CARD", "CLOSED": true, "CALCULATION_DATE": "2023-11-01", "RUBRICS": [ { "name": "provisao_TEST_card", "classification": "", "code": "", "amount": 120.08, "type": "", "business_unit": "TEST", "aggregation_key": "" } ] }, { "ENTITY_DOCUMENT": "13", "ENTITY_ID": "", "ENTITY_TYPE": "TES

Dear community gods, I am using CDH6.3.1 and I am exporting data from KUDU and it has an error as follows. I need your help now：[root@hadoop001 impalad]# sudo -u impala impala-shell -i hadoop001 -d test --query_option="SPOOL_QUERY_RESULTS=TRUE" -l -u user01--auth_creds_ok_in_clear -q "select * from test.test_table01 where length(file_data_json)/1024/1024 > 20 limit 1"Starting Impala Shell using LDAP-based authenticationLDAP password for user01:Opened TCP connection to hadoop001:21000Connected to hadoop001:21000Server version: impalad version 3.2.0-cdh6.3.2 RELEASE (build 1bb9836227301b839a32c6bc230e35439d5984ac)SPOOL_QUERY_RESULTS is not supported for the impalad being connected to, ignoring.Query: use `test`Query: use `test`Query: select * from test.test_table01 where length(file_data_json)/1024/1024 > 20 limit 1Query submitted at: 2023-11-03 21:59:04 (Coordinator: http://hadoop001:25000)Query progress can be monitored at: http://hadoop001:25000/query_plan?query_id=ed46b03a61e6b

This question has been asked before (here) but never answered so I'm hopeful that today is my lucky day and someone knows the answer.   Because unfortunately, the documentation does not cover the topic.In the UpdateAttribute there is an option to Store State of an attribute by setting the value to "Store state locally".  But to do that I first need to set the "Stateful Variables Initial Value", but the documentation doesn't say how to do that, and everything I've tried fails.   The documentation provides an example for referencing an existing attribute variable, theCount, but there is no mention of how to initialize this attribute variable. I've attempted setting the initial value using:theCount=1{"theCount":1}${theCount=1}1${setStateValue('theCount')}     <- grasping at straws with this oneNone of these work.  When I look at the value of the attribute theCount it always just shows:  "Empty string set".I'm including a scree

Hi,I have managed to download the latest Nifi 2.0.0 M1 and I'm trying to run it on my windows 10 machine. Doing some preliminary testing I ran into the following issues:1- The system requirement indicates that (https://nifi.apache.org/project-documentation.html ) indicates that at minimum I need Java 17, but when I try to start nifi using run.bat I get the following error:  Error: LinkageError occurred while loading main class org.apache.nifi.bootstrap.RunNiFi java.lang.UnsupportedClassVersionError: org/apache/nifi/bootstrap/RunNiFi has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 61.0  It turns out it needs Java 21. Not sure if the documentation has not been updated or if Im missing something.2- After upgrading to Java 21, Im able to start nifi using default configuration, the log file doesn't show any error and default username and password ar

Hi team,We faced an issue while working on Json transformation using Flink after consuming Kafka upcoming data stream. The script developed using Scala. -> The services details are:1. Server Kafka-console-consumer version: 2.5.0.7.1.7.1000-1412. Server Flink version: 14.14.0-csa1.6.2.03. Server Scala version: 12.13.10 -> The error is: java.lang.NoClassDefFoundError: org/apache/flink/api/common/serialization/DeserializationSchema  -> The Scala scripts (running using SBT):1. SBT file (build.sbt) : scalaVersion := "2.12.18"name := "kafka-flink"organization := "com.me"version := "1.0"//libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.5.0.7.1.7.1000-141"libraryDependencies += "org.apache.flink" %% "flink-scala" % "1.14.0-csa1.6.2.0" % "provided"libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % "1.14.0-csa1.6.2.0" % "provided"libraryDependencies += "org.apache.flink" %% "flink-connector-kafka" % "1.14.0-csa1.6.2.0" %

i built Apache Impala by myself, and i want use JDBC / ODBC for itcan i use Cloudera Impala JDBC / ODBC in my project for free? or does it requires any license?