Solved

Hey there,we're running CDP Private Cloud Base in 7.1.9 in 4 different Stages (Lab, DEV, QA and Prod). Is it possible to add a banner in different colours to the different WebUIs so that we can always distinguish the stages well? Like changing the colours in Cloudera Manager?Thanks and Regards,Timo

Anyone have guide how to upgrade default Zeppelin in Ambari 2.7.4, i cant find it anywhereThanks in advance.

Hi ,In Apache NIFI  I am  consuming data from one Kafka topic and publishing that data to customer Kafka topic . For this iam using ConsumeKafka and PublishKafka processor respectively.If outgoing dataflow is idle for more than 5 minutes, the Kafka connection with customer should be terminated and  reconnected when there is  any new  messages consumed .how to do this with publish kafka processor ??

I'm encountering occasional Exception when using PUTSQL processor in apache Nifi.I have already Increased innodb_lock_wait_timeout But I'm still encountering the error occasionally. How can I fix this error? And what are the optimal configurations when using putsql processor to insert large volume of records into mariadb?Variable_name;Valueinnodb_lock_wait_timeout;300Error Snippet:2023-11-24 18:04:45,837 ERROR [Timer-Driven Process Thread-51] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c7255d17-018b-1000-143e-fc839881330c] Failed to update database due to a failed batch update, java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transaction. There were a total of 1 FlowFiles that failed, 14 that succeeded, and 0 that were not execute and will be routed to retry;java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transactionat com.mysql.cj.jdbc.exceptions.SQLError.createBatchUpdateException(SQLError.java:224)

I want to set up a Nifi flow that gets data from a public RSS feed and loads it into a data lake. This RSS feed updates irregularly and when it does update it overwrites previous content. What processor(s) should I use to get data from the RSS feed (close to) when it has updated? Is it as simple as using InvokeHTTP repeatedly, checking for a change in output, then loading into data lake if the content differs from the previous invocation? Is there another way if I don't want to make the HTTP request so frequently?

Hello all, i am new to Nifi , i have Processor to InvokeHTTP and get json file ,the json format example:"{"alertDate":"2023-12-03 11:01:17","title":"XXXX","data":"XXXX, XXXX, XXXX","category":1}" "i am using the Processor ConvertJSONToSQL  to send the data to PutSQL to insert to my sql data base ,my issue is :PutSQL Failed to update database for FlowFile due to The value of the sql.args.1.value is '2023-12-02 10:54:38', which cannot be converted to a timestamp" know i need to use the JoltTransformJSON but not sure what is the correct Jolt Specification i need to put,my DB columns  are like this can you please help? - remember i am new to Nifi 🙂thanks a lot! 

Hi all,The HDFS alerted that "Stale Configuration: Refresh needed" . I followed the suggestion, but i  hesitated when the following image came out.  I want to confirm that when i click "Restart Stale Configuration" button, the entire cluster will restart? Or only serveral servers will be restarted?

I am new to Jolt, I am trying to transform JSON to a particular format using nifi, where i am moving some objects into another object, I think i am closeThis is my JSON{ "relatedParty": [ { "id": "B2BSmallandMedium" } ], "serviceOrderItem": [ { "id": "e26205f4-e144-4ded-9e64-82ede0b26f4", "action": "add", "service": { "name": "cust", "state": "active", "supportingService": [ { "id": "e26205f4-e144-4ded-9e64-82ede0b26e99", "href": "serviceInventory/v4/service" }, { "id": "7313c885-a075-4188-a6bc-c3fbfb4c7422", "href": "activationAndConfiguration/v4/service" } ] } } ]}I am new to Jolt, I am trying to transform JSON to a particular format, where i am moving some objects into another object, I think i am close.This is my JSON{ "relatedParty": [ { "id": "B2BSmallandMedium" } ], "serviceOrderItem": [ { "id": "e26205f4-e144-4ded-9e64-82ede0b26f4", "acti

Hi, I'm trying to get this JOLT working, but in some cases, in the output is printing null values. The qty of itens inside RUBRICS array is dynamic. What am I doing wrong ? Payload [ { "ENTITY_DOCUMENT": "11", "ENTITY_ID": "", "ENTITY_TYPE": "TEST_CARD", "CLOSED": true, "CALCULATION_DATE": "2023-11-01", "RUBRICS": [ { "name": "provisao_TEST_card", "classification": "", "code": "", "amount": 110.08, "type": "", "business_unit": "TEST", "aggregation_key": "" } ] }, { "ENTITY_DOCUMENT": "12", "ENTITY_ID": "", "ENTITY_TYPE": "TEST_CARD", "CLOSED": true, "CALCULATION_DATE": "2023-11-01", "RUBRICS": [ { "name": "provisao_TEST_card", "classification": "", "code": "", "amount": 120.08, "type": "", "business_unit": "TEST", "aggregation_key": "" } ] }, { "ENTITY_DOCUMENT": "13", "ENTITY_ID": "", "ENTITY_TYPE": "TES

This question has been asked before (here) but never answered so I'm hopeful that today is my lucky day and someone knows the answer.   Because unfortunately, the documentation does not cover the topic.In the UpdateAttribute there is an option to Store State of an attribute by setting the value to "Store state locally".  But to do that I first need to set the "Stateful Variables Initial Value", but the documentation doesn't say how to do that, and everything I've tried fails.   The documentation provides an example for referencing an existing attribute variable, theCount, but there is no mention of how to initialize this attribute variable. I've attempted setting the initial value using:theCount=1{"theCount":1}${theCount=1}1${setStateValue('theCount')}     <- grasping at straws with this oneNone of these work.  When I look at the value of the attribute theCount it always just shows:  "Empty string set".I'm including a scree

Hi,I have managed to download the latest Nifi 2.0.0 M1 and I'm trying to run it on my windows 10 machine. Doing some preliminary testing I ran into the following issues:1- The system requirement indicates that (https://nifi.apache.org/project-documentation.html ) indicates that at minimum I need Java 17, but when I try to start nifi using run.bat I get the following error:  Error: LinkageError occurred while loading main class org.apache.nifi.bootstrap.RunNiFi java.lang.UnsupportedClassVersionError: org/apache/nifi/bootstrap/RunNiFi has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 61.0  It turns out it needs Java 21. Not sure if the documentation has not been updated or if Im missing something.2- After upgrading to Java 21, Im able to start nifi using default configuration, the log file doesn't show any error and default username and password ar

Hello,I want to integrate data into DynamoDB from Parquet files using NiFi (which I run in a Docker container). I fetch my files from AWS S3 using the ListS3 and FetchS3Object processors and then, as I understand it, convert the files to JSON using ConvertRecord and send the data using PutDynamoDB.I've tried configuring the AvroSchemaRegistry, ParquetReader and JsonRecordSetWriter controllers, but I'm obviously doing it wrong... I've tried using an UpdateAttribute processor too but nothing works. I don't really understand if I have to add the schema and where to add it. Thanks to anyone who can help me!

I converted non-ha hadoop cluster to ha hadoop cluster using below articlehttps://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.0/fault-tolerance/content/deploying_namenode_ha_cluster.html After conversion, getting "HiveServer2Error: ImpalaRuntimeException: Error making 'add_partitions' RPC to Hive Metastore" exception while trying to upload file What can cause this problem ?

Hi there,We've been using Nifi for awhile in our projects, mostly using Kafka and AMQP Client processors for receiving data; but recently a request came up for UDP, specifically DTLS (Datagram transport Layer Security).Looking at the existing UDP Client Processors, it appears that DTLS is not supported; or at least it's not clear for us how to do it. Is that correct?Are there any alternatives then? Maybe creating a custom ListenDTLS processor based on the existing ListenUDP? Could that work?Best regards

Hi there!I'm facing a connection refused problem in my apache nifi instance trying to reach an Elasticsearch instance using a queryElasticsearchHttp processor.The elasticsearch instance is reachable using CURL or in a web browser:But when I try to execute the processor, it fails due to a connection refused errorHere is the configuration I have in the processor: The server is not secured, but anyway I also tried to access with a valid user/password, facing the same error.Here my log:  2023-11-24 09:03:14 2023-11-24 08:03:14,682 ERROR [Timer-Driven Process Thread-4] o.a.n.p.e.QueryElasticsearchHttp QueryElasticsearchHttp[id=018b15d7-f6fe-11ed-b48b-3fd75e47c7d4] Failed to read from Elasticsearch due to Failed to connect to localhost/127.0.0.1:9200, this may indicate an error in configuration (hosts, username/password, etc.). Routing to retry2023-11-24 09:03:14 java.net.ConnectException: Failed to connect to localhost/127.0.0.1:92002023-11-24 09:03:14 at okhttp3.internal

I deployed NiFi and Kafka cluster of cdp public cloud  on the azure.And  I am trying to publish data to kafka(kafka-poc) user nifi(datahub-poc).I am facing "Topic XXX not present in metadata after 0000ms" when I set 'Use transaction=false'  And I am facing "Timeout expired after 0000milliseconds while awaiting  InitProducerId" after "Invaid transition attempted from state INITIALIZING to state ABORTING_TRANSACTION"  error when i set 'Use transaction=true'    Anything should I try to solve this problems?      

I have hdp cluster 2.6.5 (kerberized) on public cloud, and i need to access hdfs from outside (public access) through hdfs cli (not webhdfs)As the external hadoop-client host can't be enrolled through ambari i just downloaded hadoop 2.7.3 package and configured core-site.xml and hdfs-site.xml as bellowcore-silte.xml   <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://datalake-cstest9</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> </configuration>   hdfs-site.xml   <configuration> <property> <name>dfs.nameservices</name> <value>datalake-cstest9</value> </property> <property> <name>dfs.ha.namenodes.datalake-cstest9</name> <value>nn1,nn2</value> </prope

Hi everybody. When I use https://github.com/cloudera/hue/blob/master/tools/ops/hue_sync_ldap_groups_cron.sh to sync ldap groups with hue on my first cluster it's works fine but when I use on a second cluster I have one error  "LD_LIBRARY_PATH can't be found, if you are using ORACLE for your Hue databasethen it must be set, if not, you can ignoreexport LD_LIBRARY_PATH=/path/to/instantclient". Cause this error,  we can't retreive de ldap_config used later in the scripts. Does someone know how to fix this issue. Thanks 😉

CREATE TABLE IF NOT EXISTS emp ( eid int, name String,salary int). . . . . . . . . . . . . . . . . . . . . . .> COMMENT 'Emp Details'. . . . . . . . . . . . . . . . . . . . . . .> ROW FORMAT DELIMITED. . . . . . . . . . . . . . . . . . . . . . .> FIELDS TERMINATED BY '\t'. . . . . . . . . . . . . . . . . . . . . . .> LINES TERMINATED BY '\n'. . . . . . . . . . . . . . . . . . . . . . .> STORED AS TEXTFILE;INFO : Compiling command(queryId=hive_20190905115646_595a00a3-9a4c-4082-a81f-c426fb3e903a): CREATE TABLE IF NOT EXISTS emp ( eid int, name String,salary int)COMMENT 'Emp Details'ROW FORMAT DELIMITEDFIELDS TERMINATED BY '\t'LINES TERMINATED BY '\n'STORED AS TEXTFILEINFO : Semantic Analysis Completed (retrial = false)INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)INFO : Completed compiling command(queryId=hive_20190905115646_595a00a3-9a4c-4082-a81f-c426fb3e903a); Time taken: 0.094 secondsINFO : Executing command(queryId=hive_20190905115646_595a00a

If a have 2 original block and 1 Parity block in HDFS according to the EC policy, then If I lose 2 of this 3 blocks, How HDFS is going to Recover the missing data. Block A  xor  Block B = Parity BlockIf I lose any one of the block, I can see HDFS will be able to create the missing block. But if I lose any 2 block, I don't see how we will recover the missing 2 block  out of 1 block. If help me understand or redirect me to some helpful resources. 

Using Nifi 1.23.2 I'm not able to resolve the YamlSlurper class in a ExecuteScript processor (Groovy 3.0.17) nor in the ExecuteGroovyScript processor (Groovy 3.0.18). With a minimal script body likeimport groovy.yaml.YamlSlurperreturnI get the following after clicking Verify Properties:Perform ValidationComponent is invalid: 'GroovyScript' is invalid because org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: Script53405bee.groovy: 1: unable to resolve class groovy.yaml.YamlSlurper @ line 1, column 392. che.nifi.logging.ComponentLog;import gro ^ org.codehaus.groovy.syntax.SyntaxException: unable to resolve class groovy.yaml.YamlSlurper @ line 1, column 392. at org.codehaus.groovy.ast.ClassCodeVisitorSupport.addError(ClassCodeVisitorSupport.java:262) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1424) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:262) at org.codehaus.groovy.control.CompilationU

I have enabled the repository Encryption with protocol version 1, after that  there is below error in Base64EncodeContent  for some of the flowfiles  and other flowfiles gets successfully processed.Can someone please help on this. 2023-08-08 19:26:07,368 ERROR [Timer-Driven Process Thread-5] o.a.n.p.standard.Base64EncodeContent Base64EncodeContent[id=7ba6e988-8bc9-392a-1ae8-f38733a69144] Processing halted: yielding [1 sec]org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find content for FlowFileat org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:3701)at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3454)at org.apache.nifi.processors.standard.Base64EncodeContent.onTrigger(Base64EncodeContent.java:135)at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)at org.apache.nifi.controller.StandardProcessorNo

Hi everyone,I have a weird behavior with one of my DBCP Controllers.I'm trying to conncet to a Progress-Database on 10.254.249.24:31510. The controller is configured as shown: When I try to connect the Database (ie. with an "Execute SQL Processor" I get the following error message:Now the weird thing is: The shown Port number in the error mesage is not the Port I configured in the DBCP Connection.I thought it's a firewall thing, but when I check the IP via netcat, everything is fine:I also can succesfully connect to the database with the exact same parameters and *.jar-Files as shown in the first Screenshot when I use a SQL Client like "SQuirreL".Any Ideas?Best regardsMaik 

I want to create a flow In NIFI where i want to consumer data from 3 Kafka topics and produce that data into 3 different Kafka topics . each topic should produce data into unique topic . for example - kafka topics 1  -->  produce to topic Akafka topics 2 --> produce to topic Bkafka topics 3 --> produce to topic Ci want to use only one Consumer processor and one producer processor . Right now i am using 3 produce Kafka processor .Can anyone suggest better approach to do so and in more optimized way. Help me to reduce three publish processors to 1 , but it should be still able to consume from multiple topics and produce to multiple topics but dynamically , like topic1 produces data to topic A only and topic 2 to topic B and so on . i tried using expression language . kafka.topic is the attribute which contains consumer topics. so i added if else condition for different consumer topic but the output was not the producer topic but whole string me

I am trying to transform a json using jolt where i am moving the array inside serviceCharacteristic to outside of service {}, eventually i plan to move to respective positions  { "serviceOrderItem": [ { "action": "modify", "id": "0965f63b-db26-4397-b222-190a891838f0", "service": { "href": "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645", "id": "adaptiveNetworkSite", "serviceCharacteristic": [ { "name": "modifyPath", "value": [ { "op": "add", "path": "$.supportingService", "value": {} }, { "op": "sub", "path": "$.supportingService", "value": {} } ] } ] } }, { "action": "create", "id": "0965f63b-db26-4397-b222-190a891838f0", "service": { "href": "serviceInventory