Q&A

Hi NIFI Team, We have currently installed NIFI with OIDC configuration which is working perfectly fine using the UI without any issues. But when using NIFI toolkit commands and providing Bearer token we are receiving ''Unauthorized error". Does NiFI provide any endpoint to get the OIDC Token separately which can then be used in the Toolkit commands? or please suggest if there is any other approach by which we can make a successful call to NIFI from Nifi Toolkit. Thanks in advance    

hello cloudera community, I have a problem in a datanode only in hdfs this datanode has 10 disks and in the hdfs settings (dfs.datanode.data.dir) is configured the 10 disks but for some time this datanode is not recognizing one of the disks configured only was already done the restart of the datanode and even so did not back to recognize looking at the datanode logs, it is not really recognizing the disk, only the 9 disks these disks are also being used by other services in the cluster and these other services are able to recognize the 10 disks without any problem, only hdfs is having this problem could you help us to solve this problem?

PutSQL is penalizing the queue and  no exceptions are found in logs2023-08-09 08:24:07,045 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] Not enough FlowFiles for transaction. Returning all FlowFiles to queue2023-08-09 08:24:07,046 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] No incoming FlowFiles.2023-08-09 08:24:07,047 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] Not enough FlowFiles for transaction. Returning all FlowFiles to queue2023-08-09 08:24:07,047 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] No incoming FlowFiles.I know if there is SQLException, Flowfile will be forwarded via failure else the flowfile will be added back to the queue for any other exce

Quoting directly from the official docu: FetchSMB: "Fetches files from a SMB Share. Designed to be used in tandem with ListSmb."My workflow is pretty simple, I have ListSMB which reads from a shared network directory, connected to FetchSMB which should get 1 file (the processor forces you to put the path of a specific file or it won't work) and finally PutHDFS to write all the files in a distributed file system (hadoop).I don't / can't understand why whenever I use List and Fetch in tandem the pipeline will just take all the files it can find on the directory specified in the List processor but won't write to the HDFS the single file I specified on the Fetch, instead it will write on the HDFS all the files from the List. The fetch, directly connected to the HDFS, should pass only a single file. What am I missing here ? What is the purpose of the fetch if it is "by-passed" by the list processor ?On top of that, all the file written to the HDFS will pick the same weight (KB or MB) of the

Good afternoon!  In planning out how we reset an environment when we have a refresh of one of our source systems, I need to ensure that any stateful processors (like QueryDatabaseTableRecord) have their state cleared so that they pick up all new records. Is there a way to find all processors storing state?  Or, even better, to just destroy the state of all processors on the NiFi instance? Thanks.

In Apache NiFi, the flow file got stuck permanently in queue when the downstream processor has some errors.I am looking to route the flow file to failure relationship so that I can handle the failure and extract the failure log. But in some cases the flow file got stuck in queue permanently which makes routing to failure relationship impossible.Is there a way to resolve the issue of flow file getting stuck? If not, is there a way to get notified that a particular flow file waits in which queue and which downstream processor in the processor group has error, so that I can track the failure?  Downstream processor's settings and configuration:   Note: Nifi is the part of the application we use to process files. We are looking for handling failure in Nifi processors without explicitly visiting NiFi UI every time (Getting to know about failures in NiFi processors from the system instead of visiting NiFi UI ). In the case mentioned above, it is impossible to route to

What are the installation steps for APache Phoenix

Has anyone figured out how to connect to SSE with OAuth2? I don't see any option in the JettyWebSocketClient or in the SSL context providers. 

Hi,We have a solution in our flow which is currently running on 1.16.3 and we are referencing parameters dynamically for some schema validations. Now we have upgraded NIFI to version 1.21.0 and this functionality of function "evaluateELString()" not working as expected. Example -  Using above NIFI expression language, we used to reference parameters dynamically but now we are getting null values although parameter is not sensitive.  I find out this was changed in NIFI version 1.19.0 to not expose sensitive parameters but why have they change it for all the parameters - NIFI-10674I can see a JIRA raised for similar issue here - NIFI-11627Can someone please help us over here ? I do not want to hardcode this thing as we have 10's of schemas in parameters and I do not want to hardcode for each one.

When using the ValidateXML processor with an XSD, how do I capture the reason for the invalid XML so I can troubleshoot instances when the incoming XML is invalid?   Unfortunately I don't have access to a validation tool in the environment where NiFi is running so I need the feedback from the ValidateXML processor. I appreciate the help.    

I am using Nifi 1.21.0.I need to get the data from Huawei Object Storage bucket using Nifi. are there any processors available in Nifi to do this? How to do this ?

I have record from kafka like this {"datetime": "2023-08-03 10:46:50", "long": 106.8451, "lat": -6.2146, "city": "Jakarta", "status": "unknown", "nickname": "unknown"} I use Convertrecord JsonTreeReader with Schema Access Strategy Infer Schema and to change Parquet before I stored in Hive But in my table in Hive the datetime change from 2023-08-03 10:46:50 to 2023-08-03 17:46:50 How to Handle that, I want to stay with same kafka record? I live in GMT+7

Hello Everyone Good Day! When I change the old password of "<property name="Manager Password">NewPassword</property>" in authorizer.xml to new password for our maintenance.I stop nifi services and I change password in authorizers.xml and login-identity-providers.xml after I change I start the nifi services but I encountered the error below.   ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Error creating bean with name 'niFiWebApiConfiguration': BeanPostProcessor before instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'metaDataSourceAdvisor': Cannot resolve reference to bean 'methodSecurityMetadataSource' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.method.configuration.GlobalMethodSe

is there is processor  from oracle  and oracle cdc now in cloudera ???

hi, i have the following problem - i am creating myself quite a few json files via AttributeToJSON processor. Here, as destination, I have set flow-file content. Now I would like to combine them into one csv file. I used MergeContent in my flow first to combine them into one json and then ConvertRecord to turn it into csv. However, it does not create one file for me, but many, I do not know if the problem lies in the MergeContent processor or ConvertRecord. Or should I approach it differently? There are my settings in MergeContent

Hi, We have 3 datanodes but the other 1 is not having expected number of blocks compared to the other 2. Also, the count of under-replicated blocks is high.  Datanode low block logs: 2023-08-01 07:31:30,134 DEBUG datanode.DataNode: isDatanode=false, isClient=true, isTransfer=false2023-08-01 07:31:30,134 DEBUG datanode.DataNode: writeBlock receive buf size 530480 tcp no delay true2023-08-01 07:31:30,134 INFO datanode.DataNode: Receiving BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972 src: /10.128.66.223:58748 dest: /10.128.64.127:98662023-08-01 07:31:30,134 DEBUG datanode.DataNode: BlockReceiver: BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972storageType=DISK, inAddr=/10.128.66.223:58748, myAddr=/10.128.64.127:9866stage=PIPELINE_SETUP_CREATE, newGs=0, minBytesRcvd=0, maxBytesRcvd=0clientname=DFSClient_NONMAPREDUCE_378619176_53, srcDataNode=:0, datanode=hdfs-datanode-0.hdfs-datanode.metering.svc.cluster.local:9866requestedChecksu

hello, everyone;    I am a freshman as a CDH manager user, My CDH version is CDH 6.0.1, my boss let me configure our CDH cluster to support "docker on yarn", but i found that feature not be supported in CDH 6.x manager, that means i have  a only way to support that with configure that by customer configure    so I decide to  edit yarn-site.xml and container-executor.cfg manually at the node of cluster，but I found that will be covered automatically by cm。    how can i modify the yarn-site.xml and container-executor.cfg and will not be covered by cm.    thx

my  CDH Cluster has been online more than half year,  all the machines and service is normal.   but recently , there is one host agent always under bad healthy, and Cloudera Manager will show the red color on this host.   then I check the agent log as below: [16/Nov/2019 02:20:11 +0000] 1142 MainThread agent WARNING Long HB processing time: 9.52771306038[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Long HB processing time: 34.8044610023[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Delayed HB: 19s since last[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Long HB processing time: 22.3401939869[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Delayed HB: 7s since last[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Long HB processing time: 36.7459609509[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Delayed HB: 21s since last[16/Nov/2019 02:22:24 +0000] 1142 MainThread agent WARNING Long HB processing t

Installed hadoop 3.3.5. On Single node server. hbase-site.xml    <configuration> <property> <name>hbase.master.namespace.init.timeout</name> <value>2400000</value> </property> <property> <name>hbase.regionserver.executor.openregion.threads</name> <value>20</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://localhost:54310/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>localhost</value> </property> <property> <name>hbase.master.ipc.address</name> <value>localhost</value> </property> <property> <name>hbase.master.hostname</name&g

I have a json {  "REQUESTID" : "379",  "REQUESTTYPE" : "V",  "REQUESTCONTEXT" : "B2B",  "CARDNUMBER" : [ "5537290000000511", "5537290000000522" ],  "EXPIRYDATE" : [ "08/20", "09/21" ],  "EMAIL" : [ "John.Jones123@abcmail.com", "Jason.456@gmail.com" ],  "PHONENUMBER" : "11234565555"}CARDNUMBER, EXPIRYDATE and EMAIL are varchar in sql table when Json is passed to covertToSQL processor, I am seeing empty string in the attribute for these columns in prepared statement (Insert)

What is the default port used by the Apache NiFi processor ListenFTP for the data channel in active FTP mode? For the control channel the default port is 2221. Is the default port for the data channel in active mode 2220? The default ports for FTP are 21 for control channel and 20 for data channel. Because ports below 1024 require root access, the NiFi default port for control channel is 2221. 

Hello everyone, we are currently working on automaticlly patching the operating system of our CDP Private Cloud Base cluster. We enabled/configured our cluster to be HA, but for two components (YARN Job History Server and Spark History Server) we couldn't find any solution. We designed our automation to be split over the timespan of three days. Starting at 8 PM each day we update one of the master nodes, one of the edge nodes (e.g. know) and one third of the worker nodes in sequential order. Our problem is how to handle an unexpected failure. Is it possible to wait until the next day or is the availabilty in danger if one server (with the mentioned compontens) isn't booting up etc.  How important are the YARN Job History Server and Spark History Server for the successful execution. As far as I understand both are only required for storing/providing logs for executed applications. Do I miss anything here? 

Hello, I´m using QueryDatabaseTable with time_stamp max value column to get in real time +2k records per day. Despite I read all docs that I found I can not understand how it works. It supose to do a select every time a new record is inserted in the table but if I take a look in the postgres log I can see that the query is executed every second in search for new records.I though that this processors works like a trigger but it seems like it doesn´t and I´m afraid that this can affect database performance.Can somebody explain how this processor internally works?  Thank you in advance  

I am using redhat 7.9 OS, and CM 7.7.1.base on the  CVE-2022-41946 security issue,https://www.postgresql.org/about/news/postgresql-jdbc-4251-4243-4238-42227jre7-security-update-for-cve-2022-41946-2551/I need to upgrade the cloudera postgresql jdbc driver files I saw the jdbc driver file is the following for the CDH-7.1.8.:/opt/cloudera/cm/common_jars/postgresql-42.2.14.jre7.6ce27816399941030df9ad06f89fa77b.jar/opt/cloudera/cm/common_jars/postgresql-42.2.24.jre7.a7b0f155b668470fb4e212e79724cc7d.jar /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/postgresql-42.3.2.jar/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/postgresql-42.3.3.jar/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/postgresql-42.2.16.jar I think just do a replacement for those jar files is a risk. How can I upgrade those file to PostgreSQL JDBC Driver version 42.2.27, 42.3.8, 42.4.3, 42.5.1, or later. Thank you!!

hi i have such a problem, i have a flow that downloads a csv file with data and there are columns like:,system name, surname, phone, mail, card name, proffesion. The file looks like this only data is more of course systemnamesurnamephonemailproffesiondepartmentABAJohnWane77777777j.wane@gmail.comaccountantaccountancyBCALillyBlack89898989l.black@gmail.comrecruiterhrI am reading this file record by record and for each record I need to query another system via REST using first name, last name, profession and department, the query looks something like this:https://system address/data/?name=${name}&surname=${surname}&proffesion=${proffesion}&department=${department}I use Invoke HTTP for thisand now this system should return me json with data:{"date": [{"name": "John","surname": "Wane","profession": "ACC","department":"ACC01"}}As you can see in the answer I get a different profession and department. How can I now create a json for each record that will contain data from the c