Q&A

Hello, After I tried to reassign some components ont ambari-server-2.2.1.0, the Web UI is stucked in this state and I don't know how to  I've restarted all the nodes but no clue what it is stucked.Thank you in advance for your help. 

Dear Sir,I am using apache nifi for  Getting file from SFTP serverBackup files another SFTP serverDelete the  all files from where getting locationsprocess file to call invokeHttp RequestSFTP images using winscp:After login folder locations:  1. Here is the diagram:2. ListSFTP properties:3. FetchSFTP properties3. Put SFTP propertiesI follow processors  bellow sequences: ListSFTP--> FetchSFTP-->PutSFTP-->UpdateAttribute --> InvokeHttpI did not get file from sftp server. What is the wrong of my configuration?   

I have a API which I need to make a call to get the response using InvokeHttpProcessor. Incase if response status code is not 200's then I need to do a retry for 3 times ( incase of 400's and 500's). I see that I can set a retry on Failure, NO retry and Retry relationships but that is not working as expected. How to configure the relationships such that there I can achieve retry of 3 times on Failure, Retry and NoRetry relationships. Also is there a way to see how many retries are performed and is nifi logging this information some where?

[begin_log]2024-01-08 18:57:00,406+0800|ERROR|pool-47-thread-1|o.s.s.s.TaskUtils$LoggingErrorHandler|Unexpected error occurred in scheduled taskorg.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown"at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100)at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170)at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:825)at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:783)at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717)at org.springf

Hi Everyone, I have been trying to explore the possibility of exploring apache ranger for centralized authorization in a microservice architecture.I'm think thinking of two possibilities:1) Integrating ranger plugin with all the services and authorize the request coming to them by intercepting the request. The drawback here is that I will have to implement this common functionality in all the applications.2) Creating a separate application for authorization, which will implement the ranger sdk to pull all the policies and configurations from Apache ranger and authorize a request using the ranger sdk, after authorization forward the request to the microservice.The drawback here is that I may not be able to use the existing plugins for Nifi, Kafka etc. when trying to use a centralized solution.What would be the best approach to implement, if someone has implemented something similar can you provide some thought on this? Or is there an even better way of doing this?

Hi guys! I have a question about Avro schema.My input flowfile are in json format and I want to convert it into xml. The method I use is using ConvertRecord Processor. In terms of ConvertRecord Processor, I use JsonTreeReader as record reader and XMLRecordSetWriter as record writer. Following picture is the properties of JsonTreeReader I set it up. Here is the json I want to convert: { "article":{ "article-meta": { "art_id": "me08001", "title" : [ { "lang" : "en", "title" : "A" }, { "lang" : "hr", "title" : "B" } ] }, "journal-meta": { "journal_id": "me9", "type" : "journal" }} If I have a lot of flowfile that contain this kind of format Json file, what Avro schema should I put inside AvroSchemaRegistry. I try a lot of online converter and it doesn't satisfy AvroSchemaRegistry when I do validation. I need the Avro schema so that the ConvertRecord can read my JSON flowfile.Thank you in advance   

Hello,In my HDF cluster of 12 Nifi nodes with version 1.8.0.3, 2 nodes takes around 40 minutes to restart. Other nodes takes only 5 minutes.Each node has 40 cores, 20gb of Heap and we have different volumes (6 for content_repository, 1 for flowfile_repository and 1 for provenance repository)I don't see any errors in logs.Do you know where this could come from?I can give you more information if needed.Thanks for your help

Hi,When my linux server reboots, Nifi automatically starts but automatically stops after few secondes.When I manually restart the service, that's okay.Could anyone help me to diagnose this issue ?OS : Cent OS 7Service :nifi.service - Apache NiFi  Loaded: loaded (/usr/lib/systemd/system/nifi.service; enabled; vendor preset: disabled)Service description (nifi.service):[Unit]Description=Apache NiFiAfter=network.target[Service]Type=simpleUser=nifiGroup=nifiLimitAS=infinityLimitRSS=infinityLimitCORE=infinityLimitFSIZE=infinityLimitNOFILE=65536PIDFile=/appli/nifi/run/nifi.pidExecStart=/appli/nifi/bin/nifi.sh startExecStop=/appli/nifi/bin/nifi.sh stop[Install]WantedBy=multi-user.targetExtract from the bootstrap log :2023-12-21 14:43:46,394 DEBUG [main] org.apache.nifi.bootstrap.Command Established connection to NiFi instance.2023-12-21 14:43:46,394 DEBUG [main] org.apache.nifi.bootstrap.Command Sending SHUTDOWN Command to port 44185I can't see any WARN/ERROR in bootstrap log or nifi log

Hi,  I am trying to download any version of CDH. As noted on the download page I must include a valid username and password associated with an active Cloudera subscription. Now, I am sure, that my company is a Cloudera Partner, so how do I know which of my colleagues has received the license key file? We are a big company so I am not sure where to start from the inside. Is this information available on the website? Thanks,Cisco

Hello, Studying the documentation below I found a good example of how to use Spark with JDBC to connect to external databases. https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html I already tested the example and it worked very well. However, when I looked carefully I saw that the password (its text) was being manipulated directly by the program. This way, from the tests I did, the password is open both in the Spark log and the connection, if it is intercepted on the network, the password will be open in open text. I'm looking for a way to make this connection safely. One mechanism that comes to mind is the hadoop credential provider that I already used with good old sqoop. So in sqoop the password was protected and no visible in any form, even if the connection was intercepted. Could anyone tell me what mechanisms I can use to make this connection securely, without exposing credentials? It can be a credential provider or something else.I didn't find anything like that in th

I changed a controller service property in a processor in a procees group which is under a version control with Nifi Registry. In the property I changed from controller service A to controller service B (the names and types doesn't really matter).Then when I check the "show local changes" I can see this change there.Then, I'm trying to "revert local changes", but nothing happens, this specific change is still there.When I'm trying to change it by myself back from controller service B to A, it does work.My question is, why the "revert local changes" doesn't revert this specific change?

NameNode is crashed when trying to restart it via CM due to: INFO Lock on /data/disk1/dfs/nn/in_use.lock acquired by nodename 26396@ip-10-2-0-224.ec2.internalWarning Encountered exception loading fsimagejava.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:251) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1166) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:757) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:642) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:956) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:931) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1666) at org.apache.hadoop.hdfs.server.namenode.NameNode.mai

Hello,I've noticed how the CML Terminal lets you install Python packages on the 'Net using pip.(A gcc compiler is even available!)Isn't that a security risk?Is there a way to only allow package installation from an in-house repository?Many thanks.

We’re working on securing NiFi and I’m coming across an issue where I intermittently get an exception. The environment is a two-node NiFi cluster which is configured to authenticate users using LDAPS, and authorise them using the FileAccessPolicyProvider / FileUserGroupProvider which we plan to replace with LDAPS. The login looks to be working. I think this is the authorisation part which is failing (although it works sometimes - before cluster voting completes). We have configured a certificate trust store for the CA (which is an internal CA), and a key store for the SSL cert. Note: We're using a san cert for SSL. Exception is… 2018-01-19 08:41:23,999 WARN [Replicate Request Thread-9] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to 10.101.50.5:8443 due to com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown 2018-01-19 08:41:23,999 WARN [Replicate Reques

 Hadoop commands take a while, usually a few seconds, to response.While the information on the webUI is presented with almost no delay.Does the information come from some APIs, logs or other?I'd like to know the data source of WebUI and how could I get it.Thanks a lot. 

Hi!We’re facing YARN-4714 and Swap memory issues, as shown on the attached printscreens.Increasing ‘spark.yarn.executor.memoryOverhead’ to default ‘0.6’ solved the issue for our spark application on NameNode.Do you think, it would be reasonable to set the value to ‘0.6-0.75’ in the file ‘yarn-site.xml’, cluster-wide for both NameNodes and DataNodes?Thank you for your response, in advance.

Good Morning,I'm flummoxed and worried. I have a significant flow that handles parquet files, and I can no longer make it work in version 2.0. Specifically, I used a ConvertRecord processor with a ParquetReader and a JsonRecordSetWriter.In the new version, 2.0, There doesn't appear to be a ParquetReader, anymore. There also doesn't appear to be any documentation about the ParquetReader being deprecated.Please help. This flow is critical to me. I am also open to suggestions on how to accomplish the same conversion in a different manner.

Attempting to add a new queue using YARN Queue Manager UI.  After saving, the following error message appears: Validation failed for Add queue operation! Error message:<?xml version="1.0" encoding="UTF-8" standalone="yes"?><RemoteException>    <exception>AuthorizationException</exception>    <message>Unable to obtain user name, user not authenticated</message>    <javaClassName>org.apache.hadoop.security.authorize.AuthorizationException</javaClassName></RemoteException> Environment:  CDP DC 7.1.4Kerberized:  YesTLS:  NoLDAP/AD:  Yes Checked Ranger logs and Audit logs, Yarn RM and security logs.  Couldn't find any related errors in those log files.  The Ranger Audit logs show the user accessing the YARN queue but does not show a denied result.  There are LDAP/AD related errors in Usersync...would that cause authentication issues for YARN Queue Manage

Hi,I have 3-node Nifi cluster using self signed certificate. and that is working fine. Now I want to run these 3 node Nifi UI (port 8443) behind Load Balancer. The changes I have in nifi.properties file is: nifi.web.proxy.host=localhost:8443,nifiinstance-group-1:8443,<load-balancer> All Nifi UI has same credential. and loadBalancer is using HTTP connection. Now, Nifi UI is coming up, but while passing the credentials it is throwing error:  there is no such error in nifi logs:3X.XX.XX.XX - - [28/Dec/2023:11:03:12 +0000] "GET /nifi/ HTTP/1.1" 200 21126 "http://3X.XX.XX.XX/nifi/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" Please guide me with the root cause if this problem and how to solve this?

There is a bug with the InvokeHTTP processor in version 2.0. I carried over my flows from version 1.23, which I upgraded to 1.24, and then I exported the flows as processing groups and imported them into version 2.0.  These flows worked fine, previously. In one, I have an InvokeHTTP processor that is sending the document to a Tika server for processing.  In version 2.0 there is a new property in this processor that was added. I did not put it there. The documentation makes no mention of it.The property was `Proxy Type` and it's value was `http`. I tried to set it as an empty string, but I continued to get a failure. The error I was getting was:18:17:26 UTC ERROR InvokeHTTP[id=dfae880c-b84a-3f2c-f1d9-941b8fc625eb] Request Processing failed: FlowFile[filename=random_20210909a.docx]: java.lang.IllegalArgumentException: Unexpected char 0x20 at 5 in header name: Proxy TypeEven after setting this property to an empty string, the error persisted. I remedied the situation by del

Hi all, I have built an Apache nifi cluster consisting of 5 nodes and a separate zookeeper server. After that I login to webui and use the ListenUDP processor. This processor opens for example 50000 port on all nodes and it can receive information on any node. For example I will send information to host1:50000, then it should distribute the information on all nodes (host2, host3, host4, host5). It does all this and everything works, but I have two questions.1) Does host1 participate in sending logs further down the processors or does it just receive and distribute?2) The Nifi cluster is called fault tolerant, but what if host1 becomes unavailable? Essentially it will not be able to receive information and distribute it to nodes. Is there any solution for this? So that when host1 goes down, the information can be transferred to host2?

at first, i want to reuse my model in HDFS, but it reported something wrong:it seems to be because the spark version is not set up correctly, I read the source code and found that spark calls sc.version when it stores the model. However, embarrassingly, the spark2.4.0 that comes with CDH6.3.2 that I use produces the following result: SPARK_VERSION is empty.   I need to use CDH spark to save and read the model, what should I do in this case, thanks very much!

Hey there,I'm trying to create a nested JSON using SQL in NiFi, but the output I'm getting has the nested part as a string. Here's the query I'm using:SELECTorder_id,JSON_ARRAYAGG(JSON_OBJECT('order_Item_Seq_Id', order_Item_Seq_Id,'product_Id', product_Id)) as order_itemFROM order_itemGROUP BY order_id;The output looks like this:[{"order_id": "10000","order_item": "[{\"product_Id\": \"10007\", \"order_Item_Seq_Id\": \"00101\"}]"}]But I want it to be a valid nested JSON like this:[{"order_id": "10000","order_item": [{"product_Id": "10007","order_Item_Seq_Id": "00101"}]}]I'm hoping for a solution using Apache NiFi or Jolt.Any help would be appreciated.Thank you!

Hi guys! I have json like that:{"id" : "20","user" : {"name" : "John","date" : ""}}I transform this date to Long using nifi langugage ${date:isEmpty():ifElse('',${date:toDate('yyyy-MM-dd HH:mm:ss'):toNumber()})} and after this i do in jolttrasformation something like[{"operation": "shift","spec": {"idTransakcji" : "id","date": "user.date",}},{"operation": "modify-overwrite-beta","spec": {"user": {"date": "=toLong(@(1,date))"}}}] and it works. But I have problem beacause when I have emptyString in this date i need to transfer it in null(not null in String) and i dont know how to do it using JoltTransformation or something else  so as not to lose what I already have and have null only when there is an empty string. 

Hi everyone,I'm currently encountering an issue related to passing request bodies in my workflow. Here's a breakdown of the situation:In my workflow, I have a total of 9 processors, all of which are InvokeHTTP Processors utilizing post requests. For each post request, I need to include a request body.The challenge lies in figuring out how to effectively pass the body using the InvokeHTTP processor. Presently, I am unable to accomplish this by utilizing the InvokeHTTP processor directly. We've attempted using GenerateFlowFile to pass the body, but it cannot be seamlessly integrated into the workflow. Specifically, I am unable to insert GenerateFlowFile between Processor 3 and Processor 4, for example.One potential solution we've considered is creating individual GenerateFlowFile instances for each processor outside the workflow. However, this approach introduces complications, especially when one processor is dependent on the output of a previous processor. For instance, if Processor 5