Solved

Hello experts, I need to connect to my customer's kafka broker from apache nifi processors. They said security protocol is SASL_SSL (where is SASL is PLAIN) and provided username and password for SASL part.But they say they dont have any ssl certificate to share for _SSL, they informed they are using clnfluent cloud and that does not need ssl certs to connect.But when I configure nifi processor with SASL_SSL it mandatorily asks for ssl certificate. Any idea how can I get this ssl certificate to connect to confluent cloud kafka broker ?Or how Nifi can connect without ssl ?Any suggestion/help much appreciated ThanksMahendra

Hello All, I had secure Nifi and Secured Nifi-registry running on Docker.Both share same certificatesAble to access both UI's individually. Given static IP to Registry in Docker and subjectalternativenames with this IP while generating certificates. Given this static_IP:<port> in registry client.created localhost user : CN=localhost, OU=Nifi in both systems and given necessary permissions.Created few buckets and given all necessary permissions..         But when I try to do version control, it says no available buckets.   As i troubleshoot, I can see the logs that shows successful authorization to registry.. 2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.5.0.1 [CN=new_admin, OU=Nifi] GET https://localhost:8443/nifi-api/flow/registries/21d17428-0189-1000-4a0e-a91cc8cd2aa7/buckets2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.

After successfully running a 3-node NiFi cluster without any issues for the past three years, I am considering setting up an additional cluster. Upon checking the resource utilization of the 3-node ZooKeeper cluster, I noticed that the nodes have very low compute utilization. I would like to know if it is recommended to stick with a setup of 3 NiFi nodes and 3 ZooKeeper nodes, totaling 6 nodes, or if it would be more beneficial to use embedded ZooKeeper.

driver class: com.cloudera.hive.jdbc.HS2Driverjar : HiveJDBC42-2.6.11.1014the kerberos needs a keytab file, i wonder if this can be set by 'org.apache.hadoop.security.UserGroupInformation' class thks.

I'm new to NiFi, as you'll realize in this post. I'm trying to troubleshoot this issue where after I'm logging in with the provided password in the log file, I don't get past the loading screen. Here are things I've noticed: in nifi.app log file, I get the error:Archive cleanup completed for container default; will now allow writing to this container.in cmd, I have this error: Command Failed to set permissions so that only the owner can read status file [path]; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file  when opening any .bat other than run-nifi.bat as administrator, the window immediately closes.This is the process that's looping over and over in the nifi-app.log file My feeling is that these are having to do with some sort of permission issue. I've tried taking the folder off read-only but that potential solution alone has been a rabbit hole of issue

I had setup Secure Nifi Registry using Docker compose.Everything worked and I am able to open UI at https://localhost:18443/nifi-registryBut when i try to add buckets, I don't even see the settings icon. As I checked the logs i can see below error:  nifi_registry_container_persistent | 2023-06-29 15:23:01,075 INFO [NiFi Registry Web Server-23] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[CN=admin, OU=NiFi], groups[] does not have permission to access the requested resource. Unknown user with identity 'CN=admin, OU=NiFi'. Returning Forbidden response.nifi_registry_container_persistent | 2023-06-29 15:23:01,129 INFO [NiFi Registry Web Server-20] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.nifi_registry_container_persistent | 2023-06-29 15:23:01,190 INFO [NiFi Registry Web Server-16] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.Illegal

Hi, I am running a query on Hive through my Spark application using HiveWarehouseConnector.I want to use a particular YARN queue for Tez job launched by HiveWarehouseConnector (custom queue configuration at application level).I have tried following two ways:1. By using Spark conf and setting spark.hive.tez.queue.name = <queue name>2. By setting tez.queue.name parameter in the hiveserver2 URL as suggested in the following thread https://community.cloudera.com/t5/Support-Questions/Setting-yarn-queue-for-hive-with-beeline/td-p/161499 I am able to set the queue for beeline using the URL option. However, any of the options is not working for HiveWarehouseConnector. Can anyone please help in this regard? 

I have NiFi installed on an EC2 instance NiFi version - 1.22.0 I have a requirement to pull files text files from s3 bucket and write them locally in NiFi as PoC. I have configured an input_port and FetchS3Object and PutFile processor as shown in the screenshot. The Nifi ec2 intance already has access to the s3 bucket and I have checked that by using "aws s3 cp s3_uri ./" command to copy files from s3 bucket to ec2 instance and it worked, However, for some reason, it seems NiFi FetchS3Object processor is running into some issue. The Nifi-app.log havent been very helpful even though I enabled "debug" level on the processor. It showed no error or warn messages. The only messages the processor showed is on the gui as shown in the screenshot. I have shown the values I configured in configuration. Only specified bucket name, object key, which I initially set to *.evtx but then changed to * to get any file and region option. Did not specify access key or se

Hi guysI'm new to nifi and I need some help if possible.I am consuming from a Kafka Consumer that, when paused for some time, returns the json with several records, and I would like to divide them into separate files 1:1ConsumeKafka ResultSet: here is my json from 1 record    { "before":null, "after":{ "uid":"80faad95-3383-4093-b4c1-e1840f076eef", "nome":"yuri", "sobrenome":"feminino", "email":"yuri.feminino@garupa.co", "celular":"53999122007", "cpf":"3", "avaliacao":"AfQ=", "created_at":1687173349000, "updated_at":1687423717000, "promocode":null, "fcm_registration_id":"edzz9eK3QDOUx-jsmpix4K:APA91bFPjXEsD7h-fYzih-pbr-TMfnF21Qk6sPDQYoa52etUwuIn3jCv1nM7ts4QYQ6f2Wcg-S88KdZsAqcEk98ZUAle-xnvFreavHUjB9_JBjj5wKDfjIYx6Z0en65Po_9knVGpz1u9", "paypal_client_id":null, "paypal_refresh_token":null, "foto":"https://garupa-dev.s3.amazonaws.com/207e9b35-c4b2-4660-af4f-cc20a629b581.jpg", "stripe_cust

Hi all, Is it possible to install Nifi in the Cloudera CDP Private Cloud Base trial? Its not available from the list of services and when I try to add the Parcel (https://archive.cloudera.com/p/cfm2/2.1.5.1000/redhat7/yum/tars/parcel/CFM-2.1.5.1000-56-el7.parcel) it requires authentication. Its strange as just about every other service is there. Thanks. 

Hi.....   We are using following commands to get the updated records count in our table after Kerberos Authentication implementation and Ranger Installation.   We get count1 variable before update the table count1=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ')   We get count2 variable after update the table count2=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ')   entries=$(echo "$count2-$count1") we unable to get the "entries" value into variable. please support to fix this.

Hello, @MattWho I upgraded my local Nifi instance from 1.21 to 1.22 using brew once its upgraded all my flows/processor groups were removed and I tried to recover by coping the flow.xml.gz  from the archive the config folder but still I'm not able to get my flows is there any other way that I can get my flows back into the new instance? ThanksVas 

Dear Community! I have a simple flow in Apache NiFi 1.20:1. QueryDatabaseTableRecord -> 2. SplitAvro -> 3. ConvertAvroToJson -> 4. LogAttribute -> 5. PutDatabaseRecord Flow descrition:1. Read records from a table order by "updated" column (a datetime with millisecs).2. put each of records into single flowfile3. Convert a record into JSON4. Log JSON into log file.5. insert a record to different databaseEach of processors have only 1 single concurrent thread on SETTING page! Everything works fine, except sometime the order of the records will be changed according to "updated" column. Could you please find me the cause? How can be ensured the order of records right? 

Hi Everyone, I want to set up an outbound  Apache Nifi SMTP email server with Outlook so that It could send the outbound email. Does anybody have any suggestions for this? I have tried with the ListenSMTP processor but not sure how to set up the "Listening Port", and what other components it would require for that I have tried to follow this page: https://medium.com/@tspann/using-nifi-as-an-smtp-email-test-server-713677ac5b67But not so sure how to configure that processor with Outlook, or even need outlook at all for that email server. Thank you, everyone, for reading. 

when i use hdfs comands it display errors:   hadoop fs -mkdir /user/,,myfile,, or   "hadoop fs -put myfile.txt /user/,,/,,"   hadoop will display"Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:dr,,,"and so on,why?who can help me?

I'm trying to add users after setting up LDAP in Grafana on CDSW.Currently, only the default Admin account exists and is required to be added with email approval.We manage users using LDAP of AD (Acrive Directory).While attempting to add LDAP user settings in Grafana, we found that the current physical path of Grafana, "/var/lib/grafana/", does not exist.I tried to add LDAP settings to grafana.ini and ldap.toml in "/var/lib/grafana", but the file does not exist even if I search the entire server.When I accessed the Grafana pod directly and checked, path and files existed.Where can I find the grafana.ini and ldap.toml files?Is there any other way to solve this problem?Thank you for any help.

I want to use RouteOnAttribute to filter on a regex. To do so, I added two attributes in the Properties: PropertyValueprivate${ip:find('^(10|127|169\.254|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168)\.')}public${ip:find('^([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(?<!172\.(16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))(?<!127)(?<!^10)(?<!^0)\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(?<!192\.168)(?<!172\.(16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$')} This processor shows two errors, both the same: `Unexpected token ':' at line 1 column 4`Why is this an error? 

Hi I get this error when i try to create a new cluster or add a new host to an exists cluster..It's the basic way to do it through the cloudera manager, but I can't figure out how to fix it, all nodes can access the internet and have been working before in another cluster that was removedThis is an screenshot   the agent log in the node show me this: Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1390, in _send_heartbeatself.cfg.master_port)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__self.conn.connect()File "/usr/lib64/python2.7/httplib.py", line 833, in connectself.timeout, self.source_address)File "/usr/lib64/python2.7/socket.py", line 571, in create_connectionraise errtimeout: timed out[07/Jul/2021 18:21:00 +0000] 1352 MainThread agent WARNING Long HB processing time: 45.0577571392[07/Jul/2021 18:21:00 +0000] 1352 MainThread agent WARNING Dela

I run NiFi on a docker engine (official image). It works fine but now I would like as part of a flow to connect to a MySQL instance.This requires me to configure a Database Driver Class Name. The value I found in an example on Internet (com.mysql.jdbc.Driver) requires a .jar (sorry, I know nothing about Java). What is the proper way to make this file available to NiFi? (not on a docker level - I have a volume mounted and can put is over there, but on a NiFi level, i.e. how to make NiFi aware of this file)

Hi,I am hitting a kryoserializer buffer issue with the following simple line in PySpark (Spark version 2.4):df=ss.read.parquet(data_dir).limit(how_many).toPandas() Thus I am reading a partitioned parquet file in, limit it to 800k rows (still huge as it has 2500 columns) and try to convert toPandas. Kryoserializer.buffer.max is already at the maximum possible value:.config("spark.kryoserializer.buffer.max", "2047m")  What other ways are there to make it run (except of reducing the amount of rows even further down)? While in stage1 he is making many steps (approx 157), in stage2 he has only one step - and thus tries to juggle with a very large object.  The following in an excerpt of the error message: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 158, localhost, executor driver): org.apache.spark.SparkException: Kryo serialization fai

we are planning to separate cluster hosts into 2 physical racks, 10 hosts for each rack. our doubt is: as we only have two physical racks, is it still recommended to use the "rack" option in cloudera manager to separate the hosts?  

联系请登陆zoom：560 656 3972其中之一的报错如下：2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: rangerkms 2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: rangerkms 2023-06-21 07:57:03,612 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x488dce1cd620027, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:745) 2023-06-21 07:57:03,612 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1035] - Closed socket connection for client /10.233.68.145:49378 which had sessionid 0x488dce1cd620027 2023-06-21 07:57:03,668 - ERR

We are having escaping issues in some expressions. Just an example (using Nifi 1.21.0):ExpressionValue{"$${key}": "value"}{"${key}": "value"}{"$${key}": "${literal('value')}"}{"$${key}": "value"}As one can see, the second expression is not escaping the dollar sign as it should (from the docs, see "Escaping Expression Language"), and just stays as "$$".Any comments/hints would be much appreciated.

we are having trouble starting the "navigator metadata server" service, in the "stderr" log we have the error below: Conflicting collector combinations in option list; please refer to the release notes for the combinations allowedError: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.can help us with this problem?

I have been using multiple record oriented processors ConvertRecord, UpdateRecord, etc. in various parts of my flow. For example, my UpdateRecord processor takes about 16 seconds to read in a 30MB flowfile, add some fields to each record and convert that data to parquet.  I want to improve performance such that this takes a lot less time.  My infrastructure that I am working on currently is a 2 node e2-standard-4 cluster in GCP with a Centos7 operating system. These two instances have 4 vCPUs and 16 GB RAM and for each repository (content, provenance, flowfile) I have separate SSD persistent drives.  A lot of the configs in NiFi are the defaults but what recommendations would anyone recommend either from an infrastructure or NiFi standpoint to improve performance on these processors.