Ask the Community

Do we have any experience on typical disk usage rations for each of the repositories (Flow file, content, and provenance)? E.g. if Content requires 200 GB of storage, the provenance and flow file would require 20 GB (for typical flows)? Trying to use this information to decide how best to slice of a NiFi server which has 12 local drives. E.g. 8 drives allocated for Content, 2 for flow file, and 2 for provenance. Appreciate any thoughts!

I am currently migrating data from MSSQL to PostgreSQL. While fetching the data from MSSQL, I use the following query to ensure the time is in UTC: SELECT "time column" AT TIME ZONE 'UTC' AS "time column"  This gives me the correct result in UTC. However, when I ingest the data into PostgreSQL using NiFi's PutDatabaseprocessor, I notice that the time is being increased by 5 hours. The destination column in PostgreSQL is of type timestamp with time zone.when I manually run an INSERT command in PostgreSQL with the same time value, the time remains unchanged. The time shift only occurs during the NiFi ingestion process. Any ideas on what could be causing this discrepancy?

We have Cloudera Private Cloud 7.1.9When trying to install Data Services, we get this error:    "unable to resolve either the CM url, CDP console, or localhost"We verified DNS settings and the hostname is resolved correctly. What can the issue be? where can we find logs with additional details?

We recently observed an unusual scenario in NiFi where timestamps like '20251229043001', '20251230043001', and '20251231043001' were generated for December 29, 2024, December 30, 2024, and December 31, 2024. These timestamps correspond to future dates, which seems abnormal. I suspect that this issue may have occurred due to a change in the timezone settings of the NiFi server.Could you please assist in investigating the root cause of this discrepancy?  

Hi All,I am trying to delete my nifi existing process group through nifi-api curl but unfortunately it gives me error that "The revision must be specified" but everything specified as per the documentation but still not working, below is the curl command what I used for delete the process group(flow) to my target system and also I can able to delete the same through UI but not works with rest api.curl -k -X DELETE "https://localhost:28443/nifi-api/process-groups/836e216e-0195-1000-d3b8-771b257f1fe6" \-H "Content-Type: application/json" \-H "Authorization: Bearer  Token" \--data '{"revision": {"version": 0},"disconnectedNodeAcknowledged": false}'ERROR: 400 - The revision must be specified.Also find the below is the output of the process_groups:{'permissions': {'canRead': True, 'canWrite': True}, 'processGroupFlow': {'id': '187acb23-0195-1000-1908-1543c761f376', 'uri': 'https://localhost:28443/nifi-api/flow/process-groups/187acb23-0195-1000-1908-1543c761f376', 'parameterContext': {'

Hello together,i am new to Apache Nifi and i try to connect to a Siemens Logo.I have added the plc4x processors and added the connection string to the processor : Plc4xListenRecordProcessor 0.12.0Connection string: s7://<myip>:102?localTSAP=0x1000&remoteTSAP=0x2000i have these datapoints:{"word1410": "%DB1.DBW1410:INT","word1406": "%DB1.DBW1406:INT","word1412": "%DB1.DBW1412:INT","word1408": "%DB1.DBW1408:INT","bit10_0": "%DB1.DBX10.0:BOOL"}My question is how to add the datapoints?When i connect to the plc using node red everything works fine. So i dont believe that i have a configuration issue on the plc.When i run the flow everything looks to be fine but no data are comming.Can anyone give me advice how to configure the flow correctly?Thanks.David   

Hi. I'm new to working with Apache Nifi. Currently, I've Apache Nifi 2.3.0I'm trying to get access_token so that I can use it in future API calls. Currently, I use Postman to test getting access_token value and it works. I want to do this via Nifi.Here is what I setup Postman to get access_tokenI'm currently using Postman in order to call an API that generates access_token that I will use later to call other APIs. I want to do the same using Apachi Nifi 2.3.0. The Apache Nifi should generate a file called tokens.txt that contains the access token. Here is what I do in Postman:POST URL: https://URL/auth/tokenHeaders:Content-Type: application/x-www-form-urlencoded;charset=UTF-8 Authorization: Basic <BASE_64_ENCODED_CLIENT_CREDENTIALS>Body:grant_type: client_credentialsThe response is in JSON format and contains the access token, expiration time of the access token.Here is an example response:{"access_token": "eyJ4NXQjUzI1NiI...KtK5elB38rcAbgFtVP9A","token_type": "Bearer","expi

Hi AllI have a list of users that i have in an array.I am trying to create a job in shell script that creates a process-group for each user and apply policy to that particular process group so that only that user and nifi's super user can access or operate in it here is how the polices config json looks like       '[{"globalAccessPolicies":[{"resource":"flow","action":"read"},{"resource":"provenance","action":"read"}],"componentLevelAccessPolicies":[{"resource":"process-groups","action":"read"},{"resource":"process-groups","action":"write"},{"resource":"operation/process-groups","action":"write"},{"resource":"provenance-data/process-groups","action":"write"},{"resource":"provenance-data/process-groups","action":"read"},{"resource":"data/process-groups","action":"read"},{"resource":"data/process-groups","action":"write"},{"resource":"policies/process-groups","action":"read"},{"resource":"policies/process-groups","action":"write"},{"resource":"data

  Hello, We are forwarding NetFlow v9 data from a core switch and VPN to NiFi. In NiFi, we are using the NetFlow Listener processor to listen to the NetFlow traffic. However, the processor is unable to process the data, and the only information we could find in the logs after enabling debug logging is the following: 2025-03-06 16:25:29,502 WARN com.cloudera.nifi.netflow.ListenNetFlow: ListenNetFlow[id=861330c9-6ffc-1e81-9978-be1bba4f3524] Sender [/0.0.0.0:00000] Cached Template ID [259] not found2025-03-06 16:25:29,502 DEBUG com.cloudera.nifi.netflow.ListenNetFlow: ListenNetFlow[id=861330c9-6ffc-1e81-9978-be1bba4f3524] Sender [//0.0.0.0:00000] Version [9] FlowSet ID [259] Length [604]   Is there anyone faced such issue and help us to identify the cause of this issue and suggest possible solutions? Thank you for your help.    

Hi, Currently i have 1 TB of data in HDFS where i am trying to migrate into S3, i am using below command, when ever i run this command job runs very fast  for 3 hours then it slows down for a week still it is running, i started last week to run this job still it is running and very slow, is this expected behavior.  nohup hadoop distcp -Dfs.s3a.access.key="$AWS_ACCESS_KEY_ID" -Dfs.s3a.secret.key="$AWS_SECRET_ACCESS_KEY" -Dfs.s3a.fast.upload=true -Dfs.s3a.fast.buffer.size=1048576 -Dfs.s3a.multipart.size=10485760 -Dfs.s3a.multipart.threshold=10485760 -Dmapreduce.map.memory.mb=8192 -Dmapreduce.map.java.opts=-Xmx7360m -m=300 -bandwidth 400 -update hdfs:<....> s3a://<.......>

Running Spring Boot application running in Docker where the application is having a jive jdbc connection and writing to hive db.The application is working fine in standalone. The issue happens in the docker container. The docker container starts , gets connected to the hive db but throws exceptionexecuting query with Exception IllegalStateException: nested jar URLs not supported ..MalformedURLException..on the Hive driver

HI All, we configured three node nifi cluster by name node1. node2 and node3, with external zookeeper z1,z2, & Z3, in a Azure Virtual machine,Now Require to add the Storage resources to these nifi nodes, we are planning for below storage to choose, But require your help in choose to best fitment.1. Azure Blob Storage2. Azure NFS Storage3. Azure external storage (Standard HDD, Standard SSD)our use case to store the below storage,a. Nifi Prevenance Registryb. Nifi content Registryc. Nifi  flow file RegistryRequest you to suggest us best fit in these case.Data expected to grow 3TB to 5TB.        

Hi,I am using JoltTransformJson to convert a plain json to nested json in Nifi and getting null in the output. The same i/p and spec is working in Jolt demo. Below are the input and the spec Input:{"ts" : "2024-09-22T12:03:13.000Z","sourceDeviceId" : 4321,"sourceName" : "light1","sourceType" : "light","locationLat" : 201.34,"locationLon" : 101.5,"locationId" : 1,"recordDate" : "2024-09-22T12:03:13.000Z","_class" : "com.prutech.smartcity.entity.LightEntity","_id" : "66f27c487bd8652abecdb9c7","status" : "INACTIVE","power" : 0,"current" : 0,"intensity" : 100,"voltage" : 0}Jolt Spec:{"operation": "shift","spec":{"ts":"ts","sourceDeviceId":"source.deviceId","sourceName":"source.name","sourceType":"source.type","locationLat":"location.lat","locationLon":"location.lon","locationId":"location.id",         "recordDate":"recordDate","_class":"_class","status":"status","power":"power","current":"current","intensity":"intensity","voltage":"voltage"}}Please help.

Trying to demonstrate that nifi can listen to multicast traffic.I have this python script:import socketimport structimport sys# Multicast group detailsmulticast_group = '224.1.1.1'server_address = ('', 10000)# Create a UDP socketsock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)# Set the time-to-live for multicast packetsttl = struct.pack('b', 1)sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, ttl)try:    # Send data to the multicast group    message = b'Hello, multicast!'    print(f"sending {message} to {multicast_group}:{server_address[1]}")    sent = sock.sendto(message, (multicast_group, server_address[1]))except Exception as e:    print(f"Error occurred: {e}")finally:    sock.close()And then my listenudp processor is:  I can rece

Hello Everyone,I have successfully configured NiFi with Azure AD authentication by following https://github.com/benkelly/NiFi-Authentication-with-Azure-Active-Directory-Setup-Guide. NiFi is working as expected.However, I am facing an issue when trying to use the REST API. When I attempt to obtain a token using the /nifi-api/access/token endpoint with an AD user’s credentials, I receive the following error:"Username/Password login not supported by this NiFi."I need guidance on how to properly authenticate and use the REST API with my current Azure AD setup.Additionally, if creating a separate file-based user is required for REST API access, what changes should I make to the current configuration to support both AD authentication and a file-based user for API access?I am new to NiFi, so I would appreciate detailed guidance on resolving this issue.Thanks in advance!

I have a CDP On prem cluster on 719 parcels , I am interested to get the HDFS service status of last 7 days . I know one way is to get from CM UI But I am interested to get from DB and want to plot a graph once I have the status of last 7 days . Its not only for HDFS service but for all the services in the cluster, Can you give me table details and query to fetch that data ?

Hi all,I am trying to connect DBeaver from my windows 10 client PC to my CDP Hive cluster, but failing with an error message of following Caused by: com.cloudera.hiveserver2.support.exceptions.GeneralException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: GSS initiate failed.My setup CDP Version : 7.1.9 CDP OS : RHEL 9.1MIT Kerberos Version : 4.1DBeaver Version : 24.3.4.202502021521KDC : Microsoft Active Directory dbeaver.ini -startupplugins/org.jkiss.dbeaver.launcher_1.0.22.202502021521.jar--launcher.libraryplugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.2.1100.v20240722-2106-vmargs-XX:+IgnoreUnrecognizedVMOptions-Dosgi.requiredJavaVersion=17-Dfile.encoding=UTF-8--add-modules=ALL-SYSTEM--add-opens=java.base/java.io=ALL-UNNAMED--add-opens=java.base/java.lang=ALL-UNNAMED--add-opens=java.base/java.lang.reflect=ALL-UNNAMED--add-opens=java

I am having a tableid | resultInt | map<string, string>Id | result1 | {”street”:”abc”, ”phone”:”123”}2 | {”country”:”def”, ”phone”:”456”}3 | {”street”:”abc”, ”zipCode”:”726”}4 | {”street”:”abc”, ”phone”:”123”}5 | {”street”:”fgh”, ”phone”:”123”}I would like to get records with street=abc and phone=123I wrote the query like below:select id, rfrom table, table.result rwhere r.key=’street’ and r.value=’abc’and r.key=’phone’ and r.value=’123’The result is empty 

/etc/cloudera-scm-agent/config.ini[General]server_host=192.168.88.54server_port=7182max_collection_wait_seconds=10.0metrics_url_timeout_seconds=30.0task_metrics_timeout_seconds=5.0monitored_nodev_filesystem_types=nfs,nfs4,tmpfslocal_filesystem_whitelist=ext2,ext3,ext4,xfsimpala_profile_bundle_max_bytes=1073741824stacks_log_bundle_max_bytes=1073741824stacks_log_max_uncompressed_file_size_bytes=5242880orphan_process_dir_staleness_threshold=5184000orphan_process_dir_refresh_interval=3600scm_debug=INFOdns_resolution_collection_interval_seconds=60dns_resolution_collection_timeout_seconds=30[Security]use_tls=0max_cert_depth=9[Hadoop][Cloudera][JDBC][Cgroup_Paths]The Internet said that the problem of auto-tls, but I did not open, try a lot of online methods can not solve, can anyone help me solve this problem

Hi, I’m currently experiencing an issue with inserting data into a Kudu table using Spark. Although I can successfully read the data via Hue, the insert operation through Spark is failing. Upon reviewing the tablet details for the table, I noticed an issue highlighted with the tablets, but I'm uncertain about the next steps to resolve it.sudo -u kudu kudu cluster ksck tstewputil1Connected to the MasterFetched info from all 3 Tablet ServersTablet 0d811e918fd94b31aed4bfe337982ea8 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]Tablet 1a27865b4f0c4dedaf43f490272c4f24 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]Tablet 260af5f38836442f873d862d04de7615 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]Tablet f738d1e1455044899230ead58c0a23e

Hello Everyone, I am a Data Quality professional at a company, and currently, I am developing automated data testing. However, at this initial stage, I am facing difficulties in connecting Python to Cloudera/Hue/Hive.In manual scenarios, I usually connect to Cloudera/Hue through a web portal like this: https://demo.gethue.com/hue/editor/?type=hive.Is there anyone here who can assist me?Thanks you 

Hi @MattWho Need your help and suggestion here as i went through many of the Nifi related article here , since my use case is different need your valuable suggestion .1. i want to know how to use SSL\TLS to have https mode for nifi url 2. in our organization already we are generating signed certificate with SAN entries asnifi-dev.x.x.net [domain we try to access via browser]nifinode1.x.x.netnifinode2.x.x.netnifinode3.x.x.netzookeepernode1.x.x.netzookeepernode2.x.x.netzookeepernode3.x.x.netpost request raised we get a file nifi.crt file with this what else certificates are required\Needed?3. as trial basis tried already creating keystore and truststore but seems some issue 4. 5. 

Hi Everyone,I’m Sachin Duggal, and I manage a bag manufacturing business. I’m currently exploring HDFS data recovery methods and have come across a question: Is it possible to recover HDFS data using only the block data? If you’ve dealt with something similar or have any suggestions on how this could be done, I’d love to hear your thoughts.Thank you in advance,Sachin Duggal

Hello everyone, I am Emmanuel Katto currently working on evaluating the disk I/O of our CDH (Cloudera Distribution for Hadoop) cluster, which consists of several hundred bare metal machines. I would like to obtain the following values for each application within a certain period of time: total_io_mb mapreduce_inputBytes mapreduce_outputBytes These values, I believe, are logged in the YARN logs, but I’m not sure how to configure YARN or the logging system to ensure these values are written in the log files. So far, through Cloudera Manager, we’ve only been able to get metrics like the yarn_application_hdfs_bytes_read_rate, but that’s not enough for evaluating overall disk I/O. Could anyone share any advice or alternatives on how to extract these specific I/O values for each application? Also, if there’s a way to configure YARN or Cloudera Manager to write these metrics into the logs, I’d appreciate your insights. Thanks in advance! Regards Emmanuel Katto

HI @MattWho Good Day, we are experiencing new error called"  Connection refused: zookeepernode2" post setting up DNS  with the keystore and Password for NIFI which we generated internally using keytool, same keystore and password we are using in zookeeper as well.Attaching the Required details, Request you to please check and provide your valuable feedbackBelow is the web UI error. we setted up the nifi cluster in below formatNIFINODE1, NIFINODE2. two azure vm'sZookeepernode1, Zookeepernode2, Zookeepernode3. three Azure vm'splease find zoo,cfg details below.#zookeeper node3 has been choosen has an electorsudo bin/zkServer.sh status/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/zookeeper/bin/../conf/zoo.cfgClient port not found in static config file. Looking in dynamic config file.grep: : No such file or directoryClient port not found in the server configsClient port found: 2181. Client address: localhost. Client SSL: false.Mode: leader____