Solved

Hello,I need your guidance on the following scenario:We have a SIEM (QRadar) infrastructure where Event Collectors receive logs from various data sources. These logs are correlated based on SIEM rules and use cases.we plan to send logs to the SIEM while also storing a copy in a Data Lake.Current Approach:We have structured the workflow as follows:DATA SOURCES → NiFi → Store in DATA LAKE & Forward to SIEM (using PUTTCP processor)Questions:Does NiFi allow segregation of data sources?Handling LEEF logs:logs reach NiFi with the original source IP but leave NiFi with NiFi’s source IP.Since QRadar first looks for the hostname in the payload (and if absent, uses the source IP), this could cause misidentification.Can NiFi be configured to retain the original source IP while forwarding logs, without modifying the original log (to comply with legal requirements)?Log Integrity & Authenticity:Does NiFi ensure log integrity and authenticity for legal and compliance purposes?LEEF Parsing:Is

I am unable to understand as I have just started with data engineering and saw a course using the HDP. I am not able to launch the image properly.Log link : https://drive.google.com/file/d/1MxawAFc68796kkh4V8I_dbILdH6bHOxQ/view?usp=sharing Configs:I changed the custom16 TCP port, host, and guest, to 1024. Please help me!

The JSON is example is :{"app": {"Id": "123","apply": [{"flag": "1","Product": [{"Code": "DZ9e","Line": "10000","status": "C",}{"Code": "EZ9e","Line": "19000","status": "C",}]}}I need output likeIDFlagCodeLineStatus1231DZ9e10000C1231EZ9e19000C I have used lateral explode/lalteral inline explode . But it is not giving value in separate rows.Anybody has any solution

Is there a way to overwrite an existing file with PutFile processor?Nifi version is 1.23.2

Hello, I'm facing some issues trying to install CM 7.13.1 on Ubuntu containers (happens in both 22.04 and 20.04) using a Postgres 10 14 server running on its own container.The install of both CM server and CM agents work fine, but when I start the cluster creation it fails during the CM agents startup. The following error is reported in the agent logs (cloudera-scm-agent.log): [11/Feb/2025 09:32:33 +0000] 8471 MainThread agent ERROR Heartbeating to node-1.cluster:7182 failed.Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/cmf/agent.py", line 1642, in _send_heartbeat response = self.requestor.request('heartbeat', heartbeat_data) File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/avro/ipc.py", line 132, in request self.write_call_request(message_name, request_datum, buffer_encoder) File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/avro/ipc.py", line 174, in write_call_request self.write_req

Hi, We are running nifi 1.24, some times when developers apply the changes it causes a discount for one of the nodes, and it is unable to sync with the other nodes.How can I have a re-sync button to re-sync the flows, to avoid the need to delete the flow directly from the affected node?

Hello,We are using a Nifi cluster with 5 nodes. Each node is a machine with 48 cores and 280-300 Gi of available memory. The issue we currently have is Nifi having trouble keeping up when a high number of tasks is required, 1,000,000+ tasks. The files are eventually transferred but errors do pop up in the mean time that seem to be irrelevant and more related to the flow trying to keep up. It's a simple flow with a GetFile > UpdateAttribute > PutSFTP > LogMesssage. I have increased concurrent tasks and batch sizes but this has little to no effect. 

I have been struggling with this for a week now.Here are my files:Authorizers.xml<?xml version="1.0" encoding="UTF-8" standalone="yes"?><authorizers><userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">cn=localhost,ou=ngc</property><property name="Initial User Identity 2">CN=localhost,O=ngc,L=San Diego,ST=California,C=US</property></userGroupProvider><userGroupProvider><identifier>ldap-user-group-provider</identifier><class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class><property name="Authentication Strategy">SIMPLE</property><property name="Manager DN">cn=localhost,ou=ngc</property><property n

@MattWho Is there anywhere I can find what types of filters I can use in the "User Search Filter" or the Group one?  I am trying to widdle down the amount of objects that are populated from LDAP into my Users. In my AD Server, users are in 2 different OU's.  So, for example, I'll have users in ou=Admin Accounts,ou=Domain Admins,dc=my,dc=network,dc=com but I'll also have users in ou=Regular Users,ou=Testers,dc=my,dc=network,dc=com.  And the only way to really pull both are to have my base search set to dc=my,dc=network,dc=com.  Doing this pulls computer accounts and all users, even users that would never need access (like service type accounts). I can filter out the computer accounts by using this filter: (objectCategory=cn=Person,cn=Schema,cn=Configuration,dc=my,dc=network,dc=com) But I'd like to also filter out the accounts, or OU's that aren't needed.  Or, if possible just use the 2 NiFi groups I have in AD (Nifi Admins/Nifi Users) wi

The error from my Spark job is++++Failing this attempt.Diagnostics: Application application_1738011234567_0014 initialization failed (exitCode=255) with output: main : command provided 0main : run as user is xxxxmain : requested yarn user is xxxxUser xxxx not found++++I read this post <https://community.cloudera.com/t5/Support-Questions/MapReduce-job-failing-after-kerberos/td-p/160273>. My group mapping configuration is hadoop.security.group.mapping = org.apache.hadoop.security.LdapGroupsMapping. I kinited xxxx before the job run. I added the AD user xxxx to an AD group hadoop. But I still got the same error.This online doc might be appliable <https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/security-authorization/topics/cm-security-authorization-ldap-group-mappings.html#ariaid-title3>I might need to add the flag -Dcom.cloudera.cmf.service.config.emitLdapBindPasswordInClientConfig=true to the variable CMF_JAVA_OPTS flag. But the documentation is for CDP 7.1.8 and does

Hi Team,I’m seeking your support in converting a string to a timestamp with microseconds in Hive. When using unixtimestamp, I notice that the milliseconds are being dropped. Could you please advise on how to capture milliseconds (and microseconds) correctly, and which data type I should use to store the value in Hive?SELECT from_unixtime(unix_timestamp('2024-11-05-10.03.17.872195', 'yyyy-MM-dd-HH.mm.ss.SSSSSS')) as times,'2024-11-05-10.03.17.872195' as valeFROM  

Hello,I have setup NiFI 2.1.0, and am trying to figure out how to allow users in a NiFi Microsoft AD Group to authenticate to the GUI.  I have it setup so users can log in if I add their LDAP account.  But, I'd rather just have users in the group I created in AD be able to login instead of having to go in and add new users to NiFi each time.Here are the configs I have setup for my authorizers.xml: <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Initial User Identity 1">ldapuser</property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authenticat

Hello Team,We are planning to upgrade our use of NiFi from 1.x to 2.x, and we are looking very much forward to try the new version.In NiFi 1.x we are using the "file-manager.sh" script in the toolkit for making backup and restore. It appears that this file is no longer present in NiFi 2.x.1) Has this file been removed from NiFi 2.x permanently, or do you know of any plans for adding it to NiFi 2.x in a future release? 2) If the file is not coming to NiFi 2.x will there be any replacement file for making backup/restore, or should we collect files for backup ourselves such as flow.json.gz?Best Regards

Hi,We want to stop Cloudera environment. After we stopped it via Management Console, we discovered there are liftie-***-liftie-infra-NodeGroup instances still running on AWS. Liftie Nodes are part of Scaling Group.Is it safe to stop those instances by hand via AWS console?We would be able to start them also by hand and later start Cloudera env.

Hello! I'm trying to convert float values on my flow files into integers. My Jolt Specification looks like this:[{"operation": "modify-default-beta","spec": {"progress": "=toInteger","setup_value": "=toDouble","proposed_setup_cost": "=toDouble","real_setup_cost": "=toDouble","invoiced_value": "=toDouble","last_recognized_value": "=toDouble","starting_date": ["=toDateTime", "yyyy-MM-dd HH:mm:ss", "coerceOrNull"],"last_progress_change": ["=toDateTime", "yyyy-MM-dd HH:mm:ss", "coerceOrNull"],"end_date": ["=toDateTime", "yyyy-MM-dd HH:mm:ss", "coerceOrNull"]}}] But somehow my "progress" value is still being outputted like a float. If I have the value 100.0, it is not transformed to 100, for example. I also tried using "=toInteger(@(1,value))", but that also didn't worked.

Hi, all,To improve the throughput of nifi, I enlarged my standalone nifi to a three nodes cluster. And, configured my data flow accordingly, now it works but the load balance doesn't work as expected.The load balancing is configured as round-robin, and it seems that all flow files are dispatched to three nodes at the beginning. However, later I noticed that the cluster slowed down dramatically, with only one node being processing and the other two running nothing. However, I think the unfinished flow files in the queue should be dispatched to other nodes.192.168.1.200 is the master node and the slowest node in the cluster. My nifi version is 1.12.1,  is it the expected behavior of round robin on this version?I want to maximize throughput, please tell me how I can do that.

Hello, Im currently setting new cluster.  When I try to start some role I get notifications that: Role is missing Kerberos keytab. Go to the Kerberos Credentials page and click the Generate Missing Credentials button.So I try to Generate Missing credentials and I got: failed script:/opt/cloudera/cm/bin/merge_credentials.sh due to:/opt/cloudera/cm/bin/merge_credentials.sh failed with exit code 1 and output of <<+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf-merged-out15742810764060677617.keytab+ shift+ ktutil+ chmod 600 /var/run/cloudera-scm-server/cmf-merged-out15742810764060677617.keytabchmod: cannot access '/var/run/cloudera-scm-server/cmf-merged-out15742810764060

I use CDP Private Cloud Base 7.1.7 and just enabled Kerberos security. I followed the setup documentation but could not proceed further than this step <https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-kerberos-authentication/topics/cm-security-kerberos-enabling-step7-prepare-cluster.html>.In short, I lost "supergroup" access to hdfs. Here are details.* I created an AD account mysuperuser@example.com and an AD group mysupergroup@example.com.* After Kerberos is enabled, I changed dfs.permissions.superusergroup=mysupergroup, and restarted the cluster. Certainly,  "mysupergroup" and "mysuperuser" do not exist anywhere in Hdfs POSIX permission settings.* I kinited mysuperuser@example.com, but got hdfs permission denied error. It looks like that Kerberos could not understand AD groups associated with the kinited account.* Then I changed dfs.permissions.superusergroup=mysuperuser, restarted all services, but still got permission denied error.I intended to use Ranger

Hi All, I want to find out what are the roles / groups assigned to a DB / Schema in Hive, how I can get those details ? please suggest. Thank you. Regards,PK

In Apache Atlas, I am trying to model the data flow of different processes. The issue I am having is that some of these processes share common DataSets but I don't necessarily want the different processes I am modeling to appear to be connected to each other.For example, in this lineage model, I want to show that there is an input of an XML Data source file into a process that outputs and transferred to another computer.{"entity": {"typeName": "datasystem_datatransfer","attributes": {"id":"b75af137-9279-4c73-be9f-0e37b686dde5","qualifiedName": "b75af137-9279-4c73-be9f-0e37b686dde5@datasystem_datatransfer","displayName": "Data Transfer Use Case 1","inputs": [{"uniqueAttributes":{"qualifiedName": "25b60fe5-891c-4c94-87ab-b075d838ec30@datasystem_datasource"},"typeName": "datasystem_datasource"}],"outputs": [{"uniqueAttributes":{"qualifiedName": "21781e1b-4b94-435b-be0a-141776267c4e@datasystem_computer"},"typeName": "datasystem_computer"}],"description": "Data transfer from Data Source to

Good morning,Is there a way to automate the creation of users, groups, and/or policies in NiFi? For example, I have configured LDAP authentication in NiFi -- is there a way to manage not only authentication but also authorization? It is not necessary for this to be LDAP-based.I was considering the possibility of using a script with the API, but I wanted to know if there is an "out-of-the-box" solution available.Thank you.

Hello all,I'm trying to convert many records from json to parquet with ConvertRecord processor.Most succeed, but convert fails for some files with this error. org.apache.parquet.schema.InvalidSchemaException: Cannot write a schema with an empty group: optional group pop_pools  I assume this is because some json files contain the following field. "pop_pools": {} As source record is log data, we can't modify it.Is there any way to avoid this error and convert the records to parquet format?Thanks,

Hello,I' trying to merge over 20000 parquet files into 1 file with MergeRecord processor.But it generates 7 files.It doesn't seem to be caused by Bin size, Bin age or number of records limitation.Looking into the generated file, it looks that similar log records are in the same file. So I assume MergeRecord processor looks into source record content and determine the target file to be merged into.Is this expected behavior?I don't set value in Correlation Attribute Name field.Thanks,

We have created new java service for establishing connection with hive of Cloudera kerborized cluster, but stuck because of below error. Error remains the same even we generate kerberos ticket for the hive user through its keytab at remote machine. Tried establishing manual connection using the hive beeline utility as well after generating Kerberos ticket but facing the same issue after several hit and trail .If any one having authentic document or steps to establish jdbc or beeline connection with remote hive for the kerboriszd cluster please share.Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)Detail error message is below (with masked host and server deatils>> -bin]$ ./beelineSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/maxiq/apache-hive-3.0.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/ma

This issue occurred right after I enabled TLS on my CDP Private Cloud Base 7.1.7. The client call to HBASE Thrift API failed at TLS hanshake.Below is the connection test output with the handshake failure.++$ openssl s_client -connect mycompany.com:9191CONNECTED(00000003)write:errno=0---no peer certificate available---No client certificate CA names sent---SSL handshake has read 0 bytes and written 287 bytesVerification: OK---New, (NONE), Cipher is (NONE)Secure Renegotiation IS NOT supportedCompression: NONEExpansion: NONENo ALPN negotiatedEarly data was not sentVerify return code: 0 (ok)---++My Thrift API port is 9191 (not the default 9090). This port worked well before TLS was enabled.There should be no certificate/ca issue because the Thrift (on the same node) UI over TLS works just fine. Below is the connection test output showing a successful handshake.++$ openssl s_client -connect mycompany.com:9095CONNECTED(00000003)depth=2 CN = MYROOTCA...---Certificate chain...---Server cer