Solved

Hello,Recently I setup NiFi 2.0.Everytime I start NiFi, I get below warning messages in /var/log/messages. Dec 13 12:07:58 myserver nifi.sh[208489]: WARNING: A restricted method in java.lang.foreign.Linker has been calledDec 13 12:07:58 myserver nifi.sh[208489]: WARNING: java.lang.foreign.Linker::downcallHandle has been called by org.apache.lucene.store.PosixNativeAccess in an unnamed moduleDec 13 12:07:58 myserver nifi.sh[208489]: WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this moduleDec 13 12:07:58 myserver nifi.sh[208489]: WARNING: Restricted methods will be blocked in a future release unless native access is enabled Is there any way to supress these messages?Here are product versions.Red Hat Enterprise Linux 8.8NiFi 2.0.0Java OpenJDK 23.0.1.0.11Regards,

Hi Team, I am implementing the CSFLE(Client Side Field Level Encryption) in the Nifi for the Kafka data. I have written Groovy script to get the logic like Consume/deserialize/Encrypt the data. The script is working as standalone. However, if i place it in ExecuteScript processor ,i am getting Caused by: javax.security.auth.login.LoginException: No LoginModule found for org.apache.kafka.common.security.plain.PlainLoginModule Error. It seems it is compatibility issue in the kafka-client jar file. However, i could not find the fix. Please provide your input on this.My groovy script is below.  import org.apache.kafka.clients.consumer.ConsumerRecordimport org.apache.kafka.clients.consumer.ConsumerRecordsimport org.apache.kafka.clients.consumer.KafkaConsumerimport org.apache.kafka.common.serialization.StringDeserializerimport java.time.Durationimport org.apache.kafka.common.security.plain.PlainLoginModuleimport javax.security.auth.login.Configurationimport javax.security.auth.logi

Hi Team, HDFS was working before kurberos enabled , noww post enabling kurberos HDFS is not working. Getting error in HDFS GUI, HDFS is running in cloudera manager.Name node is showing, Test of whether the NameNode is in safe mode.```Bad : This NameNode is in safe mode.```Error:```HTTP ERROR 502 Process information not available.URI:STATUS:MESSAGE:SERVLET:/cmf/process/1546341404/logs502Process information not available.Spring MVC Dispatcher Servlet```

In your article:https://docs.cloudera.com/management-console/cloud/environments-azure/topics/mc-using_azure_database_for_postgresql_flexible_server.htmlIn the middle of this article, you mentioned:"In general, when registering an Azure environment and creating a Data Hub, you can choose to use Flexible Server, Flexible Server with Private Link, Flexible Server with Delegated Subnet (deprecated), Single Server (deprecated), or Single Server with Private Link (deprecated),..."I believe that the Flexible Server with Delegated Subnet is NOT deprecated.

I have 2 NIFI cluster, one NIFI cluster which is running as helm in EKS and another NIFI cluster which is deployed  with 2 linux servers, We want to enable Site to Site connection between EKS NIFI cluster to Remote NIFI clusterwhat is the best way to connect via Remote Processor group, what configuration to be set for RPG

Hi everyone,I am facing an issue while running a Sqoop import job. The process gets stuck at:INFO mapreduce.Job: map 0% reduce 0%The job does not progress further. Additionally, I see the following message in the logs:INFO conf.Configuration: resource-types.xml not found. INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.Despite this, there are plenty of available resources in the cluster, so resource allocation should not be the problem. I also have 3 healthy nodes. I’ve tried various troubleshooting steps, but I haven’t been able to resolve the issue.Could the "resource-types.xml not found" warning be causing this problem?If so, where should the resource-types.xml file be placed in a Cloudera setup?Any help or suggestions would be greatly appreciated!

All OS disks on the cluster server are broken,Therefore, I need to install a new OS and set up hdfsHowever, all data disks are considered availablename node has "edits" file and "fsimage" file,There are "blk" and "blk meta" files in the data node.Is it possible to restore hdfs data based on block data in this situation?

Hi guys,I want to convert Nested JSON to Flattened JSON using JOLT transform.Nested JSON:{"data": {"getBusinessRelationListing": {"edges": [                {                    "node": {                        "id": "1964730",                        "name": null,                        "code": "CN02",                        "bejoCompany": true,                        "legacyVendorId": [],                        "locatonType": [                            "Commercial"  &n

Hi Everyone,Struggling with transforming the JSON flow file, would appreciate help from the community!input json flow file:[{"firstName": "John","LastName": "Malkovich","MiddleName": "J","nameType": {"type1": "Other""type2": "Primary"}}, {"firstName": "Johnny","LastName": "Cage","MiddleName": "M","nameType": {"type1": "Basic""type2": "Primary"}}]expected output json flow file:{"fileName": "file.xml","count": "2","individuals": [{"firstName": "John","LastName": "Malkovich","MiddleName": "J","nameType": {"type1": "Other""type2": "Primary"}}, {"firstName": "Johnny","LastName": "Cage","MiddleName": "M","nameType": {"type1": "Basic""type2": "Primary"}}]​}Jolt  specification :[ { "operation": "default", "spec": { "fileName": "file.xml"", "count": "=2" } }, { "operation": "shift", "spec": { "*": { "@": "individuals[]" } } } ] gettingError:TransformException: The Spec provided can not handle input that is a top level Json Array. 

Hi , My Nifi loads all the AD groups from my LDAP server , but it doensn't load any member information of it . As per the image , one can see that the Members list is empty.How is it possible to get the users in the group too ? I can add it manually , but I would like the ldap to be able to retrieve the users as per the group. 

hello,I'm trying to read a parquet file using the ConvertRecord processor and I'm getting the error:ConvertRecord[id=e599cd8f-9a1d-3134-4daf-af7bc91cdd57] Failed to process FlowFile[filename=objectTable_tract_5074_DC2_2_2i_runs_DP0_2_v23_0_1_PREOPS-905_step3_31_20220314T212509Z-part0_output.parquet]; will route to failure: org.apache.avro.SchemaParseException: Illegal initial character: 0In my file the columns are numeric and the first one starts with 0 (zero).

Hi All,So I have a flow that at the end changes particular status through API. The direct upstream processor is putEmail, which is supposed to notify dynamically generated contacts with data.After this e-mail is sent, the requirement is to wait 24 hours before using the API call. I have had a hard time understanding the Wait processor and its relation to signal release. ControlRate does not really work (or I cannot set it up properly). What would be a suggested way to deal with such a scenario?

Hello,I'm triyng to sort record in parquet file like below in order of timestamp.{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:58.440995\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message AAA\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:58.429579\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message BBB\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:08.441709\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message CCC\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:08.428501\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message DDD\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:48.440624\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message EEE\"}I confugured the following query in QueryRecord processor and it passed validation in the processor.SELECT * from flowfile ORDER BY JSON_EXTRACT(Src_obj__event_metadata, "$.t

I am facing an issue with Notify and Wait starting from version 2.0.0. In order for them to listen to each other in version 1.28, I need to configure them with the MapCacheClientService 2.0.0 and MapCacheServer 2.0.0 services. However, in version 2.0.0, although I can still find both services in the documentation, in practice, the configuration only includes the MapCacheClientService 2.0.0 service, and I can no longer find the MapCacheServer 2.0.0 service.Using Wait and Notify in version 2.0.0 seems much more difficult compared to previous versions. I spent two days trying to figure it out, but I haven't made any progress. If possible, could you please guide me on the correct configuration so that Wait and Notify can work properly, like they did in the older versions?

How to configure OIDC authentication in nifi-registry 1.19.1?When configuring secure access ssl + oidc does not display the UI when trying to access. In the log I don't see errors just warnings, one of them is this:o.s.l.core.support.AbstractContextSource Property 'userDn' not set - anonymous context will be used for read-write operationsIs it possible to access this version through OIDC?

My NiFi flow fails when encountering a CSV with a column containing a double quote within a string, such as:"Protection from Abuse Order on file against dad Raul Martinez Lopez." NO CONTACT WITH DAD. 12/16/2014 kb.The error is occurring at the Record Reader stage. Has anyone else successfully handled CSV data with embedded double quotes?From csv:My record reader config  

Hi at all,I got a timeseries table with existing range partitions based on timestamp field, that covers years from 2022 to 2024 month by month. The problem is that last partition has an open bound (PARTITION 1717200000000 <= VALUES) and contains data from "1717200000000" (Saturday, June 1, 2024 0:00:00) to now, and is still storing live data with obviously performance loss during query.From the docs I haven't found a command that lets me split last partition in other partitions in order to update them, but the only way seems to drop last partition and continue adding since this time, and the problem is that by dropping last partition I will lose three and more months of data, that is obviously not possible.Is there an alternative to prevent data loss and adding range partitions to improve performances?Thanks in advance

Hi Team,I am trying to implement for the scenario where the events should sleep for 5 minutes and let all the flow files are queued up, then check the count of the files if it is greater than 10 route to failure else route to SUCCESS.I did in Execute script processor as below. However, i am trying to avoid the executescript for this and try to use Native nifi processors.====== Executescript ===============Thread.sleep(300000)def flowFiles = session.get(100)if (!flowFiles || flowFiles.size() <= 10) {session.transfer(flowFiles, REL_SUCCESS)} else {session.transfer(flowFiles, REL_FAILURE)}  

We have a Nifi cluster with 3 nodes running in docker containers that are always processing something. I included keys in the truststore.jks file of each node and for the changes have effect, I need to restart each node at a time, so that Nifi could still be running and we don't lose any data . There a way to restart them safely? ChatGPT said that it's possible to execute inside the container the command nifi.sh stop, but I don't know if this will cause data loss.

Hello!Input flowFile has column "date_time" with timestamp-millis type The task is to update each value of column, by substracting 3 hours from datetime value. In fact I need to transform datetime value from timezone GMT +3 to timezone GMT +0For example, value "2024-11-24 19:43:17" to be transformed in "2024-11-24 16:43:17"I tried to use UpdateRecord with following script:           format(toNumber((/date_time):minus(3600)), "yyyy-MM-dd HH:mm:ss")And, as expected, transformation failed with exception: "Unrecognized token 'minus'..."Operation "minus" applied to attributes values. And "date_time" is not an attribute.As an alternative, I considered next way:1. SplitText for splitting one FlowFile into N FlowFiles, where N - is the number of records.2.UpdateAttribute to add new attribute "dateTime" for each of N flowFiles by: dateTime <- /date_time3. UpdateRecord for each of N flowFiles with script: /date_time <- ${dateTime:toNumber:minus(

Background: I'm a new admin to our existing CDP environment. Current prod set up uses Nifi, Ranger, Impala, HMS, Hive on Tez, Isilon/HDFS on Dell Powerscale.  New directive is to create databases and Iceberg tables via S3 using buckets on the same Powerscale in a different access zone and redirect Nifi processes writing to hdfs to write to S3 buckets.Steps: I update the core-site.xml and hive-site.xml files with the following properties - "fs.s3a.secret.key", "fs.s3a.access.key" and "fs.s3a.endpoint".Success: I validated connection to the S3 bucket with aws cli and can run Nifi PutS3Object and the file is in the expected S3 bucket directory.Challenge: I'm getting a error in Impala when I attempt to create an Iceberg table in the same bucket to use with the Nifi PutIceberg processor."ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:CAUSED BY: MetaException: Got exception: org.apache.hadoop.fs.s3a.AWSClientIOException getFileStatus on s3a://ce-dev-bucket-3/s3

Hi,I am trying to create json, that is always the same, {"status":"APPROVED"}I have used replace text, used alwaysreplace and put that text there.Here is the specification of the invokehttp Am I missing something? 

I have a cluster, we are running INSERT OVERWRITE QUERY through HIVE CLI which fails with OutOfMemoryError: Java heap space.I have tried multiple set parameters and updated the machine config but no luck.set hive.exec.max.dynamic.partitions=4000;set hive.exec.max.dynamic.partitions.pernode=500;set hive.exec.dynamic.partition.mode=nonstrict;SET mapreduce.map.java.opts=-Xmx3686m;SET mapreduce.reduce.java.opts=-Xmx3686m;SET mapred.child.java.opts=-Xmx10g;set hive.tez.container.size=16384;set tez.task.resource.memory.mb=16384;set tez.am.resource.memory.mb=8192;set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager;set hive.support.concurrency=false;set hive.vectorized.execution.enabled=true;set hive.vectorized.execution.reduce.enabled=true;set hive.exec.orc.split.strategy=BI;set hive.exec.reducers.max=150;Error thrownStatus: FailedVertex failed, vertexName=Reducer 4, vertexId=vertex_1731327513546_0052_5_08, diagnostics=[Task failed, taskId=task_1731327513546_0052_5_08_00004

my cluster is cdh5.14  impala 2.11 kudu 1.6.0impala on kudu,When union all is used to merge more than three select join queries, the third and later select must perform full table scanning. How can I solve the problem of full table scanning?A full table scan is not performed when the select join statement is executed alone.sql as follows:SELECTmb.unitno,mb.materialcode,mb.starttime,mb.length,ib.equipmentcode,ib.defectclass,ib.defectname,ib.sideFROMsc1.tba1 mbLEFT OUTER JOINsc1.tbb1 ibONmb.materialcode = ib.materialcodewhere mb.starttime>'20240827'and mb.starttime<'20240828' )UNION ALLSELECTmd.unitno,md.materialcode,md.starttime,md.length,id.equipmentcode,id.defectclass,id.defectname,id.sideFROMsc1.tba2 mdLEFT OUTER JOINsc1.tbb2 idONmd.materialcode = id.materialcodewhere md.starttime>'20240827'and md.starttime<'20240828' )UNION ALLSELECTme.unitno,me.materialcode,me.starttime,me.length,ie.equipmentcode,ie.defectclass,ie.defectname,ie.sideFROMsc1.tba3 meLEFT OUTER JOINsc1

We have a hive table in cloudera data platform and we need to export the same to oracle.hive:create external table temp_fns .ABC(account_id NUMBER,`1234` number) stored as orc; oracle : create table schema.ABC(account_id decimal(28,0),"1234" decima;(28,0)) ; sqoop export command :sqoop export  --connect jdbc:oracle:thin:@//server:1521/xyz --username pravin --P --num-mappers 5 --hcatalog-database temp_fns --hcatalog-table ABC -table schema.ABC -- --batch The ORA error I am getting is "Error: java.io.IOException: java.sql.SQLSyntaxErrorException: ORA-01747: invalid user.table.column, table.column, or column specification"We cannot alter the oracle structure and most important thing is the same sqoop export works from HDP but it fails while exporting from CDP.THe error is because of numeric column name "1234" when sqoop generates insert into statements like insert into schema.ABC (account_id,1234) values (1,234)it fails with ORA error  ORA