Q&A

Hi! I installed CDP Private Cloud 7.1.7 trial version in which Kafka 2.5.x is installed. Is there anyway I can upgrade to the latest CDP 7.1.9 in which I want to try with Kafka 3.x or upgrade kafka service only? Thanks for any suggestion!

Hi there,I am an absolute NIFI beginner, and I would like to implement the following data integration (DI) pipeline using Apache NIFI:Scenario: I have two independent data sources:A CSV file.A database.Both contain streams of sorted lines with the same column structure. Each record, except of other columns, also includes:A primary ID.A calculated hash, generated from the content of the line.Goal: I want to compare the records based on their ID columns from both sources and classify them as:New: The ID is present in the CSV file but not in the database.Changed: The ID exists in both sources, but the hash values differ.Same: The ID exists in both sources, and the hash values are the same.Deleted: The ID is missing from the CSV file but exists in the database.The output flow of this "diff" process should produce a stream where each line is enriched with a status flag ("new", "changed", "same", "deleted") indicating the result of the comparison.Background: I have successfully implemented t

Hi Everyone,How do you run NIFI - Ambari as root user ?

We are using nifi as api gateway. Whenever we pass any attributes in request header it should be captured as http.headers.xxxx in flowfile but when checked in flowfile ; header value is coming in small case though we have passed it in caps. In our implementation we have 2 instances of nifi running. One instance doesn't have any issue where as other has . Please help if there is any configuration I'm missing. 

I have 5 Nodes:NN1NN2DN1DN2DN3Three Apache Services:ZookeeperHadoopYarnHBaseWill need MapRed tooFor authentication-Companywide my client is using LDAP, AD. For application purpose they setup Kerberos and Active Directory as KDC.So, its AD Kerberos. I don't handle KDC by myself - Security team does.Currently I have setup an open-source self-managed Hadoop cluster and its configured to 'SIMPLE AUT'I want to use the authentication, so how do i proceed sir? Can someone please help with detailed steps and instruction?

Some Scripts suddenly stopping without ending for some days

Hi All, We have been facing an issue with building a flow in Apache NiFi (version 1.18) to receive events from Azure EventHub.We had a Process Group that would read events from Azure EventHub using ‘GetAzureEventHub’ processor. It was running successfully, but in March 2024 it stopped working without any changes to NiFi server or on the Azure EventHub. We received the following error - “Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException: reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException - Caused by: java.lang.InterruptedException”Since then, we have not been able to connect to Azure EventHub from NiFi. We have tried ‘ConsumeAzureEventHub’ processor as well, but received “Connection is disposed. Cannot get management instance” error. Azur

I am currently facing an issue while using the ImportTsv command to load data into an HBase table. I am running the command and the job is failing with a non-zero exit code 1. ./hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns="HBASE_ROW_KEY,cf:name,cf:age,cf:gender" -Dimporttsv.separator="\t" my_1 /hbase/test2.txtERROR LOG:SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/super/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/super/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]2024-09-02 09:48:21,871 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where appl

Hi Team,i am using 1.27 NIFI, For QuerySalesforceObject processor i am unable to find proxy configuration service for instance url, We have Proxy configuration service for OAuth2 Access token but not for salesforce url, Please suggest for proxy service enablement for salesforce instance URL as well. Sample screenshot of existing configuration below.Thanks,Krish 

I have a nifi flow to read from 1 table and write to another. But when it inserts  it thows the error: PutCassandraRecord[id=bf945542-ef3f-1b23-0000-0000463e16ae] Unable to write the records into Cassandra table due to com.datastax.driver.core.exceptions.InvalidQueryException: You must use conditional updates for serializable writes- Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: You must use conditional updates for serializable writes: {} putCassandraRecord processor: I have tried both serial and local_serial, and none, but all give same error. I am able to run this if i use putCassandraQL but that means i also have to have a EvaluateJsonPath and a ReplaceText  to create the insert statement. I am trying to make it more efficient by using the record aware processor. Any help would be appreciated. Thank you! 

We are working on converting the Unix time into date-time using the QueryRecord processor. We understand that QueryRecord is built with the calcite SQL syntax and tried to use the DATE_FROM_UNIX_DATE function. We are receiving the below error. Please help us resolve this issue with your expertise.We can achieve this conversion using the update attribute but we wanted this as part of the Query record as we wanted to apply this conversion on the completed dataset.Query :SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFileError :QueryRecord[id=2cc603e4-0191-1000-f3c7-ef830564cf72] Unable to query FlowFile[filename=convertfunctionsample.csv] due to org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile]- Caused by: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile]- Caused by: org.apache.calcite.runtime.Ca

I am reading oracle table using pySpark. my oracle table contains data type NUMBER and it contains 35 digits long value. when I read this column using spark, it seems spark assumes more precision than original (+10 precision)and end up in throwing following error:java.lang.ArithmeticException: Decimal precision 45 exceeds max precision 38`please note: I am inferring schema from spark data frame as I don't want to explicitly pass schema.code:df_rdbms_data = spark.read.jdbc(url=source_jdbc_url, table=source_table_name, properties=source_properties)df_rdbms_data.printSchema()df_rdbms_data.show()Is there any way to resolve this issue apart from not passing explicit schema or explicit casting?Thanks in advance.

I'm trying to Insert into a table located in /warehouse/tablespace/managed/hive from a table located in /warehouse/tablespace/external/hive using beeline. But I'm getting: ERROR : Status: FailedERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1707433862019_0104_2_00, diagnostics=[Vertex vertex_1707433862019_0104_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: countries_src_ap initializer failed, vertex=vertex_1707433862019_0104_2_00 [Map 1], org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=16, exceptions:2024-08-02T20:24:47.396Z, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68708: Call to HOST/IP HOST:16020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed row 'countries_src,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=HOST_NAME,16020,1722553888133, seqNum=-1 at org.apache.hadoop.hbase.client.RpcRetryin

Hi all!i faced out with the problem.I need to convert xml file to json.here is my flowConvertRecordXMLReaderJsonRecordSetWritterEverything is ok, but the problem is in convetring digit-valuesfor examplexml data<document_code> <code>000</code></document_code>converted to "document_code": { "code": 0} another one is:<service-code>0012388833456</service-code>converted to "service-code": 12388833456 this is critical data, so i want Ni-Fi to convert them as string.all values as string - is what i need.Help please.

Hi all,After establishing a connection to Cloudera Virtual Warehouse from PowerBI Desktop using Hive LLAP connector, I'm not able to load Iceberg tables. I receive the following error:DataSource.Error : ODBC : ERROR [42000] [Microsoft][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: Error while compiling statement: FAILED: Runtime Exception org.apache.hadoop.hive.ql.metadata.HiveException: Table XXXX is an Iceberg table format which is not currently supported.I tried also installing the Hive ODBC driver 2.7.0. I configured the User DSN and test was successful. When I try to connect form PBI Desktop, I can establish the connection but I receive the same error if I try to load an Iceberg table:DataSource.Error: ODBC: ERROR [42000] [Cloudera][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: Error while compiling statement: FAILED: Runtime Exception org.apache.ha

hi cloudera community,creating a full template using the nifi webui mode works perfectlywe need to create the template, export and import using the nifi api, has anyone managed to configure this?

Our group has tried connect janusgraph with Cloudera HBase, but it fails.  In the documentation of CDP 7.2.18, these two Spark features: GraphX and GraphFrames are not supported by Cloudera. Is there any other Graph Data Base or Graph computation tool is supported by Cloudera?

We are trying to connect with cdh hdfs using java program and it's failing. With pre authentication failed error. I tested the hadoop fs commands all are working fine. We tried soo many way nothing is working fine.  Can you please help me with this issue. 

Greetings, I am trying to migrate from Nifi 1.16.2 to 1.16.3.When I try to use my flow file form my 1.16.2 installation, the application goes up but the controller service StandardRestrictedSSLContextService 1.18.0, shows a problem with the sensitive properties that are not supported.After updating anything inside the controller, everything starts working as expected. But if a restarts Nifi, the application does not restarts.I try to find some errors in the  nifi-app.log but none is logged, just Warnings, here some of them  2022-11-23 17:01:00,812 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 9, col 21: cvc-complex-type.2.4.a: Invalid content was found starting with element 'description'. One of '{url}' is expected.2022-11-23 17:01:00,825 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 71, col 19: cvc-complex-type.2.4.d: Invalid content was foun

Using NiFi, I am processing a file. Once processing is done, I want to shift it to an archive folder. I want it to keep only say 30 days files and older files to be deleted. How to do. One more question is how to have timestamp name appended to file name. Like if file name is file1.txt then how to have it renamed like file1_09_09_2024_15_54.txt where we have 09_09_2024 as date and 15_54 as time(hour and mins)

HiI have a question on our Nifi flow design and specifically about the usage of DuplicateFlowFile processor.The latest documentation mentions that, it is intended for load test.Does that mean, it can not be used in regular production flows? Please confirm.https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/nifi-2.0.0-M4/org.apache.nifi.processors.standard.DuplicateFlowFile/index.html"Intended for load testing, this processor will create the configured number of copies of each incoming FlowFile." Please verify the current and improved designs below and advise if any implications with the new approach (where DuplicateFlowFile processor is used to clone flow files)Current Design:  New Design is below (with DuplicateFlowFile processor)        

Hi,I am testing nifi 2.0.0-M4 in cluster mode (secured mode).But whenever I start the second node I get the following error when connecting to nifi UI. 024-09-07 13:50:16,976 WARN [Replicate Request Thread-107] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to desktop:11443java.net.ConnectException: Failed to connect to desktop/192.1.1.39:11443at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)at okhttp3.internal.http.RealI

Hello Experts,We have couple of Eventhub consumers running in nifi 1.16.3 version.The output connection is configured with default backpressure - 10,000 messages and 1GB.But I see 'ConsumeAzureEventhub' is keep on consuming data even after crossing backpressure.What is the reason for this behavior and how to fix this?@mattw  ThanksMahendra  ThanksMahendra

Say I have a file having 100 records. In this file I want to ignore the first line as it's data I do not want to process (It contains some info like file name, date and number of records). I want to process from second line. How to do that?

I create draft from ReadyFlow Hello World and start test session. Then error showed up as below