Ask the Community

My NiFi flow fails when encountering a CSV with a column containing a double quote within a string, such as:"Protection from Abuse Order on file against dad Raul Martinez Lopez." NO CONTACT WITH DAD. 12/16/2014 kb.The error is occurring at the Record Reader stage. Has anyone else successfully handled CSV data with embedded double quotes?From csv:My record reader config  

Hi at all,I got a timeseries table with existing range partitions based on timestamp field, that covers years from 2022 to 2024 month by month. The problem is that last partition has an open bound (PARTITION 1717200000000 <= VALUES) and contains data from "1717200000000" (Saturday, June 1, 2024 0:00:00) to now, and is still storing live data with obviously performance loss during query.From the docs I haven't found a command that lets me split last partition in other partitions in order to update them, but the only way seems to drop last partition and continue adding since this time, and the problem is that by dropping last partition I will lose three and more months of data, that is obviously not possible.Is there an alternative to prevent data loss and adding range partitions to improve performances?Thanks in advance

Hi Team,I am trying to implement for the scenario where the events should sleep for 5 minutes and let all the flow files are queued up, then check the count of the files if it is greater than 10 route to failure else route to SUCCESS.I did in Execute script processor as below. However, i am trying to avoid the executescript for this and try to use Native nifi processors.====== Executescript ===============Thread.sleep(300000)def flowFiles = session.get(100)if (!flowFiles || flowFiles.size() <= 10) {session.transfer(flowFiles, REL_SUCCESS)} else {session.transfer(flowFiles, REL_FAILURE)}  

We have a Nifi cluster with 3 nodes running in docker containers that are always processing something. I included keys in the truststore.jks file of each node and for the changes have effect, I need to restart each node at a time, so that Nifi could still be running and we don't lose any data . There a way to restart them safely? ChatGPT said that it's possible to execute inside the container the command nifi.sh stop, but I don't know if this will cause data loss.

Hello , I am trying to access Nifi in a way that all users in the AD group memberof=CN=AG-Y-DEF-LJKGH,OU=Groups,DC=corp1,DC=ad1,DC=xyz,DC=net  can have access to it, since it is getting quite tedious for the admin to keep adding the users one by one and assigning them policies.  My authorizers.xml looks like : <?xml version="1.0" encoding="UTF-8" standalone="yes"?><authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./auth-conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 0">CN=my-nifi-0.my-nifi-headless.nifi-test.svc.cluster.local, OU=NIFI</property> <property name="Initial User Identity 1">CN=ABC123,OU=Service Users,OU=User Accounts,

I've been racking my brain for the past couple of days attempting to connect to a Hive server with a Python client using pyhive on Windows. I'm new to Hive (pyhive too for that matter), but am a reasonably experienced Python dev. Invariably I get the following error:(pyhive-test) C:\dev\sandbox\pyhive-test>python test.pyTraceback (most recent call last):File "test.py", line 3, in <module>conn = hive.Connection(host='192.168.1.196', port='10000', database='default', auth='NONE')File "C:\Users\harnerd\Anaconda3\envs\pyhive-test\lib\site-packages\pyhive\hive.py", line 192, in __init__self._transport.open()File "C:\Users\harnerd\Anaconda3\envs\pyhive-test\lib\site-packages\thrift_sasl\__init__.py", line 84, in openraise TTransportException(type=TTransportException.NOT_OPEN,thrift.transport.TTransport.TTransportException: Could not start SASL: b'Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2'when executing the following script:from py

Hello!Input flowFile has column "date_time" with timestamp-millis type The task is to update each value of column, by substracting 3 hours from datetime value. In fact I need to transform datetime value from timezone GMT +3 to timezone GMT +0For example, value "2024-11-24 19:43:17" to be transformed in "2024-11-24 16:43:17"I tried to use UpdateRecord with following script:           format(toNumber((/date_time):minus(3600)), "yyyy-MM-dd HH:mm:ss")And, as expected, transformation failed with exception: "Unrecognized token 'minus'..."Operation "minus" applied to attributes values. And "date_time" is not an attribute.As an alternative, I considered next way:1. SplitText for splitting one FlowFile into N FlowFiles, where N - is the number of records.2.UpdateAttribute to add new attribute "dateTime" for each of N flowFiles by: dateTime <- /date_time3. UpdateRecord for each of N flowFiles with script: /date_time <- ${dateTime:toNumber:minus(

Does Apache Nifi 2.0 supports IBM MQ 9.2 using JMS processor?As per this article from IBM, MQ 9.3 added support for Jakarta messaging. That is working fine for Apache Nifi 2.0 using JMS processor. However, it is failing for MQ 9.2 and throwing class cast exception as shown below2024-11-22 17:26:01,822 ERROR [Timer-Driven Process Thread-5] o.a.n.j.cf.JMSConnectionFactoryProvider JMSConnectionFactoryProvider[id=5375f4df-0193-1000-3023-0c7084045874] Failed to configure JMSConnectionFactoryHandlerjava.lang.ClassCastException: class com.ibm.mq.jms.MQConnectionFactory cannot be cast to class jakarta.jms.ConnectionFactory (com.ibm.mq.jms.MQConnectionFactory is in unnamed module of loader org.apache.nifi.nar.InstanceClassLoader @5a30ab46; jakarta.jms.ConnectionFactory is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @383790cf)

Background: I'm a new admin to our existing CDP environment. Current prod set up uses Nifi, Ranger, Impala, HMS, Hive on Tez, Isilon/HDFS on Dell Powerscale.  New directive is to create databases and Iceberg tables via S3 using buckets on the same Powerscale in a different access zone and redirect Nifi processes writing to hdfs to write to S3 buckets.Steps: I update the core-site.xml and hive-site.xml files with the following properties - "fs.s3a.secret.key", "fs.s3a.access.key" and "fs.s3a.endpoint".Success: I validated connection to the S3 bucket with aws cli and can run Nifi PutS3Object and the file is in the expected S3 bucket directory.Challenge: I'm getting a error in Impala when I attempt to create an Iceberg table in the same bucket to use with the Nifi PutIceberg processor."ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:CAUSED BY: MetaException: Got exception: org.apache.hadoop.fs.s3a.AWSClientIOException getFileStatus on s3a://ce-dev-bucket-3/s3

Hello everyone, I wanted to know if it is possible to connect to a server in GCP to start using the ML offered by Cloudera. Thanks!

Hi, Is there any probnlem in 6.21 mirror ,server2 jmx http://10.10.61.35:10002/jmxexception: HTTP ERROR 401,Problem accessing /jmx. Reason:Unauthorized access, Powered by Jetty:// 9.3.25.v20180904

Hi,I am trying to create json, that is always the same, {"status":"APPROVED"}I have used replace text, used alwaysreplace and put that text there.Here is the specification of the invokehttp Am I missing something? 

I have a cluster, we are running INSERT OVERWRITE QUERY through HIVE CLI which fails with OutOfMemoryError: Java heap space.I have tried multiple set parameters and updated the machine config but no luck.set hive.exec.max.dynamic.partitions=4000;set hive.exec.max.dynamic.partitions.pernode=500;set hive.exec.dynamic.partition.mode=nonstrict;SET mapreduce.map.java.opts=-Xmx3686m;SET mapreduce.reduce.java.opts=-Xmx3686m;SET mapred.child.java.opts=-Xmx10g;set hive.tez.container.size=16384;set tez.task.resource.memory.mb=16384;set tez.am.resource.memory.mb=8192;set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager;set hive.support.concurrency=false;set hive.vectorized.execution.enabled=true;set hive.vectorized.execution.reduce.enabled=true;set hive.exec.orc.split.strategy=BI;set hive.exec.reducers.max=150;Error thrownStatus: FailedVertex failed, vertexName=Reducer 4, vertexId=vertex_1731327513546_0052_5_08, diagnostics=[Task failed, taskId=task_1731327513546_0052_5_08_00004

Hello,I have deployed Apache NiFi and NiFi Registry on a Kubernetes cluster, but I am unable to retrieve the buckets in the NiFi interface. However, when I set the buckets to "public", I can see them in the "Network" tab of the browser’s developer tools.I have been troubleshooting this issue for over a month without success. If anyone has encountered a similar situation or has any suggestions on what might be causing this, I would greatly appreciate any help or advice.Thank you in advance for your assistance!@MattWho 

when I enable HDFS Federation RBF, Hive tez job will throw error as follows: hive-server2.log: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable (null) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] I see https://issues.apache.org/jira/browse/TEZ-4032, and my tez version is 0.10.1,I didn't get any help from it. Is there someone who can help me?

my cluster is cdh5.14  impala 2.11 kudu 1.6.0impala on kudu,When union all is used to merge more than three select join queries, the third and later select must perform full table scanning. How can I solve the problem of full table scanning?A full table scan is not performed when the select join statement is executed alone.sql as follows:SELECTmb.unitno,mb.materialcode,mb.starttime,mb.length,ib.equipmentcode,ib.defectclass,ib.defectname,ib.sideFROMsc1.tba1 mbLEFT OUTER JOINsc1.tbb1 ibONmb.materialcode = ib.materialcodewhere mb.starttime>'20240827'and mb.starttime<'20240828' )UNION ALLSELECTmd.unitno,md.materialcode,md.starttime,md.length,id.equipmentcode,id.defectclass,id.defectname,id.sideFROMsc1.tba2 mdLEFT OUTER JOINsc1.tbb2 idONmd.materialcode = id.materialcodewhere md.starttime>'20240827'and md.starttime<'20240828' )UNION ALLSELECTme.unitno,me.materialcode,me.starttime,me.length,ie.equipmentcode,ie.defectclass,ie.defectname,ie.sideFROMsc1.tba3 meLEFT OUTER JOINsc1

Specifies the character to use as a delimiter between fields when query results are printed in plain format by the -B option. Defaults to tab ('\t'). If an output value contains the delimiter character, that field is quoted, escaped by doubling quotation marks, or both. Added in Impala 1.0.1.My  output value contains  some ('\t')，So，I tried using  \`field\` OR “field” and so on,  But all of  them doesn't work.Does anyone know how to solve it?Impala Shell v3.4.0-SNAPSHOT

I am trying to run use db_name;show tables and it is failing. Can you please help me with it [root@cupid ~]# sudo -u hive beeline -u "jdbc:hive2://charon.devemc.com:2181,cupid.devemc.com:2181,portia.devemc.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;" --showHeader=false --outputformat=tsv2 --delimiter=';' -e "use default;show tables;"scan complete in 2msConnecting to jdbc:hive2://charon.devemc.com:2181,cupid.devemc.com:2181,portia.commvault.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;Connected to: Apache Hive (version 0.14.0.2.2.9.0-3393)Driver: Hive JDBC (version 0.14.0.2.2.9.0-3393)Transaction isolation: TRANSACTION_REPEATABLE_READError: Error while compiling statement: FAILED: ParseException line 1:11 missing EOF at ';' near 'default' (state=42000,code=40000)

We have a hive table in cloudera data platform and we need to export the same to oracle.hive:create external table temp_fns .ABC(account_id NUMBER,`1234` number) stored as orc; oracle : create table schema.ABC(account_id decimal(28,0),"1234" decima;(28,0)) ; sqoop export command :sqoop export  --connect jdbc:oracle:thin:@//server:1521/xyz --username pravin --P --num-mappers 5 --hcatalog-database temp_fns --hcatalog-table ABC -table schema.ABC -- --batch The ORA error I am getting is "Error: java.io.IOException: java.sql.SQLSyntaxErrorException: ORA-01747: invalid user.table.column, table.column, or column specification"We cannot alter the oracle structure and most important thing is the same sqoop export works from HDP but it fails while exporting from CDP.THe error is because of numeric column name "1234" when sqoop generates insert into statements like insert into schema.ABC (account_id,1234) values (1,234)it fails with ORA error  ORA

HTTP ERROR 400 Invalid SNIURI:STATUS:MESSAGE:SERVLET:CAUSED BY:URI: /nifiSTATUS: 400MESSAGE: Invalid SNISERVLET: -CAUSED BY: org.eclipse.jetty.http.BadMessageException: 400: Invalid SNICaused by:org.eclipse.jetty.http.BadMessageException: 400: Invalid SNIat org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:266)at org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:207)at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1594)at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)at org.eclipse.jetty.util.thread.st

I have executed the command hdfs namenode -reformat on my hadoop cluster and the namenode data files have newly generated VERSION file.This created inconsistency as the datanode and namenode VERSION files have different cluster-ID.I manually changed the cluster-id in the datanode VERSION file to match the namenode one. This resulted in succesful start of the cluster but after the start there is a new block create with new blockpoolID which is empy. All my data is in the old datafile with old blockpoolID but it is currently not mapped to my hadoop filestystem.How can I recover my old data and make it accessible in my hadoop cluster again?Can I somehow just start over with my old datafile from datanode?

We're attempting to execute a basic Spark job to read/write data from Solr, using the following environment:CDP version: 7.1.9Spark: Spark3Solr: 8.11Spark-Solr Connector: opt/cloudera/parcels/SPARK3/lib/spark3/spark-solr/spark-solr-3.9.3000.3.3.7191000.0-78-shaded.jarWhen we try to interact with Solr through Spark, the execution process hangs indefinitely, without any errors or results. Other components, such as Hive and HBase, integrate smoothly with Spark, and we’re using a valid Kerberos ticket that successfully authenticates with other Hadoop components. Additionally, we’ve tested REST API calls to Solr via both curl and Python’s requests library, and we’re able to retrieve data with the Kerberos ticket.The problem appears isolated to Spark’s connection with Solr, as all other systems interact as expected. Has anyone experienced a similar issue or have ideas on what might be causing this?Here’s the Spark code we’re trying:solr_options = {"zkhost": "zkURL-01.orgis.ie:2181,zkURL-02.o

using commandALTER TABLE test table DROP PARTITION (START_DATE = DATE_ADD('2023-09-13', -13) )error: SQL Error [500051] [HY000]: [Cloudera][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code: 40000, SQL state: TStatus(statusCode:ERROR_STATUS, info Messages:[*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:78 cannot recognize input near 'DATE_ADD' '(' ''2023-09-13'' in constant:17:16Need Result : '2023-08-31'

Hi Team,How can I replace the custom ConsumeASB Processor with the InvokeHttp Processor in NiFi? I am currently using a Topic subscription with my ConsumeASB Processor to read messages from an Azure Service Bus endpoint.How can I configure the InvokeHttp Processor to achieve the same functionality?Could you please provide some inputs.

Hello!Xml file has the following structure:  <root> <nested1> <nested2> <nested1>value</nested1> </nested2> </nested1></root>           EvaluateXPath processor type_rec  property:  name(/*/*[1])And I got the following error:      EvaluateXPath[id=ce4e0c05-169f-387f-520d-77e2e664bacc] Input parsing failed FlowFile[filename=69de5f2c-a887-4042-8734-beca03e3d27b]: org.apache.nifi.xml.processing.ProcessingException: Parsing failed- Caused by: org.xml.sax.SAXParseException; lineNumber: 6; columnNumber: 5; The element type "nested2" must be terminated by the matching end-tag "</nested2>".As I understand, parser except, at 4 level closed tag </nested1> , because it'v met open tag with the same name at level 2 (<nested1>).   But at level 4 file has a new tag with the same name. I tried to use ReplcaseText processor for