Q&A

Hi, I am using UnpackContent to extract files from a zip and then load them to another location. I am seeing corruption in the files going to that other location that I think is caused by some race condition on the extracted files. The zip files have a directory and file structure that look something like thisfile1.zip -> data/file.txtfile2.zip -> data/file.txtwhere the contents of file.txt in the two zip files is different but they are named the same. We use UnpackContent with Packaging Format 'zip' and File Filter 'data' to only get files in the data directory.This works fine when processing individual files, but now that we have scaled this up, there appears to be an issue where the file extracted from file1.zip gets overwitten by the extracted file from file2.zip and then when we copy the file over, the content is corrupted between the two. Looking at the properties after the UnpackContent, I see absolute.path is something like <NiFi Location>/data/file.txt after e

Getting SSL Certificate error while from Power BI to impala server. ODBC connector is working fine,  however while doing the direct connection getting error. SSL certificate not found or missing.

I am working in NiFi 1.16 and trying to use ExecuteSQL processor to write out a Parquet file.  My challenge is the decimal format is being changed to "LogicalType(STRING: StringType())".The data is coming from an MS SQL server.  We have tried to set the precision and scale, but still continues to come out in a string instead of a decimal.  

What does this message mean on an invalid processor:I see it for processor FetchXMLFile.On scheduling tab, execution dropdown has 'Primary Node' selected. On changing it to 'All Nodes', error goes away.I am on 1.22.0(with some nars from 1.10.0 also added to lib folder).

I am configuring a new 3 node NiFi cluster on three AWS EC2 instances.  I'm attempting to use the tls-toolkit.sh utility to generate the SSL keystore.jks and truststore.jks for each of the 3 nodes.  It generates them and I can start the cluster but when I load the UI it seems like it's working but then I get this message.An unexpected error has occurred (log out) (home)javax.net.ssl.SSLPeerUnverifiedException: Hostname my.elb.name not verified: certificate: sha256/LhH2H...hkIE= DN: CN=node1.server.name, OU=NIFI subjectAltNames: [node1.server.name]The nifi-app.log has basically the same to say about it.WARN [Replicate Request Thread-7] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to my.elb.name:port due to javax.net.ssl.SSLPeerUnverifiedException: Hostname my.elb.name not verified: ... subjectAltNames: [node1.server.name]I would have expected the subjectAltNames to include my.elb.name because I included it in my tls-tool

I'm having an issue with a Spark-Hive application running on a Kerberos cluster. I receive a javax.security.sasl.SaslException: GSS initiate failed error, which appears to be caused by not finding any Kerberos tgt. Here's the error log:     23/08/04 22:56:55 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.    23/08/04 22:56:55 INFO HiveClientImpl: Attempting to login to Kerberos using principal: hdfs01@HDP.COM and keytab: hdfs01.keytab-2ca1f730-bef7-4166-90ce-67317c75c793    23/08/04 22:56:55 INFO UserGroupInformation: Login successful for user hdfs01@HDP.COM using keytab file hdfs01.keytab-2ca1f730-bef7-4166-90ce-67317c75c793    23/08/04 22:56:55 INFO metastore: Trying to connect to metastore with URI thrift://master3.abc.xyz.com:9083"    23/08/04 22:56:55 ERROR TSaslTransport: SASL negotiation failure    javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExce

For my work, I had to copy nifi 1.10.0 nars into lib folder of nifi 1.22.0. I do this as part of docker image build process.This work seems to support this.First, container launch has issues with multiple jetty nars, so I dont't copy them, and jetty issue is gone. But container launch still fails with:java.lang.IllegalStateException: Multiple framework NARs discovered. Only one framework is permittedIs this due to nar versions(1.10.0 and 1.22.0) or do I need some more configuration to make it work?  

Hi Team,I am facing following issue(error) to start region and master servers. Anyone is having any idea on this.023-09-01 10:51:24,807 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed construction RegionServerjava.lang.UnsatisfiedLinkError: failed to load the required native libraryat org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:54)at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:186)at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:37)at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)at

Hi, I have just started using Apache NiFi, so things are pretty new to me. What I want is to migrate our existing NARs(created for nifi 1.10) to version 1.22.We actually tried it with 1.15.1 too but we found that there were new processor properties which were required, and so things didnt go well. So I want to know if there is some well documented way of achieving it?

when I am using PutFile Processor and ExecuteScript Processor,the object in the flow between them stuck for longer time than expected and processed many times.this case happend only between these 2 processors only.even if I set the expiration time for the flow and set it to 1 second, and scheduled the run of ExecuteScript to 4 sec, still processing the flow at least 2 times.which cause sending the file to the API many times.the version of the NIFI I am usnig is: NIFI v1.18, also I tried to upgrade and use version 1.22, but this issue still as is. The script I am using in ExecuteScript is:import org.apache.commons.io.IOUtilsimport org.apache.nifi.processor.io.OutputStreamCallbackdef flowFile = session.create()def filePath = 'E:/tmp/tmp_file.txt'def file = new File(filePath)if (file.exists()) {def fileContent = IOUtils.toByteArray(new FileInputStream(file))flowFile = session.write(flowFile, { outputStream ->outputStream.write(fileContent)} as OutputStreamCallback)session.transfer

I have a secure hadoop cluster with HDP3.1，I recently tried to interconnect this cluster with the Knox component to implement a secure proxy. This cluster has kerberos, Ldap, and https enabled, I create one config like this: <topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>main.ldapRealm</name> <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value> </param> <param> <name>main.ldapContextFactory</name> <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value> </param> <param> <name>main.ldapRealm.contextFactory</name> <value>$ldapContextFactory</val

Problem Summary---------------------------------------------------Is Cloudera have estimation time for jobs completion.Problem Description---------------------------------------------------Hi Support,I need to understand and have information on any time estimation of completion of jobs running in Hadoop respective of map reduce or spark jobs?

如何下载CDH6.3.2？（免费帐户）谢谢

Logs from solr.log WARN  [c:ranger_audits s:shard1 r:core_node2 x:ranger_audits_shard1_replica_n1] org.apache.hadoop.security.authentication.server.AuthenticationFilter (AuthenticationFilter.java:565) - Authentication exception: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) Hadoop.secuorg.apache.hadoop.security.authentication.utill.signerexception invalid signature.Pfa 

We have a delimited flatfile which has special characters ingested into HDFS. We have created a Hive External table on top of the HDFS path and visualized the data in Hue & Beeline. But the special characters are displayed as question mark (?). Please advise the solution.

Dears,  I need help. I trying to version my flows, but is show this error: (Failed to register flow with Flow Registry due to Error creating snapshot: An unexpected error has occurred. Please check the logs for additional details) Looking in the logs, I found this error:  org.apache.nifi.web.NiFiCoreException: Failed to register flow with Flow Registry due to Error creating snapshot: An unexpected error has occurred. Please check the logs for additional details. ERROR o.a.nifi.web.StandardNiFiServiceFacade The flow has been created, but failed to add a snapshot. Returning the created flow information. Last week it was working perfectly but now this error started. Can you help me?   Thanks!  

I have a flowfile that is a list of files,that gets split into individual flowfiles (10 in this case)I am using FetchSFTP to fetch the file from a sftp location, the content of this fetched file is then split and is processed and stored.In this example: I want to start processing ep.txt only after i have processed bulk.txt after that names.txt and so on.I was referring to these links:NiFi workflow monitoring – Wait/Notify pattern with split and merge – Pierre Villardand How to wait for all fragments to be processed, then do something? (ijokarumawak.github.io)I was able to implement something like this. but this is not what I was looking for, I want to implement a wait before fetchSFTP I tried release signal identifier with filename , but that was stopping any flowfile from moving forward. Please advise, thank you 

After deleteing all version of a flow from nifi registry, restarting a node cannot connect cluster, It always display below error. How can I resolve it? 2021-12-16 22:13:17,520 ERROR [Timer-Driven Process Thread-149] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=d059cc81-0176-1000-0000-0000089d93e4,name=整机CMES] with Flow Registry because could not retrieve version 3 of flow with identifier 5d63465f-9893-4f93-8d27-b8c258519d68 in bucket fbcc346b-ee7c-4ec4-ba7d-68c0f4c306b1org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving flow snapshot: The specified flow ID does not exist in this bucket. at org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:117) at org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient.get(JerseyFlowSnapshotClient.java:97) at org.apache.nifi.registry.flow.RestBasedFlowRegistry.getFlowContents(RestBasedFlowRegistry.java:213)

Does HDFS perform load balancing on the storage of data blocks when the disk space of each node is different? 

Hello, We are facing issues regarding KDC,  on CDP HBase and Impalad wont start and here below the logs: The Keytab file is generating kerberos ticket correctly  (using the command line kinit) and the service Keytabs have been regenerated several times but still the same issue. This issue is blocking: Impala, HBase and Yarn Nodemanagers Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVERat sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:226)at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:237)at sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:477)at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:340)at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:314)at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:169)at sun.se

I am executing python script in ExecuteStreamCommand and my code last line dumps the json key values. My goal is to read those key values into PutDatabaseRecord processor and insert the values in my database. However, I am unable to read content of the flow file of ExecuteStream. I have used ConvertAvrotoJson but it is showing output as "not a avro data file". Is there any workarounf of it? Thanks.

Hello,we are performing Hive queries with PySpark using the HWC in JDBC_CLUSTER mode.Everything is running fine and we get the results for the queries, but we also receive a warning message saying that connection has been closed:23/08/16 09:59:05 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist23/08/16 09:59:05 WARN transport.TIOStreamTransport: Error closing output stream.java.net.SocketException: Connection or outbound has closedat sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1181)at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)at java.io.FilterOutputStream.close(FilterOutputStream.java:158)at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)at org.apache.thrift.transport.TSocket.close(TSocket.java:235)at org.apache.thrift.transport.TSaslTransport.close(TSaslTransport.java:400)at org.apache.thrift.transport.TSaslC

I have api username and password which is stored in secret now i need to access them in invoke http processor through environment variable but i was unable to access because the fields doesn't allow expression language. how do i access these values from secret?

Hi,how to define "Table Name Pattern" for more than 1 table in ChangeCaptureMySql processor configuration file.  I tried below patterns where "product_types" and "user_subscription" table names. I want to read data only from these tables.[product_types|user_subscription]_ or [product_types|user_subscription]% 

Hi folks! I’m trying to create a connection between Dbeaver and Hive on a CDP 7.1.6 from a Mac OS Ventura 13.1 (22C65). I used the Apache Hive connection option; Put this sring on Host path on the main page…: host:port,host:port,host:port,host:port,host:port/default;httpPath=cliservice;password=xxxxxx;principal=hive/_HOST@host.domain;serviceDiscoveryMode=zooKeeper;transportMode=http;user=xxxx;zooKeeperNamespace=hiveserver2 …and Host/IP, user and password on the SSH page (It is working fine). On the Driver library added the jars files:log4j-slf4j-impl-2.10.0.jarslf4j-log4j12-1.7.30.jar But I’m keep receiving this message: Error: Can't create driver instance  Error creating driver 'Hadoop / Apache Hive' instance.Most likely required jar files are missing.You should configure jars in driver settings. Reason: can't load driver class 'org.apache.hive.jdbc.HiveDriver'  Error creating driver 'Hadoop / Apache Hive' instance.Most likel