Q&A

I receive a response from InvokeHTTP as a folder->another_folder->file.json.gz and I want to do some operations on the json file using SplitJson processor, so, I need to decompress the file.json.gz I tried to UpdateAttribute to rename the parent folder to be folder.zip then UnpackContent but it's not supporting .gz format  Then, I tried instead of UnpackContent, the ExecuteStreamCommand with unzip command with a plan to use PutFile then ListFile to get the file.json.gz and pass to SplitJson but I got this error in ExecuteStreamCommand step Failed to write flow file to stdin due to Broken pipe: java.io.IOException: Broken pipe  What's the solution for this? I tried to set Connection Timeout propriety to 120s but still the same error, What is the best way to get file.json.gz, decompress it and pass it to SplitJson processor? 

Dear Community,   NiFi locally installed on my laptop was working fine until today that I got the following error in the URL.    An unexpected error has occurred. Please check the logs for additional details   This is the error that I am seeing in the log file.     2022-06-17 09:42:45,569 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@759a678a{nifi-error,/,file:///C:/Users/alshahin/nifi-1.15.3/work/jetty/nifi-web-error-1.15.3.war/webapp/,AVAILABLE}{.\work\nar\extensions\nifi-server-nar-1.15.3.nar-unpacked\NAR-INF\bundled-dependencies\nifi-web-error-1.15.3.war}     Any help is appreciated. Thanks Regards Alireza  

Hi, I am new to NiFi, I wanted to understand the how can we connect to the oracle database through NiFi?Do we need to download the driver for the same and paste it to NiFi server ? or we have to install the database on the same server and after using that driver then only we can connect to the database. Please help me.  

Hi guys,I build a secured NiFi cluster authenticated by LDAP server.I am able to use NiFi Rest Api to access the secured cluster via Bearer Token.I have build a Nifi flow that need to call */nifi-api/* every 20 seconds and I have a problem is the Token is expire after 8 hours (based on my config) and i have to update Bearer Token manually after 8 hours :((I was wondering whether some way that I can get a permanet Bearer Token to Call REST API to access a secured NiFi cluster?Thanks.

Hi, I have done migration from CDH to CDP 7.1.7. We were using Apache Flume previously. But since Flume is removed from CDP, our data flow is broken. I want to know if there is any documents available for setting up standalone Flume that can connect CDP Kafka consumer and save data to HBase tables.Also want to note that I am using Kerberos for my cluster.  

Hello Team, We had a situation where one application consumed over 1 TB of disk space which eventually flooded the disk space. We had to kill this application for freeing the space on this disk. Due to this not happening in the future, we want to limit the storage consumption of the YARN application. Could you please share how to configure this? Best Regards

Hi  do you have any ide what  how i solve the above error

I was trying to setup Nifi cluster.All these hosts pings one another.I have tried everything i could on the web, but no success.This is my configuration for master node:Thanks.MASTERNODE:nifi.state.management.embedded.zookeeper.start truenifi.zookeeper.connect.string =masternode:2181,workernode02:2181,workernode03:2181nifi.cluster.is.node=truenifi.cluster.node.address=masternodenifi.cluster.node.protocol.port=9991nifi.cluster.node.protocol.threads=10nifi.cluster.node.event.history.size=25nifi.cluster.node.connection.timeout=5 secnifi.cluster.node.read.timeout=5 secnifi.cluster.firewall.file=nifi.cluster.protocol.is.secure=falsenifi.cluster.load.balance.host=masternodenifi.cluster.load.balance.port=6342nifi.remote.input.host=masternodenifi.remote.input.secure=falsenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.web.http.host=masternodenifi.web.http.port=9443------------------------------------------------------WORKE

Hi ,Nifi is not able to form a cluster after restart , this was working fine for months without any issues, only after restart i am seeing issues , can anyone please help ? i have below zookeeper configuration and nifi.properties and nifi-app.log information. we are using external zookeeper. zoo.cfg initLimit=100autopurge.purgeInterval=24autopurge.snapRetainCount=30syncLimit=50tickTime=2000dataDir=/ngs/app/xxx/zookeeper-data-356_1admin.enableServer=trueadmin.serverPort=9990standaloneEnabled=falseserver.1=node1:2887:3887;2184server.2=node2:2886:3886;2182server.3=node3:2889:3889;2183quorum.cnxn.threads.size=204lw.commands.whitelist=mntr,stat nifi.properties nifi.state.management.configuration.file=./conf/state-management.xml# The ID of the local state providernifi.state.management.provider.local=local-providernifi.state.management.provider.cluster=zk-providernifi.state.management.embedded.zookeeper.start=falsenifi.state.management.embedded.zookeeper.properties=./conf/

I have a list of json records : [{}, {},{},{}]I splitted them to get : {},{},{},{}i need to load them to postgres as json datatypethe result in postgres should be like that |                data                               ||---------------------------------------|| { "name":"alex", "surname":"rolfi"}|| { "name":"rolf", "surname":"syon"}|| { "name":"med", "surname":"akil"}|i did that with the following processors : but Putdatabaserecord did not solve the problem

Hi. I'm trying to reproduce a typical edge-node submission pattern for PySpark jobs using the CDE Jobs UI. To provide a module with custom Python functions that are declared as UDFs, one can specify:spark_session.sparkContext.addPyFile("python_utils.py")or use the --py-files argument with spark-submit: spark-submit --py-files python_utils.py pyspark_main.py Pure Python functions can then be imported in this way: import pyspark.sql.functions as Fdef add_udf_column(df):  from python_utils import python_func  python_udf = F.udf(python_func, StringType())  df = df.withColumn("udf_column", python_udf(df["src_column"]))  return df Attempting something similar using the CDE Jobs UI, Spark cannot seem to find the custom module. My settings are:Application File: pyspark_main.pyArguments: --py-files python_utils.pyAdvanced Options:Python, Egg, Zip files: Added python_utils.py The error I'm getting is:ModuleNotFoundError: No module named 'python_utils'

While querying postgres database using ExecuteSQL Processor 1.16.1 converts the date into UTC format. However my target system requirement is to store data in local timezone. This happens only if you set the AVRO logical data type to True. If Avro logical types set to false then it returns the localtime. I have few 100 date fields and i cannot convert each of these from the source. How do i force the ExecuteSQL with AVRO type set to return the date as-is from the source.  When Avro Type set to false : "install_date" : "2018-07-20 13:31:25.733"When Avro Type set to true:  "install_date" : "2018-07-20T05:31:25.733Z", 

 Hi  What does the eye symbol signify - is it just a view? Regards Philip

If i have to send a mail via oozie containing a spreadsheet for some data alert , how can i do it ?

Hi, I need help. I have successfully installed NiFi on Cooperates Linux Machine. But while accessing this NiFi server getting "This site can’t be reached" error. We checked everything but still persisting the issue. Please help me.   

Hi,I am getting a csv without a header then adding a header with ReplaceText to it after the ReplaceText the header is added but the rest is crashed.PS: This problem happen with this file only I tried the flow with other csv files and worked fine, can this be a csvReader problem or configuration on the encoding ? Before :The replace :  After :

We had an HDP 2.5.3.0 cluster that had an HDFS balancer CLI command that is not found with HDP 3.1.5.0.  I am adding new nodes the the 3.1.5 cluster and it appears there is some HDFS balancing going on, albeit slowly.  Did the CLI go away with 3.1.5.0 or am I missing something?

hi I am using CI/CD to deploy nifi cluster in kubernetes with rolling deploy strategy and having issue with updating flow.xml file, looks like after 1st node updated with new flow file it is going in sync with old cluster and getting old flow.xml from old master and after second deploying and getting same old file e t c, could you help with solutions to avoid this ?

Hi,  I guess this is more a feature request :  to be able to edit  CML runtime catalog (delete runtimes, to keep the catalog clean) thanks

I had issue with puthive3streaming log below.PutHive3Streaming[id=9209aacb-0180-1000-397f-3fe8b9f11781] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost: java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHostPlease give me advice 

How can I use InvokeHTTP without SSL verification? I followed this article on how to import the public certificate but that cert is not CA-signed, and InvokeHTTP doesn't work. Below is the error I get.https://community.cloudera.com/t5/Support-Questions/Is-it-possible-to-provide-options-to-InvokeHTTP-nifi/m-p/236239  In InvokeHTTP, I'm using the StandardSSLContextService with the following configs:   Thanks.Valentin  

 I need a help, I am new to NiFi. I am using python script. I wanted to add output stream relationship result of ExecuteStreamCommand processor into original flow file. In original file I am getting body and the script is generating users information. Please help me

Hi, I 'am trying to delete all the files that there are into the path from S3 Bucket. For example I have Buketname/Input_file and on this I have:File1.txtFile2.txtFile2.txt....I need to delete all the files but not the bucket.If I conect ListS3 process to DeleteS3Objet process I can delete all the files but it delete the path too.If I connect ListS3 process --> FetchS3Object-->RouteonAttribute (to obtain the file name and discard the path that came as file name too using matched and not matched) -->DeleteS3Object to do the delete action. All the process work well and I don't have error but the files is not delete.Could somebody tell me, how can resolve this issue?There are some topic that talk about how delete the files in s3 but without delete the path?Thanks

Hi I'm using pyspark and currently I am encountering an issue that I had not seen before when trying to write a dataframe to HDFS as a table. This is an example of the code I'm running:   df.write.mode('overwrite').format('parquet').saveAsTable('{}.{}'.format(target_schema, table_name))   And the stacktrace of the error I'm getting is the following:   Py4JJavaError: An error occurred while calling o1042.saveAsTable.: org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159) at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:502) at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.saveDataIntoTable(createDataSourceTables.scala:217) at org.apache.spark.sql.execution

I am trying to run Nifi in a docker container on an EC2 instance.  However, whenever my container starts I am seeing the following error.  2022-06-14 02:23:51,093 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.java.io.IOException: Failed to bind to ec2-xx-xx-xx-xx.us-east-2.compute.amazonaws.com/xx.xx.xx.xx:8082at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)at org.eclipse.jetty.server.Server.doStart(Server.java:401)at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)at org.apache.nifi.web.server.JettyServer.start(JettyServer.j