Solved

Hello, Hello,I've been trying to setup NiFi with single user authentication but got stuck in a "Access Unknown: Certificate and Token not found" error, and I'm unable to find a solution. Any help would be greatily apreciate.So, I'm trying to deploy in a remote machine with Docker, with authentication provided by NiFi, using a Nginx as reverse proxy.Maybe I have to pass p12 file to NiFi, but I have no idea how to proced.My docker-compose file is as follows version: '3.9'x-base: &base  image: apache/nifi:1.16.2  networks:    - nifix-nifi-environment: &nifi-environment  NIFI_WEB_HTTPS_PORT: 8443  NIFI_CLUSTER_IS_NODE: "true"  NIFI_ZK_CONNECT_STRING: "zookeeper:2181"  NIFI_ELECTION_MAX_WAIT: "30 sec"  NIFI_ELECTION_MAX_CANDIDATES: 2  NIFI_SENSITIVE_PROPS_KEY: "my-random-string"  NIFI_WEB_PROXY_CONTEXT_PATH: "//"  NIFI_CLUSTER_NODE_PROTOCOL_PORT: 8082  NIFI_WEB_PROXY_HOST: "nifi0:8443,nifi0,nifi0.mysite.com,n

HiHow can I make Impala  read only required partitions not all partitionswe are using the results from a sql query RegardsPhilip

NIFI version 1.16.1 nifi-hive3-nar-1.16.2   While loading data to hive tables using PutHive3Streaming, some of the tables loads are getting failed with errors. I tried to change commit size, it didn't help.  out of 50 hive table loads only 5 gets failed with this error and this is repeatable. Record count for these tables are less than 1Million.   2022-06-21 09:53:23,114 ERROR [Timer-Driven Process Thread-21] o.a.n.processors.hive.PutHive3Streaming PutHive3Streaming[id=9c9e3916-2000-1698-a0da-2dc44149819f] Aborted transaction cannot be committed: Transaction txnid:10346416 already abortedorg.apache.nifi.processors.hive.PutHive3Streaming$ShouldRetryException: Aborted transaction cannot be committed: Transaction txnid:10346416 already abortedat org.apache.nifi.processors.hive.PutHive3Streaming.lambda$onTrigger$0(PutHive3Streaming.java:512)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:360)at org.apache.hadoop.s

Hello Team, We recently upgraded the CM from version 7.2.X to 7.6.1. Since this is a production cluster we didn't do a restart of Cluster yet. However, we are getting alert with YARN HistoryServer. After checking the logs, we couldn't see any issue from the YARN HistoryServer but we found issues in the Cloudera agent. Please check the logs as shown below.  [16/Jun/2022 07:50:44 +0200] 3249 GM JOBHISTORY throttling_logger ERROR (4 skipped) Error fetching metrics at 'https://xxx.xxx.com:19890/jmx'Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/generic/metric_collectors.py", line 223, in _collect_and_parse_and_return self._adapter.safety_valve)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/url.py", line 305, in urlopen_with_retry_on_authentication_errors return function() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/generic/metric_collectors.py", line 245, in _op

I am using Nifi 1.6.0. I am trying to copy an S3 file into redshift. The json file on S3  looks like this:[{"a":1,"b":2},{"a":3,"b":4}]However this gives an error because of '[' and ']' in the file (https://stackoverflow.com/a/45348425).I need to convert the json from above format to a format which looks like this:{"a":1,"b":2}{"a":3,"b":4}So basically what I am trying to do is to remove '[' and ']' and replace '},' with '}'.The file has over 14 million rows (30 MB in size)How can I achieve this using Nifi?

Dear Community,   NiFi locally installed on my laptop was working fine until today that I got the following error in the URL.    An unexpected error has occurred. Please check the logs for additional details   This is the error that I am seeing in the log file.     2022-06-17 09:42:45,569 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@759a678a{nifi-error,/,file:///C:/Users/alshahin/nifi-1.15.3/work/jetty/nifi-web-error-1.15.3.war/webapp/,AVAILABLE}{.\work\nar\extensions\nifi-server-nar-1.15.3.nar-unpacked\NAR-INF\bundled-dependencies\nifi-web-error-1.15.3.war}     Any help is appreciated. Thanks Regards Alireza  

Hi, I am new to NiFi, I wanted to understand the how can we connect to the oracle database through NiFi?Do we need to download the driver for the same and paste it to NiFi server ? or we have to install the database on the same server and after using that driver then only we can connect to the database. Please help me.  

Hi guys,I build a secured NiFi cluster authenticated by LDAP server.I am able to use NiFi Rest Api to access the secured cluster via Bearer Token.I have build a Nifi flow that need to call */nifi-api/* every 20 seconds and I have a problem is the Token is expire after 8 hours (based on my config) and i have to update Bearer Token manually after 8 hours :((I was wondering whether some way that I can get a permanet Bearer Token to Call REST API to access a secured NiFi cluster?Thanks.

Hi, I have done migration from CDH to CDP 7.1.7. We were using Apache Flume previously. But since Flume is removed from CDP, our data flow is broken. I want to know if there is any documents available for setting up standalone Flume that can connect CDP Kafka consumer and save data to HBase tables.Also want to note that I am using Kerberos for my cluster.  

I was trying to setup Nifi cluster.All these hosts pings one another.I have tried everything i could on the web, but no success.This is my configuration for master node:Thanks.MASTERNODE:nifi.state.management.embedded.zookeeper.start truenifi.zookeeper.connect.string =masternode:2181,workernode02:2181,workernode03:2181nifi.cluster.is.node=truenifi.cluster.node.address=masternodenifi.cluster.node.protocol.port=9991nifi.cluster.node.protocol.threads=10nifi.cluster.node.event.history.size=25nifi.cluster.node.connection.timeout=5 secnifi.cluster.node.read.timeout=5 secnifi.cluster.firewall.file=nifi.cluster.protocol.is.secure=falsenifi.cluster.load.balance.host=masternodenifi.cluster.load.balance.port=6342nifi.remote.input.host=masternodenifi.remote.input.secure=falsenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.web.http.host=masternodenifi.web.http.port=9443------------------------------------------------------WORKE

I have a list of json records : [{}, {},{},{}]I splitted them to get : {},{},{},{}i need to load them to postgres as json datatypethe result in postgres should be like that |                data                               ||---------------------------------------|| { "name":"alex", "surname":"rolfi"}|| { "name":"rolf", "surname":"syon"}|| { "name":"med", "surname":"akil"}|i did that with the following processors : but Putdatabaserecord did not solve the problem

Hi. I'm trying to reproduce a typical edge-node submission pattern for PySpark jobs using the CDE Jobs UI. To provide a module with custom Python functions that are declared as UDFs, one can specify:spark_session.sparkContext.addPyFile("python_utils.py")or use the --py-files argument with spark-submit: spark-submit --py-files python_utils.py pyspark_main.py Pure Python functions can then be imported in this way: import pyspark.sql.functions as Fdef add_udf_column(df):  from python_utils import python_func  python_udf = F.udf(python_func, StringType())  df = df.withColumn("udf_column", python_udf(df["src_column"]))  return df Attempting something similar using the CDE Jobs UI, Spark cannot seem to find the custom module. My settings are:Application File: pyspark_main.pyArguments: --py-files python_utils.pyAdvanced Options:Python, Egg, Zip files: Added python_utils.py The error I'm getting is:ModuleNotFoundError: No module named 'python_utils'

While querying postgres database using ExecuteSQL Processor 1.16.1 converts the date into UTC format. However my target system requirement is to store data in local timezone. This happens only if you set the AVRO logical data type to True. If Avro logical types set to false then it returns the localtime. I have few 100 date fields and i cannot convert each of these from the source. How do i force the ExecuteSQL with AVRO type set to return the date as-is from the source.  When Avro Type set to false : "install_date" : "2018-07-20 13:31:25.733"When Avro Type set to true:  "install_date" : "2018-07-20T05:31:25.733Z", 

 Hi  What does the eye symbol signify - is it just a view? Regards Philip

If i have to send a mail via oozie containing a spreadsheet for some data alert , how can i do it ?

Hi, I need help. I have successfully installed NiFi on Cooperates Linux Machine. But while accessing this NiFi server getting "This site can’t be reached" error. We checked everything but still persisting the issue. Please help me.   

Hi,I am getting a csv without a header then adding a header with ReplaceText to it after the ReplaceText the header is added but the rest is crashed.PS: This problem happen with this file only I tried the flow with other csv files and worked fine, can this be a csvReader problem or configuration on the encoding ? Before :The replace :  After :

We had an HDP 2.5.3.0 cluster that had an HDFS balancer CLI command that is not found with HDP 3.1.5.0.  I am adding new nodes the the 3.1.5 cluster and it appears there is some HDFS balancing going on, albeit slowly.  Did the CLI go away with 3.1.5.0 or am I missing something?

hi I am using CI/CD to deploy nifi cluster in kubernetes with rolling deploy strategy and having issue with updating flow.xml file, looks like after 1st node updated with new flow file it is going in sync with old cluster and getting old flow.xml from old master and after second deploying and getting same old file e t c, could you help with solutions to avoid this ?

Hi,  I guess this is more a feature request :  to be able to edit  CML runtime catalog (delete runtimes, to keep the catalog clean) thanks

I had issue with puthive3streaming log below.PutHive3Streaming[id=9209aacb-0180-1000-397f-3fe8b9f11781] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost: java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHostPlease give me advice 

How can I use InvokeHTTP without SSL verification? I followed this article on how to import the public certificate but that cert is not CA-signed, and InvokeHTTP doesn't work. Below is the error I get.https://community.cloudera.com/t5/Support-Questions/Is-it-possible-to-provide-options-to-InvokeHTTP-nifi/m-p/236239  In InvokeHTTP, I'm using the StandardSSLContextService with the following configs:   Thanks.Valentin  

 I need a help, I am new to NiFi. I am using python script. I wanted to add output stream relationship result of ExecuteStreamCommand processor into original flow file. In original file I am getting body and the script is generating users information. Please help me

Hi, I 'am trying to delete all the files that there are into the path from S3 Bucket. For example I have Buketname/Input_file and on this I have:File1.txtFile2.txtFile2.txt....I need to delete all the files but not the bucket.If I conect ListS3 process to DeleteS3Objet process I can delete all the files but it delete the path too.If I connect ListS3 process --> FetchS3Object-->RouteonAttribute (to obtain the file name and discard the path that came as file name too using matched and not matched) -->DeleteS3Object to do the delete action. All the process work well and I don't have error but the files is not delete.Could somebody tell me, how can resolve this issue?There are some topic that talk about how delete the files in s3 but without delete the path?Thanks

Hi I'm using pyspark and currently I am encountering an issue that I had not seen before when trying to write a dataframe to HDFS as a table. This is an example of the code I'm running:   df.write.mode('overwrite').format('parquet').saveAsTable('{}.{}'.format(target_schema, table_name))   And the stacktrace of the error I'm getting is the following:   Py4JJavaError: An error occurred while calling o1042.saveAsTable.: org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159) at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:502) at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.saveDataIntoTable(createDataSourceTables.scala:217) at org.apache.spark.sql.execution