Find answers, ask questions, and share your expertise
Recently active
hi I am using CI/CD to deploy nifi cluster in kubernetes with rolling deploy strategy and having issue with updating flow.xml file, looks like after 1st node updated with new flow file it is going in sync with old cluster and getting old flow.xml from old master and after second deploying and getting same old file e t c, could you help with solutions to avoid this ?
Hi, I guess this is more a feature request : to be able to edit CML runtime catalog (delete runtimes, to keep the catalog clean) thanks
I had issue with puthive3streaming log below.PutHive3Streaming[id=9209aacb-0180-1000-397f-3fe8b9f11781] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost: java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHostPlease give me advice
How can I use InvokeHTTP without SSL verification? I followed this article on how to import the public certificate but that cert is not CA-signed, and InvokeHTTP doesn't work. Below is the error I get.https://community.cloudera.com/t5/Support-Questions/Is-it-possible-to-provide-options-to-InvokeHTTP-nifi/m-p/236239 In InvokeHTTP, I'm using the StandardSSLContextService with the following configs: Thanks.Valentin
I need a help, I am new to NiFi. I am using python script. I wanted to add output stream relationship result of ExecuteStreamCommand processor into original flow file. In original file I am getting body and the script is generating users information. Please help me
Hi, I 'am trying to delete all the files that there are into the path from S3 Bucket. For example I have Buketname/Input_file and on this I have:File1.txtFile2.txtFile2.txt....I need to delete all the files but not the bucket.If I conect ListS3 process to DeleteS3Objet process I can delete all the files but it delete the path too.If I connect ListS3 process --> FetchS3Object-->RouteonAttribute (to obtain the file name and discard the path that came as file name too using matched and not matched) -->DeleteS3Object to do the delete action. All the process work well and I don't have error but the files is not delete.Could somebody tell me, how can resolve this issue?There are some topic that talk about how delete the files in s3 but without delete the path?Thanks
Hi I'm using pyspark and currently I am encountering an issue that I had not seen before when trying to write a dataframe to HDFS as a table. This is an example of the code I'm running: df.write.mode('overwrite').format('parquet').saveAsTable('{}.{}'.format(target_schema, table_name)) And the stacktrace of the error I'm getting is the following: Py4JJavaError: An error occurred while calling o1042.saveAsTable.: org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159) at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:502) at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.saveDataIntoTable(createDataSourceTables.scala:217) at org.apache.spark.sql.execution
I am trying to run Nifi in a docker container on an EC2 instance. However, whenever my container starts I am seeing the following error. 2022-06-14 02:23:51,093 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.java.io.IOException: Failed to bind to ec2-xx-xx-xx-xx.us-east-2.compute.amazonaws.com/xx.xx.xx.xx:8082at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)at org.eclipse.jetty.server.Server.doStart(Server.java:401)at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)at org.apache.nifi.web.server.JettyServer.start(JettyServer.j
Hello! So I can't access NiFi UI because the server if probably down exactly because of amount of files that have been stuck at queue. So the question is how can I find out the size of the queue for the moment using may be logs or maybe there are some other ways to do so? Thank you beforehand!
Hello, On our use case, we're receiving data in JSON format and it must be converted into Avro using a schema that has X and Y includes mandatory fields and everything else (Z and W) should go into a Custom map. Here is our Avro schema:"fields": [{"name": "X", "type": "long"},{"name": "Y", "type": "string"},{"name": "Custom", "type": { "type": "map", "values": "string" } }] And an example of data coming in:{"X":123,"Y":"ABC","Z":"zzz","W":"www"} We have implemented a JoltTransform to do the needed transformation to get Z and W into a Custom array.Now the issue is on doing the conversion from JSON into Avro since we get an error stating it cannot convert an java.lang.Object into a Map. How can we properly process this JSON and put that array into a Map so it can be properly stored using that Avro schema? Thank you.Best regards,Miguel
HI , I have a requirement where I need to do case-insensitive joins across the system and I don't wish to apply upper/lower functions. I tried setting TBLPROPERTIES('serialization.encoding'='utf8mb4_unicode_ci') at Table level but still the comparison is happening considering case sensitivity. PSB - drop table test.caseI;create table test.caseI(name string, id int)TBLPROPERTIES('serialization.encoding'='utf8mb4_unicode_ci');insert into test.caseI values ('hj',1); drop table test.caseI_2;create table test.caseI_2(name string, id int)TBLPROPERTIES('serialization.encoding'='utf8mb4_unicode_ci');insert into test.caseI_2 values ('HJ',1); select * from test.caseI iinner join test.caseI_2 i2 on i.name=i2.name;--No ResultTried with encoding 'SQL_Latin1_General_CP1_CI_AI' but got same result as above.Any help would be appreciated, thanks!
Hi,I want to send specific flowfiles to a specific remote MiNiFi Java.Testing with 1 MiNiFi works great, out from NiFi, into MiNiFi for processing.I'm prepping the MiNiFi flow on NiFi, templating, exporting, converting to yaml, modifying any security settings i need and restarting MiNiFi to accept the updated config.yml. Now i'm adding additional MiNiFis.On NiFi I've setup a PG with a RouteOnAttribute (Attribute is the specific MiNiFi name) to send to a specific S2S Output Port for that MiNiFi. Up to here, all seems fine. The problem i have is in the MiNiFi template.I have the RPG going into a processor, and that's when i specify the From Output Port which is specific to the MiNiFi.How can i template/generalize/variabilize/other the From Output so that i don't need to make separate template for each and every MiNiFi instance?I could theoretically move the RouteOnAttribute from the NiFi to the MiNiFi side but at that point it means i already sent a ton of data to all the MiNiF
Hello everyonei have a content in Json (an array of id)[{"id":7532233},{"id":755163}] and i need to convert this whole array into one attribute so that i can after that refer to it in an ExecuteSql processor with statement :update column 1 where id exists(${attributeName})
I am using the utility of wait and notify and for that I have used Distributed map cache client service and Distributed map cache server, but somehow I am getting an error in service "too many open files", I have even increased the "nofile" in node. In the same instance but in another nifi flow I am able to use this functionality but if I do configure this service in another parallel flow I face issues. Below I am attaching the screenshot of conf and error. any help would be appreciated.
I am using the latest version of impala JDBC Driver 2.6.27.1032, when my sql is "upsert into my.testtable(id,name,insert_time) values(?,?,?)" and using preparestatement, then the column "insert_time" will be replaced to "upsert_time", so my sql be changed to "upsert into my.testtable(id,name,upsert_time) values(?,?,?)" and i get a error "errorMessage:AnalysisException: Unknown column 'UPSERTby' in column permutation",。How can I solve this problem?please!
cdp7.1.6SQL_STREAM_BUILDER-1.12-csa1.4.1.0-cdh7.1.6.0-297-15479260
Hi I have created a new entity and I would like to show all the attributes by default while doing the 'Create new entity' process. At present, the UI mask shows only the Required attributes. How can I do this?
Hi,I want to insert data to Ms. Sql's server from nifi flow.Where can I find a sample or a guide?Also, where have I must install sqljdbc-4.0.jar?
We recently added a two nodes to our cluster through ambari wizard, we installed datanode, nodemanager, Metrics Monitor, LogFeederThe datanode/nodemanager are starting correctly by not livetopology_mappings.data was updated in both mnode and cnodes cat /etc/hadoop/conf/topology_mappings.data[network_topology]cnode2.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.172=/default-rackcnode5.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.169=/default-rackcnode4.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.175=/default-rackcnode3.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.67=/default-rackcnode1.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.188=/default-rackcnode6.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.9=/default-rack datanodes have 2 external disks to store hdfs data [root@node6 ~]# df -hFilesystem Size Used Av
Hi Team,From last few days, data03 node was not working, so we moved it to decomissioned state and restarted all our services. Now i need to recomission and start the services here as well.I selected the host and under actions clicked on End maintenace, but it is failing.Please help me out with what shall i do to resolve this?
Hi everyone, I am having some troubles with Nifi Cluster configuraition (version 1.16.2). Actually, the nodes are up and I can access them via https. But then: When i checked the logs, the error was:WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered WARN [Heartbeat Monitor Thread-1] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine leader for role 'Cluster Coordinator'; returning null I tried a lot of possible solutions, but i still got the problem My nifi.properties configuration:nifi.cluster.is.node=truenifi.cluster.nod
Can someone point me how to get it configured? What's the reason HDP/Cloudera dropped the embedded view from Ambari to begin with? It was a useful tool to see the details of a hive/tez job, good for troubleshooting.
We are planning a production cluster and right now I already have the Mysql being replicated but only have one Cloudera Manager . However we need to patch etc and sometimes have to take downtime of the CM . Is there a way to add a 2nd cloudera manager to the same database ?
Hi Team,Can anyone please help me with the below file: cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpmi am unable to find it as the below site is giving 404 error:https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/Regards,Jessica Jindal
Hi,after having deleted tera bytes of data from HDFS (1/4 of the total capacity), the block count among data nodes did not decrease as expected. It is still over the critical threshold.How could it be solved?Thank you
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.