Find answers, ask questions, and share your expertise
Recently active
I run NiFi on a docker engine (official image). It works fine but now I would like as part of a flow to connect to a MySQL instance.This requires me to configure a Database Driver Class Name. The value I found in an example on Internet (com.mysql.jdbc.Driver) requires a .jar (sorry, I know nothing about Java). What is the proper way to make this file available to NiFi? (not on a docker level - I have a volume mounted and can put is over there, but on a NiFi level, i.e. how to make NiFi aware of this file)
I have NiFi pipeline which get data from a csv and transform into a JSON. Then I want to send a post API request from the JSIN created.I need help setting up the InvokeHTTP processor in NiFi. API authorization is using an API key. I need assitance of how to define the api keys, url and JSON body inside InvokeHTTP processor
Hi,I am hitting a kryoserializer buffer issue with the following simple line in PySpark (Spark version 2.4):df=ss.read.parquet(data_dir).limit(how_many).toPandas() Thus I am reading a partitioned parquet file in, limit it to 800k rows (still huge as it has 2500 columns) and try to convert toPandas. Kryoserializer.buffer.max is already at the maximum possible value:.config("spark.kryoserializer.buffer.max", "2047m") What other ways are there to make it run (except of reducing the amount of rows even further down)? While in stage1 he is making many steps (approx 157), in stage2 he has only one step - and thus tries to juggle with a very large object. The following in an excerpt of the error message: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 158, localhost, executor driver): org.apache.spark.SparkException: Kryo serialization fai
I am getting following error in NiFi `ExecuteScript` processor. ```java18:07:09 UTCERROR9e1d9c5e-0188-1000-ffff-fffffbc2581710.42.62.250:8080ExecuteScript[id=9e1d9c5e-0188-1000-ffff-fffffbc25817] ExecuteScript[id=9e1d9c5e-0188-1000-ffff-fffffbc25817] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: java.lang.IllegalStateException: java.lang.IllegalStateException: StandardFlowFileRecord[uuid=40d42213-2c7a-49a2-8c5d-5a7fc58bad2d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1686706804350-73290, container=default, section=586], offset=6556, length=39642],offset=0,name=40d42213-2c7a-49a2-8c5d-5a7fc58bad2d,size=39642] already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed in <script> at line number 37- Caused by: javax.script.ScriptException: java.lang.IllegalStateException: java.lang.IllegalStateException: StandardFlowFileRecord[uuid=40d4
Hello,I wanted to install Impala-shell for remote querying my CDP cluster, but looking with no much success I found the latest impala-shell client is in: archive.cloudera.com/p/cdh7/7.1.2.1/redhat7/yumnothing about 8 version, nothing about my CDP version: 7.1.8... Finally going with this "old" version but now I'm facing these issues:I already have python2 and 3But not sure what more to try... no information at all in cloudera docs, even in apache impala web page 😞
we are planning to separate cluster hosts into 2 physical racks, 10 hosts for each rack. our doubt is: as we only have two physical racks, is it still recommended to use the "rack" option in cloudera manager to separate the hosts?
联系请登陆zoom:560 656 3972其中之一的报错如下:2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: rangerkms 2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: rangerkms 2023-06-21 07:57:03,612 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x488dce1cd620027, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:745) 2023-06-21 07:57:03,612 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1035] - Closed socket connection for client /10.233.68.145:49378 which had sessionid 0x488dce1cd620027 2023-06-21 07:57:03,668 - ERR
We are having escaping issues in some expressions. Just an example (using Nifi 1.21.0):ExpressionValue{"$${key}": "value"}{"${key}": "value"}{"$${key}": "${literal('value')}"}{"$${key}": "value"}As one can see, the second expression is not escaping the dollar sign as it should (from the docs, see "Escaping Expression Language"), and just stays as "$$".Any comments/hints would be much appreciated.
we are having trouble starting the "navigator metadata server" service, in the "stderr" log we have the error below: Conflicting collector combinations in option list; please refer to the release notes for the combinations allowedError: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.can help us with this problem?
I have been using multiple record oriented processors ConvertRecord, UpdateRecord, etc. in various parts of my flow. For example, my UpdateRecord processor takes about 16 seconds to read in a 30MB flowfile, add some fields to each record and convert that data to parquet. I want to improve performance such that this takes a lot less time. My infrastructure that I am working on currently is a 2 node e2-standard-4 cluster in GCP with a Centos7 operating system. These two instances have 4 vCPUs and 16 GB RAM and for each repository (content, provenance, flowfile) I have separate SSD persistent drives. A lot of the configs in NiFi are the defaults but what recommendations would anyone recommend either from an infrastructure or NiFi standpoint to improve performance on these processors.
Can I know how to fetch AWS secret manager credentials with NiFi? Which processor I have to use? Even though there is a controller called "AwsSecretsManagerParameterProvider" where do I need to use this? My requirement is to connect with the AWS hosted Database
I'm new to exploring Cloudera services, and I have a question: to use the data warehouse service, what components do I need to install and what are the minimum configurations?I plan to install it on VMware vSphere. I've learned that it requires Cloudera CDP Cloud Base and CDP Cloud Data Service (to be deployed on OpenShift with a minimum of 5 nodes). In my case, how do I determine the number of CDP Cloud Base nodes required? Do I need to install OpenShift for CDP Cloud Base?
Hello,I'm trying to edit Yarn Queue Manager queues and it seems i'm not able to edit them, there is not the option to edit them and there are no information in the doc about it...it's supposed to have this options panelBut this is not my case.the only information found it is to set the queue manager service in yarn, that's all...Any other ideas?
I have a problem when sending JSON data to Elastic from Nifi - Nifi puts [ square brackets ] around a JSON array, and so Elastic doesn't recognise the data. The current fix is to use a SplitJSON processor, which removes the square brackets, but this will become unworkable when the number of flow files gets a bit big in number. Is there a way of creating an array of JSON that is recognised by Elastic that does not contain square brackets?I am also using an older version of Nifi - Ver 1.12.1
Hi,Would need some information on how to connect to Kubernetes pods of DW, DE DF ML on AWS. Do we need to make some configurations etc, what are the ways that we can connect? Any docs /guide would be helpful. Thanks Wert
I am setting up a stand-alone CDP cluster. However, getting errors when doing the agents installation steps. I have used this command but it still error:
Hello, I've been stuck on this issue for weeks and am currently on a wit's end. For the past few weeks, I've been trying to get my Apache Nifi to be able to post messages to a MsTeams channel. I've set up a MsTeams webhook and tried to use the 'InvokeHTTP' Processor to initiate a POST request to the MsTeams webhook. I've also tried using 'ExecuteStreamCommand' processor to invoke a curl Command to send a message to my Webhook URL, but its not working. This is what's inside my curl_command.sh, curl -H "Content-Type: application/json" \ -d "{\"text\": \"Hello World from curl\"}" \ <Webhook URL> Any help would be greatly appreciated
Embedded zookeeper is failing to start while setting up a three node cluster for NIFI.The error message indicates that there is an issue with starting the embedded Zookeeper server while starting NiFi on your three-node cluster. The specific error causing the problem is "Address already in use," which suggests that the port required by Zookeeper is already occupied by another process. How can I solve this issue?
Hello, I was trying to install feast python library for which protobuf upgraded version is required. But protobuf 3.19.4 is part of available packages in ML runtime addons, so installing protobuf upgraded version is not shown in pip list command. Any suggestion on upgrading libraries which are part of ML runtime addons ?
We have a cluster running OS opensuse 12sp5. I wonder if CDH 5.14.2 supports this OS version. I‘v seen the CDH release notes, but still want to make sure.
I've been stuck for so long, I don't know what to do. Help❤️ I have installed Hive using ambari,When I try to insert a piece of data into the table, I get a Tez error,Here's what I've done:hive services and Tez services are normal. The user name I use is root, and encryption-free login has been configured on all nodes.root has its own /user/root directory and /tmp/hive/root on hdfsroot is a hive administrator and has all permissions on related databases and tablesI use yarn application-list to verify that no tasks are runningI use yarn node-list to confirm that all nodes are RUNNINGI use hdfs dfsadmin-report to confirm that all nodes are Normal and in normal stateI use hadoop fsck confirmed /warehouse/tablespace/managed/hive/root.db/test is healthyMy question is detailed below:I have a list like this0: jdbc:hive2://hdp0:2181,hdp1:2181,hdp2:2181> desc test;INFO : Compiling command(queryId=hive_20230614090425_2f598338-0603-4ba5-b4a2-8b77796b6d1b): desc testINFO : Semantic Analy
Hi to all,hope someone can point me on the right direction.I have something like that:[ { "reference" : "PB", "date" : "20230613", "barcode" : "C337287V28490010", "shipment_number" : ""}, { "reference" : "PB", "date" : "20230613", "barcode" : "C337287V28490011", "shipment_number" : ""}, { "reference" : "PB", "date" : "20230613", "barcode" : "C337287V28490012", "shipment_number" : ""} ]I need to update only a specific record with a met condition.Speaking on a SQL language it should be something like that:UPDATE FLOWFILE SET shipment_number = '${Shipment_Number_Attribute} ' WHERE barcode = '${Barcode_Attribute}'Unfortunately this statement in QueryRecord doesn't work, maybe UPDATE not supported?Moreover is there any way to achieve the same with UpdateRecord? Many thanks for any help.Appreciate.
I have a csv file which a new record comes and appends every 15 minutes. I need to extract only the latest record from the csv file and save as a JSON
Dear all, My NIFI version is 1.16.3. I am trying to fetch the status of some processors (NIFI-API) within NIFI it self using InvokeHTTP processor.My NIFI-API is protected and need authentication to be used. I have a technical user with username and password that could get a token from the token API endpoint:https://nifi.company.com/nifi-api/access/token.So I defined a controller service OAuth2AccessTokenProvider to generate access tokens. From next defined API endpoint, I want to get the proprieties of a NIFI processor by its IDhttps://nifi.company.com/nifi-api/processors/ae91429e-es3a-1eq9-fe53-b13680cff1e5I used for that the InvokeHTTP processor with a GET HTTP method and by linking the OAuth2 Access Token provider field to the previously defined OAuth2AccessTokenProvider controller service.But I am getting such error InvokeHTTP Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the
Hello Experts,My current platform is build based on below components.CDP 7.1.7Hive 3Spark 2.4.7DeltaLibraryHadoop 3.1 All our data tables are hive external tables in parquet format.At present we have pyspark streaming solution in place to read json data from a golden gate feed and process through pyspark and deltalibrary solution in cyclic fashion.For each cycle, when streaming data is processed on delta path, we merge the data from delta path to hive external path but during this update on hive path, if there is someone who performs a query on this table gets an error that HDFS I/O file not found error.To solve this, we wanted to perform automic update on hive external table such that read queries are not failed when updates happening. We have tried below ways but didn't work.1) spark.sql2) hive.sql3) df.write (using jdbc connection)4) Regular df.write with overwrite mode and overwrite = True None of these worked, however it looks like "insert into overwrite .. " query works
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.