Q&A

We have 3 node apache nifi cluster, built on same versions of nifi i.e Apache nifi 1-.16.0 and Open JDK 11. When trying to delete a funnel from below loop, one of the node gives an error on UI "###### node is is unable to fulfill this request due to: An unexpected error has occurred. Please check the logs for additional details."Nifi infinite loopEnabled debug logs to see more details  and unfortunately it ended up in  java.lang.StackOverflow error from the below logs. Seems like flow ended in weird state.  We tried deleting work directories on cluster and restarting cluster. It din't help though ERROR [NiFi Web Server-34943] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.StackOverflowError: Delayed StackOverflowError due to ReservedStackAccess annotated method. R eturning Internal Server Error response.91796 java.lang.StackOverflowError: Delayed StackOverflowError due to ReservedStackAccess annotated method91797 a

Hi,    We are trying to authenticate Nifi with single-user-authorizer using the auto-generated username and password itself. We thought of changing the token expiration time. But, we couldn't find any configuration for token expiration for this default authentication. There is an api which gives the expiation time of the token but the token is still active even after the expiration time we got from the api. Is there any configuration to change the duration of active token?  Or Is there a way to get the actual token expiration time?Where can I find the document for working of token expiration for nifi default authentication?

Need help  to convert a date from below JSON file from mm/yy to yyyy-mm format using NIFI{    "book_id": "3791"    "Key": {        "value": [            {                "Number": "553",                "Date": "05/20"            },            {                "Number": "5537290000000522",                "Date": "03/22"          

I moved a working flow that is populating Solr indexes from one process group to another. In the original, the SolrLocation property of the PutSolrContentStream processor is populated using two parameters:#{solr_url}/#{solr_index_name_a}It's done this way because a QueryRecord processor is used to split the record set into two groups, and one path uses the "a" index, and the other, the "b" index. However, in the new flow, I have to append a year value (i.e., "2023") to the name of the index depending on earlier processing. To accomplish this, I am holding the name of the index in an attribute instead of a parameter. At the appropriate time in the flow, I use an UpdateAttribute processor to append the correct year to the index name. Then further down, I have the PutSolrContentStream processor, and I populate the SolrLocation property like this:#{solr_url}/${solr_index_name_a}This fails with an "HTTP ERROR 404 NOT FOUND". It took a lot of trial an error, but I have discovered if I h

Hello,we are running the following Kudu cluster configuration:1 master3 tserversVersion: 1.16The total number of live tablet replicas is 552, as inspected via tserver UI.Tables are partitioned with both range and hash partitioning, following Kudu guidelines.Most of tablets are bound to historical tables, i.e. data is continuously inserted without being further modified.On the other hand, we have some "lookup" tables keeping only the last value of other tables, i.e. a lot of updates is continuously performed on these tablets.What we noticed from this scenario is that most of the disk space is occupied by Kudu metadata which seems to be unused during normal insert/scan operations. And this metadata seems to have an high impact, occupying around 50% of our total disk space (which is 30GB per instance, right now).After having rebooted one of the instances, we have noticed that Kudu is executing a sort of cleanup/data compression that allows to free around 40% of disk space, in particular a

Hello,due to a lack of memory on a DE cluster in DataHub while executing Spark jobs, we are evaluating the resizing of worker/compute nodes with more performant EC2 instance (no need to add more nodes, just increase the available memory).At the moment we have 3 worker nodes and 4 compute nodes, running on m5.4xlarge instances.Is it better to resize both the nodes type or we can resize, for example, only the compute nodes? Thank you,Andrea 

In Cloudera Data Flow, I'm using the `QueryDatabaseTable` processor with a maximum-value column to keep track of which records have already been processed. When I'm in Flow Designer testing the flow, I can see that it's maintaining state. Does this state persist if I deploy it, or does the deploy clear the state? I expect it would be the latter, but want to check.Follow up question: Say I make changes to the flow and redeploy. Is the state cleared this time, or does it use the last deployments flow? I'd like to be confident that making changes to the flow doesn't cause it to reprocess all the data again.

I am trying to train a random forest on a dataset containing around 1 Million records and 218 variables (size 70 MB), with a split train (70%) test (30%) and save the quality metrics calculated on the test set (accuracy and f1 score).  I managed to run the random forest if I set the number of trees on a low level, use not very deep trees or only use a sample of the data, but as soon as I try to run a randomforest with 150 trees with a higher maxdepth (depth of the tree) on the entire train set, I get an error: Py4JJavaError: An error occurred while calling o1945.evaluate.: java.lang.OutOfMemoryErrorThis error seems to be a problem related to the driver memory: in the spark web ui there seems to be no problem, no failed job, just it stopped to run the next jobs and my spark session in the cdsw can no longer communicate with the cluster and the spark application. And when I am running the randomforest with the same parameters on a 10% sample of the data, it works but

Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://accounts.google.com/.well-known/openid-configuration": sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target;this is error which im gettingmy nifi.properties file is as follows :- nifi.security.autoreload.enabled=falsenifi.security.autoreload.interval=10 secsnifi.security.keystore=./conf/keystore.jksnifi.security.keystoreType=jksnifi.security.keystorePasswd=tyiImfLpqM5/qX1l0VsjK7sxhQx5YZFOZgstogGz8Eknifi.security.keyPasswd=tyiImfLpqM5/qX1l0VsjK7sxhQx5YZFOZgstogGz8Eknifi.security.truststore=./conf/truststore.jksnifi.security.truststoreType=jksnifi.security.truststorePasswd=vFF0nH+45S5ESohcQsH37s/bqJ/Kmy5RiHpL+ZWo2VUnifi.security.user.authorizer=managed-authorizernifi.security.allow.anonymous.authentication=falsenifi.security.user.login.identity.

I need to retrieve all flowfiles in a specific queue in NiFi, but the API only returns up to 100 results at a time. My queue contains 358 flowfiles, so I need a way to retrieve all of themBelow are the API am using for get the flowfiles.API : https://myserver.com:myport/nifi-api/flowfile-queues/ba619122-3c65-3279-a2ba-b3ad89f8a429/listing-requests/903d485f-d4aa-102a-0000-0112e4a15ea {"listingRequest": {"id": "903d485f-d4aa-102a-0000-0112e4a15ea","uri": "https://myserver.com:myport/nifi-api/flowfile-queues/ba619122-3c65-3279-a2ba-b3ad89f8a429/listing-requests/903d485f-d4aa-102a-0000-0112e4a15ea","submissionTime": "10/18/2023 17:07:57.632 IST","lastUpdated": "17:07:57 IST","percentCompleted": 1,"finished": true,"maxResults": 100,"state": "Completed successfully","queueSize": {"byteCount": 3792,"objectCount": 350},"flowFileSummaries": [{"HERE IS THE FLOW FILES TILL 100TH POSITIONS": 100}]}}

Hi,I have been using the cdswctl tool for a while for creating sessions that can be accessed by VS Code from Windows.Today i tried to create a session from a Mac using the same command and it throws an error complaining that the API key is not valid.   I have checked the key and same key works fine from Windows.The command i am using is :./cdswctl login -n USERNAME -u http://cdsw.mydomain.xxxx/ -y APIKEYCan anyone offer any advice?Thanks

Hello,I created several Ranger Hive policies via APi or GUI, but some of them throw error "Another policy already exists" during UPDATE policy via API or GUI too.I am able to drop and create policy, but update should work too.Have you any idea, what is the problem?Best regardsKamiled

Hi when i'm performing calculations for "Importo" attribute, it computing correctly , but since it is an array i'm getting repetitive of  "Importo" : "." for each item level.  Can you help in get rid of this unwanted  "Importo" : "." values.INPUT:{"FatturaElettronicaBody": {"DatiGenerali": {"DatiGeneraliDocumento": {"Numero": "130889","Data": "31.10.2023","Causale": "22122013260655367-000180 DEL 20/12/2022","Divisa": "EUR","ImportoTotaleDocumento": "25369.31","TipoDocumento": "${TipoDocumento}"},"DatiDDT": [{"NumeroDDT": "287421-30.10.2023"},{"NumeroDDT": " 287419-30.10.2023"},{"NumeroDDT": " 287423-30.10.2023"},{"NumeroDDT": " 287422-30.10.2023"},{"NumeroDDT": " 287420-30.10.2023"}]},"DatiBeniServizi": {"DettaglioLinee": {"AltriDatiGestionali": {"RiferimentoTesto": "22122013260655367-000180 DEL 20/12/2022","RiferimentoData": "22122013260655367-000180 DEL 20/12/2022","TipoDato": "INTENTO"},"NumeroLinea": [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,"addnewline_1"],"Quant

Hi, i have cluster with knox installed and kerberos autentication enabled, i want to access the knox admin ui to make changes in the topologies, the thing is that i dont really know what is the admin-ui credentials, i tried using my admin from the kerberos but it doesn't work. my credentials from the cloudera manager also dont work. Everything is installed in the same cluster so i wanted to see if someone hava any idea what im doing wrong or  any idea of what credencials am i supposed to use.Thanks in advance

I started DistributedMapCacheClientService on Controller Services.when i try to start FetchDistributedMapCache processor got an error "AnnotatedConnectException: Connection refused".Do you have any comments on this situation?@MattWho 

HelloI was able to install CDS (spark) 3.3 with CDP 7.1.7-SP2. However, I am not able to do it with CDP 7.1.7.Can someone confirm the latest spark supported bu following CDP(s):1. CDP 7.1.7 - ??2. CDP 7.1.7 SP1 - ??3. CDP 7.1.7 SP1 - CD 3.3 works here.

How to use FFmpeg in NIFI . I want to convert Video to Tiff .

I have a flow that queries data from a source table and loads it into an iceberg table. The source data includes a Date type which gets read in as a string. The destination table has the same field where the schema is also a Date. When I try to use the PutIceberg processor, I get the following error:org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Failed Conversion of Field [null] from String [2023-10-30] to LocalDate- Caused by: java.lang.NumberFormatExceptionThe flow file is Avro and I'm using an AvroReader service for the PutIceberg. Here's a sample of the Avro:[ {"scan_id" : "1698665963-82DD686DBC1979E6FF6C5443BA82456621DB5004","queue_seq_id" : 558745163,"flight_airline_code" : "AC","flight_number" : 8663,"flight_departure_date" : "2023-10-30","scan_airport_code" : "YHZ","scan_checkpoint_name" : "Transborder","scan_stage_name" : "S2 - Screening Lines","manual_entry" : "N","wait_time_pax_queue" : "Main Queue","wait_time_seconds" : 93,"included_in_counts" : "

Trying to load and parse YARN application metrics and store it for performance analysis by using REST api call. Data returned is a single json entry with multiple application entries nested in a complex format. Can someone suggest or provide correct DDL how to map all data points to a hive external table since JsonSerde can parse and tabulate it?Sample json data format: {"apps":{"app":[{"id":"application_282828282828_12717","user":"xyz","name":"xyz-4b6bdae2-1a0c-4772-bd8e-0d7454268b82","queue":"root.users.dummy","state":"finished","finalstatus":"succeeded","progress":100.0,"trackingui":"history","trackingurl":"http://dang:8088/proxy/application_282828282828_12717/","diagnostics":"session stats:submitteddags=1, successfuldags=1, faileddags=0, killeddags=0\n","clusterid":282828282828,"applicationtype":"aquaman","applicationtags":"ABC,xyz_20221107070124_2beb5d90-24c7-4b1b-b977-3c9af1397195,userid=dummy","priority":0,"startedtime":1667822485626,"launchtime":1667822485767,"finishe

I am new to Apache NiFi and  have a fresh apache NiFi instance on my local system and i want to create 3 users and authenticate them with openID/google oAuth from their email ID( instead of seeing the usual NiFi login screen, i want to see the google authentication window where i can use my email id to login).i tried to follow the bryan bende's article on it but it feels very outdated for now. (https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect)please help me with this and provide me with a step-by-step breakdown on how can i achieve this? Thanks

Hi Team,I have Nifi connectivity with ioT Hub. In ioT hub i have device under which publisher and subscriber is there. Nifi is configured with ioT hub and its working fine however intermittently the connectivity breaks between ioT and nifi.Do we have any mechanism or process in nifi to identify when connectivity breaks it retry the connection. If restart the nifi service manually then again connection establishes. I want to automize this process either from ioT hub or from nifi side if connectivity breaks then it should retry connection and restart the nifi service automatically.@cotopaul @Former Member  @ApacheNifi 

I have a problem like this.I am using ApacheNifi nifi-1.23.0 version.Generate flowFile with GenerateFlowFile processor as follows.When I view the content, I see that it is as I shared below.As far as I understand, it mixes the content I triggered with a different content.What could be the reason for this?    

Nifi Error : failed to import data from buffer | unable to create content claim due to Java io EOFException.Please help to resolve this nifi UI issue .  

[root@b7-40 ~]# hiveSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]

I started the Cloudera training a few years back, but did not complete it.As technologies have change over the years, is the Cloudera training on Coursera still relevant to study Cloudera's current offerings?If not, does Cloudera have plans to do anything with the materials on Coursera at this point ?Are there systems to replace the Cloudera VM for the training lessons on Coursera which may or may not come from Cloudera ?