Q&A

Hi Team, We are currently facing an issue in NIFI with OIDC as an authentication mechanism. Below is what we are seeing when NIFI URL redirects to OIDC(keycloak) url. OIDC works perfectly well with 1.20.0 version but when upgraded to 1.21.0 we are getting below error. ERROR: Unauthorized error="invalid_scope", error_description="Invalid scopes: openid address email web-origins profile user phone microprofile-jwt roles groups offline_access" Please suggest if there is any significant change made from 1.20.0 to 1.21.0 nifi oidc configuration which might have caused the issue. Appreciate your help in advance.

Hi Team, I am new to Nifi servers.I am currently working on a Development project where i developed a job to fetch changes from Mysql databases and apply them in a warehouse. I have 70 similar but different databases and therefore 70 Nifi jobs. The final aggregated database is around 100 GB. and daily some 1GB changes from all databases. I am currently working on development environment and using a single server.Now when i move to production with 70 Nifi jobs running all the time, do I need to think of a Nifi cluster ? or it is fine with single server ? What factors decide cluster vs stabdalone ? Also How can I handle upgrades to Nifi in future, Is it better for me to use Docker ? or having a normal instance is better ? Please help me with your valuable knowledge..

Is it possible to select with the QueryRecord processor also child fields of an given JSON structrue? E.g. following JSON: { "name" : "Max", "id" : 1, "location" : { "city" : "Frankfurt" }} And I would do the following query: SELECT name, location.city FROM FLOWFILE But this throws an error: QueryRecord[id=a4744340-017a-1000-6944-73fe4d362d90] Unable to query StandardFlowFileRecord[uuid=706e8ae8-f76c-4ebb-9c65-c46cb7026c89,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1626251405764-2, container=default, section=2], offset=790026, length=67],offset=0,name=55b72cfd-5308-4e83-9304-166193e76478,size=67] due to java.sql.SQLException: Error while preparing statement [SELECT "name", "location.city" FROM FLOWFILE]: org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Error while preparing statement [SELECT "name", "location.city" FROM FLOWFILE] Is there something that I'm doing wrong? Here is the configurati

I have issue in importing the data from dataframe converted to csv then uploading it into hive but its not loading properly .My procedure:1st I took a Data frame from database and converted into a csv ,which has 343 columns and 24 lakhs rows 2nd I took the csv file to hive and I loaded the data to hive using load data code to table which i created directly by connect the hive to same database .this is what ,I am doing.In this case , my issue is for some rows it taking proper values but for some is null  or 0.then i took a sample of 5 rows and I checked manually then i find out in csv file some rows there are some extra comma .so I manually removed and tried ,it worked but this cant be happening in real-time .so pls help me on this by giving some suggestion.

I am trying to roll my own search_after pagination query by saving the last sort values to a db and then retrieving them on the next time I run the query.  Thing is if there are no sort values from the db this means this is a new query and I skip the search_after clause in my query. If I do retrieve a set of sort values from the db then I need to plug them into the query. I am having trouble with the EL statement. This is what I have:   { "query": { "bool": { "must": [ {"term": {"key1": "value1"}} ], "must_not": [ {"exists": {"key2": "value2"}} ], "filter": [ {"range": {"@timestamp": {"lte": "now-1w"}}} ] } } ${search_after:isEmpty():not():ifElse(', \"search_after\": "${search_after}"', '')}, "sort": [ {"@timestamp": "asc"}, {"value3": "asc"} ]}   The ${search_after} variable defined as  '[ts int milliseconds, "a key value string"]'  Including the sq brackets. The error I am receiving is:Unknown key for a VAL

Hi, I am running CDH 7.1.7p0.  I have a external ORC database on S3 bucket.  I did not configure this fs.s3a.server-side-encryption-algorithm in hadoop core-site.xml.  However, I will randomly get this error when running hive query.  Rerun the same query may success or fail with same error message.  ERROR : Vertex failed, vertexName=Map 14, vertexId=vertex_1684599244994_0012_7_02, diagnostics=[Vertex vertex_1684599244994_0012_7_02 [Map 14] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: ss1 initializer failed, vertex=vertex_1684599244994_0012_7_02 [Map 14], java.lang.RuntimeException: ORC split generation failed with exception: java.lang.RuntimeException: java.io.IOException: Cannot find password option fs.s3a.bucket.sg-cdp.fs.s3a.server-side-encryption-algorithmThe database and query was part of the hive benchmark tool from github:https://github.com/hortonworks/hive-testbenchCan you help me to resolve this problem.   Thank you.&

Dear community,I am writing to you because I have the following problem: How can I connect to impala tables from Cloudera AI Workbench, if I am setting up a session where R 3.x or R 4.x is used. I managed to make the connection using python, but I need to do it using R as well. I have the connection host, the port, the authentication mechanism. For example, the following python code manages to connect: "from impala.dbapi import connect conn = connect(host='host_name', port=21050, auth_mechanism='GSSAPI', use_ssl=True, kerberos_service_name='impala') cursor = conn.cursor()"  But I need to achieve the connection now using R in Cloudera Data Science Workbench. Does anyone know how to translate the above code to R 3.x or R 4.x? I already tried using the 'RImpala' library, but it was removed from the CRAN repository according to https://cran.r-project.org/web/packages/RImpala/index.html. If anyone has an idea how to help me I would be very grateful.

Set "OAuth2 Access Token provider" in Invokehttp prosessor with grant type as client credentials as well as client id and client secret, thinking no need to Set Authorization Bearer token in this processor, the processor will dynamically set retrieved token with post request to get remote API response. But always to 401 error back. Not able to tell it is error from token provider or from remove API to send post request. However, if using Postman, in a post  request, I can do it successfully to get remote API response. So I am sure the Grant type, client id and clietn secret are correct. What I am missing to get it work? WIl be very appreciate if NiFi experts guide me to configure invokeHttp processor for oauth2 seeteing to resolve my problem?

Is there a way to convert use expression language to convert the current time to a specific timezone.  Here is what I have currently but that's GMT not GMT+10:00. ${now():format("yyyy-MM-dd'T'HH:mm:ss.SSSXXX", "GMT")}

I am storing several JSON Schemas as Parameter Contexts, and I need to access the schemas dynamically to validate flowfiles.  But which schema depends on the flowfile, so I want to access a particular schema in the Parameter Contexts dynamically using an attribute value.  How do I do that? Here is what I've tried:#{${schema.name}}${schema.name:evaluateELString()} The second technique I am using creates the exact string that I need #{Blue_JSON_Schema}, and then I try to perform an eval on it.  But unfortunately it doesn't work either. Hopefully someone knows how to do this.  Thanks for taking the time to read this.      

I'm attempting to transition from using PutBigQueryBatch to PutBigQuery. However, the latter doesn't seem to evaluate the dataset and table attributes as expected.In the provided example, one processor has the values hardcoded, which works correctly, while the other processor relies on retrieving those values from the flowfile's attributes. However, it appears that the attributes are not being properly evaluated. The example flow (using Nifi 1.21.0):The flowFile used: The left PutBigQuery processor has hardcoded values, and it works:The right PutBigQuery processor fails because it seems it's using the EL definition but it's not evaluating it: Any help would be appreciated.

I have a multinode NiFi Cluster running in Docker Containers and I configured the DistributeMapCacheClient and DistributedMapCacheServer to point to a particular container's hostname and set the persistence directory property such that each node in the cluster uses the same Map Cache. I want to make that the MapCache persists even after I restart the container or create a new container. How would I configure that to happen?

Hi, I am trying ro read from a json file and create flowfiles with the kews as attributes and one flowfile per value in the list. The json file looks something like this:{"key1": null,"key2": null,"list": ["foo", "bar"],"key3": 1} This is my execute script:  import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallbackfrom org.apache.nifi.processor.io import InputStreamCallbackflowFile = session.get() if (flowFile != None): stream_content = session.read(flowFile) text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8) json_data = json.loads(text_content) for i in json_data["list"]: flowFile2 = session.create() flowFile2 = session.putAttribute(flowFile2, "key3", str(json_data["key3"])) flowFile2 = session.putAttribute(flowFile2, "key1", json_data["key1"]) flowFile2 = session.putAttribute(flow

Hi,Please let me know how to connect the Google Drive in nifi and read the files.Which Processor to be used.Please help in this regards,Best Regards,Rana

Hi, we get the following Ranger error - maybe you can help me to fix it as soon as possible?! (We activated MIT Kerberos). Thanks in advance! How can I check that the password of the keystore file is correct? And where can I change it? Feb 17, 2020 4:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startINFO: Provided Kerberos Credential : Principal = rangeradmin/pdeluh0004392.hub.deluh.example.com@RDDL.PROD.EXAMPLE.COM and Keytab = /etc/security/keytabs/rangeradmin.service.keytabFeb 17, 2020 4:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer$1 runINFO: Starting Server using kerberos credentialFeb 17, 2020 4:29:57 PM org.apache.coyote.AbstractProtocol initINFO: Initializing ProtocolHandler ["http-bio-6182"]Feb 17, 2020 4:29:57 PM org.apache.coyote.AbstractProtocol initSEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-6182"]java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineL

Hello NiFi Community, I have an AvroSchemaRegistry controller service and it references a bunch of schemas stored in a parameter contexts variables.  Both the AvroSchemaRegistry and Parameter Contexts are at the same scope level. However, I have a very large number of schemas so for organizational purposes I'd like to organize them into separate Parameter Contexts: red_schemas, green_schemas,  etc...But I'm still using just the one AvroSchemaRegistry controller service.   Is it possible to create child Parameter Contexts (red, green, etc...) that inherit from the main Parameter Context and then reference the variables from a single controller service, which is at a higher scope?  If so, what is the syntax for referencing variables in a different scope than the controller service? Thanks for reading.

Hi Team, I want to setup kerberos client in my Mac Laptop having MacOS Monterey (version 12.6.5). I have put the krb5.conf file at below paths./etc/krb5.conf/Library/Preferences/edu.mit.kerberosBut when I try to run kinit, i get gelow error.--kinit -kt /Users/banshidhar_sahoo/Desktop/POC_KEYTAB/test.headless.keytab test@EXAMPLE.COMkinit: krb5_get_init_creds: unable to reach any KDC in realm EXAMPLE.COM, tried 0 KDCs-- I have also set the ENV Variable as below:KRB5_CONFIG=/etc/krb5.conf But getting same error while doing kinit. Can you please suggest how to point to krb5.conf so that it can reach out to the correct kdc server. Regards,Banshi. 

I have two processes1. consumerkafkarecord --mergerecord  --puthdfs2. consumerkafkarecord --mergecontent--puthdfswhen I use process 1, I have files in ndfs on the output readable spark, database, python libraries without problems, but the files are not larger than 200mb, all different sizes, although 500mb is set, but it is not filledwhen I use process 2 with the same parameters in mb and the number of lines, the files I get are exactly 500mb, but these files do not open, not by spark, not by any database, not by python librariesquestion why?I also want a large file always 500mb and so that it can be read without problems as in process 1mergecontent settingsMerge Strategy Bin-Packing AlgorithmMerge Format Binary ConcatenationAttribute Strategy Keep Only Common AttributesCorrelation Attribute Name No value setMinimum Number of Entries 10000Maximum Number of Entries 1000000Minimum Group Size 100 MBMaximum Group Size 500 MBMax Bin Age No v

How do I restrict the access to an endpoint provided by HandleHttpRequest to specific users? I did setup successfully a HandleHttpRequest-HandleHttpResponse chain, with Client Authentication = "Need Authentication". I provided a key- and truststore and users with a client certificate trusted by CA cert in the truststore can access the endpoint. However, I do not want everybody in my company to access this endpoint, I want to restrict this to a small group of users (actually to one single server). For testing purposes I created an empty truststore and imported into this truststore only my client certificate. However, now I cannot access the endpoint anymore (ERR_BAD_SSL_CLIENT_AUTH_CERT). How do I configure the JKS truststore to only trust one specific client? Or how do I configure HandleHttpRequest processor? If this is not possible, are there any other possibilities?Could I somehow use a Nifi-REST-API-Token?Could I somehow authenticate with the help

hello cloudera community, we check in the graph "total_kudu_on_disk_size_across_kudu_replicas" and there are tables with 500GB with that, we need to know what is the recommended size for a kudu table?

Nifi provenance data is not visible and remains at 80% in searching for provenance events after upgrading from 1.13.0 to 1.15.3 Below are the error logs : 2022-04-05 23:51:22,905 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Eventsorg.apache.lucene.store.AlreadyClosedException: this IndexWriter is closedat org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at jav

Hi, does anyone knows whether there are any restrictions on using external shuffle services on CDP?For instance, are the following external shuffle services supported?:- uber rss- apache uniffle

Hi team, We were in process of migrating the flow from HDF to CDP. We migrated few and while testing the flow they are working fine on new home we encountered an issue. We were working on flow in which we encountered and issue with processor 'InferAvroSchema 1.18.0.2.2.6.0-260' where as procssor 'InferAvroSchema 1.9.0.3.4.0.22-1' which was working fine with the data. While working with new processor we encountered an issue which is listed below:  ERROR b11042ff-c573-1d90-0000-0000338c6dea rit-dev-nifi-nifi0.regn-cld.fxd6-2a7z.cloudera.site:8443InferAvroSchema[id=b11042ff-c573-1d90-0000-0000338c6dea] Processing halted: yielding [1 sec]: java.lang.NoSuchMethodError: org.apache.avro.Schema$Field.<init>(Ljava/lang/String;Lorg/apache/avro/Schema;Ljava/lang/String;Lorg/codehaus/jackson/JsonNode;)V Nifi Verson: 

While attempting to migrate data from MSSQL to Postgres with nearly a thousand tables, I am currently retrieving the list of tables (LDT) and splitting it into individual tables using splitAvro. Now, I'm seeking a solution to efficiently map these tables to their corresponding names in the target system and transfer the data to the tables that share the same names in both the source and target databases. Additionally, I would like to know if Nifi offers a feature to convert and migrate schemas between different databases.Thank you.

Hi Team, I trying to configure Apache Nifi LDAP authentication, Here is my nifi.properties. -----nifi.security.autoreload.enabled=falsenifi.security.autoreload.interval=10 secsnifi.security.keystore=/var/tmp/mks/certs/keystore.jksnifi.security.keystoreType=PKCS12nifi.security.keystorePasswd=************nifi.security.keyPasswd=nifi.security.truststore=/var/tmp/mks/certs/keystore.jksnifi.security.truststoreType=JKSnifi.security.truststorePasswd=**************nifi.security.user.authorizer=managed-authorizernifi.security.allow.anonymous.authentication=falsenifi.security.user.login.identity.provider=ldap-providernifi.security.user.jws.key.rotation.period=PT1Hnifi.security.ocsp.responder.url=nifi.security.ocsp.responder.certificate=------ Here is my "login-identity-providers.xml" -------><provider><identifier>ldap-provider</identifier><class>org.apache.nifi.ldap.LdapProvider</class><property name="Authentication Strategy">LDAPS&