Solved

Hello! I am creating a view `view_business` with IMPALA: since the source table `table_a` is a parquet formatted table with complex data types such as array and struct, I have to always run an query option statement "SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL" to load the arrays; otherwise those array columns are showing NULL value. DROP VIEW IF EXISTS discovery_database.view_business;SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL;CREATE VIEW discovery_database.view_business AS ( SELECT * FROM table_a)Once the view is created, if I want to query the view, I still need to run the query option statement first as follows: SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL; SELECT * FROM discovery_database.view_businessHowever, the query option statement above is not supported in ODBC connector when I tried to use the statements above to link the curated table `view_business` as a data source in a power BI report.  My question is: is it possi

Hi Team, We have 2 node managers of each having cpu 16 . we restrict CPU limit of YARN by set yarn.scheduler.maximum-allocation-vcores & yarn.nodemanager.resource.cpu-vcores to 12. But CPU utilization shows 100% utilization. May i know how to troubleshoot this issue. Though we limit cpu to 12 why it is taking 100% of cpu i.e..,16. Immediate suggestions much appreciated. Regards,Hanu 

Hi Team, When we are running Msck Repair Table command it is hanging , can anyone please help me on this issue?? Regards,Hanu

Good afternoon all, and thank you in advance.I have a Apache NIFI solution that is pulling from API a pair of dates, i've grabbed out the relevant bit from the json:"arrival" : "2023-07-24T14:33:00.263Z", "departure" : "2023-07-24T14:21:15.263Z"And i'm successfully getting those values down to attributes via extract attributes.However when i perform a toNumber() on them to convert to miliseconds since EPOCH to determine the time between the dates, i return an empty condition upon review of my flow in the queue.I've gone so far as to arbitarily grab "NOW()" and attempt to toNumber() it but with the same result as trying to modify the above attributes.  incidentally NOW() returns a format like: Mon Jul 24 15:39:21 UTC 2023 - its different that my UTC formatted items, but it also doesn't want to toNumber() - returning again an empty valueShows me attempting to convert the attributes:and this shows the flow in the queue after hitting that updateattrs.  The datetimestamps are

Hello experts, I am working on a project where I have two Gitlab instances available in two different networks i.e., A and B. I need to design a solution using NiFi to read data from certain branches of different Gitlab repos available in Network A and transfer the data in real time to the corresponding branches of the Gitlab instance in Network B. I will greatly appreciate if you could give me some high level idea to develop the solution using NiFi. If relevant, please mention some standard NiFi processor which can help read in this regard. Thank you

Hi everyone,I'm trying to use NiFis Rest API to start and stop a Processor, but evertime I get the response The desired state is not setHere is my curl-Command which I use:curl -k --header "Authorization: Bearer $token" -i -H 'Content-Type: application/json' -XPUT -d'{ "revision": { "version": 18, "clientId": "01891026-da00-199f-9dc9-ae3a37dce067" }, "status": { "runStatus": "RUNNING" }, "component": { "id": "0189100e-1e29-115c-ec96-b857cd155f4a", "state": "RUNNING" }, "id": "0189100e-1e29-115c-ec96-b857cd155f4a" }' https://localhost:8443/nifi-api/processors/0189100e-1e29-115c-ec96-b857cd155f4a/run-statusHas anyone an idea what's the problem?Best regardsMaik

Hi ,          i've a flow  where there two flows one for xml file and other for corresponding Pdf file. Currently i'm merging those using Mergecontent and generating the zip file out of it.  by making sure minimum entries 2.if input files load are heavy the merge is not matching  corresponding file means (same invoice file should be merged) but sometimes other invoice file merging due to big load from input at a time. i already implemented 10 secs delay  for each flow and merging two files. is there a mechanism to merge files based on common attribute before mergecontent. Thanks in advance

I'm using the NIFI ExtractText Processor and I'm trying to come up with the regular expression to extract values from a JSON String that is in the flowfile-content coming from a response of an API (of course I cannot change how the API responds). JSON Data returned from the REST API is: {  "Result" : {    "SessionID" : null,    "ResultData" : {      "DATAENTITYNAME" : "List",      "CURRENTCURSORPOSITION" : 2,      "DATARECORD" : [ {        "DATAFIELD" : [ {          "FIELDNAME" : "name",          "FIELDLABEL" : "Name",          "DATATYPE" : "VARCHAR",          "FIELDVALUE" : "JSMITH",          "ISKEYFIELD" : true        }, {          "FIELDNAME" : "namedesc",          "FIELDLABEL" : "Name Desc.",

I need to convert the the request with Apache-nifi, can you help me how to do that? curl --location --request POST 'https://localhost:8080/token' \--header 'Content-Type: application/x-www-form-urlencoded' \--data-urlencode 'client_id=xxxxxxxx' \--data-urlencode 'client_secret=yyyyyyyy' \--data-urlencode 'type=EEEEEEEEEEE' \--data-urlencode 'scope=ZZZZZZ'  Thank for your help!!

Hello, I've been trying to use Spark jdbc to connect to a CDP cloud virtual warehouse using the Hive jar (hive-jdbc-3.1.0-SNAPSHOT-standalone.jar) but I have the following errorCaused by: java.sql.SQLFeatureNotSupportedException: Method not supportedat org.apache.hive.jdbc.HivePreparedStatement.addBatch(HivePreparedStatement.java:78)Looking into the jar sources I see that the addBatch method throws the exception by default. Spark needs to use this method to insert the data into hive. Is there any other driver that can be used to overcome this problem.If no, does cloudera will support spark to connect to CDP virtual warehouse. Thanks in advance Best regards,Diego

Hi everyone, We are unable to get the record count into the variable  after Kerberos Authentication implementation and Ranger Installation. we are using following command but it's showing count by printing. Following is we used commandI need to get the count into the variable. anyone please help me! count=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ')

How to disable/remove auto TLS certificates and create self-signed certificate in Cloudera version 6.2. The Cloudera version I am using is 6.2 having self-signed certificates that expired a few days back. now the Cloudera manager is not restarting. I want to remove the existing ones and create a new self-signed certificate and apply it to the cluster. can anyone help?

Hello Community, I can assign an attribute's value using a reference to a parameter context value ( e.g. #{variablle_abc} ). But when I attempt to do this same thing within the "Advanced" window in an UpdateAttribute processor, it fails.  I've verified the Process Group Parameter Context is set.   Anyone know how to make this work in the Advanced window of an UpdateAttribute processor?

Hello all!Is there a way to use PGR and input/output ports in the same nifi instance?The aim of that is to reuse the same processor multiple times by only reference to one PGR.If not, are there any other suggestions to achieve the same idea؟

Hello everyone, can someone please take 15 minutes with me to solve some issues in apache nifi ? i just started learning apache nifi and there is some mistakes that i don't know how to solve. i'm reay to prepare a meet for that. if anyone is available it would be so grateful !! Thank you.

Problem Summary---------------------------------------------------Behavior of auth_user table in CDH 6.3.4Problem Description---------------------------------------------------SymptomUser's first name states only one letter in hue schema under user tableQuestion:Is this the default behavior of know issue in CDH 6.3.4?User's first name has only first letter in Hue auth_user table of Hue(configured LDAP)+-----+----------+------------------+-------------+--------------------------+| id | username | first_name | FirstLetter | last_name |+-----+----------+------------------+-------------+--------------------------+| 22 | mycp2cqj | T | T | Khoo || 90 | mycp2fep | W | W | Ng || 101 | cim26384 | M | M | Mohd Baharudin || 243 | mycp1sfs | J | J | Wong || 264 | mycp22yf | G | G | Lee || 272 | mycp2hg4 | W | W | Ab Rahman || 282 | mycp1vk8 | Hui Sue | H | Khoo | <<<<<<<<<<| 284 | phcc1yb9 | R | R | Rochelle Anne C de Matta |

Hello Experts, I am using ConsumeAzureEventHub processor to consume messages in Nifi 1.16.'Concurrent Tasks' field of this processor is disabled so I am not able to speed up the consume processor.What is the reason for this limitation and is there any work around for this ?(we usually run kafka consumer with higher concurrency matching partitions) Thanks,Mahendra

I had set the properties in zoo.cfg below to enable TLSv1.3 on zookeper 3.8.0 , but in the logs I see that zookeeper is connected to other zookeeper node on TLSv1.2 that is the default version. I want to know how can I enable TLSv1.3 on zookeeper 3.8.0zookeeper.ssl.protocol=TLSv1.3zookeeper.ssl.quorum.protocol=TLSv1.3

unable add hive aux jars from CM

Hi Team, Job failing due to below error, can any one help me on this. could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation 

 hello, I have the following problem - I have a flow and I have a name attribute in it, I need to change it to uppercase. I used UpdateAtribute processor for this, as property I have name and as vaule I have:${name:toUpper()}unfortunately it doesn't work and now instead of having name in capital letters I have Empty string set, anyone know why this is happening?         

I need to load the data from big query table using nifi.I used executesql nifi processor to load data from big query table.when i was executing processor, i got following error.java.lang.NoSuchMethodError: 'void com.google.api.gax.rpc.RequestUrlParamsEncoder.(com.google.api.gax.rpc.RequestParamsExtractor)'configure DBCPConnectionPooling Service in following class name - com.simba.googlebigquery.jdbc42.DriverI am using nifi1.21.0 version. download latest version of big query jdbc driver(version 4.2) and other all dependencies and put into nifi/lib folder I also used correct connection url when configuring pooling service.How to solve this error?

Hi,I'm new to nifi, I downloaded nifi, I'm trying to log in, but I can't enter the address:https://127.0.0.1:8443/ 2023-07-10 16:16:34,682 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.java.lang.IllegalStateException: Invalid nifi.web.https configuration in nifi.propertiesat org.apache.nifi.web.server.connector.FrameworkServerConnectorFactory.<init>(FrameworkServerConnectorFactory.java:89)at org.apache.nifi.web.server.JettyServer.configureConnectors(JettyServer.java:738)at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:207)at org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)at org.apache.nifi.NiFi.<init>(NiFi.java:164)at org.apache.nifi.NiFi.<init>(NiFi.java:83)at org.apache.nifi.NiFi.main(NiFi.java:332)Caused by: org.apache.nifi.security.util.TlsException: The truststore properties are not validat org.apache.nifi.security.util.SslContextFactory.getTrustManagers(SslContextFactory.jav

Hello experts, One of our client has data stored in delta table(delta lake), I am totally new to delta lake part.Do we have any option in nifi to read data from delta lake or is there any sample 3rd party implementation available for the same for nifi? Thanks,Mahendra

Our CDP version is 7.1.7 and enabled kerberos, after we set up DAS service on it, we found that: queries only submitted by hive.keytab can be collected to DAS, queries submitted by other users' keytab such like hdfs.keytab/generic account keytab cannot be collected.So, does DAS only support to collect queries submitted by hive.keytab?If not, how can I config this?