Q&A

Hi, I'm trying to query a Hive database on NiFi, but I get the following error, both Hive and NiFi are hosted on the same Docker container.I downloaded and put 'nifi-hive-nar-1.25.0.nar' and 'nifi-hive-services-api-nar-1.25.0.nar' files in the lib folder to add the processor as It was not built in for my installation.I'm trying the following configuration :This is the part of the schema where I'm encountering this issue:Maybe the issue is with the connection URI ? Regardless any help to steer me to the solution would be nice, thanks.

How we  can get the content of file from Extract Email Attachments process   ?  if we see there are Many process coming after ExtractEmailAttachments process and in the end the put file process create file but the content of file is the value of variables for process which before the putfile processConsumeIMAP -> ExtractEmailHeaders -> ExtractEmailAttachments  -> UpdateAttribute -> ExecuteSQL -> ConvertAvroToJSON -> EvaluateJsonPath -> put file

hiveserver2 web UI page opens but cannot see any details like no of active users and all .I am using cdp 7.1.3 in which kerberos and ranger both are enable..how to resolve this ?

Below is the error in bulletin board: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archive cleanup. Total number of filescurrently archived = 1My flows are simple and this is just test environment, so not much flow files as well may be some 10-20 only. My content_repository is mounted on a disk. it is just 8MB occupied. Disk has some 100 GB storage.  below are properties of content repository:  # Content Repositorynifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepositorynifi.content.claim.max.appendable.size=50 KBnifi.content.repository.directory.default=./content_repositorynifi.content.repository.archive.max.retention.period=7 daysnifi.content.repository.archive.max.usage.percentage=50%nifi.content.repository.archive.enabled=truenifi.content.repository.always.sync=falsenifi.content.viewer.url=../nifi-content-viewer/  I am co

Hi .. I have nifi running with ldap-user-group-provider integrated with AD. However, there are some usernames with mixed case sAMAccountName, lets say `sAMAccountName=Michael`.Logging into Nifi as `Michael` works (permission already given to an AD group which this user is a member of), but when logging into NiFi as `michael` or any username that does not match with the exact case, while the login is successful, i'm getting insufficient permission error.have tried to set group membership enforce case sensitivity to false, but seems like did not work, any steps I might have missed?and extract from authorizers.xml:<property name="Page Size">300</property><property name="Group Membership - Enforce Case Sensitivity">false</property><property name="Sync Interval">10 mins</property><property name="User Search Base">DC=company,DC=org</property><property name="User Search Scope">SUBTREE</property><property name="User Identity A

I want to do error handling of publish kafka processor in case of kafka goes down in apache nifi by retry mechanism. Retry mechanism needs to be implemented in separate flow(remove load from main flow).Nifi version = 1.18.0So, in case of failure of publishKafka processor should pass flow file in the failure relationship. Which is working properly in case of low count of incoming flow files.Example- Initial image generated few flow files for failure caseFinal image after some time Flow files successfully directed to failure relationship-As the number of incoming flow files increases the flow files are not able to routed to the failure relationship by publishKafka processor.(which is the real life scenario, if kafka goes down then publishKafka processor will apply back pressure to incoming flow files.)Here is my publishKafka processor configsI tried using inbuilt retry mechanism of nifi processorBut still the issue is not resolved, publish kafka processor not forwarding flow fi

Hi sir,I am using ListSFTP 2.0.0-M1+FetchSFTP  processor  and move file another location. I have two issues1. Every time it make a folder and inside the folder keep the file. Here is image: 2. I need to move file using filename like filename+uniqueid.xmlIf my file name is test.xml , I will rename file name like test++uniqueid.xml.I have used  below path for move destination directiory/myfolder/${filename}Here is the image of FetchSFTP :What is wrong of my configuration?Please help me..   

Hi,Getting below error while calling Slack channel from Nifi. Used putSlack Processor."o.apache.nifi.processors.slack.PutSlack : Failed to open connectionjavax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target "I have added slack cert in to my truststore.I am trying to get it tested in local (haven't deployed in to environment server). Added the channel name and webhookURL  (which generated under "companyName.slack.com") Is there anything i need to add it in local truststore to work?.Please help@MattWho @SAMSAL @cotopaul  

Example : I have data in http://20.30.06.08/files/test/and from the API's have a many csv data like : Name                                lastmodified              size test-20240625.csv    2024-06-26 08:14      3.9Ktest-20240626.csv    2024-06-27 08:14      2.9Ktest-20240627.csv    2024-06-28 08:14      1.9Khow to get all data csv from folder test, with apache nifi (invokedHTTP processor)

I upgraded from nifi-2.0.0-M1 to nifi-2.0.0-M3 on a new Server.On my old Server the QueryDatabaseTable Job works fine, maximum value would be increased on every Job-Runtime as expected.On the new Server maximum value is just increasing by 1. If my initial.maxvalue.hist_num is set to 114887000 i can see in State hist_num-value: 114887001And every job-runtime it starts from 114887001. I tried so much different settings in QueryDatabaseTable but cant get it done, to work.Maybe someone has an idea, how i can fix this. Would be very nice. Eventually logs where i can see error messages or something like that?Thanks 

Hello,Context :- Cluster of 3- NiFi 1.23.2 - Registry 0.7.0 on external serverAfter the upgrade from 1.13.2 to 1.23.2 I have a strange reaction from the cluster of 3. I have sometime a node that disconnect with the only error that NiFi can't delete a processor "because it's destination of another component". The thing is this processor don't mean to be delete. My guess it's because I have a diff between local change between node in the same cluster.For example if I voluntary create a process to create some local change, when I check on a node I have 1 local change but on another can be 501 local change. This mostly delete then add the same processor.If I try to revert the local change to the last version of bucket, I always have a node that disconnect. So I shutdown it server side and can't start it up again because the error of desynchronisation in local flow. Even if I copy the flow.xml.gz and flow.json.gz from a healtier node, same error.The only way I have to make it up a

Hi there,Maybe you guys can shed a light on an issue we’ve been having since we went live with one of our flows using alot of processors running on the primary node. We are using a 2 node cluster where we run a couple of getsqs (time driven, 1min) and executesql (cron scheduling, spread out during the day both on primary node. We do this so only one task is ran resulting in just on trigger flow file. Other wise we get 2 sqs events for the same event and 2 executesqls queries.What we have been seeing is that the jvm heap utilization on the primary node is above 70%, this increased gradually the past few days from 25%, so didn’t peak straight to 70%. The secondary node on the other hand is on a stable below 10%. So quite clear that there is a correlation between the processors running only on primary and the heap used.Our questions are:1. When a primary node processor triggers, will the flow file be processed by the following processors in the flow only on the primary node? Or can t

I have running hive locally using this docker compose image. However when trying to acecss admin role by docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/' -n hive and doing set role admin; I am getting this error ERROR : DDLTask failed, DDL Operation: class org.apache.hadoop.hive.ql.ddl.privilege.role.set.SetRoleOperationorg.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException: kaif doesn't belong to role adminversion: '3.9'services:postgres:image: postgresrestart: unless-stoppedcontainer_name: postgreshostname: postgresenvironment:POSTGRES_DB: 'metastore_db' POSTGRES_USER: 'hive' POSTGRES_PASSWORD: 'password' ports:- '5432:5432' volumes:- hive-db:/var/lib/postgresqlnetworks:- hivemetastore:image: apache/hive:4.0.0depends_on:- postgresrestart: unless-stoppedcontainer_name: metastorehostname: metastoreenvironment:DB_DRIVER: postgresSERVICE_NAME: 'metastore' SERVICE_OPTS: '-Xmx1G -Djavax.jdo.option.ConnectionDriverName=org.postgresql.Driv

I have a json[ { "orderId": "100", "orderItems": [ { "orderItemSeqId": "101", "externalId": "56789" }, { "orderItemSeqId": "102", "externalId": null } ] }, { "orderId": "200", "orderItems": [ { "orderItemSeqId": "201", "externalId": "78990" }, { "orderItemSeqId": "202", "externalId": "83937" } ] }]Using QueryRecord processor in NiFi, I wish to select the orders having any item with null externalId.So, in the output we should have only the first map with "100" orderId as it has one item with null externalId.I tried this SQL with RPATH but doesn't work:SELECT * FROM flowfile WHERE RPATH_STRING(orderItems, '[*][isEmpty(./externalId)]') = 'true'isEmpty function of RecordPath returns true if the string is empty or null else false.

HiWhen I run kudu cluster ksck , return this error: Corruption: master consensus error: there are master consensus conflictsAll Tables in this Cluster Kudu is ok (Tablet Server), but the problem is in Kudu Master.Follow the Master Sumary:UUID | Address | Status----------------------------------+-----------------------------+---------42e0db81612f46a4bb5fee541b1316eb | node02 | HEALTHY7b6a3355b027464c98fe5f77c7548350 | node04 | HEALTHY83e343ec25aa48e2b22f0de244e42a6d | node03 | HEALTHYAll reported replicas are:A = 42e0db81612f46a4bb5fee541b1316ebB = 83e343ec25aa48e2b22f0de244e42a6dC = 7b6a3355b027464c98fe5f77c7548350The consensus matrix is:Config source | Replicas | Current term | Config index | Committed?---------------+--------------+--------------+--------------+------------A | A B C | 218 | -1 | YesB | A B* C | 218 | -1 | YesC | A B* C | 218 | -1 | YesThe problem is in Master Node 03 (B)? 

HiAny expected date for a possible SP1 for cdp 7.1.9 ?Is 7.1.9 / 7.11.3 compatible with RHEL 8.10 ?

For now, I can use ImpalaODBC to download data from Hive in SASgrid, but when I tried to upload data in SASgrid to Hive, it raised up a error messageDoes anybody has any idea about it? Thanks in advanced.

I want to query all tables/columns metadata in hive. The way this is done in other databases like mssql, postgresql is to query information_schema views, but i don't see anything like that in hive. Can someone confirm is information_schema doesn't exists in hive. If it doesn't exists what would be a way to query metadata ??Tried querying information_schema, but didn't get anything.

Here is my Input Json,[{"PortfolioSharing": [],"CustomFields": [{"PrimaryId": 2069165,"ColumnName": "TRX_PORTFOLIO_CUSTOMFIELD2","FieldName": "Use Bank Account From Customer?","DataType": "nvarchar","Value": "Yes","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "UseBankAccountFromCustomer?","Type": "dropdown","Id": 103}],"Id": 2069165,"Guid": "0f2aa667-5d6d-4e8a-9ed1-7ef89db03d6d","Name": "N.M. SANGANI"},{"PortfolioSharing": [{"PortfolioId": 2074065,"CustomerId": 2159935,"PersonalId": "1006631","ReasonId": 810,"ReasonProgId": "OWNER","ReasonName": "Company owner","Comment": null,"TypeId": 808,"TypeProgId": "FULL","TypeName": "Full"}],"CustomFields": [{"PrimaryId": 2074065,"ColumnName": "TRX_PORTFOLIO_CUSTOMFIELD2","FieldName": "Use Bank Account From Customer?","DataType": "nvarchar","Value": "Yes","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "UseBankAccountFromCustomer?","Type": "dropdown","Id": 103}],"Id": 2074065,"Guid": "b276ec7e-b770-4dae-a9b7-2390652ec

Hi, I am looking to use invokehttp with StandardOAuth2AccessTokenProvider controller to automatically get Access Token and make the API call with the token. The requirement to get token has an extra field compared to standard OAUTH token :Below is the curl command. It includes extra key called auth_chain. curl --location 'https://xyedfrg.xom/access_token' \--header 'content-type: application/x-www-form-urlencoded' \--data-urlencode 'username=test' \--data-urlencode 'password=test' \--data-urlencode 'client_id=F1234rdcfhkv' \--data-urlencode 'client_secret=test-secret' \--data-urlencode 'grant_type=password' \--data-urlencode 'auth_chain=OAuthLdapService' But this is how the StandardOAuth2AccessTokenProvider looks:If I add auth_chain as property it errors out saying :How can I make this OAUTH call to get token using invoke http.*please note, cannot use the option provided here (https://co

I have downloaded some huge LLM Model files from huggingface to local filesystem.How can i shared them between CML Projects，so that it looks like exist under /home/cdsw/models?

Hello,We are on CDP PB 7.1.9 and goal is to monitor YARN applications and performance along with few HDFS metrics on Grafana via dashboards and ultimately trigger alerts. Metrics has to be collected via Prometheus agent and shared with Grafana.At this point, I have downloaded a Prometheus agent and created a below yml configuration. I understand this will collect all the metrics, which is intentional to start with.lowercaseOutputName: truerules: # All Gauge type Hadoop Metrics - pattern: 'hadoop<name=.*><>(Count|Value)' name: hadoop_${1}_gauge type: GAUGE # All Counter type Hadoop Metrics - pattern: 'hadoop<name=.*><>(Count|Value)' name: hadoop_${1}_counter type: COUNTERPrometheus agent jar and this config file is stored at /var/lib/prometheus_jmx_config/.For now testing this only on resource manager instances to verify if metrics are getting collected.Following few articles and Grafana documentation, I understood that I will need to run Promethe

How can we add Local Users in Hue when HUE is integrated with AD? Attempting to add a user using add/sync LDAP user returns an error stating no user found in LDAP. We have some users who are not on AD we would them to be added to HUE as Local Users. Have searched but could not get any information on thisCM 7.6.1Cloudera Runtime - 7.1.7 Appreciate any input on this. Thanks 

Hi,since a few weeks, we have regular warnings on our datanodes :"This role's host has been out of contact with Cloudera Manager for a concerning amount of time."cm-agent is consuming alot of CPU , in particular MonitorDaemon-R (cm-agent PID is 25598):top -H -b -n1 -p 25598top - 14:08:21 up 22 days, 6:17, 1 user, load average: 5,10, 5,07, 4,98Threads: 35 total, 2 running, 33 sleeping, 0 stopped, 0 zombie%Cpu(s): 20,1 us, 21,5 sy, 0,0 ni, 56,5 id, 0,0 wa, 0,0 hi, 1,9 si, 0,0 stKiB Mem : 26370784+total, 7788424 free, 38057140 used, 21786227+buff/cacheKiB Swap: 31457276 total, 31349500 free, 107776 used. 22445827+avail MemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND25833 root 20 0 3085980 407988 10016 R 87,5 0,2 124:55.80 MonitorDaemon-R25598 root 20 0 3085980 407988 10016 S 0,0 0,2 28:07.22 cmagent25748 root 20 0 3085980 407988 10016 S 0,0 0,2 2:49.93 cmagent25754 root 20 0 3085980 407988 10016 S 0,0 0,2 0:14.20 Audit-Plugin25755 root 20 0 3085980 407988 10016 S 0,0 0,2 0:1

Hi Team,We are using Hadoop version 2.10.2 version and facing below error while starting  server. Due to company security policy no execution permission is enabled on /tmp directory.   Is there any better alternate way by which we can resolve this?  I am new to Hadoop project and help is much appreciated. Many Thanks in advance.Logs Message:  2024-06-13 09:57:00,871 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManagerjava.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-7115603708108814032.8: /tmp/libleveldbjni-64-1-7115603708108814032.8: failed to map segment from shared object]at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.