Q&A

Hi everyone,I have a requirement for Apache Nifi. In a nifi flow if there are files stuck in queue for more than some specified time let say 5 mins , I should get an email saying that file is stuck in the nifi flow.Is that possible ? Please let me know if there is any possible solution.Thanks in advance !!

Hallo,When to enable Kerberos via ambari, I am facing the following window popup at the time of Testing client after client installation sayingin my log ambari-server listed below2024-06-04 06:27:43,380 WARN [agent-report-processor-2] ActionManager:162 - The task 76 is not in progress, ignoring update2024-06-04 06:27:43,861 INFO [ambari-client-thread-6248] AmbariManagementControllerImpl:4086 - Received action execution request, clusterName=hadoop, request=isCommand :true, action :null, command :KERBEROS_SERVICE_CHECK, inputs :{HAS_RESOURCE_FILTERS=true}, resourceFilters: [RequestResourceFilter{serviceName='KERBEROS', componentName='null', hostNames=[]}], exclusive: false, clusterName :hadoop2024-06-04 06:27:44,149 WARN [ambari-client-thread-6248] KDCKerberosOperationHandler:329 - Failed to kinit as the KDC administrator user, admin/admin@HADOOP.COM: ExitCode: 1 STDOUT: STDERR: kinit: Server not found in Kerberos database while getting initial credentials2024-06-04 06:27:44,151 ERRO

Hello Community,I am working on a data ingestion pipeline using Apache NiFi, and I am using the `ExecuteSQL` processor to read data from a database table and ingest it into my data flow. I want to ensure that all records from the table have been successfully ingested, but I am unsure how to verify this within NiFi. Here are the details of my current setup and the specific challenges I am facing:Current Setup:1. Database Source: I am querying a relational database.2. NiFi Processor: I am using the `ExecuteSQL` processor to execute a SQL query and fetch data from the table.3. Downstream Processors: The flow files generated by `ExecuteSQL` are processed by subsequent processors such as `ConvertRecord`, `PutHDFS`, etc.Specific Challenges:1. Ensuring Completeness: How can I verify that all rows from the table have been ingested? Is there a mechanism within NiFi to confirm that no records have been missed?2. Record Counting: Is there a way to count the number of records fetched by `ExecuteSQ

The new cumulative hotfix 7 for Cloudera Runtimes 7.1.9 points to CHF6 instead 😉https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/runtime-release-notes/topics/chf7-pvcb-overview.html(probably a copy/paste issue 😉 )

i am trying to check the files that are created under data folder when we export table to hdfs location.I have single column table with 5000 rows but after export it created only one data file named as 000000_0I have another table with partition, 2 partitions with 95 rows each approx. and for this table export is see 2 folders each for partition and then under it only one file. Can you please help me to know how are the data files for the exported table are created and how is it distributed? I mean is it only one data file for each table without partitioned? or it will create multiple files based on specific row count or size count or on what criteria it is split?

Hello,Has anyone encountered an issue were Knox is not writing audit logs of specific topology. We have below topologies created including few of them migrated from HDP, however, necessary modifications were done and are listed in Knox UI.cdp-proxycdp-proxy-apicdp-proxy-tokenhealthtokenexchangeuser1 - created for user grouptopo1 - created for user group and migrated from HDPtopo2 - created for user group and migrated from HDPapp - Used by app accountsKnox is successfully writing Ranger audit logs in HDFS for only cdp* topologies which were created by Cloudera during setup of Knox service and not for other. Written logs are visible in access tab of Audit section in Ranger Admin UI. We have total of 3 clusters and this is the case 2 clusters, for 1 cluster everything works fine. I have compared the configuration and also topology xmls and all seems correct (except for instance details which is obvious).Would it be anything related to Ranger or Solr configuration for Knox? However, i

Hello,We have a 3 node cluster where one of the nodes is only 64 cores 128 GB RAM and other two machines are each 128 cores 128 GB RAM. I am using Round Robin load balancing in the flow as Listfile --> LB --> FetchFile, as a result of which load is getting divided (almost) uniformly among all the machines, which is expected behavior. However, the load average on smaller machine is exceeding 64 (this machine is sweating!) whereas the load average on other two machines is almost 50.So my question is there a way in NiFi to distribute load in such a way that the load on the smaller machine can be maintained to stay less than 64, and give the other two machines some more work to do?I tried using DistributeLoad processor, but not sure which strategy to use because Round robin would distribute load equally, next available would distribute when the next node is available. However, how do I configure DistributeLoad such that it divides load equally among two bigger machines but lesser loa

We have 2 instances of nifi-registry with git as flowProvider, when we make some changes and commit from nifi instances, sometimes appears everything is ok, sometimes appears "there is a new version of the flow". I think one of the nifi-registry instances it's not synced.¿How can we force the sync between multiple nifi-registry instances?

Example:Flowfile databranches,name,id,acc,serialNoparent1,abc,123,1234567788,1child1,eee,345child2,rrr,678parent2,mno,786,97865662762,2child1,ttt,764child2,ryy,456output:parent1,abc,123,1234567788,1child1,eee,345,1234567788,1child2,rrr,678,1234567788,1parent2,mno,786,97865662762,2child1,ttt,764,97865662762,2child2,ryy,456,97865662762,2I have to add the parent last two columns details to Childs of the same parent like parent1 last column details to child1, child2 rows in the flow file.Please suggest a possible way to use the processor.

Input Json,==============[{"CustomFields": [{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD2","FieldName": "Business Registration No","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "BusinessRegistrationNo","Type": "inputfield","Id": 99},{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD3","FieldName": "Type of Business ","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "TypeOfBusiness","Type": "dropdown","Id": 100},{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD4","FieldName": "Birth Certicate No / Other","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "BirthCerticateNoOther","Type": "inputfield","Id": 101},{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD5","FieldName": "Passport No","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "PassportNo","T

Hello,I have a JSON data in the ClickHouse database with a type of string. The JSON data looks like this: [{ "Name": "Gta V" "Type": "Xyz" "content": "{"Game":{"Subject":[{"Time":{"@value":"201511021057"}}]}}" }] The output I get using Nifida ExecuteSqlRecord is like this. I can remove the \ character using the ReplaceText process. The output looks like this:  [{ "Name": "Gta V" "Type": "Xyz" "content": "{"Game":{"Subject":[{"Time":{"@value":"201511021057"}}]}}" }] However, the value part for the content key is still a string. How can I make its type as JSON?The desired output is like this: [{ "Name": "Gta V" "Type": "Xyz" "content": {"Game":{"Subject":[{"Time":{"@value":"201511021057"}}]}} }] 

Hi Team, Zepplin is Installed in CDP which does not contain Kerberos.I tried with default username and password as admin/admin, zeplin is not working in GUI.Let me know the default password of zeplin to login 

Hello, Is there a straight documentation available that would help to side car migrate Oozie jobs (50+) from HDP to CDP PB? I know of the properties file that might need some modification to point to CDP RM and relevant servers, however, facing it hard to understand / map properties file as we have 50+ workflows to be migrated. Thankssnm1523

Hello,Can you assist me in retrieving the values stored in the "customfield_10161" field from JSON input? My expectation is to consistently receive the values and name it "DEFECT_ROOT_CAUSE" , whether they are stored as a list or as a single value."How to write JOLT which will handle the both the case. let say If I will get list then extract last element of it, if not list then record as it is.?Extracting List: (works fine when input is list)when input is not list: then DEFECT_ROOT_CAUSE column is missing My expectation to get the records in DEFECT_ROOT_CAUSE columns whether its list or not.     

HelloI am trying to set up a cluster with NiFi 2.0-M3 but have an error. My docker-compose file is here - https://hastebin.com/share/woqezesefi.yamlError:2024-06-02 08:10:54,535 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 248301603f5c:8082; will use this address for sending heartbeat messages2024-06-02 08:10:54,536 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at 248301603f5c/<unresolved>:8082. Will send Cluster Connection Request to this address2024-06-02 08:10:54,571 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchorsThe same docker-compose is working fine with

2024-05-31 19:23:17,927 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [file:/user/history/done]at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:701)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:637)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:598)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:97)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:150)at org.apache.hadoop.service.AbstractService.init(AbstractService

After setting custom kerberos princ for hive: Unable to read HiveServer2 configs from ZooKeeper (state=,code=0). But if i use default kerb principal i am able to logon to hive.followed this dochttps://docs.cloudera.com/cdp-private-cloud-base/7.1.9/security-kerberos-authentication/topics/hive-custom-kerberos-principal.html SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Connecting to jdbc:hive2://im19-vm4.eng.cohesity.com:2181,im19-vm5.eng.cohesity.com:2181,im19-vm6.eng.cohesity.com:2181/default;password=root;principal=khive/_HOST@im19-vm4.realm;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=9xSdfOCgyzHPGv24yUYU9QOob4sKxiey5GsTnZgK7oB;trustStoreType=jks;user=root;zooKeeperNamespace=hiveserver2Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 conf

From last few days I am received alert from SERVICE_MONITOR_HEAP_SIZE in our CDP cluster (version 7.1.7). we are getting frequent alerts on the SERVICE_MONITOR_HEAP_SIZE.How can I fix this issue? Monitored Entities are :logs :   

curl -H   "Content-Type: application/JSON ; Authorization: Bearer {token here}" -d "{"revision":{"clientId":"ac153723-44ff-17fd-e42e-f975cb01817b"},"processors":{"id":"ac15a47e-44ff-17fd-ffff-fffffc691b51","running":"true"}}" -X PUT "https://aaa.com:443/nifi-api/processors/ac15a47e-44ff-17fd-ffff-fffffc691b51/run-status" we are trying to run NIFI processor using NIFI API as above, but its giving UNAUTHORIZED error always. Can any one help in getting this fixed. Are we using correct code? We are able to get the token from NIFI.

Hello,I have Two Questions1. In case of error , GenerateTableFetch is not getting routed to Failure.  2.  GenerateTableFetch keep on trying .  I need it should do this only once in case of error.Thanks,Riyaz

Hello All, Can anyone provide me the most challenging real time issues solved on Spark, YARN and Hive ? @RangaReddy 

I am trying to create a flow to convert a JSON object to Avro and then consume the produced data from Avro to JSON.This is the sample data I am using When I check the data that is finally consumed and converted back to JSON , It is coming as array instead of object :[ {"Vehicle" : "BigVehicle","Fromuser" : "User1","Timestamp" : "2024-03-19T13:00:00+0530"} ]convertRecord configuration for JSON toAvro conversion:Convert Record from Avro to JSON: 

We have changed our postgresql DB instance to a new port and migrated all the locally available data from older DB instance to a newer oneUpdated the port every place needed in cloudera manager and db.propertiesHowever while starting the hive metastore server is failing with below errorP.S : We are using postgres as external database ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [TThreadPoolServer WorkerProcess-20]: Retrying HMSHandler after 2000 ms (attempt 3 of 10) with error: javax.jdo.JDOUserException: Exception thrown while loading remaining rows of queryat org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:659)at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:310)at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)at org.datanucleus.store.rdbm

I created a table and in beeline (hive) and it worked quickly.   # Movies tableCREATE EXTERNAL TABLE movies ( movieId INT, title STRING, genres STRING)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES ( "separatorChar" = ",", "quoteChar" = "\"", "escapeChar" = "\\")STORED AS TEXTFILELOCATION '/user/hive/warehouse/movielens/movies'TBLPROPERTIES ("skip.header.line.count"="1");# Ratings tableCREATE EXTERNAL TABLE ratings ( userId INT, movieId INT, rating DOUBLE, rating_timestamp BIGINT)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES ( "separatorChar" = ",", "quoteChar" = "\"", "escapeChar" = "\\")STORED AS TEXTFILELOCATION '/user/hive/warehouse/movielens/ratings'TBLPROPERTIES ("skip.header.line.count"="1");   I am attempting:    CREATE TABLE avg_movie_ratings ASSELECT movieId, AVG(rating) AS avg_ratingFROM ratingsGROUP BY movieId;   which starts a

Hello Experts,I see this red color high lighted number "2(1)"on Apache nifi processor.Is this something related to background process (processor thread) failing or something?I face the issue of this custom processor getting stuck once in a while, trying to understand the issue.This processor just invokes an http post endpoint to upload a file.Any help/suggestion is appreciated. Thanks,Mahendra