Q&A

When  trying to connect to a dblink to user the QueryDatabaseTable, NiFi is returning this error:• 14:24:36 BRT ERRORQueryDatabase Table[id=c3576641-0184-1000-ad73-c81ab57c11bb]QueryDatabase Table[id=c3576641-0184-1000-ad73-c81ab57c11bb] failed toprocess session due to Cannot invoke method public default java.sql.Connectionorg.apache.nifi.dbcp.DBCPService.getConnection(java.util.Map) throwsorg.apache.nifi.processor.exception. Process Exception on Controller Service withbecause the Controlleridentifier bf90fd3d-0184-1000-4a92-d8b25677a705Service's State is currently ENABLING; Processor Administratively Yielded for 1sec: org.apache.nifi.controller.service.Controller Service DisabledException: Cannotinvoke method public default java.sql.Connectionorg.apache.nifi.dbcp.DBCPService.getConnection(java.util.Map)throwsorg.apache.nifi.processor.exception. Process Exception on Controller Service withidentifier bf90fd3d-0184-1000-4a92-d8b25677a705 because the ControllerService's State is curr

Hi, I want ask a shared variable between non-associated Processors. I have 2 separate Process Groups.One group monitors the status of the API called log manager, and changes the variable 'Circuit_Break' to 'enable' through UpdateAttribute when the API status is bad. Another group receives syslog as udp and put it in an API called log manager. At this time, if the variable called 'Circuit_Break' is 'enable' through RouteOnAttribute in the middle, alert with slack and try to execute ControlRate. thisis Group 2 thisis Group 1  Is it possible to use variable shared by different process groups?If not available, are there any other good ways? I would like to periodically monitor the API and give a slight delay according to the API status.When the API is in a bad state, I want to give it time to recover by adjusting the throughput(by ControlRate).  

I am getting below input flow file: Input:[ {"original" : {"Id" : "100008","Date" : "2022-09-22"},"enrichment" : {"Record" : "Record 11","Description" : "Invalid values for Id or Date"}}, {"original" : {"Id" : "100009","Date" : "2022-09-02"},"enrichment" : {"Record" : "Record 2","Description" : "Invalid values for Id or Date"}} ] Expected output:[{"original" : {"Id" : "100009","Date" : "2022-09-02"},"enrichment" : {"Record" : "Record 2","Description" : "Invalid values for Id or Date"}}] and Output I am getting is the below using Groovy script:["{\"original\":{\"Id\":\"100009\",\"Date\":\"2022-09-02\"},\"enrichment\":{\"Record\":\"Record 2\",\"Description\":\"Invalid values for Id or Date\"}}"]  Groovy Script:import org.apache.commons.io.IOUtilsimport java.nio.charset.StandardCharsetsimport groovy.json.*import java.util.ArrayListdef flowFile = session.get()if(!flowFile) returntry {flowFile = session.write(flowFile,{ inputStream, outputStream ->def text = IOUt

Hi , I am new to the groovy... I want to know, how json array can be formed for dynamic variables from json flow file thorough groovy script.. please let me know, if you have an idea 

Can you help me to find where do Nifi stores templates?I am running Nifi on a docker container.PS: there is no dir under the folder conf.Here is the WORKDIR: 

SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]2022-12-05 13:22:59: Starting HiveServer2WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]S

I have set up Cloudera Manager (Trial) 7.4.4  in Ubuntu 20.04.4 LTS (Focal Fossa) as a single node cluster in AWS EC2.   All the services HDFS, Hue , Hive , Spark , Livy , Solr , ZooKeeper , Yarn are in good health and running.   I can login to Hue editor and open spark notebook. But I'm getting the issue as The Spark session could not be created in the cluster when trying to execute notebook.    I The Spark session could not be created in the cluster: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(Spark

I have requirement to execute a PostgreSQL procedure in Nifi, I tried to use Put Sql processor , but ended up getting error. Could someone let me know what I'm doing wrong. This is how my entire flow looks like. -querying from a table and putting the record to another stage table using the first 'PutSql' processor. And afterwards the second putsql processor to call my stored procedure.   

I am trying to re-create a ranger-audits collection for Infra-Solr with numShards=2 instead of 1.     # curl -k --negotiate -u : "https://fqdn.example.net:8995/solr/admin/collections?action=CREATE&name=ranger_audits&collection.configName=ranger_audits&numShards=2" {"responseHeader":{"status":400,"QTime":1730},"failure":{"fqdn.example.net:8995_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at https://ufqdn.example.net:8995/solr: Error CREATEing SolrCore 'ranger_audits_shard2_replica_n2': Unable to create core [ranger_audits_shard2_replica_n2] Caused by: Could not initialize class java.time.zone.ZoneRulesProvider"},"success":{"fqdn.example.net:8995_solr":{"responseHeader":{"status":0,"QTime":1278},"core":"ranger_audits_shard1_replica_n1"}},"Operation create caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: ranger

I want to collect the metadata from HDFS. But when i searched , it looks like there is no atlas hook for HDFS available like it is there for kafka , hive and other data sources. Can I get the HDFS hook  install steps or the steps to create a custom hook. 

Hello guys, I having a problem with Airflow on CDE. This make me boring almost a week. I am using the ECS containers on Data Services 1.3.4-b211. On CDE, I can run a Spark job, open a Grafana UI, but Airflow UI can not. Thanks for viewing my issue Any advice would be greatly appreciated

Hello, unfortunately I couldn't find an answer for this question, although it seems to be easy enough. I have a json file of this format, which I want to generate separate flowfiles from singularizing the values in NestedKey2: My input:{"Key1" : [ {"NestedKey1" : "Value1","NestedKey2" : [ "Value2", "Value3", "Value4", "Value5"],"NestedKey3" : "Value6"} ]} Desired output:Flowfile 1:{"Key1" : [ {"NestedKey1" : "Value1","NestedKey2" :  "Value2","NestedKey3" : "Value6"} ]} Flowfile 2:{"Key1" : [ {"NestedKey1" : "Value1","NestedKey2" :  "Value3","NestedKey3" : "Value6"} ]} ... and so on. Thanks for any suggestions.

I have installation of ECS (Embedded Container Service / Experience Compute Service )  v 1.3.3 , CDP 7.1.7.1000. Cloudera ML and DWX were ok to start and use. But, CDE (Data Engineering XP) service always fail to start with log saying {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstalllationFailed","Attempt":"3"}.   Not sure what is best way to start troubleshooting with such log displyed only.

Hi guys, I am able to connect to my Hive2 server with the following command:  beeline jdbc:hive2://<hostname>:10001/;ssl=true;sslTrustStore=/etc/hadoop/conf/hive-truststore.jks;trustStorePassword=<redacted>;transportMode=https;httpPath=cliservice However, when I try to connect with Hue I get the following error: 500 Server Error: Server Error for url: https://<hostname>:10001/cliservice/ <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /cliservice/. Reason: <pre> java.lang.IllegalArgumentException: Invalid sign, original = <long string> current = <another long string></pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.25.v20180904</a><hr/> </body> </html> (error 500) I

I need to change the column order in a Hive table, but I want to make sure I can do it without deleting any of the existing data.

Hello,We are able to login to Hue UI locally with default user name / password, the issue arises when choosing AD Authentication it does not accept the username / password. Error in Hue Access log[25/Nov/2022 14:00:29 -0800] DEBUG Initiating TLS[25/Nov/2022 14:00:29 -0800] DEBUG search_s('DC=mtest,DC=com', 2, '(&(sAMAccountName=%(user)s)(sAMAccountName={0}))') returned 0 objects:[25/Nov/2022 14:00:29 -0800] DEBUG Authentication failed for User1_dev: failed to map the username to a DN.[25/Nov/2022 14:00:29 -0800] WARNING 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" --- Failed login for user: User1_dev[25/Nov/2022 14:00:29 -0800] INFO 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" returned in 1062ms 200 114908[25/Nov/2022 14:00:40 -0800] DEBUG 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" -Error in Runcpserver.log[25/Nov/2022 14:43:01 -0800] resource ERROR Error logging return call POST http://localhost:7187/api

Hi everyone I have spent several days trying to solve the following case. I need to join two flowfiles (master-detail) using Nifi flow, I tried using queryrecord and mergerecord but i couldn't did it. I have some json based files in a HDFS (Apache Spark) i need to make a join and put de result in a database table (using nifi v1.11.4). I can read HDFS files but i cant' perform a join.   Example: master_file content: {   "FALLA_ID" : 40217,  "FALLA_FECHA" : "1998-01-01 00:00:00",  "FALLA_RPT_NRO" : 25,  "FALLA_FRECUENCIA" : "60.0000" }   detail_file content: { "FALLA_ID" : 40217,"PRINCIPAL" : 1,"FALLA_DTL_ID" : 1,"FALLA_CLASE" : 1,"TPF_FALLA_ID" : 1 }, { "FALLA_ID" : 40217,"PRINCIPAL" : 1,"FALLA_DTL_ID" : 2,"FALLA_CLASE" : 2,"TPF_FALLA_ID" : 5 }   Expected_File content: {   "FALLA_ID" : 40217,  "FALLA_FECHA" : "1998-01-01 00:00:00",  "FALLA_RPT_NRO" : 25,  "FALLA_FRECUENCIA" : "60.0000",   "PRINCIPAL" : 1,   

Hi all, I'm having problems getting the JavaScript I wrote for ExecuteScript to work properly. What I'm trying to do:My flowfiles contain rows of data from a SELECT statement. Each flowfile is searched line by line for non-ASCII characters, and all non-ASCII characters that are found are replaced with empty strings before the data is loaded to a target database. I'd like to grab the value for the table's unique identifier column and write it to a new table, say "nifi_replaced_data" so that I can easily look in the source database and see the row with the non-ASCII character that NiFi replaced in the target database. For example, I might be processing a table with the columns:column_1, column_2, unique_id, column_4And the flowfile, which is tab delimited, might look like:somedata    bad__character   AA11BB22   someotherdataotherdata no_bad_character CC33DD44 someotherdata The process:Each flowfile is checked in it's entirety for no

Hi, There is a kafka producer converting decimals to bytes using this:import java.math.BigDecimal;import java.math.RoundingMode;import java.nio.ByteBuffer;import org.apache.avro.Conversion;import org.apache.avro.Conversions;import org.apache.avro.LogicalType;import org.apache.avro.LogicalTypes;import org.mapstruct.Mapper;@Mapperpublic class BigDecimalMapper { public ByteBuffer bigDecimalToByteBuffer(BigDecimal bigDecimal) { if (bigDecimal == null) return null; Conversion<BigDecimal> conversion = new Conversions.DecimalConversion(); LogicalType type = LogicalTypes.decimal(15, 4); BigDecimal scaledBigDecimal = bigDecimal.setScale(4, RoundingMode.HALF_UP); // Schema parameter is not used by Conversions so pass as null return conversion.toBytes(scaledBigDecimal, null, type); }We are using the confluent schema registry and this is the schema:{"type": "record","namespace": "event","name": "Event","fields": [{"name": "maxApprovedAmount","type": "bytes","logicalT

I need to create a flow that update rows in my database based on csv file.I have :1- GetFilestatusclient_idActive123Inactive456 2- UpdateAttribute to update the Baobab_file (the schema is already in the AvroSchemaRegistry (collection) 3- SplitRecord to get each line statusclient_idActive123 4- ExecuteSQL : update table_client set current_status = status where id = client_id How can I do this I tried ${status} and \status but no luck Thank you

 Cloudera Manager (Trial) 7.4.4Cloudera Runtime 7.1.7 (Parcels)  Cloudera SCM log: 2022-07-28 19:19:59,113 ERROR CommandPusher-1:com.cloudera.cmf.command.CommandHelpers: GenerateCredentials - Execution error:java.io.IOException: Encountered error with /opt/cloudera/cm/bin/gen_credentials.sh: Cannot access generated keytab file /var/run/cloudera-scm-server/cmf2117942825672975512.keytabat com.cloudera.cmf.security.GenerateCredentialsCommand.generateKeytab(GenerateCredentialsCommand.java:533)at com.cloudera.cmf.security.GenerateCredentialsCommand.createKeytab(GenerateCredentialsCommand.java:637)at com.cloudera.cmf.security.GenerateCredentialsCommand.generateMissingKeytabs(GenerateCredentialsCommand.java:350)at com.cloudera.cmf.security.GenerateCredentialsCommand.access$300(GenerateCredentialsCommand.java:70)at com.cloudera.cmf.security.GenerateCredentialsCommand$2$1.run(GenerateCredentialsCommand.java:745)at com.cloudera.cmf.security.GenerateCredentialsCommand$2$1.run(

Yarn's application priority can be found in yarn 8088 resource manager's website,How do the priority work in yarn? Now the version of CDH I use is 6.3.2.Hadoop 3.0.0 Hive 2.1.1 I use hive on spark.Can I use the config to manage the application priority? For some reason some hive sqls should have high application priority when those sqls are appending and run ahead of the other.  Instead of running the same time and equal share the compute resources.   When I set the following config in Hive, it seems do not work well in Yarn.MapReduce"-Dmapreduce.job.priority=xx"Flink"-yD yarn.applicaiton.priority=xx"Spark"spark.yarn.priority=xx" In hive sql."set spark.yarn.priority=10;"It does not work ...  

Orginal data:{"data": [{"id": "1234569","Date": "2022-08-22"},{"id": "1234567","Date": "2022-08-22"},{"id": "34567","Date": "2022-08-22"}]}API Response:[ {"Record 2" : "Invalid values for emp info"}, {"Record 3" : "Invalid values for emp info"} ]Where Record 1,Record 2,Record 3 is array of json object index. Here we have three records in orginal data, Record 1 passed successfully and remaining two failed and due to that in response we are getting Record 2 and Record 3i.e. if index of orginal data = value of Record (for example original data's index 1 is passed successfully so it should not merge. In orginal Response Index 2 failed, so it should merge with Record 2 and so on.. ) then final output should look like below:{"data": [{"id": "1234567","Date": "2022-08-22","Record 2" : "Invalid values for emp info"},{"id": "34567","Date": "2022-08-22","Record 3" : "Invalid values for emp info"}]}How can I do that? Please help.

Hi Team, I have an requirement: Input:{"headers": {"comments": "3"},"data": [{"Id": "100000","Date": "2022-09-22"},{"Id": "100001","Date": "2022-09-02"}]} Jolt Spec:[{"operation": "shift","spec": {"data": {"*": {"Id": "[&1].Id","Date": "[&1].Date","$": "[&1].array_Ind"}}}}, {"operation": "modify-default-beta","spec": {"*": {"Record": "=concat('Record ',@(1,array_Ind))"}}}] I am getting below output:[ {"array_Ind" : "0","Id" : "100000","Date" : "2022-09-22","Record" : "Record 0"}, {"array_Ind" : "1","Id" : "100001","Date" : "2022-09-02","Record" : "Record 1"} ] Expected output: [ {"array_Ind" : "1","Id" : "100000","Date" : "2022-09-22","Record" : "Record 1"}, {"array_Ind" : "2","Id" : "100001","Date" : "2022-09-02","Record" : "Record 2"} ]   Can you please help 

I am getting info messages like the one below while reading and writing files in hdfs.can anyone explain the meaning of this message, will it cause any issues? I have 3 Namenode(open-source Hadoop) in HA. INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-errorat org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:108)at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2094)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1550)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3342)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1208)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(Clien