Q&A

Please keep in mind I am in no way savvy in "this stuff" at all, so please bare with me.Issue: I am still receiving "Your connection is not private" / "NET::ERR_CERT_AUTHORITY_INVALID" when accessing the NiFi web UI that I have installed on a Linux server even though I set it up with a certificate provided by my company (I believe did something wrong here).Goal: Anyone who tries to access the web UI will be met with the NiFi Login screen (this part is already setup with LDAP) without having to import a certificate instead of the warning/secure ("Your connection is not private") page. I think it's important to know that I have it working fine with self-signed certificate and importing the certificate into my browser.Summary (Please read this knowing that my understanding of the subject is very minimal):1. I generated a CSR and keystore.jks (from what I understand contains the private key) with the following command:keytool -genkey -alias server -keyalg RSA -keysize 2048 -keystore keysto

I'm trying to use a ListenUDPRecord processor to parse syslog messages as input, using SyslogReader as Record Reader and JsonRecordSetWriter as Record Writer.The solution is working as I am getting a json message as output with the following fields: priority, severity, facility, version, timestamp, hostname, body.The json must then be indexed in a solr collection and the problem is that I get a timestamp field of the form: Nov 16 12:36:32 but I would need a unix timestamp fomat field (e.g. 1637062592000) or an output like that: "2021-11-16 12:36:32".I tried to specify the "Timestamp Format" field of the JsonRecordSetWriter service (i.e. "yyyy-MM-dd HH:mm:ss" or "MM/dd/yyyy HH:mm:ss") but output does not change.How can I change the structure of the timestamp field of my output json message?

Our license is getting expired on Jan 1st, 2022. Will it be possible to use Cloudera Manager to remove Cluster and Parcels post this date? 

Tried testing the ranger service connection and it is getting failed  with message "JAAS configuration missing or not correct in Ranger"   Ranger-admin log: 2021-11-16 12:00:58,661 ERROR org.apache.ranger.services.kafka.RangerServiceKafka: <== RangerServiceKafka.validateConfig Error:java.lang.Exception: JAAS configuration missing or not correct in Ranger Kafka Service...bootstrap.servers,security.protocol,sasl.mechanism,kafka.keytab,kafka.principal2021-11-16 12:00:58,661 ERROR org.apache.ranger.biz.ServiceMgr: TimedCallable.call: Error:java.lang.Exception: JAAS configuration missing or not correct in Ranger Kafka Service...bootstrap.servers,security.protocol,sasl.mechanism,kafka.keytab,kafka.principal2021-11-16 12:00:58,662 ERROR org.apache.ranger.biz.ServiceMgr: ==> ServiceMgr.validateConfig Error:org.apache.ranger.plugin.client.HadoopException: java.lang.Exception: JAAS configuration missing or not correct in Ranger Kafka Service...bootstrap.servers,security.protocol,

Impala is not supporting org.apache.hadoop.hive.serde2.OpenCSVSerde SerDe class. Please suggest the solution. Table aggregate_test is created using Hive and org.apache.hadoop.hive.serde2.OpenCSVSerde SerDe class. Following query is not working:SELECT * from open_aggregate_test.aggregate_test ; Above query gives following error:SQL Error [500051] [HY000]: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: Failed to load metadata for table: 'open_aggregate_test.aggregate_test'CAUSED BY: TableLoadingException: Could not load table open_aggregate_test.aggregate_test from catalogCAUSED BY: TException: TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL, error_msgs:[TableLoadingException: Failed to load metadata for table: open_aggregate_test.aggregate_testCAUSED BY: InvalidStorageDescriptorException: Impala does not support tables

I am trying to use the oozie java action to run a hello world java program to test the functionality but no matter what I do it seems oozie does something and passed but does not run my java program I confirmed this by entering garbage in the 'Main class' field and it still passes and doesn't do anything useful?  This passes?? So what is oozie doing? and how do i get it to run the jar? (it is uploaded to hdfs) These are the logs which show no attempt at running my jar Log Type: stdout Log Upload Time: Thu Nov 04 21:57:01 +0200 2021 Log Length: 3809 log4j: Trying to find [container-log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@5e2de80c.log4j: Using URL [jar:file:/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p48.16116877/jars/hadoop-yarn-server-nodemanager-3.1.1.7.1.4.48-1.jar!/container-log4j.properties] for automatic log4j configuration.log4j: Reading configuration from URL jar:file:

Hello,I have a question regarding the concept of parameter context. The admin guide says : A Process Group can only be assigned one Parameter Context, while a given Parameter Context can be assigned to multiple Process Groups. OK, let's assume you have internal kafka or database, and some external system you connect to. You might want to have a parameter "group"for the internal kafka connection, and another parameter contexts for each external system connection.However, your flow, clearly a part a process group, reads data from external system and stores data internally. Thus you need actually 2 connections in the flow,it means 2 parameter contexts, which is not possible. To solve that, you assign the connections (controller services) to the parent group and the parameters.Finally, for connections which are common to many flows, like your kafka, databases, and HttpSSL you land in the parent NiFi Flow, having parameter context which containsa bunch of pretty unrelated paramete

I installed a new hdfs+hbase cluster. But after started for few hours, hbase regionservers all shutdown. There isnt any data writing. versions:HBase  1.2.0-cdh5.10.0Hadoop  2.6.0-cdh5.10.0zookeeper 3.4.5-cdh5.102021-11-09 21:15:51,928 WARN [ResponseProcessor for block BP-899263853-9.180.152.33-1634099649590:blk_1073742680_2626] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-899263853-9.180.152.33-1634099649590:blk_1073742680_2626java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075)2021-11-09 21:15:51,928 WARN [DataStreamer for file /hbase/WALs/jfhbase03,60020,1636456250380/jfhbase03%2C60020%2C1636456250380.default.1636463451629 blo

Hi everyone,I'm newbie to Nifi. I have already configured and build my test server. I have different DB connections in my NiFi. However, our DB connection passwords are dynamic in my company. Because we are using CyberArk security system, our DB user passwords are not constant. Our applications send a kind of query to cyberark system to fetch the password on the fly and use that password to connect to DBs. My question is how to do this on a NiFi system. Can we configure the sensitive data on the controllers dynamically?

Hi All, Currently we are using Cloudera 6.3.1 enterprise edition. From past two days i have observed that few node managers are not responding properly and they failing to launch applications with error Unable to find 'resource-types' please find the attachment for the logs and please provide me the solution for this .  ThanksSri . Thanks 

we need to query an array of struct and we can see that more than 1 level of dots are not accepted in the SQL query and it throws error when such queries are executed.e.g., select abc, def, a1.b1 from xxx where so n so --> This query executes successfully. but the following:e.g., select abc, def, a1.b1.c1 from xxx where so n so --> This query fails irrespective of us defining the correct arguments in the from section.-------------------------------Can someone tell how many such levels of dot notations are allowed/supported in Impala. I was unable to find any Impala document which was explaining this.

Hi  Is there a way to configure multiple LDAP servers (LDAP HA) in the hive-site.xml ?As of now we have only configured one LDAP server using below property:hive.server2.authentication.ldap.url <property> <name>hive.server2.authentication.ldap.url</name> <value>LDAP_URL</value></property> Our requirement is to config two LDAP servers to provide HA. Thanks 

I am using Rstudio and I need to save some tables in my sandbox, however I cannot specify the location, because by default we send the tables to the path: hdfs: //haprod/warehouse/tablespace/managed/hive/cld_ml_bi_eng.db/iris How can I specify the location when writing the table is in the LOCATION '/sandbox/CLD_ML_BI_ENG/iris' library(DBI)library(odbc)con_Hive <- dbConnect(odbc::odbc(), "Hive_produccion", database = "cld_ml_bi_eng", encoding = "latin1")colnames(iris) <- gsub(pattern = "\\.", replacement = "_", x = colnames(iris))dbWriteTable(conn = con_Hive, name = Id(schema = "cld_ml_bi_eng", table = "iris"), value = iris)LOCATION '/sandbox/CLD_ML_BI_ENG/iris' 

Hi,  What is the difference in impala executors and impala coordinators ?  Which one shall I increase in order to run my query faster ? Thanks in advance

The Atlas service was successfully installed and I could see the web UI port is open by using netstat. But I can't connect the Atlas Web UI(503 error), in the /var/log/atlas I didn't find any suspicious information as well. Any ideas why the web UI didn't work? 

Hello all, We have an application Spark Structured Streaming (version 3.0.1.3.0.7110.0-81) which crashed after 1hours and 15 minutes.This application read data from input kafka topic, run some transfomation and write the result in an another output kafka topic.The application use the mode "foreachBatch" with a trigger interval set to 30 secondes and maxOffsetsPerTrigger set to 7000.We get this exception :java.lang.OutOfMemoryError: Java heap spaceat java.util.Arrays.copyOf(Arrays.java:3332)at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)at java.lang.StringBuilder.append(StringBuilder.java:136)at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:203)at scala.collection.TraversableOnce.$anonfun$addString$1(TraversableOnce.scala:369)at scala.collection.TraversableOnce$$Lambda$90/910599202.apply(Unknown Source)at scala.collection.immutable.List.foreach(List

My cluster is Ambari 2.7.5.0, HDP 3.1.5, all nodes RHEL 7.9. When I look at the Cloudera Support Matrix and drill down to Oracle JDK 8, it specifies a minimum level of 1.8.0_77.  1.8.0_311 is out now, and I'd like to update my cluster to use that. Is there a maximum level of Oracle JDK 8 I should use?  Are there Ambari/HDP issues with the latest JDK?  Thanks for your consideration,Mike

We are using cdp 7.2.x and ADLS GEN2 public cloud . I have a requirement ,related to which one team wants to consume the data via rest Api . 1. They expose their rest api and then using some tool or code we can send data from Hive layer to their rest api . Can we do this via NIFI? 2. we create rest api on top of our Hive layer in ADLS and the other team can consume the data from this API.  Can you Please suggest how to proceed on both options . Are they feasible?  

Hello Community, This Is my first post with you  my problem is i want extract the header of a table from oracle database and based on the header i want to execute a create statement to store the header into postgresql  and all that using NIFI is there any solution for this? Thank you 

Hi everyone,I have installed CDP 7.1.6 cluster, with kerberos and TLS enabled.All services works fine except Atlas I get the error:HTTP ERROR 503 Service UnavailableURI: /STATUS: 503MESSAGE: Service UnavailableNeeded kafka topics, hbase tables and Solr collections all are created, needed policies on Ranger too are goodI cannot neither enable the service Atlas for HBaseI don't have any Error on atlas application.log :Any idea ?2021-10-24 21:03:45,634 INFO - [main:] ~ Attempting to retrieve password for key password from cert.stores.credential.provider.path configured credential provider path localjceks://file//var/run/cloudera-scm-agent/process/1546354754-atlas-ATLAS_SERVER/conf/atlascreds.localjceks (SecurityUtil:63)2021-10-24 21:03:45,638 DEBUG - [main:] ~ backing jks path initialized to file:/var/run/cloudera-scm-agent/process/1546354754-atlas-ATLAS_SERVER/conf/atlascreds.localjceks (AbstractJavaKeyStoreProvider:169)2021-10-24 21:03:45,638 DEBUG - [main:] ~ initialized local file as

Hi, i have an issue that had also time ago and dont remember how i fixed it.Basically when try to connect to some hbase tables, it gives error : "org.apache.hadoop.hbase.NotServingRegionException:  table XXX is not online on worker04"My hbase is 2.0.Another thing is that if i restart hbase, i have like 200 Regions in transit, but they disappear if i do "restart all region servers" . But it doesnt fix the problem.I recall that i did something with zookeeper, with the ZkCli, but cant remember what.I think the problem is with the Meta table. But since i'm in a Production env. i must be very careful.Thanks for your help!

I got an error invalid conversion requested in put database record processor while reading the data from file and loading into table.  I observed one of the table field datatype created as char. When I modify it to varchar datatype I can able to load data using PutDatabaseRecord processor.  Nifi version  I am using is 1.13.2 . Kindly advise me how to load char datatype values via nifi without altering table. 

In yarn    Impala Admission Control 

Hi, I want the first letter to be a capital letter and the rest lower case: UPPER(LEFT(fieldname,1))+LOWER(SUBSTRING(fieldname,2,LEN(fieldname))) AS fieldname I keep getting this error:Error while compiling statement: FAILED: ParseException line 25:10 cannot recognize input near 'LEFT' '(' 'fieldname' in function specification Please help

I am using the ConsumeKafkaRecord_2_6 processor to consume from kafka. I am trying to use a custom LoginModule. I have both a custom LoginModule and a custom CallbackHandler. I put my two classes in a jar then loaded the jar on the Nifi system. When the processor starts, I get this error: Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: com.eek.my.auth.krb.SauKrbLoginModuleWrapper. I am not sure how to add the jar so that the processor will find my classes. The dynamic property I am adding to reference the custom class is sasl.jaas.config. So the value of that looks like this: sasl.jaas.config=com.eek.my.auth.krb.SauKrbLoginModuleWrapper required krbProvider=com.sun.security.auth.module.Krb5LoginModule;