Q&A

Hello, I have an expired certificate for Kafka on my server, my Kafka runs from Cloudera Manager, as follows: But my Kafka Server still working just like my consumers and producers connections via SSL.Can anyone help me to know if it's a bug or misconfiguration?

Hello Experts, I am looking for a regex to configure in "RouteOnContent" processor of Nifi so that based on specific string in flow file content I can route FF. FF content :● rclone.service - Rclone(file transfer tool) Remote Service Loaded: loaded (/etc/systemd/system/rclone.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-02-03 09:46:01 UTC; 6h ago Main PID: 23647 (rclone) Tasks: 13 Memory: 61.3M CPU: 5.143s CGroup: /system.slice/rclone.service └─23647 /usr/bin/rclone rcd --rc-no-auth --rc-job-expire-duration=60m --cache-tmp-upload-path=/tmp/rclone/upload I want to search for the string "Active: active (running)" in flow file content, if present route to different path. I tried few regex but did not work as expected. Any suggestion would be much appreciated. ThanksMahendra

Hello,I am still a newbie in NiFi. I Json flowfile that i want to persist in a database using ConvertJsontoSql & PutSql.I have no problem if the names of keys in Json flowfile match with the columns name in the database.But if the keys names are different from column names whilst referring to same thing.For example:Flowfile attribute:  surnameDatabase column: lastnameI have no access to alter database. And i have tried JoltTransform which didn't work.Please, how do i map json keys to columns names , even if the key names change it would still point to the same column in the database.Thank you.  

Hi, I am seeing some situations where I have two Hive SQL commands running concurrently and I'm getting a lost update.  I am running Hive 2.3.6 on EMR with hive.support.concurrency = true and I believe this shouldn't be happening based on what I understand about Hive table locking.  (I am not using ACID transactions but the table locking should still prevent lost update as far as I know;) Specifically I have a "load data" statement loading data into table T from an S3 location.  I have an "insert overwrite T select * from T" table running concurrently from another Hive connection that deletes some rows from T but should not be affecting rows from the load data statement.  I am seeing that the data from the load data statement disappears after the insert overwrite finishes.  My understanding is that the load data and insert overwrite should create an exclusive table lock on T so they should allow each other to finish before reading or writing data from T.&n

Hey Folks, I'm unable to get 1.14.0 to run on my linux box, it appears to be unhappy with configuring SSL services.  Any advice or suggestions are welcome.  2021-08-03 18:54:06,172 WARN [main] o.a.n.d.html.HtmlDocumentationWriter Could not link to org.apache.nifi.ssl.RestrictedSSLContextService because no bundles were found for ListenFTP2021-08-03 18:54:06,172 WARN [main] o.a.n.d.html.HtmlDocumentationWriter Could not link to org.apache.nifi.ssl.SSLContextService because no bundles were found for ListenFTP2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size2021-08-03 18:54:07,231 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size2021-08-03 18:54:07,234 WARN [main] net.schmizz.sshj.Defaul

Hi guys,    Is there a way to isolate impala daemon resources in one cluster?    eg. 6 daemons in 1 Impala Cluster. Is there a method to isolate 2 daemons' resources for testing, 4 daemons' resources for production? Testing queries only use the memory resource in their daemons.    HAProxy can forward requests to different daemons, but memory resources are not isolated. If query from testing takes too much memories, production query may failed while executing.    The only way I can figure out is setup another impala cluster. I know impala provides admission control, what if the query execution plan estimate memory cost wrong, the actual memory cost is far more than the estimation(eg. 3G VS 150G)? What parameters limit the actual memory usage?

Hello All, I am novice to CDP. I have an existing CDH 6.3.3 setup where we have a 6 node cluster and have deployed various services such as Hive, Impala, Kudu, Spark, HDFS, Sqoop, Yarn . We need to migrate to CDP 7.1.6. Could anyone please provide a detailed steps that can be followed for the migration. 

from impala.dbapi import connectimport pandas as pdclass IMPALA: def __init__(self, host, port, user, db): self.host = host self.port = port self.user = user self.db = db def __GetConnect(self😞 if not self.db: raise (NameError, "没有设置数据库信息") self.conn = connect(host=self.host, port=self.port, user=self.user, database=self.db) cur = self.conn.cursor() if not cur: raise (NameError, "连接数据库失败") else: return cur def ExecQuery(self, sql): cur = self.__GetConnect() cur.execute(sql) resList = cur.fetchall() self.conn.close() return resList def ExecNonQuery(self, sql): cur = self.__GetConnect() cur.execute(sql) self.conn.commit() self.conn.close()user = IMPALA('172.17.200.112', 21050, 'q_cluster', 'rawdata')sql = 'select count(DISTINCT(user_id)) as cnt from events where date = CURRENT_DATE() /*SA(production)*/'re = user.ExecQu

Hi,  I am trying to run this import command: sqoop import --connect jdbc:postgresql://[host]:[port]/[db] --username [user] --password-file [path_to_pass] -table [table] --hive-import -hive-drop-import-delims --as-parquetfile --parquet-configurator-implementation hadoop --hive-table district_startdates_test --target-dir [target_with_db] -m 1 --hs2-url="jdbc:hive2://[cdp]/;ssl=true;transportMode=http;httpPath=[path_to_proxy];user=[user];password=[cluster_passw]" --verbose but it gets stuck at   DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection. ...any idea?thanks

Hello.I would like to use the kafka connect to connect the kafka brokers provided by cdp and on-premise kafka brokers. To do that, I need to set the configuration value "connector.client.config.override.policy".But I could not find the menu to set custom configuration of kafka connect. Could you let me know how to set the custom configuration of kafka connect especially "connector.client.config.override.policy".?

Hello,We need advise about advanced Kafka configuration in Cloudera We are using cloudera env installed on 6 servers per following distribution:Namenode1-2Datanode1-3Application node kafka service is installed on data nodes 1-3 and we would like to know:How can we configure Kafka port per hostname and not per service?we would like to have following outcome for example:datanode1 - remain with kafka default port 9092datanode2 - will be configured with new port #datanode3 - will be configured with new port # ThanksNoam 

Sometimes the repo has all the versions of logredactor, sometimes it only has 2. This varies every time I hit refresh. I'd guess there's servers out of sync with each other behind a load balancer maybe? https://repository.cloudera.com/artifactory/cloudera-repos/org/cloudera/logredactor/logredactor/Our project relies on 1.0.3 for a Spark dependency, is there any chance of this being fixed or should we start just hosting it locally?

Hi. I was wondering if anyone can confirm or deny that kerberized hive can (how?) work with not kerberized Zookeeper (+kerberized HDFS and Yarn). Unless I overlooked something (which is very possible), it seems like in line 93 of attached log hive is trying to auth against ZK, and since there is any ZK security enabled this attempt failed. If I'm right, is there any property that I can set up to disable auth attempts on hive side? hdfs/yarn seems to have zero problems working with unsecured ZK.Or perhaps we need to enable kerberos on ZK nodes because hive won't work otherwise, period?  

Hi All, I am trying to achieve incremental sqoop import into hive when a new record is created as well when updates happen to the existing records. We have tables having both created date column and modified date column. But the modified date column will be updated only when there is any update in the existing records and will remain null when a new record is inserted. Only the created date will be updated when new records are created.  So, when we try the incremental lastmodified option it always looks for the modified date column but our source tables doesnt have values in the modified date column always as it keeps NULL for new records and only updates the value when an existing table gets modified.  So, for this kind of source DB tables with such columns is there any solution in sqoop import.  Also, sqoop incremental with lastmodifed doesnt work with hive tables and we have to create a table already and then run this incremental lastmodified&

Example : Input : 10/12/2019 I need to get subsequent Sunday of given input date

I saw this kind of notation/style using double curly braces everywhere in HDP(Ambari) or CDP (CMS) UI. Below is a configuration value under zeppelin.shiro.knox.main.block for Zeppelin configuration. (This is a random sample I picked and this question is not about Zeppelin.)++krbRealm.signatureSecretFile={{CONF_DIR}}/http_secret++I understand that I can simply overwrite {{CONF_DIR}} with the actual path. However, I wonder whether {{CONF_DIR}} an ansible variable? If yes, how do I define the variable CONF_DIR in CDP Cloudera Manager?https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#defining-simple-variables Regards,

I am a beginner in NIFI. And i would like to read live logs from a remote server to a Nifi flow on another system. I have no access to install Nifi on that remote server, please what is the best method to implement the above scenario.Thanking you so much for your reply.

We're trying to upgrade from TLSv1.0 to TLSv1.2 for the below configuration but getting error as attached.These are the installed versionsHDF: 3.1.0Nifi: 1.5.0.3.1.0.0-564Ambari: 2.6.2.2HDP: 2.6.5.1100. we have tried following the steps from this page and it didn't worked as well.https://community.cloudera.com/t5/Support-Questions/Nifi-SSL-TLS-qestion/td-p/285528Kindly help me to resolve this issue.

In the guide "Cloudera Enterprise Reference Architecture for Azure Deployments" ( https://docs.cloudera.com/documentation/other/reference-architecture/topics/ra_azure_deployment.html), it mentions " Using deployment scripts hosted on the Azure GitHub Repository".  However, the link of "Azure GitHub Repository" (https://github.com/Azure/azure-quickstart-templates/tree/master/cloudera-on-centos) is not valid anymore.   Do you have the valid link or could you fix the documentation? Thanks.

Hi I am trying to build the cluster by following the documenthttps://docs.cloudera.com/documentation/director/latest/PDF/cloudera-director.pdf#page=49&zoom=100,68,200   However I am facing issue while Installing Altus Director Server and Client on the EC2 Instance. I have attached the authentication error. I have tried with admin and cloudera as username and password. Can you please suggest troubleshoot? sudo wget "http://username:password@archive.cloudera.com/p/director6/6.3/redhat7/cloudera-director.repo"   Authentication error 401

Hi Impala Team, SAS BI Server - Impala DB connection set up established and end users can connect to Impala DB, Access data but often analysts are reported below scenario and error. Scenario 1:  Analyst assigns SAS Impala DB library in SAS EG then run the code with assigned Impala DB library in reference to read data, that first query executes fine and when analyst wait for complete execution lets say for 5-10 min and re run the same program then gets below error. Scenario 2.  Analyst expand SAS impala DB library in SAS EG then open table to have look at data, close the table then if try to open the same table or other table from same library after 5-10 min then gets below error. ERROR: CLI prepare error: [Cloudera][ImpalaODBC] (450) Error when calling the Impala Thrift API ExecuteStatement: SSL_write: BrokenPipe Does anyone experience the above scenario and error? can you please suggest what could be the root cause and fix for this?&nb

Hi experts,Our hadoop cluster has an old version of log4j and we were wondering how to properly upgrade log4j?Can we just replace the log4j jar file with an upgraded version? Currently this is one of the log4j files in our hadoop cluster. /usr/hdp/2.6.1.0-129/hadoop/client/log4j-1.2.17.jar Any help is much appreciated. Thanks,

Hi,   I am trying to create 5 node cluster using AWS AMI - Cloudera Director CSC (ami-c318a8a8). However, while launching it via launcher node the error pops up in configuration as below:- Found errors in cluster configuration:* No parcel found for product CDH   as I checked in the configuration file, the parcel repository is: parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.4.7/"]   Can you please suggest the troubleshoot?   Thanks

When trying to access the kudu metrics from a master or worker node, in the browser, Kerberos authentication is needed.In the browser thats no problem, after navigating tohttps://<kudu_master>:8050/metrics Im asked for my credentials via pop-up, after entering them and then it shows me the metrics. I'd like to connect the KUDU API via NIFI and use a InvokeHTTP processor to access the metrics. Of course i get the error that authentication is needed. entering username and pw in the GET request header did not help. How do I make acquire the metrics via a GET request?

Dear experts, I notice when I try to load a hbase data in pyspark, it tells mejava.io.IOException: Expecting at least one region for table : myhbasetable at org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatBase.getSplits(MultiTableInputFormatBase.java:195) at org.locationtech.geomesa.hbase.jobs.GeoMesaHBaseInputFormat.getSplits(GeoMesaHBaseInputFormat.scala:43) at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:130)  It looks like it is telling me the table has to at least have some data in at least 1 region. This is the relevant piece of code: -->https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java   try (Connection conn = ConnectionFactory.createConnection(context.getConfiguration())) { while (iter.hasNext()) { Map.Entry<TableName, List<Scan>> entry = (Map.Entry<TableName, List<Scan>>) iter.next()