Q&A

I'm trying to use NiFi expression language with a custom property on the ExtractText processor. It appears that the ExtractText processor interprets the value literally and does not interpolate the value. Is there a way to achieve this?    java.util.regex.PatternSyntaxException: Illegal repetition near index 2${start}^at java.base/java.util.regex.Pattern.error(Pattern.java:2028)at java.base/java.util.regex.Pattern.closure(Pattern.java:3374)at java.base/java.util.regex.Pattern.sequence(Pattern.java:2229)at java.base/java.util.regex.Pattern.expr(Pattern.java:2069)at java.base/java.util.regex.Pattern.compile(Pattern.java:1783)at java.base/java.util.regex.Pattern.<init>(Pattern.java:1430)at java.base/java.util.regex.Pattern.compile(Pattern.java:1095)at org.apache.nifi.processors.standard.ExtractText.onScheduled(ExtractText.java:366)at jdk.internal.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)at java.base/jdk.internal.reflect.DelegatingMethodAccessorImp

I'm trying to read tables and columns and columns' type from Metastore DB. I checked "COLUMN_V2" table and it doesn't have TBL_ID, but it has CD_ID ! Then, I released that there is a table named CDS having one column which is "CD_ID" and the values are as the same as TBL_ID in TBLS table. I want to know if below query using CD_ID is a reliable query or not.Select tbls.TBL_NAME, cv2.COLUMN_NAME,cv2.TYPE_NAME from TBLS tblsJOIN COLUMNS_V2 cv2 on tbls.TBL_ID = cv2.CD_ID And I would be grateful if I can know more about below concerns:1. What is the purpose of CDS table?2. why TBL_ID was not enough and CD_ID was needed?3. What does "CD" stands for?4. Does CD_ID has the same value as TBL_ID all the time? or there are scenarios that they can be different? 

Hello everyone. So the problem is next. I have Apache Hive 2.3.6. on my cluster. I start multiple sessions simultaneously and independently from each other. And then temporary .pipeout files with directories are created and if certain session end up with an error this empty files keep accumulating because they are not deleted if the session end up not right. The similar issue was here, but it was not solved - https://issues.apache.org/jira/browse/HIVE-6091.So the question is how can I find the list of active sessions on my cluster or how can I check whether a session is active or not by sessionId in Hive? I need this information because I want to delete those .pipeout files and in order to do that I need to know which file are still in use (belongs to active session) and which are not! Thanks!

Hi All, Need some help. We have datamart where we have Hive (90%), HBASE (5%), (Impala/Parquet) etc and we want to mask these data sources. Can any one please suggest a TOOL to mask these data sources? 

what is the procedure to migrate HDFS DATA  from one data centre to another data centre?

hive-site.xml <property> <name>system:java.io.tmpdir</name> <value>/usr/local/hive/tmp</value> </property> <property> <name>system:user.name</name> <value>${user.name}</value> </property> <property> <name>hive.server2.webui.host</name> <value>active-nn</value> <description>The host address the HiveServer2 WebUI will listen on</description> </property> <property> <name>hive.server2.webui.port</name> <value>10002</value> <description>The port the HiveServer2 WebUI will listen on. This can beset to 0 or a negative integer to disable the web UI</description> </property> core-site.xml <property> <name>hadoop.proxyuser.hive.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hive.hosts</name&

Dear all,  I'm installing CDP private cloud base and i have installed Hive, Hive on Tez, Impala and Kudu services on the cluster But for my use case i don't want impala to see hive database because i will use impala to work with kudu and i will use hive on tez to work with hive. So, how can i separate impala from hive ?  

Dear Team, can you please advise why varchar is not acceptable when replicating from Oracle to Kudu though I can create table with varchar on Kudu? Error from our CDC tool.Error while writing batch on table impala::cbs.cb_ad_x Reason account_type_v isn't [Type: string], it's varchar. Cause: account_type_v isn't [Type: string], it's varchar CREATE TABLE cbs.cb_address_xm (addressid varchar(65535) ,account_link_code_n BIGINT ,account_type_v varchar(65535) ,address_type_n INT ,account_type varchar(65535) , address_format varchar(65535) ,address_type INT , building varchar(65535) , city varchar(65535) ,city_desc varchar(65535) , country varchar(65535) , country_desc varchar(65535) ,district varchar(65535) , district_desc varchar(65535) , floor varchar(65535) , landmark varchar(65535) , postal_code varchar(65535) , po_code varchar(65535) , street varchar(65535) , street_desc varchar(65535) , sub_locality_code varchar(65535) , sub_locality_desc varchar(65535) , op_insert_date

Hi All, I am new to Cloudera Environment, and i've setting Cloudera with CDSW services but i found error when running spark in session cdsw project , here's the error : 21/09/24 07:56:47 436 ERROR SparkContext: Error initializing SparkContext.java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:892) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.inv

Hello,   I started a test CDP 7.1.7 cluster, added some files in HDFS and created tables in Hive. However Atlas is empty. I expected to find at least some hive_table and hdfs_path items.   The result is the same if I use a custom user (with Atlas ADMIN profile) or with the predefined admin user.   No error is displayed.   Is there something I was supposed to do I missed ? What is my next action to populate Atlas or to find what is missing ?   Thank you, Jérôme

Dears,How send data from nifi cluster to 3 minifi ( same config ) "Data" -> NiFi Cluster -> RPG ->  MiNiFi 1, MiNiFi 2, MiNiFi 3 (receive same "Data" in 3 minifi) Thank you in advance.Best Regards,

Hi Team,How do we identify hive metastore performance and which tables/dbs causing more pressure on Hivemetastore and number of connections on HMS? Regards,naresh.B

从MQTT接收到的数据，要读入json数组Spoiler (Highlight to read){"device_name":"Device37","device_id":"GanghuaLianXian","date":"2021-09-24 17:45:44","sensor":[{"sensor_id":"Data05_37_alarm","value ":"0"},{"sensor_id":"Data06_37_alarm","value":"0"},{"sensor_id":"Data07_37_alarm","value":"0"}, {"sensor_id":"Data08_37_alarm ","value":"0"},{"sensor_id":"Data09_37_alarm","value":"0"},{"sensor_id":"Data10_37_alarm","value":"0"},{"sensor_id ":"Data11_37_alarm","value":"0"}, {"sensor_id":"Data12_37_alarm","value":"0"},{"sensor_id":"Data13_37_alarm","value":"0"},{"sensor_id":"Data14_37_alarm","value":"0"},{"sensor_id":"Data15_37_alarm", "value":"0"}, {"sensor_id":"Data16_37_alarm","value":"0"},{"sensor_id":"Data17_37_alarm","value":"0"},{"sensor_id": " Data18_37_alarm","value":"0"},{"sensor_id":"Data19_37_alarm","value":"0"},"0"},Data15_37_alarm","value":"0"}, {"sensor_id":"Data16_37_alarm","value":"0"},{"sensor_id":"Data17_37_alarm","value":"0"},{" sensor_id":"Data18_37_alarm","value":"0"},{"sensor_id":"Data1

Hi All As I know Impala is similar to SQL but I have a requirement where need to convert SQL server Stored Procedure into Impala code. Is this possible? This procedure has more than 500 lines that have multiple temporary tables, different joins, delete, update, and dynamic SQL. Is there any way this can be converted to Impala?

Hi,   I am using CM 6.3.1 with CFM. As I am exploring NIFI -> ControllerStatusReportingTask and I wanted to log the output to a log file. I realised the logback.xml is auto generated by Cloudera Manager, how am I going to add customize logback configuration for the ControllerStatusReportingTask into the logback.xml?   From the CM, there is a NIFI Node XML Override (logback_safety_valve). Should I use this? It seems that, the input require a complete logback XML file. Just wanted to be sure of the approach as I will be upgrading on my production system.  

Hello Team,I need help in geting the below issue fixed.I have ftp server and in the ftp server i have a path where we will keep our daily files in a folder Path: /user/Mahesh/testFolder under the path : 202100923,20210924 Now i need to transfer the files and the folders and place them in s3 bucket How do i get the folder name. Since i have so many folders do the below command work in update attribute processor ${absolute.path:getDelimitedField('4','/')} please advise any other way to extract folder name. Remember this folder name changes daily. and  can have  multiple folders also at a time.   

Hi, I am trying to connect to impala form dbweaver and constantly getting the below error. I have imported all the necessary drivers. [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: java.net.ConnectException: Connection refused: connect.java.net.ConnectException: Connection refused: connectjava.net.ConnectException: Connection refused: connectConnection refused: connectConnection refused: connect connection string format:- jdbc:impala://hostname:21050/pmmcounter_db

Hi, I have downloaded the latest version of Nifi 1.13.2. In one of my flows Im using an Execute script processor that utilizes python engine  which I know it was working without problems in an older (version 1.11.4). I noticed when the processor executes for the first time it works then if you stop it and re run again it hangs and the flow files get stuck in the upstream queue. Initially I thought something in my script  is causing the issue  so I change it to something as simple as this flowFile = session.get() if flowFile != None:     # All processing code starts at this indent     session.transfer(flowFile, REL_SUCCESS)Even with this simple script the processor hangs and the flow files remain in the queue. This is a major issue and its causing me not to upgrade. Not sure if you are aware of it and if if there is anything that can be done to mitigate. please advise. 

Is it possible for Hive Metastore to support multi-tenancy?I'm trying to build a multi-tenant feature in Hive Metastore. It means that there will be multiple user and each user can create databases, tables, etc. Also, the database or tables created by that user cannot be accessed by another users.I've thought maybe by providing an authentication to Hive will give a multi-tenancy feature. But after tried it with Kerberos, it seems that different users cannot create database with the same name, for example when user1 have a database named dbA then user2 cannot create database named dbA. I've not tried using authorization tools like Ranger, but it looks like it will not give me multi-tenancy feature (different user can create database with the same name).There are several solutions that I can think of, that is:- Each user have their own Metastore. It seems inefficient to have a lot of hive metastore, but it is the most straight forward solution to me.- Include username in database name. D

Failed to regenerate the missing credentials in cm, the error log is as follows:Key log information:com.cloudera.cmf.command.CmdExecException: java.lang.IllegalArgumentExceptionCaused by: java.lang.IllegalArgumentExceptionERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 61246(GenerateCredentials) has completed. finalstate: CANCELLED,success:false,msg:java.lang.IllegalArgumentException It is worth noting that the import of kerberos account manager credentials in CM is successful.And the keytab in the /var/run/cloudera-scm-server directory can be kinit successfully /var/log/cloudera-scm-server/cloudera-scm-server.log The log error is as follows:Key log information:ERROR Staleness-Detector-1: com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Failed to check staleness for service client configs : DbService{id=75,name=atlas} com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs Caused by: java.la

We are migrating from Hortonworks (HDP) to Cloudera (CDP) and having difficulties connecting to the hbase.The following code works when using a HDP host:   from thrift.transport import TSocket, TTransport, THttpClientfrom thrift.transport.TSSLSocket import TSSLSocketfrom thrift.protocol import TBinaryProtocol, TCompactProtocolimport app.hbased.Hbase as Hbasehost = "hdp.mycompany.com"port = 9090sock = TSocket.TSocket(host, port)saslServiceName = "hbase"transport = TTransport.TSaslClientTransport(sock, host, saslServiceName, mechanism="GSSAPI")protocol = TBinaryProtocol.TBinaryProtocol(transport)client = Hbase.Client(protocol)transport.open()table_names = client.getTableNames()print table_names   However, when changing the host to a CDP server:host = "cdp.mycompany.com"we get the following error: Traceback (most recent call last):File "C:\Users\myuser\apps\app\test\test.py", line 20, in <module>table_names = client.getTableNames()File "C:\Users\my

I am trying to learn the basics in cloudera and Impala and by reading this documentation :https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_cluster_util_custom.html#concept_jp4_4bh_hxI saw that with tsquery I can monitor Impala. I want to ask :1) where can I run these kind of queries?2) where can I find the attributes like category and clusterName in cloudera? Thanks in advance.

Hi,We have recently made our external ZooKeeper's implement SSL, Now Nifi is failing to startup.  2021-09-23 15:46:01,711 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut

Hello, We are running single node nifi in 8 core machine and configured 'Maxmum Timer driven thread count' as 300.Once in a while Nifi get totally stuck (wont accept flowfiles) and also no app log will generate till we restart nifi manually. So wondering if its due to nifi thread starvation... In below article 'NiFi Thread starvation' section its mention that system will stuck in resource competition cycle.https://community.cloudera.com/t5/Community-Articles/NiFi-HDF-Dataflow-Optimization-Part-2-of-2/ta-p/245113 So any idea if nifi needs more threads than system available core then will there be any chance of nifi to get stuck ?  @Wynner   Thanks,Mahendra

We have a job that was running fine in 7.1.4, but is now failing after we went to 7.1.6.  It fails with permission issues writing to a directory in /user/yarn.  The job runs as a different user, and we are wondering why it started writing what looks like temporary output to a directory under /user/yarn, that the user running job doesn't have access to?