Q&A

Hi team,I update airflow to v2.6.3, and find I can't connect to mssql 2008 by using JDBC.Following are my setting：Docker Desktop：v4.19.0airflow：v2.6.3JAVA：openjdk-11-jdkJDBC Driver：jtds-1.3.1.jarConnection URL：jdbc:jtds:sqlserver://xxx.xxx.xx.xx:1433/xxxxDriver Path：/usr/local/airflow/jars/jtds-1.3.1.jarDriver Class：net.sourceforge.jtds.jdbc.Driver The connection was failed.  I wonder why the message show the connection string as below：   jdbc://jdbc:jtds:sqlserver:  why not only  jdbc:jtds:sqlserver:  Any help will be appreciated, thanks!   

Hello all,We are having an issue while adding airflow service to the CDH 5 cluster.Problem: Executing the command "Upgrade Airflow DB" on the Airflow service in a Cloudera cluster on GCP (CentOS 7, CDH 5.16.2, Airflow 1.10.1) results in an error: "Could not create process: null". The attempt to upgrade the Airflow database fails while adding airflow service to the CDH cluster.Context:The same infrastructure and parcel configurations have been successfully used on 6-7 on-premises clusters.Logs are empty, and the scm-server log provides the same error message, indicating a failure to create the process.Environment:Operating System: CentOS 7CDH Version: 5.16.2Airflow Version: 1.10.1Steps Taken:Added the Airflow parcels and csd in the proper location with cloudera-scm owner.Added the Airflow service.Created Mysql DB for airflow.Added Mysql lib/client on the all hosts.Added Redis server to manage the airflow in cloudera.Checked logs for detailed error information, but logs are empty.Verifie

Hi everyone,I've been exploring Ranger for access control with Trino and while doing some tests I found out, that for one of the use cases I need, an user could be in multiple groups. Each of the groups would have a custom row level filter in the same table. Example of row level filters:Select Group = Group_UK, Access Type = Select, Row Level Filter = (country = 'UK')Select Group = Group_DE, Access Type = Select, Row Level Filter = (country = 'DE')If I have a row level policy defined as shown in the example and my user is in both of this groups I will only be able to see the data with the row level filter (country = 'UK'). All other filters coming after the first one won't work for any of the users that are in multiple groups.Is there a way to enable this functionality in Ranger? or can I only give one filter to one user per table? e.g. Select User = user_a, Access Type = Select, Row Level Filter = (country in ('UK', 'DE')) I came up with a workaround to this which

Hi, is it possible to execute an exe file in nifi ?

I came across this:Starting with the NiFi 1.0 release, a Zero-Leader Clustering paradigm is employed. Each node in a NiFi cluster performs the same tasks on the data, but each operates on a different set of data. Apache ZooKeeper elects a single node as the Cluster Coordinator, and failover is handled automatically by ZooKeeper. All cluster nodes report heartbeat and status information to the Cluster Coordinator. The Cluster Coordinator is responsible for disconnecting and connecting nodes. Additionally, every cluster has one Primary Node, also elected by ZooKeeper.If nifi follows zero-leader paradigm, then what is the purpose of primary node? And how does it differ from other nodes?

Hi Team,We got critical vulnerability CVE-2014-0014 found in CDP 7.1.7 SP1 commons-fileupload-1.3.3.jar, could you please check and confirm if Apache Struts is used in the Cloudera 7.1.7 SP1? Thanks.Path:./jars/commons-fileupload-1.3.3.jar./lib/atlas/extractors/lib/azure-adls/commons-fileupload-1.3.3.jar./lib/atlas/extractors/lib/aws-s3/commons-fileupload-1.3.3.jar./lib/atlas/server/webapp/atlas/WEB-INF/lib/commons-fileupload-1.3.3.jar./lib/search/lib/commons-fileupload-1.3.3.jar./lib/solr/server/solr-webapp/webapp/WEB-INF/lib/commons-fileupload-1.3.3.jar./lib/hbase-solr/lib/commons-fileupload-1.3.3.jar./lib/oozie/oozie-sharelib-yarn/lib/spark/commons-fileupload-1.3.3.jar./lib/search/lib/search-crunch/commons-fileupload-1.3.3.jar

Hi ,I am using cloudera CDP Public cloud with CDP Runtime version 7.2.17.As Apache storm is not a part of cloudera stack, i downloaded from apache website and configured kerberos manually with the help of storm documentationhttps://storm.apache.org/releases/2.6.1/SECURITY.htmlI am using kafka spout and hdfs bolt , I was able to connect to kafka spout with kafka Spout configuration properties,But i faced SIMPLE authentication is not enabled. Available:[TOKEN] error for hdfs boltby going through storm documentation related tohdfs bolt( https://storm.apache.org/releases/2.1.0/storm-hdfs.html ) i configured some of properties in yaml filestorm.yaml storm.zookeeper.servers:- "hari-cluster-test1-master0.avinash.ceje-5ray.a5.cloudera.site"storm.local.dir: "/home/venkataavinash_kesineni/storm/apache-storm-2.6.1/data"nimbus.seeds: ["hari-cluster-test1-master0.avinash.ceje-5ray.a5.cloudera.site"]supervisor.slots.ports:- 6700- 6701- 6702- 6703- 6704- 6705storm.thrift.transport

Hi folks,trying to build a local data engineering environment based around HBase and Phoenix. I've manually downloaded and configured all the packages and I'm trying to connect using to Phoenix via the PQS. The PQS is running but when I try and connect I'm getting the following error: [hduser@dataengineering ~]$ sqlline-thin.pyPicked up _JAVA_OPTIONS: -Xmx4096mSetting property: [incremental, false]Setting property: [isolation, TRANSACTION_READ_COMMITTED]issuing: !connect -p driver org.apache.phoenix.queryserver.client.Driver -p user "none" -p password "none" "jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF"Connecting to jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUFjava.lang.IllegalArgumentException: Cannot fetch parser for Response with missing class nameat org.apache.calcite.avatica.remote.ProtobufTranslationImpl.getParserForResponse(ProtobufTranslationImpl.java:332)at org.apache.calcite.avatica.remote.ProtobufTranslationImpl.parseResponse(

Hi I have a table where I have multiple conditions using operators or conditions. so, for each of the rows I need to get the variable name which has _ CONS_GRP = 'CCA'HL_GROUP_IN in ('1_HNO', '2_HO_W') and trim(PROD_SEG_MODL_DE) = 'VASO'CONS_SVC_GRP = 'CCA' and olb_cust ne 'A'output:for row1:   Cons_grpfor row2: HL_GROUP_IN PROD_SEG_MODL_DE for row3: CONS_SVC_GRP               olb_cust  I am okay if the output is in a row or column. appreciate any help.  

Can we run Cloudera Data Platform (CDP) on Amazon Elastic Container Service (ECS) within an AWS environment?

Hi,I require some assistance with CDP -PC CDE cli command. What I am trying to achieve is to run cde jobs via cde cli with the current days date.My command line is - cde job run --name generate_post_view --vcluster-endpoint https://my.cluster.com/dex/api/v1 --user my.user_id Would need assistance on how to pass date argument in this command line so that I don’t have to go to CDE UI and edit the date for running every job.Appreciate any help in thisThanks 

Is it possible to set bootstrap.conf properties(java.arg.2, java.arg.3) in kubernetes config yaml?

I'm trying to get a connection to Hive after enabling SSL. I can already connect with Kerberos, but it's the addition of SSL specifically that's giving me trouble.The application I need to connect is using JDBC in Java. I don't have much control over the internals, just the connection string. I'm trying to get a working connection via JSQSH first as a proof of concept.As a simplest possible case I've tried just logging into the cluster as the hive user and running `hive`. That's able to connect, and I can see the Beeline connection info:Connecting to jdbc:hive2://cluster-head.domain.com:2181,cluster-standby.domain.com:2181,cluster-w-1.domain.com:2181/default;password=hive;principal=hive/_HOST@REALM.COM;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=...;trustStoreType=jks;user=hive;zooKeeperNamespace=hiveserver224/04/30 06:40:03 [main]: INFO jdbc.HiveConnection: Connected to cluster-head.domai

I have a flow file which needs to be saved to Database. Size of flow file content can be <= 2 MB. I have tried the PutSQL but it throws  ORA-01704 as the insert statement with CLOB in it becomes too large. I have tried using PutDatabaseRecord, but it doesn't store complete flowfile in teh clob column, Information that it stores is this "apRecord[{responseMetadata=MapRecord[{responseType=ACTIVITY, activityDataExist=true, partNo=1, noOfParts=6}], responseData=MapRecord[{responseList=[Ljava.lang.Object;@676af33}]}]" @cotopaul @SAMSAL @MattWho @steven-matison @TimothySpann  Can you pls look into thislooks like it is wrapping the Json with MapRecord. Can someone pls suggest how to insert a flowfile content into a CLOB column in DB. I cannot use ExecuteScript to use PL/SQL ( ExecuteScript is not approved to use by Nifi team in my company).

Hello,We are trying to run NiFi + Registry 2.0.0-M2 on OpenShift, and no matter what combination of USER, chgrp, chmod, etc. we use we still get"sed: couldn't open temporary file /opt/nifi/nifi-current/conf/sedYird4M: Permission denied"and pod fails to start. We followed the OpenShift guidance of adding this to Dockerfile to no avail:RUN chgrp -R 0 /opt/nifi && chmod -R g=u /opt/nifiPlease advise, this has been a blocker for a few days now!

For nifi, we have jvm options in bootstrap.conf:java.arg.2=-Xms512mjava.arg.3=-Xmx512mThen we also have env variables(got to know on google search,not from official documentation) that can be set from outside(like from kubernetes config yaml):NIFI_JVM_HEAP_INITNIFI_JVM_HEAP_MAXWhat's the connection between these 2 sets of properties?

I've created a cluster in Cloudera and now I want to add Hive, but I need it to utilize the Tez execution engine. So, I'm adding Hive on Tez and the Tez service. However, I'm confused about assigning server IPs to roles.In the Hive service section, I see options for gateway, hiveserver2, metastore, and WebHcat.In the Hive on Tez section, I see options for gateway and hiveserver2.In the Tez section, only gateway is listed.Now, I'm confused. Should I assign the same IP to all gateways and hiveserver2, or should I assign different IPs? Where should I assign hiveserver2, in Hive or in Hive on Tez? And how many servers can I assign to hiverserver2, metastore, and gateway?Moreover, when the Hive section covers gateway and hiverserver2 options, why do the Hive on Tez and Tez sections also show these options?Additionally, I would appreciate clarification on how to configure Hive high availability in this setup. I'm seeking guidance to proceed and would appreciate any help from the Cloudera com

I want to poll an Impala table using SQL queries and only want to retrieve data records added since I last polled. This would be using a Date/Timestamp column in the table. Does anyone know how to achieve this?I am happy to hold the last Date/Timestamp retrieved, in my source system

Hi,I'm not sure if this has been asked before but Im finding it strange that there is not much info or discussion about it. Basically I have scenario where at some point of time I believe I was getting some corrupted data from an API call. When I went to verify that by executing the API call after few hours from the error I dont see the corrupted data. How do I prove\disapprove  this? if I can search the data provenance for that particular response flowfile I would be able to see what did i get at that time after the call. The problem is the out of the box search provenance criteria doesn't provide a way to search against the content or the flowfile custom attributes and it only allows to search against system fields attributes that I dont store. Is there a way to perform such search even by creating some dataflow using certain processors in nifi or using some scripting language?@MattWhoor any body who can help with I would really appreciated. 

Hi,I had install the CDP PVC and use a PostgreSQL embeded database. First, when create cluster only add the Cloudera Management Service, SOLR, HBASE, Atlas, Ranger, Zookeeper and HDFS. Then I add Hive, Tez and Hove on Tez service. After add Hive service and grant access at Ranger I test to create database and table but Atlas not listing the table. I did some thing to resolved it. First I followed this article https://community.cloudera.com/t5/Support-Questions/Atlas-is-empty-after-hive-table-creation/m-p/324386. After that I update parameter atlas.hook.hive.synchronous value to true. But the Atlas still didn't show the hive table I create. Any workarround for this case? Thanks

We are looking for the values that need to be populated for the best practices recommended for below error along with the eventqueue.size=20000.ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(41,WrappedArray())  1. Run with less executors, if necessary2. Run with less cores per executor3. Disable dynamic allocation if this is an option and start out with a static number of executors

 Failed Provisioning while provisioning CML work space. It was succeeded when the first time I tried.However failed provisioning, after re-porovisioning after delete workspace on my test-bed.And also It happened on customer site for POC. I tried version 1.5.2 and 1.5.3, 1.5.3-h1 on RHEL 8.4.Version of Cloudera Manager Cloudera Manager 7.11.3 (#50275000 built by jenkins on 20240213-1404 git: 14e82e253ab970bfd576e4f80d297769a527df18)Tried DataServices Version1.5.2-b886 / 1.5.3-b279 / 1.5.3-h1-b2 Version of screen shot 1.5.2-b886 I have to update the Data services on customer site.Please give any advices.

Hello,I tried to install a trial CDP Cluster following the given steps: https://docs.cloudera.com/cdp-private-cloud-base/latest/installation/topics/cdpdc-trial-installation...The Cloudera Manager executed is https://archive.cloudera.com/cm7/7.4.4/cloudera-manager-installer.bin It was installed successfully.The Cluster installation is failing with the messages:Uninstalled. View Failed to install openjdk8 package. Error: Cannot find a valid baseurl for repo: cloudera-manager This is the complete log:Uninstalled. View Failed to install openjdk8 package. View/tmp/scm_prepare_node.pCEw7JVQusing SSH_CLIENT to get the SCM hostname: 100.200.196.179 59194 22opening logging file descriptorStarting installation script...Acquiring installation lock...BEGIN flock 4END (0)Detecting root privileges...effective UID is 1000BEGIN which pbrunEND (1)BEGIN sudo -S id/usr/bin/which: no pbrun in (/home/opc/.local/bin:/home/opc/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/l

Hello everyone.Since CentOS 8 has been discontinued more than a year ago, and Rocky Linux / Alma Linux have been left occupying the same role as free RHEL's mirror distributions, I would like to know if Cloudera has already a date scheduled in the near future to start supporting any of these distributions as a base operating system for CDP base and related products. Thanks in advance

Greetings, everyone.I'm kind of new to NiFi and trying to implement some simple authorization method that is not default (single-user). All I want to do is a couple static users with some permissions and login screen asking for credentials - 2 roots and 2 read-only will be more than enough for example.I've tried to search for my answer in other topics both in cloudera and documentation, but didn't find any useful info. As far as I'm concerned, Nifi currently support 3 methods - single-user, LDAP and Kerberos.Do I really need to use some LDAP to make this happen or there's more simplified solution in that case? If so - can anyone share some ideas what exactly I need to do?I use Nifi 2.0.0-M2 at the moment. Let me know if some additional info is needed.Thanks a lot in advance and have a great day!