Q&A

Hello, After CDP deployment we observed a suspecious traffic from diffrent internet IPs to CDP Agentport UDP 7191After an investigation it comes from /opt/cloudera/cm-agent/bin/floodPlease could anyone give us more details about this traffic ? RegardsYasine L

kafka version cdh 6.3.4console producer for single message from command line works fine but issue with below commandconsole producer command :1. kafka-console-producer --broker-list <bokerhost:port> --topic <my-topic> --producer.config source-broker-1.properties < sampleMsg.txt2. cat sampleMsg.txt | kafka-console-producer --broker-list <bokerhost:port> --topic <my-topic> --producer.config source-broker-1.properties 3. echo sampleMsg.txt | kafka-console-producer --broker-list <bokerhost:port> --topic <my-topic> --producer.config source-broker-1.properties  my property file looks like:security.protocol=SASL_SSLsasl.kerberos.service.name=kafkassl.truststore.location=/cloudera/security/jks/jks file and set jaas.conf correctly

Hi, we recently started evaluating apache nifi for our use cases. Here is one of such sample pipe line that we have constructedwe have two processed one is Getfile which checks for files present in a local folder. Once they are available it sends them to the execute stream command.  In execute stream command we want to send file that we received from getfile step to a remote linux server. The destination linux server doesn't have an FTP server hence cant use putFTP. This is the command I want to execute to transfer the filesshpass -p '123' rsync -avz -e ssh --ignore-existing "${filename}" test@172.172.172.172:/home/test/nifioutput Here is the configuration for execute stream command log fileAny help would be greatly appreciated.  RegardsLakshmi Tharun. 

Impala version : impalad version 2.12.0-cdh5.16.2As shown above, I have two tables; Customer table and Calendar Table, wich columns I describe below:Customer Table:[Customer] : Customer ID[Date] : Creation date[Num_Days] : Number of working days to calculate the [next_wkday] in the query result.Calendar Table:[civil_util] : when (1) -> workday; when (0) -> non working day I need to get the query_result with the three columns of Customer Table and a calculated date [next_wkday], representing the number of working days [Num_Days] after the Creation date, jumping the zeros in the [civil_util] column. The query below calculates the [next_wkday] using the Lead() function. But is not a solution because the offset parameter must be a constant, and we need to use the [Num_Days] value for each Creation Date:   select *, Lead (to_date(ref_date),5) OVER (ORDER BY to_date(ref_date)) AS next_wkday, datediff(Lead (to_date(ref_d

Hi, when I'm trying to run NiFi (both for versions 1.20.0 and 1.21.0) with JDK 1.8.0_202 on a Windows 64bit machin I keep on receiving the same error message:2023-04-23 01:00:45,455 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2023-04-23 01:00:45,455 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: C:\nifi\NIFI-1~1.0 2023-04-23 01:00:45,455 INFO [main] org.apache.nifi.bootstrap.Command Command: C:\java\JDK18~1.0_2\bin\java.exe -classpath C:\nifi\NIFI-1~1.0\.\conf;C:\nifi\NIFI-1~1.0\.\lib\javax.servlet-api-3.1.0.jar;C:\nifi\NIFI-1~1.0\.\lib\jcl-over-slf4j-2.0.6.jar;C:\nifi\NIFI-1~1.0\.\lib\jetty-schemas-5.2.jar;C:\nifi\NIFI-1~1.0\.\lib\jul-to-slf4j-2.0.6.jar;C:\nifi\NIFI-1~1.0\.\lib\log4j-over-slf4j-2.0.6.jar;C:\nifi\NIFI-1~1.0\.\lib\logback-classic-1.3.5.jar;C:\nifi\NIFI-1~1.0\.\lib\logback-core-1.3.5.jar;C:\nifi\NIFI-1~1.0\.\lib\nifi-api-1.20.0.jar;C:\nifi\NIFI-1~1.0\.\lib\nifi-framework-api-1.20.0.jar;C:\nifi\NIFI-1~1.0\.\lib\nifi-nar-utils-1.

I ran into an issue last week, which seems to be an issue related to the Cloudera ODBC Driver for Impala. First I thought it was an issue with pyodbc, but after submitting the issue on their github we figured out that it was a driver issue. For reference: pyodbc github issue 1197 Simply put, executing query1 and query2 should produce the same results, but it doesn't: query1 = """SELECT col1, col2 as col2_orig, (CASE WHEN col2 IS NOT NULL THEN col2 ELSE 0 END) AS col2_1, (CASE WHEN col2 IS NULL THEN 0 ELSE col2 END) AS col2_2 FROM sandbox.jk_test"""query2 = f"""WITH wrapped_table AS ({query1}) SELECT * FROM wrapped_table""" It results in two different sets: ['col1', 'col2_orig', 'col2_1', 'col2_2']['1 ', '1 ', '1 ', '1 ']['2 ', 'None ', 'None ', '0 ']['col1', 'col2_orig', 'col2_1', 'col2_2']['1 ', '1 ', '1 ', '1 ']['2 ', 'None ', '0 ', '0 '] The 2nd result is the expected

Hi, I’m running spark on k8s and would love to access files that are submitted with —files on the driver (not executor). I know files are uploaded to /tmp/spark-XXXX/ folder that is random, but I can’t find any environment variable that I can use in driver to access them. the reason I want this is to pass a config file in —files with some parameters , and I want to access this file before I initialize spark context.

Hello.Is there a way in ranger policies to  add permissions for users that are only in both of domain group1 and domain group2 (logical AND between groups). Because there is always a logical OR between users and groups in ranger policies??

Hi All,Could you please help in getting below value to be splitted across the array , the value outside of arrray Input:{"DatiRiepilogo": {"Natura": "N3.5","AliquotaIVA": ["22.00","5.00"],"ImponibileImporto": ["215.43","6.70"],"Imposta": ["47.39","0.34"]}} SPEC:[{"operation": "shift","spec": {"DatiRiepilogo": {"Natura":{"*":{"@(2,Natura)": "FatturaElettronicaBody.DatiBeniServizi.DatiRiepilogo[&1].Natura"}},"AliquotaIVA": {"@": "FatturaElettronicaBody.DatiBeniServizi.DatiRiepilogo[&1].AliquotaIVA","*": {"@": "FatturaElettronicaBody.DatiBeniServizi.DatiRiepilogo[&1].AliquotaIVA"}},"ImponibileImporto": {"@": "FatturaElettronicaBody.DatiBeniServizi.DatiRiepilogo[&1].ImponibileImporto","*": {"@": "FatturaElettronicaBody.DatiBeniServizi.DatiRiepilogo[&1].ImponibileImporto"}},"Imposta": {"@": "FatturaElettronicaBody.DatiBeniServizi.DatiRiepilogo[&1].Imposta","*": {"@": "FatturaElettronicaBody.DatiBeniServizi.DatiRiepilogo[&1].Imposta"}}}}}] Expected

Hello everyone,I am currently facing an issue with Cloudera Manager on my CDH 6.3.4 installation. I have noticed that my licenses are no longer valid, which is preventing me from accessing the Cloudera Manager graphical interface. However, I want to emphasize that I can still use the API to manage and control the various services.I have attempted to restart Cloudera Manager and my nodes, but the problem persists. I am not sure why my licenses have become invalid, as they were functional until recently.Can you please advise me on the best approach to resolve this issue? Are there specific steps I should follow to update or reactivate my licenses? Has anyone else encountered this issue with CDH 6.3.4 and could share their experience?I appreciate your assistance and guidance in advance. If you need more information to help me address this problem, please feel free to ask.Best regards.

Hi Team,I have below requirement to convert the SQL server queries into hive. But i am not sure how to achive that as there is no pivot and unpivot available in hive.could you please help me with the hive equivalent queries for the below SQL queries? Query 1-----------------SELECT x,y,zFROM dbo.tbl1 UNPIVOT (split FOR check IN (1, y2, y3, y4, y5, y6, y7, y8, y9, y10)) AS splnpivotQuery 2----------------SELECT *FROM(SELECT a,b,c FROM tbl2) SRCPIVOT (SUM(TOTAL_AMOUNT) FOR cbn_TYPE IN ([ONE TQ FOUR], [going loss], [COSTS LEAVING team sales], [profit], [check money])) AS PIV;Query 3--------------NULLIF (ISNULL(abc.Tc, 0) + ISNULL(abc.YR, 0), 0); Thanks in advance,Wasim 

Hi all, I would like to execute "INSERT OVERWRITE" from managed table to Hive Transaction Table which is EMRFS. What I tried was to first create managed table with s3 location (CSV), and executed the following: INSERT OVERWRITE TABLE tobe_tblSELECT * FROM asis_tbl LIMIT 10; but I got error as below:...Total jobs = 1Launching Job 1 out of 1FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask... I noticed that there was an error saying it is writing in Avro not ORC. What commands or configurations could I make to properly INSERT OVERWRITE to Hive Transaction Table?

I m trying to use HWC (HiveWarehouseConnector) with Spark- to append to an existing Hive ACID table- to overwrite an existing Hive ACID table- to append to a new Hive ACID table that does not exist before- to overwrite to a new Hive ACID table that does not exist beforeWriting to a new Hive ACID table works but writing to an existing table does not work using append/overwrite. Please let me know how to use HWC to append/ overwrite existing ACID tables. List of steps that I followed in CDP 7.1.7 SP1 single node cluster is given below:1) Ran kinit username2) Ran these below commands in beeline hive client command line.drop database testDatabase cascade;drop database tpcds_bin_partitioned_orc_1000 cascade;create database tpcds_bin_partitioned_orc_1000;use tpcds_bin_partitioned_orc_1000;create managed table web_sales(ws_sold_time_sk bigint, ws_ship_date_sk bigint) stored as orc;insert into web_sales values(80000,1), (80001,2), (80002,3);create database testDatabase;3)I prepared a

Hi, I am looking to develop a visualization dashboard from hdfs or hive tables in grafana, as far i found that it is used only for telemetry. is that true ? what are other options?

I'm using the Zeppelin 0.10.1 container. The first launch of the container is successful.If you stop the container and start it again, it does not startcontainer log on startzeppelin | INFO [2023-10-09 16:09:59,673] ({main} ZeppelinLocationStrategy.java[locate]:44) - Load configuration from /opt/zeppelin/conf/zeppelin-site.xmlzeppelin | INFO [2023-10-09 16:09:59,681] ({main} ZeppelinLocationStrategy.java[locate]:44) - Load configuration from /opt/zeppelin/conf/zeppelin-site.xmlzeppelin | INFO [2023-10-09 16:09:59,796] ({main} ZeppelinConfiguration.java[create]:135) - Server Host: 0.0.0.0zeppelin | INFO [2023-10-09 16:09:59,796] ({main} ZeppelinConfiguration.java[create]:139) - Server Port: 8088zeppelin | INFO [2023-10-09 16:09:59,796] ({main} ZeppelinConfiguration.java[create]:141) - Context Path: /zeppelin | INFO [2023-10-09 16:09:59,797] ({main} ZeppelinConfiguration.java[create]:142) - Zeppelin Version: 0.10.1zeppelin | INFO [2023-10-09 16:09:59,821] ({main} Log.java[initialized]:16

Hi, we are trying to create an alert to get notification when no row was inserted into a given HBase table in given period of time.  The following command was used: IF (SELECT total_write_requests_rate_across_regionservers WHERE serviceName=$SERVICENAME AND max(total_write_requests_rate_across_regionservers ) = 0 AND entityName = "hbase" AND category = "SERVICE") DO health:badbut the name of HBase table is missing. The `htableName` property is mentioned under the https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_dg_tsquery.html#concept_d2g_k11_dk__section_cdx_sbp_3l chapter, but it does not work - the command return empty results after putting the `htableName` attribute to WHERE condition. Could I please to an advice, how to get a notification when entities are not inserted to the HBase table?

I need some inputs on relationship definitions in a custom processor code:1. Do I need to define all three(success, failure, original) relationships in processor class?2. Do I need to handle all three relationships(using session.transfer(....))?3. Why do I need to transfer original flowfile as it has been processed?

Any hive bucketing experts here?I have this scenario where my each of my individual partitions needs to be bucketed at a different level. I have tried the following scenarios but it doesn't work.Created a bucketed table and created a partition (date_id='2022-10-22') and set the bucketing number to 64.Result: No of files created in the target partition is equivalent to no of buckets in the source table (6). Expected file count is 64.Created a non bucketed table and created a partition (date_id='2022-10-22') and set the bucketing number to 64. Result: No of files created the target is not related to source files count eitherPlease share your thoughts.Thanks

Zookeeper canary failed and Quorum membership failed, I have 3 ZK nodes, I test the connection between the nodes using telnet port 218,4181 are openoutput of the log file:2023-09-29 23:21:12,409 INFO org.apache.zookeeper.server.PurgeTxnLog: Removing file: Sep 22, 2023 11:21:12 PM /var/lib/zookeeper/version-2/snapshot.1ef000000062023-09-29 23:21:12,409 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task completed.2023-09-30 23:21:12,401 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task started.2023-09-30 23:21:12,404 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: zookeeper.snapshot.trust.empty : false2023-09-30 23:21:12,407 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task completed.2023-10-01 07:48:51,957 WARN org.apache.zookeeper.server.persistence.FileTxnLog: fsync-ing the write ahead log in SyncThread:1 took 1479ms which will adversely effect operation latency. File size is 134217744 bytes. See the ZooKeeper trouble

When a try this script, i have  this error raise Exception('No LDAP URL was specified') on  hue/apps/useradmin/src/useradmin/ldap_access.py at master · cloudera/hue · GitHub . So i decided to read  the  following lines : ldap_url = ldap_config.LDAP_URL.get()username = ldap_config.BIND_DN.get()password = desktop.conf.get_ldap_bind_password(ldap_config)ldap_cert = ldap_config.LDAP_CERT.get()search_bind_authentication = ldap_config.SEARCH_BIND_AUTHENTICATION.get()if ldap_url is None:raise Exception('No LDAP URL was specified')Its means that ldap_url is None, so i tried to prind ldap_config and i have :"<class 'desktop.lib.conf.BoundContainerWithGetAttr'>(config=<desktop.lib.conf.ConfigSection object at 0x7f5b80fb0250>, bind_to={'auth': {'jwt': {}}, 'database': {}, 'query_database': {}, 'task_server': {}, 'session': {}, 'knox': {}, 'smtp': {}, 'websockets': {}, 'raz': {}, 'ozone': {}, 'kerberos': {}, 'tracing': {}, 'metrics': {}, 'slack':

I have a usecase which involves creating a new Record(actually several records) from a record in input flowfile, and writing it in output flowfile.PS: I would use avro reader to read input and avro writer to write into output.What is idiomatic way of doing it in nifi?

Hi,in a Unix environment, I get this error while connecting to Hive via Kerberos, using the library pyHive in a python script:thrift.transport.TTransport.TTransportException: Could not start SASL: b'Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/LOCAL.IT@EXAMPLE.IT not found in Kerberos database)'I am able to connect to Kerberos using "kinit -kt user user.keytab", and also via Hive ODBC driver. I use the same krb5.conf file, with Default Realm = EXAMPLE.IT.With kinit, I obtain, correctly:Default principal: user@EXAMPLE.ITValid starting Expires Service principal09/28/23 11:05:16 09/28/23 11:05:16 krbtgt/EXAMPLE.IT@EXAMPLE.ITThe error is only using pyHive library. In the error, the library uses the domain LOCAL.IT instead of the one specified in krb5.conf, that is EXAMPLE.ITMy connection in pyHive:conn = hive.Connection(host="host.domain.it",

Hi,I want to read data from Ranger Rest API ...assets/accessAudit...Works fine but need to set the startDate like I can do it in the Ranger WebUI in the Audit part.Is there any docu where to find how to do that? Example: ...server:port/service/assets/accessAudit?startDate=06/02/2023&resourceType=@table&accessType=select&sortBy=eventTime&sortType=descending&excludeServiceUser=falseIn this example the startDate does not work, only getting the data from the actual day.If I use the startDate in the Ranger WebUI it works fine. BRRoland

Hi Team,I am a newbie for nifi. I am trying to connect mssql 2008 by using DBCPConnectionPool.The following are my setting： Docker Desktop：v4.19.0Nifi：v1.22.0JDBC Driver：jtds-1.3.1.jarJDBC Driver location：/opt/nifi/nifi-current/lib/jtds-1.3.1.jar DBCPConnectionPool Configure Controller Service： Database Connection URL：jdbc:jtds:sqlserver://xxx.xxx.xx.xx:1433/xxxxxDatabase Driver Class Name：net.sourceforge.jtds.jdbc.DriverDatabase Driver Location(s)：/opt/nifi/nifi-current/lib/jtds-1.3.1.jarDatabase User：xxxPassword：xxxValidation query：Select 1 as test And I try to Verify Controller Service Configuration get the message bellow： org.apache.nifi.dbcp.DBCPConnectionPool DBCPConnectionPool[id=67dc5ad3-0189-1000-09ac-b19f6c25590c] Failed to establish Database Connectionorg.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create PoolableConnectionFactory (Network error IOException: Connection timed out (Connection timed out

I want to transfer data from HDP 2.6 with kerberos authentication via spark to CDP with  kerberos authentication. I am trying to run 2 kerberos ticket in a server but not able to run. One kerberos ticket will be used for reading data from HDP and second kerberos ticket for writing data in CDP cluster. Anyone has any better idea apart from spark please let me know