Q&A

Hi, I have been struggling to get my oozie jobs that run a single spark action to run through without an error. I am using client mode. When I run the spark job via the command line with spark-submit there is no issue and from what I understand this is because I am able to set the driver memory settings but via Oozie the launcher hosts the driver and it so this new container now needs to match the memory settings of my the driver or it will fail. There are many blogs and articles I have read but none of these have helped:https://stackoverflow.com/questions/24262896/oozie-shell-action-memory-limithttps://community.cloudera.com/t5/Support-Questions/Oozie-Spark-Action-getting-out-java-lang-OutOfMemoryError/td-p/39732http://www.openkb.info/2016/07/memory-allocation-for-oozie-launcher-job.htmlhttps://stackoverflow.com/questions/42785649/oozie-workflow-with-spark-application-reports-out-of-memoryWhere the issue comes in partly is I don't know where in the XML these settings sh

Hello Everyone, I would like to understand how I can define the retention period for Ranger Audits in CDP Infra Solr. Ranger Audits are filling up the disk space on my nodes and I would like to configure a retention period for the same. Is there any configuration setting in CDP-INFRA-SOLR or Ranger Service in Cloudera Manager that can be configured? Thanks,Megh

Dear experts, I built an application that runs on HDF-3.4.1.1, to be accurate, a jar file runs in Storm supervisor(s), so I need to get the Ambari 2.7.3.0  with the supported HDF stack and packs binaries, apparently, all the binaries that were for Hortonworks changed with the new links that can't be accessed publicly this is my problem. may you kindly help me in this to provide an alternative so I can get the binaries?   Appreciated in advance @HDF  @GangWar @MattWho  @smdas @Shelton    

I have 5 Data nodes, where 1 Data node crashed. Name node is still replicating the data blocks of the crashed Data node. What happens when I recover the crashed Data node. The data blocks which is under replicated will be reported but what happens to already replicated blocks. Will they be marked as corrupted blocks or will be deleted/removed since there will be 4 replicas?

Hello everybody,   I am looking for ways to increase speed of data ingestion in my Company's Datalake with, specifically, use of Oracle Data Connector in Sqoop. Our HortonWorks(Now Cloudera) environment uses Sqoop 1.4.6. Following theses docs in sqoop site I asked our dba's Team a user with this features: A) create and alter session privileges B) Object privileges: select on dba_segments — Required for Sqoop imports only select on dba_constraints — Required for Sqoop imports only select on v_$database — Required for Sqoop imports only select on v_$parameter — Required for Sqoop imports only Playing with this Sqoop feature I ran this command: Sqoop import-Doraoop.jdbc.url.verbatim=true--direct--connectjdbc:oracle:thin:@myserver.myserver.myserver:1521/mydb--usernamemyuser--password-alias********--tablemyschema.mytable--target-dir/mypath/mytable--verbose And get this output: >>> Invoking Sqoop command line now >>>2021-06-04 18:05:04,323 [main] WARN org.apach

I'm sending data from a MiNiFi agent to a NiFi Cluster. I set port 1026 for the nifi.remote.input.socket.port in the site to site properties section of the nifi.properties file in the NiFi cluster to get these data. I want to add another MiNiFi agent for sending some different data to the previously mentioned NiFi Cluster. So I want to add another port in the nifi.remote.input.socket.port for distinct receiving values in the NiFi cluster from MiNiFi agents. How can I add multiple values for nifi.remote.input.socket.port and if it is not possible, how can I handle receiving different values in the same NiFi cluster from MiNiFi agents.Nifi version: 1.8.0MiNiFi version: 0.5.0 java

Hi team, currently I have chosen the HDP Version 3.1.5 to gain insights from structured and unstructured data, but I feel confused because of Cloudera announcement about moving to a subscription model, Does All of technology about HDP remain, free and open source?  if I have a cluster running HDP 3.1.5 without a subscription but have the local repo, does it allow me to use it for free? 

Hello, I am trying Apache Nifi for testing purpose. I create a flow that need to unzip a file that contains an .xml file. I chain those processors: ListFIle => IdentifyMimeType => UnpackContent. All files I have test are failled. And I have a java error on my logs: 2021-04-09 17:56:40,839 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.UnpackContent UnpackContent[id=b746ede2-0178-1000-dd20-22c2c827cbb7] Unable to unpack StandardFlowFileRecord[uuid=cfe7807c-d6ad-4127-b779-75b2f57c0ba6,claim=,offset=0,name=data.zip,size=0] due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from UnpackContent[id=b746ede2-0178-1000-dd20-22c2c827cbb7]: net.lingala.zip4j.exception.ZipException: Could not fill buffer; routing to failure: org.apache.nifi.processor.exception.ProcessException: IOException thrown from UnpackContent[id=b746ede2-0178-1000-dd20-22c2c827cbb7]: net.lingala.zip4j.exception.ZipException: Could not fill bufferorg.apache.ni

I have setup a 3-node nifi cluster (i.e. nifi2, nifi3, and nifi4) using version 1.13.2 and I have enabled security and TLS using nifi tool kit. Also, I have configured the nifi cluster to authenticate using openid connect against azure ad. During the login, the browser was redirected to the azure portal for authentication, after successful login, it was redirected back to the nifi node but got "Untrusted proxy error" with following stack trace on every node:...2021-06-04 17:49:09,091 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null2021-06-04 17:49:09,102 DEBUG [NiFi Web Server-17] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.2021-06-04 17:49:09,102 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null2021-06-04 17:49:09,107 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifi4.{valid_

Hello Team, We have scheduled the command below kinit -R -kt /opt/striim/streamset.keytab streamset/RB-HADOOP-03@INNOV.LOCAL Does this refresh the ticket? We have an ETL tool that will replicate data to Kudu. The user on the ETL tool will receive authentication keys and save this in "keytab"? then Ranger will allocate this user a ticket? How do we analyze this ticket? What is its purpose? Thanks, Roshan

Dear all, After migrating an historical flow from 1.11.4 to 1.13.2, without any changes, after a while, the limits are raised but the content repository is never cleaned. I restart nifi, and the content-repository is cleaned. Thx for your helpJean-Louis OS : Ubuntu : 20.04.2Java : OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)Nifi properties: # Content Repositorynifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepositorynifi.content.claim.max.appendable.size=10 MBnifi.content.claim.max.flow.files=100nifi.content.repository.directory.default=/data3/nifi_repositories/content_repositorynifi.content.repository.archive.max.retention.period=12 hoursnifi.content.repository.archive.max.usage.percentage=50%nifi.content.repository.archive.enabled=truenifi.content.repository.always.sync=falsenifi.content.viewer.url=/nifi-content-viewer/nifi.content.repository.encryption.key.provider.implementation=. 

Hi,I've been using a flow to get events from Azure Event Hub. I have decided to move from GetAzureEventHub to ConsumeAzureEventHub. But ConsumeAzureEventHub returns following: "failed to process session due to EventHub names must conform to the following rules to be able to use it with EventProcessorHost: Must start with a letter or number, and can contain only letters, numbers, and the dash (-) character. Every dash (-) character must be immediately preceded and followed by a letter or number; consecutive dashes are not permitted in container names. All letters in a container name must be lowercase. Must be from 3 to 63 characters long." The EventHub name contains underscore which is OK for EvenHub itself and for GetAzureEventHub as well. I'm unable to change it because I'm just a "reader" ... Is there any possibility how to use ConsumeAzureEventHub for such name?

We want to hide templates and workflows created by one user to the other users in the system. So the workflow should be visible to only its owner.

How to do error handling for ListSFTP processor (eg. if it fails to connect to SFTP server due to any reason)?  The only connection it shows is "Success".

Hello,I am getting the below error when trying to check logs via CM for Hue Kerberos Ticket Renewer.  CM / CDH - 6.3.3I checked / /var/log/hue but did not find kt_renewer.log. Appreciate any assistance to fix this. ThanksAmn

On a kerberized HDP cluster when I try to run dfsio it fails with below error##########################################org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:563)at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)Caused by: org.apache.hadoop.security.AccessControlException: User hdfs does not have permission to submit application_1622150236559_0013 to queue defaultat org.apache.hadoop.yarn.server.resourcemana

Hello Can we install the below Cloudera applications on the IBM Power System with RHEL 7.x or 8.x OS?  1) Spark Streaming 2) Sqoop 3) Nifi 4) Impala Best regardsKannan

I am using cloudera quickstart 5.13.This is a phenomenon that the start and restart buttons of CM disappear as soon as you press add service to use kudu table and download kudu parcel(s).If you click the service arrow such as HDFS or Impala and click restart, the following error occurs.How can I solve this?Help Me!

Hi All, We are getting the below error intermittently for some tables and we have hive 3.1 . I am passing a sample query like insert into table test as select * from some table in Nifi PutHiveSQL processor .  Hive target table is managed table and stored as avro format Error :PutHive3QL[id=d1652e76-54f7-30cf-8dc8-b1934cee3c26] Failed to update Hive for StandardFlowFileRecord[uuid=7447e7f0-4d1e-40fa-9965-5c98f7d11341,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1618924236521-4, container=default, section=4], offset=154416, length=3631],offset=0,name=20210421_test_branch_info,size=3631] due to java.sql.SQLException: Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask. MetaException(message:Cannot change stats state for a transactional table ra_test.test_branch_info without providing the transactional write state for verification (new write ID -1, valid write IDs null; current state

CDP 7.1.6 with Isilon OneFS v8.2.2.0,  AD kerberos enabled. While installing RangerKMS service, it failed to start up with the following errors:   [root@n04 ~]# less /var/log/ranger/kms/ranger-kms-n04.gz.local-kms.log2021-06-11 08:30:32,179 INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp: -------------------------------------------------------------2021-06-11 08:30:32,181 INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp: Java runtime version : 1.8.0_232-b092021-06-11 08:30:32,185 INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp: KMS Hadoop Version: 3.1.1.7.1.6.0-2972021-06-11 08:30:32,185 INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp: -------------------------------------------------------------2021-06-11 08:30:32,208 INFO org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil: getFilesInDirectory('/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/ranger-kms/ews/webapp/WEB-INF/classes/lib/ranger-kms-plugin-impl'):

I am getting following error. I correctly put the path in server.properties, zookeeper.properties and zoo.cfg cmd>C:\kafka_2.11-2.0.0> .\bin\windows\kafka-server-start.bat .config\server.propertiesAccess is denied.log4j:ERROR setFile(null,true) call failed.java.io.FileNotFoundException: \logs\server.log (The system cannot find the path specified)    

Hello Team, Not able to change my company's name in my profile. By mistake during account creation, selected incorrect company's name. Request you to kindly fix it. Thanks,Alex

Hi guys, How to get user specified configuration data in hive udf? what is your solution?i think there are basically three ways to achieve this goal:1. use a hdfs file to store the configuration data in xml/json/txt format, and then read the hdfs file in udf, users need to change the hdfs file when they want to change specific configuration parms;2. use a local xml/json/txt file and pack it into the udf jar, then read the local file in udf, users need to change the local file and repack it into the udf jar when they want to change specific configuration parms;3. use a hive table to store the configuration data, then read the table using hive sql dml statements in the udf (I know this sounds strange, normally we don't issue sql queries to hiveserver2 in udf, but this should also be possble),  users can use hive sql dml to change specific configuration parms when they need to.4. use a hive table to store the configuration data, then read the underneath hdfs file to get configur

I have run manually minor and major compaction. Table data is referring to the new delta and base folder but the old delta and base folders are not removing and it consumes the hive storage. Moreover, No running job is referring to this table.Queries:  create table compaction_check.check_file(name string, age int);insert into compaction_check.check_file values('user 1', 10),('user 2', 20),('user 3', 30);insert into compaction_check.check_file values('user 4', 10),('user 5', 30),('user 6', 15);ALTER TABLE compaction_check.check_file COMPACT 'major';  Check input files: select distinct INPUT__FILE__NAME FROM compaction_check.check_file; output:  Actual table directory : Why stale the base and delta folders are not removed?  

Hello,I am getting following error when trying to start Hive server = 2021-05-20 12:21:50,088 - Will retry 7 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)2021-05-20 12:22:00,098 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}2021-05-20 12:22:00,780 - call returned (1, '')2021-05-20 12:22:00,781 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)2021-05-20 12:22:10,791 - Process with pid 12962 is not running. Stale pid file at /var/run/hive/hive-server.pid  I referred to similar threads at this forum but didn't help.Pls suggest how to fix this issue thank you.Regards,Amey.