Solved

Hello,   I am facing issue with resources for individual jobs in a queue.My 'Dev' queue details =     How can I make sure that individual job can take unlimited resources in the particular queue?  For now I have increased resources from 30% to 80% for 'Dev' queue as 'Maximum user limit 100%' is not helping.   Regards, Amey.  

I'm looking at a possible use of Hadoop ecosystem with the high rate and high volume scientific data.Data that needs to be stored is arriving in a stream updating ~20 times a second, with each data stream delivering anywhere from a single value to an array of 500k values (could be integers, doubles,..). Streams have a name, data is binary (not text), and comes with timestamps. There could be millions of such streams to handle. I would look into store these input streams with the help Avro to the HDFS. From the client perspective I would like to preferably work with python (not really looking for SQL-like access at the moment). User should query for data using stream name and be able to fetch data from different time slices. Assuming I would be able to scale up the node count and storage space as required, is this use case something that Hadoop ecosystem would be good at? Are there any use cases like this out there? Any benchmarks I can look at? Thank you in advance!

I have a pyspark job with these configs: self.spark = SparkSession.builder.appName("example") \.config("hive.exec.dynamic.partition", "true") \.config("hive.exec.dynamic.partition.mode", "nonstrict") \.config("hive.exec.max.dynamic.partitions", "5000000") \.config("hive.exec.max.dynamic.partitions.pernode", "1000000") \.enableHiveSupport() \.getOrCreate()   I can not find anywhere how to set a configuration to increase the max row size to 150mb. I found the command only in impala. Can you please help me ? Thanks in advance.

I need to setup a sandbox environment with hortonworks schema registry & SSL authentication enabled. Any help on how to achieve the same ?

Hello.I have to support a cluster with Cloudera Data Science Workbench (v1.6), that is installed across two nodes (each with 60GB of memory).I need to understand how CDSW use Docker and Kubernetes. I can see that there are a lot of containers and k8s pods running in the background.Is it possible to automate the removal of idle containers?How can I monitor the memory consumption of each node and each container?CDSW v1.9 has a Grafana Dashboard  for monitoring. Can I get these metrics by a CLI?

I am trying to submit the job to the Hadoop cluster in the CDSW terminal. I see the job get killed before submitting. Can you please let me know how to increase memory for the terminal. cdsw@fecbq09gv6giyrvl:~/h2o-3.32.1.3-cdh6.3$ hadoop jar h2odriver.jar -nodes 1 -mapperXmx 1g -extdriverif $CDSW_HOST_IP_ADDRESS -driverif $CDSW_IP_ADDRESS -driverport $CDSW_HOST_PORT_0 -disownWARNING: Use "yarn jar" to launch YARN applications.Killedcdsw@fecbq09gv6giyrvl:~/h2o-3.32.1.3-cdh6.3$

Hi folks  We installed Streams Messaging Manager (SMM) on CDP 7.1.4, and when access de UI we received this error:"An exception with message [Not authorized] was thrown while processing request." We checked ranger privilegies, but we believe that some privilege is necessary. thanks for help. Eduardo

I'm having somes issues when i run some hive queries, i had one HiveMS on node "monde2" then i added a new HiveMS and HiveServer2 on "mnode4", the issue appears only when the metastore when "mnode4" is selected. Example :Running this query on MS mnode2 finish without any issuehive -hiveconf hive.root.logger=ERROR,console -hiveconf mapreduce.job.reduces=200 -hiveconf hive.execution.engine=mr -hiveconf hive.metastore.uris="thrift://mnode2.<fqdn>:9083" -e "SELECT 'eu' region, CONCAT('eu_', clecommande) order_id, * FROM replication_prd.eu_facturation_commande;" But if i change the MS to mnode4, the query fails with some errorshive -hiveconf hive.root.logger=ERROR,console -hiveconf mapreduce.job.reduces=200 -hiveconf hive.execution.engine=mr -hiveconf hive.metastore.uris="thrift://mnode4.<fqdn>:9083" -e "SELECT 'eu' region, CONCAT('eu_', clecommande) order_id, * FROM replication_prd.eu_facturation_commande;"2021-07-08 18:27:21,027 WARN [main] conf.HiveConf: HiveConf o

Hi,I am using invokeHTTP processor for calling REST API when that REST api gives error like 422 or other it is also sending reason in response body. I want to retrieve that message and send it on mail but on this kind of error flow go into retry or no retry which gives me original flow file only. Can anyone pls help me to retrieve response body in this case instead of original flow file. Thanks in Advanced.

Hello Team, can you please advise if there is an equivalent of the function on Kudu below to extract address and location from a table with XML data type? select TMP_ACCOUNT_CODE_N,decode(EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'),'.',null,EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'))||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@SUB_LOCALITY_DESC')||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@CITY_DESC')New_installation_address from tmp_address_xml@cbsstandbywhere address_type_n = 4 Regards, Roshan

Hello,Rather new to Hadoop, so this might seems like simple question with a straightforward answer. 🙂We are presently facing the following error:2021-06-19 03:31:07.614: Watching for process completion/termination.2021-06-19 03:31:31.997: Task ReceivingAgent failed: RuntimeException: MapReduce job failed - please look in application logs for details. Cause:Task failed task_1621791333957_1544505_m_000016Job failed as tasks failed. failedMaps:1 failedReduces:0AttemptID:attempt_1621791333957_1544505_m_000015_0 Info:Error: com.podiumdata.base.error.PodiumFault: utils.error.code.HADOOP_SPLIT - error processing hadoop file split : unable to find split record boundary for position:2013265920 after 2625536 bytes.you have specified a simple terminated record with no field enclosurescheck your specified recordTerminator:ANY_NEWLINE and your specified fixedFieldCount:4@at_01 com.nvs.utils.stream.hadoop.CsvSplitInputStream.throwUnableToLocateRecordBoundary(CsvSplitInputStream.java:49)@at_01 com.n

Hi,   We are not able to connect to impala from squirrel /workbench client using JDBC. At the same time, We are able to successfully connect using ODBC driver. There is no configuration issue from Kerberos side as we are able to fetch the data using ODBC.   We have installed 1) CDH 6.1 cluster 2) MIT Kerberos enabled 3) SSL enabled for Impala 4) Tested using the Impala JDBC driver version 2.5.45(ClouderaImpalaJDBC4_2.5.45)      and 2.6.2 (impala_jdbc_2.6.2.1003)   We are getting the below error   [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failedAlso, could not send response: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification

I am trying to  format the date field which contains spaces in replace text processor from a csv file.  Got the error as it is unable to parse the date column which is spaces for first record. Please let me know how to handle this Error message: Replace text failed to process session due to Cannot parse attribute value as a date; date format ddMMyyyy; attribute value:       Input csv: 1,        , 1232,02091997,234 Search value : (.{1}),(.{8}), (.{3}) Replacement value : $1, ${'$2':toDate("ddMMyyyy") :format("yyyy-MM-dd HH:mm:ss.SSS") }, $3Replacement strategy : Regex ReplaceEvaluation mode : Entire Text 

Hello Team, is there a way we can schedule a sql statement in Impala (hue)? For example I wish to schedule delete of a table. delete cdr.tab1; Thanks, Roshan

Hi team, Could you please tell me why do we segregate the compute node from storage node in the hadoop world, In this way are we not breaking the data locality's  philosophy so in this way we are achieving the intra/inter rack data locality not the local data locality and what is the hurdle we faced in the previous design putting the both(compute and data node) on the same node). Thanks

Hello Gurus, I am having Ranger KMS test connection failed, it is POC test.CDP 7.1.6 with Isilon OneFS v8.2.2.0, AD kerberos enabled. Ranger KMS is up and runningdefault policy login as keyadmintest connection failed Already added following lines in kms-site.xml ( added in Ranger KMS -configuration )hadoop.kms.proxyuser.rangeradmin.hosts=* hadoop.kms.proxyuser.rangeradmin.groups=* hadoop.kms.proxyuser.rangeradmin.users=* Ranger KMS debug:2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: ==> RangerPluginClassLoader.deactivate()2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: <== RangerPluginClassLoader.deactivate()2021-06-26 06:51:38,420 ERROR org.apache.hadoop.crypto.key.kms.server.KMS: Exception in getkeyNames.org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS'2021-06-26 06:51:38,420 WARN org.apache.hadoop.crypto.key.kms.serve

When I do "hdfs dfs -cat file1" from the command line, I got the exception saying that it Cannot obtain block length for LocatedBlock. How should we handle this case?

Hello,I would like to write Bash script to run across all of our Cloudera hosts and to monitor if some envs are going to be expired via GUI, i'm able to see it on http://<cloudera host>:7180/cmf/license/settings page does anyone know how can i get result of Cloudera license expiration date via command line? Thanks,Noam

Hi All, I have a requirement like, I need to create hive policy through REST API in CDP Env with two groups .Here , we are giving permission at group level not at user level.one group with "ALL" permissions  and 2nd group with "select" permission. I have created policy through REST API with one group with "all" permissions but how to mention 2nd group with "select" permission in same create policy command.I have tried the below method but did not worked.   CURLURL="https://host:6182/service/public/v2/api/policy/"CURLDATA='{ "isEnabled":true,"service":"cm_hive","name":"policy_test1","description":"Policy for employees database access","isAuditEnabled":true,"resources":{"database":{"values":["DBNAME"],"isExcludes":false,"isRecursive":false},"table":{"values":["*"],"isExcludes":false,"isRecursive":false}},"policyItems":[{"accesses":[{"type":"ALL","isAllowed":true}],"users":[""],"groups":["GROUP1"],[{"type":"SELECT","isAllowed":true}],"users":[""],"groups":[

Hello,I need some assistance / guidance related to Impala Audit & Lineage Logs.In our cluster I  see Impala Audit Logs (enable_audit_event_log) Un-checked, yet on the node level we see logs getting generated.[service1@myserver audit]# ls -ltrhtotal 215M-rw-r--r-- 1 impala impala 4.2M May 27 14:45 impala_audit_event_log_1.0-1622078304372-rw-r--r-- 1 impala impala 3.9M May 27 15:34 impala_audit_event_log_1.0-1622083537006-rw-r--r-- 1 impala impala 4.3M May 27 16:44 impala_audit_event_log_1.0-1622086468134-rw-r--r-- 1 impala impala 4.3M May 27 17:18 impala_audit_event_log_1.0-1622090691805-rw-r--r-- 1 impala impala 4.3M May 27 18:26 impala_audit_event_log_1.0-1622092726630-rw-r--r-- 1 impala impala 4.2M May 27 20:07 impala_audit_event_log_1.0-1622096799288-rw-r--r-- 1 impala impala 4.0M May 27 21:17 impala_audit_event_log_1.0-1622102850640-rw-r--r-- 1 impala impala 3.7M May 28 01:36 impala_audit_event_log_1.0-1622107032783-rw-r--r-- 1 impala impala 2.7M May 28 14:34 impala_audit_

Hi, is there a way we can change the datatype from int to double for a column on Impala? Table size is around 3 billion records. I do not plan to drop and recreate it. ALTER TABLE cbs.gprs_home_cdrs CHANGE percentage_val_n percentage_val_n double;AnalysisException: Cannot change the type of a Kudu column using an ALTER TABLE CHANGE COLUMN statement: (INT vs DOUBLE)impalad version 3.4.0-SNAPSHOT RELEASE (build 134517e42b7b6085e758195465f956f431e0e575) Built on Sat Dec 12 11:15:02 UTC 2020Version: Cloudera Enterprise 7.1.3 (#4999720 built by jenkins on 20200805-1701 git: fa596184790377f07ba80e9cd4da8b875237939c)Java VM Name: OpenJDK 64-Bit Server VMJava Version: 11.0.10 Thanks, Roshan

Dear Team, how can I mask last 5 digits in field below in Kudu? on Ranger?Thanks, Roshan

I wanted to know about this everchanging technology and its core advantages which can benefit a business

I'm getting this error Error while processing statement: Cannot modify hive.msck.path.validation at runtime. It is not in list of params that are allowed to be modified at runtime how can I fixed it or configure it on hive config in CM?

Hello, I am writing on behalf of Eustema Spa (partnership@eustema.it) which is currently a bronze level partner with Cloudera.I wanted to ask if it is possible to receive the certificate attesting to the partnership (as it was sent to us for the previous deadline before renewal) Thank you  Alessandro