Q&A

Hi all,I upgrade my nifi from 1.10 to 1.13.2 and i faced an error as the image below:I tried find and fix with a lot solution, but i cant put the json into my mysql database.Any one can help me?  ThanksDuongNT

use script ：  /opt/module/cm/cm-5.16.2/share/cmf/schema/scm_prepare_database.sh  Error：  Mon Jul 12 17:30:10 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'temp' and jdbc url 'jdbc:mysql://localhost/?useUnicode=true&characterEncoding=UTF-8'com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet successfully received from the server was 287 milliseco

Hello,   I am facing issue with resources for individual jobs in a queue.My 'Dev' queue details =     How can I make sure that individual job can take unlimited resources in the particular queue?  For now I have increased resources from 30% to 80% for 'Dev' queue as 'Maximum user limit 100%' is not helping.   Regards, Amey.  

I'm looking at a possible use of Hadoop ecosystem with the high rate and high volume scientific data.Data that needs to be stored is arriving in a stream updating ~20 times a second, with each data stream delivering anywhere from a single value to an array of 500k values (could be integers, doubles,..). Streams have a name, data is binary (not text), and comes with timestamps. There could be millions of such streams to handle. I would look into store these input streams with the help Avro to the HDFS. From the client perspective I would like to preferably work with python (not really looking for SQL-like access at the moment). User should query for data using stream name and be able to fetch data from different time slices. Assuming I would be able to scale up the node count and storage space as required, is this use case something that Hadoop ecosystem would be good at? Are there any use cases like this out there? Any benchmarks I can look at? Thank you in advance!

I have a pyspark job with these configs: self.spark = SparkSession.builder.appName("example") \.config("hive.exec.dynamic.partition", "true") \.config("hive.exec.dynamic.partition.mode", "nonstrict") \.config("hive.exec.max.dynamic.partitions", "5000000") \.config("hive.exec.max.dynamic.partitions.pernode", "1000000") \.enableHiveSupport() \.getOrCreate()   I can not find anywhere how to set a configuration to increase the max row size to 150mb. I found the command only in impala. Can you please help me ? Thanks in advance.

I need to setup a sandbox environment with hortonworks schema registry & SSL authentication enabled. Any help on how to achieve the same ?

Hello.I have to support a cluster with Cloudera Data Science Workbench (v1.6), that is installed across two nodes (each with 60GB of memory).I need to understand how CDSW use Docker and Kubernetes. I can see that there are a lot of containers and k8s pods running in the background.Is it possible to automate the removal of idle containers?How can I monitor the memory consumption of each node and each container?CDSW v1.9 has a Grafana Dashboard  for monitoring. Can I get these metrics by a CLI?

I am trying to submit the job to the Hadoop cluster in the CDSW terminal. I see the job get killed before submitting. Can you please let me know how to increase memory for the terminal. cdsw@fecbq09gv6giyrvl:~/h2o-3.32.1.3-cdh6.3$ hadoop jar h2odriver.jar -nodes 1 -mapperXmx 1g -extdriverif $CDSW_HOST_IP_ADDRESS -driverif $CDSW_IP_ADDRESS -driverport $CDSW_HOST_PORT_0 -disownWARNING: Use "yarn jar" to launch YARN applications.Killedcdsw@fecbq09gv6giyrvl:~/h2o-3.32.1.3-cdh6.3$

Hi folks  We installed Streams Messaging Manager (SMM) on CDP 7.1.4, and when access de UI we received this error:"An exception with message [Not authorized] was thrown while processing request." We checked ranger privilegies, but we believe that some privilege is necessary. thanks for help. Eduardo

I'm having somes issues when i run some hive queries, i had one HiveMS on node "monde2" then i added a new HiveMS and HiveServer2 on "mnode4", the issue appears only when the metastore when "mnode4" is selected. Example :Running this query on MS mnode2 finish without any issuehive -hiveconf hive.root.logger=ERROR,console -hiveconf mapreduce.job.reduces=200 -hiveconf hive.execution.engine=mr -hiveconf hive.metastore.uris="thrift://mnode2.<fqdn>:9083" -e "SELECT 'eu' region, CONCAT('eu_', clecommande) order_id, * FROM replication_prd.eu_facturation_commande;" But if i change the MS to mnode4, the query fails with some errorshive -hiveconf hive.root.logger=ERROR,console -hiveconf mapreduce.job.reduces=200 -hiveconf hive.execution.engine=mr -hiveconf hive.metastore.uris="thrift://mnode4.<fqdn>:9083" -e "SELECT 'eu' region, CONCAT('eu_', clecommande) order_id, * FROM replication_prd.eu_facturation_commande;"2021-07-08 18:27:21,027 WARN [main] conf.HiveConf: HiveConf o

Hi,I am using invokeHTTP processor for calling REST API when that REST api gives error like 422 or other it is also sending reason in response body. I want to retrieve that message and send it on mail but on this kind of error flow go into retry or no retry which gives me original flow file only. Can anyone pls help me to retrieve response body in this case instead of original flow file. Thanks in Advanced.

Below error can be seen in host monitor log. Is there a way resolve it?   Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[HOSTMONITOR], EXCEPTION_TYPES=[java.io.IOException], HOST_IDS=[host_id], STACKTRACE=[java.io.IOException: Server returned HTTP response code: 403 for URL: http://cloudera_manager_ip:7180/cmf/j_spring_security_checkat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:285)at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:596)at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:686)at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64)at com.cloudera.cmon.firehose.Main.main(Main.java:392)

ERROR 2021-05-26 15:43:41.000453 [main] - Unable to set property on handler 'kafkahandler' (oracle.goldengate.handler.kafka.KafkaHandler). Failed to set property: TopicName:="BUBB-MX_ALL_SOFT" (class: oracle.goldengate.handler.kafka.KafkaHandler).oracle.goldengate.util.ConfigException: Failed to set property: TopicName:="BUBB-MX_ALL_SOFT" (class: oracle.goldengate.handler.kafka.KafkaHandler).

Hello Team, can you please advise if there is an equivalent of the function on Kudu below to extract address and location from a table with XML data type? select TMP_ACCOUNT_CODE_N,decode(EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'),'.',null,EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'))||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@SUB_LOCALITY_DESC')||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@CITY_DESC')New_installation_address from tmp_address_xml@cbsstandbywhere address_type_n = 4 Regards, Roshan

Hello,Rather new to Hadoop, so this might seems like simple question with a straightforward answer. 🙂We are presently facing the following error:2021-06-19 03:31:07.614: Watching for process completion/termination.2021-06-19 03:31:31.997: Task ReceivingAgent failed: RuntimeException: MapReduce job failed - please look in application logs for details. Cause:Task failed task_1621791333957_1544505_m_000016Job failed as tasks failed. failedMaps:1 failedReduces:0AttemptID:attempt_1621791333957_1544505_m_000015_0 Info:Error: com.podiumdata.base.error.PodiumFault: utils.error.code.HADOOP_SPLIT - error processing hadoop file split : unable to find split record boundary for position:2013265920 after 2625536 bytes.you have specified a simple terminated record with no field enclosurescheck your specified recordTerminator:ANY_NEWLINE and your specified fixedFieldCount:4@at_01 com.nvs.utils.stream.hadoop.CsvSplitInputStream.throwUnableToLocateRecordBoundary(CsvSplitInputStream.java:49)@at_01 com.n

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/LogicalTypeat org.apache.sqoop.manager.DefaultManagerFactory.accept(DefaultManagerFactory.java:67)at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184)at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:272)at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:96)at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:616)at org.apache.sqoop.Sqoop.run(Sqoop.java:147)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)at org.apache.sqoop.Sqoop.main(Sqoop.java:252)Caused by: java.lang.ClassNotFoundException: org.apache.avro.LogicalTypeat java.net.URLClassLoader.findClass(URLClassLoader.java:382)at java.lang.ClassLoader.loadClass(ClassLoader.java:418)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)at java.lang.ClassLoader

Hi,   We are not able to connect to impala from squirrel /workbench client using JDBC. At the same time, We are able to successfully connect using ODBC driver. There is no configuration issue from Kerberos side as we are able to fetch the data using ODBC.   We have installed 1) CDH 6.1 cluster 2) MIT Kerberos enabled 3) SSL enabled for Impala 4) Tested using the Impala JDBC driver version 2.5.45(ClouderaImpalaJDBC4_2.5.45)      and 2.6.2 (impala_jdbc_2.6.2.1003)   We are getting the below error   [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failedAlso, could not send response: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification

I am trying to  format the date field which contains spaces in replace text processor from a csv file.  Got the error as it is unable to parse the date column which is spaces for first record. Please let me know how to handle this Error message: Replace text failed to process session due to Cannot parse attribute value as a date; date format ddMMyyyy; attribute value:       Input csv: 1,        , 1232,02091997,234 Search value : (.{1}),(.{8}), (.{3}) Replacement value : $1, ${'$2':toDate("ddMMyyyy") :format("yyyy-MM-dd HH:mm:ss.SSS") }, $3Replacement strategy : Regex ReplaceEvaluation mode : Entire Text 

Hello Team, is there a way we can schedule a sql statement in Impala (hue)? For example I wish to schedule delete of a table. delete cdr.tab1; Thanks, Roshan

Hi team, Could you please tell me why do we segregate the compute node from storage node in the hadoop world, In this way are we not breaking the data locality's  philosophy so in this way we are achieving the intra/inter rack data locality not the local data locality and what is the hurdle we faced in the previous design putting the both(compute and data node) on the same node). Thanks

Hello Gurus, I am having Ranger KMS test connection failed, it is POC test.CDP 7.1.6 with Isilon OneFS v8.2.2.0, AD kerberos enabled. Ranger KMS is up and runningdefault policy login as keyadmintest connection failed Already added following lines in kms-site.xml ( added in Ranger KMS -configuration )hadoop.kms.proxyuser.rangeradmin.hosts=* hadoop.kms.proxyuser.rangeradmin.groups=* hadoop.kms.proxyuser.rangeradmin.users=* Ranger KMS debug:2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: ==> RangerPluginClassLoader.deactivate()2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: <== RangerPluginClassLoader.deactivate()2021-06-26 06:51:38,420 ERROR org.apache.hadoop.crypto.key.kms.server.KMS: Exception in getkeyNames.org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS'2021-06-26 06:51:38,420 WARN org.apache.hadoop.crypto.key.kms.serve

When I do "hdfs dfs -cat file1" from the command line, I got the exception saying that it Cannot obtain block length for LocatedBlock. How should we handle this case?

Hello,I would like to write Bash script to run across all of our Cloudera hosts and to monitor if some envs are going to be expired via GUI, i'm able to see it on http://<cloudera host>:7180/cmf/license/settings page does anyone know how can i get result of Cloudera license expiration date via command line? Thanks,Noam

Hi - We are laoding data from sqoop to Hive table in parquest format where the dates are showing properly in hadoop. We are distcping the data to AWS S3 and creating the tables in Athean using Glue Crawler. When we query both tables(Hadoop & AWS) we are seeing different values for timestamp & date columns. Hadoop Query Data:SELECT effective_dt FROM "dbname"."table" where Pk_id='78393904'output0001-01-03 07:00:00.000 Athean Query DataSELECT effective_dt FROM "dbname"."table" where Pk_id='78393904'output0001-01-01 12:00:00.000 Any suggestions how to overcome this issue.

Hi All, I have a requirement like, I need to create hive policy through REST API in CDP Env with two groups .Here , we are giving permission at group level not at user level.one group with "ALL" permissions  and 2nd group with "select" permission. I have created policy through REST API with one group with "all" permissions but how to mention 2nd group with "select" permission in same create policy command.I have tried the below method but did not worked.   CURLURL="https://host:6182/service/public/v2/api/policy/"CURLDATA='{ "isEnabled":true,"service":"cm_hive","name":"policy_test1","description":"Policy for employees database access","isAuditEnabled":true,"resources":{"database":{"values":["DBNAME"],"isExcludes":false,"isRecursive":false},"table":{"values":["*"],"isExcludes":false,"isRecursive":false}},"policyItems":[{"accesses":[{"type":"ALL","isAllowed":true}],"users":[""],"groups":["GROUP1"],[{"type":"SELECT","isAllowed":true}],"users":[""],"groups":[