Q&A

Hi All, We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down. Need urgent solution on this. Please help ---Version---Ambari version -> 2.6OS -> Linux (centos 7.9.2009)HDP-2.6.5.1100Attaching the required NN log 2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current; 2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogres

Hi all, i saw this guide: https://community.cloudera.com/t5/Support-Questions/how-to-change-password-for-ambari-db-post-install/td-p/126685 but it was for Ambari 2.2.0 and i was wondering if do you think is the same for Ambari 2.7.4.I have in ambari.properties: - server.jdbc.user.passwd=/etc/ambari-server/conf/password.datDoing "cat /etc/ambari-server/conf/password.dat " i see the default password (bigdata) in plain, so NO PASSWORD ENCRYPTION ENABLED.I'm using Postgres 9.3Thanks!!

Hello Community, I am going through this tutorial https://www.cloudera.com/tutorials/setting-up-a-spark-development-environment-with-java.html and I am trying to submit it through Oozie work flow. In Hue I am going to Query > Scheduler > Workflow then dropping Java program to actions and then uploading jar file then adding the main class which is Hortonworks.SparkTutorial.Main. When I click to run the Oozie work flow job I keep getting the error of: "Caused by: java.lang.ClassNotFoundException: Class Hortonwork.SparkTutorial.Main not found". I'm using intelliJ to do this project so I hold down ctrl and hover over Main and it takes me to MANIFEST.MF file and it says Main-Class: Hortonworks.SparkTutorial.Main so I'm getting the main class definition right I feel like. I cannot figure out why it is saying it can't find my class.  

Hello Team, In a customer cluster facing the issue with too many open file descriptors. The CDP version is 7.1.5  Bad : Open file descriptors: 31,364. File descriptor limit: 32,768. Percentage in use: 95.72%. Critical threshold: 70.00%.  Can you please share how to mitigate this issue? Is it okay to increase the Maximum Process file descriptors and what would be the recommended value? Best Regards  

we need to block the DDL (create, drop, insert, update) for whoever is using IMPALA, is it possible to do this type of blocking? ps: user should only be able to SELECT with IMPALA and nothing else. DDL should only work via HIVE. Is it possible to do this type of blocking configuration?

HiFrom an API I receive a parquet as an Array[Byte] (stored here as parquetPayload), and want to convert that into a DataFrame. My current function is below, and includes a write to disk  val tempFilePath = new Path("/tmp/", java.util.UUID.randomUUID().toString + ".tmp")val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)val stream = fs.create(tempFilePath, true)try { stream.write(parquetPayload, 0, parquetPayload.length)} finally { stream.close()}val df = spark.read.parquet(tempFilePath.toString)   This works, but I would like to avoid the write to disk, and keep this all in memory. Is this possible? 

Hi Team, I am ex-employee of CLoudera/Hortonworks and completed HDPCA certification in Dec'19. Registered mail ID was smahendrada@cloudera.com. I missed to save my certificate and I dont have the certificate copy. Could you please help me. Thanks,Sruthi

Hi Guyz, please help me to build the python script for cleaning HDFS dir which are older than 3 days.give me the suggestions 

Hello !I have a ListenFTP processor opened on a port and when i am trying to connect to it via FileZila i have an error "Failed to retrieve directory listing". The connection seems to be establish first but then this error occurs. Nifi is hosted on an ubuntu server running in a docker image ListenFTP processor is opened on port 2221 I tried to change some configuration in FileZila but nothing worked. The connection works well on localhost. Somone has already encoutered that kind of issue ?

[root]# hiveSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://

hello all,I am currently running a python version of 3.6.8, Can I still install spark 3 on my cdp cluster?

Hi,I am not able to permanently delete tombstone in Hbase.Tombstone markers are flushed for non-MOB column family but it is not flushed for the MOB column family. Any solution to delete the same so that we save some hdfs space. Regards,Satya

Hi, I am trying to export large hbase table into sequence files. Our requirement is to split the sequence files with specific size limit. We have tried multiple options but output part files are getting generated with large size like 6-7 GB. We used the below sample export command which trying to generate sequence file in a range of 100mb - 135mb,  but didn't work as expected.   hbase org.apache.hadoop.hbase.mapreduce.Export -D dfs.blocksize=1048576 -D mapred.min.split.size=102400 -D mapred.max.split.size=1048576 \table_name /output/hbase/   Could you please help me here to export the table into small chunks ?  

Hi,every one:    I have a question.I found there are many "cmd=get_database" log in hadoop-cmf-hive-HIVEMETASTORE.out file.But I haven't done this much.Is it because there are so many other actions that trigger it? 

If I took an export of the main flow (NiFi Flow) from a server via the Export to JSON option on the right-click menu, how can I import that into a new server as the top-level flow?  I know I can create new process groups with the export.  But I want to restore the controller services and parameter contexts. I know I could import into a process group, and then move things out again.  But, that would also require re-creating all the controller services back at the top level...which would create new UUIDs...and require updating 400-500 processors manually to map them to the new controller services. I've seen various posts about using NiFi registry and the toolkit.  But even when I look at that, I don't see how that works with the top-level group.  If there is some way to handle this that will require a minimum of reconfiguration on services, processors or process groups after the import, it would be appreciated.

2021-05-18 16:50:45,740 INFO scm-web-112:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/testConnection2021-05-18 16:50:45,744 INFO scm-web-112:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HueTestDatabaseConnection with sensitive arguments. Host: DbHost{id=2, hostId=4f19de0c-8fc0-4c39-9ed8-fb65346048ba, hostName=cn2.hadoop.com}2021-05-18 16:50:45,854 INFO scm-web-112:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/testConnection, Status:2002021-05-18 16:50:48,768 INFO scm-web-113:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult2021-05-18 16:50:48,779 INFO scm-web-113:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:2002021-05-18 16:50:51,770 INFO scm-web-105:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operat

Hello, I am trying to install Cloudera Manager 7.4.4 trial from CDP Private Cloud Base Free Trial | Cloudera. The install script worked as expected and I am able to access the CM portal through port 7180. My trial licence is valid for remaining 46 days.  But when trying to set up a cluster it constantly fails with the following error /tmp/scm_prepare_node.e4VFgyneFailed to download package. Status code: 404 I am new to Cloudera and already tried to contact sales but no feedback so far. :-(((

I have one Json file and I have validate it, Its a valid json.I am creating Hive table on that file and then select data but its showingError: java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object- JSONException: A JSONObject text must end with '}' at 2 [character 3 line 1](state=,code=0)when I tried to set below property.ALTER TABLE test_rahul SET SERDEPROPERTIES ( "ignore.malformed.json" = "true");Then It showing Null values for all the fields. Can someone Kindly Help me.json File:{"buyer": {"legalBusinessName": "test1 Company","organisationIdentifications": [{ "type": "abcd","identification": "test.bb@tesr"},{"type": "TXID","identification": "12345678"}]},"supplier": {"legalBusinessName": "test Company","organisationIdentifications": [{"type":"abcd","identification": "test28@test"}]},"paymentRecommendationId": "1234-5678-9876-2212-123456","excludedRemittanceInformation": [],"recommendedPaymentInstructions": [{"executionDate": "2022-06-12","

Hi I see that Cloudera Manager and CDP Private Cloud Base versions 7.6.7 and 7.1 can be install on redhat 8.4. I have servers redhat 8.4 but I did a yum update and now are redhat 8.5, my question is, can I install CDP in this linux version? redhat 8.5?Thanks

I'm having difficulty with this processor and making sure I get all the security stuff correct. I've got no problem with everything downstream, but when it comes to Keystore/Truststore stuff I'm like a potato in a forgotten cellar.  For now, just to test, I have this (nifi 1.15.3): And the SSL Context:Where my jks was generated using (with testing values)keytool -genkey -alias nifi-ftps -keyalg RSA -keypass MyKeyPass312 -storepass MyStorePass312 -keystore nftp.jks When I log on with Filezilla it gets to this point:Status: Connecting to my.server.ip...Status: Using username "nifitest".Status: Access deniedError: Authentication failed.Error: Critical error: Could not connect to server Which I assume means the ports etc are all good and it can connect to the Processor - but I'm missing something in my SSL settings. I have tried changing FileZilla to use a key file but it only uses PPK or PEM files. This is my java version: sudo java -versionopenjdk

I have written python function and run in pyspark - cloudera shell but it throwing error while running it. But when I run the same function in my pyspark - local shell it's working perfectly fine. Complete Code:from pyspark.sql.functions import *df=spark.createDataFrame([('America/New_York','2020-02-01 10:00:00'),('Europe/Lisbon','2020-02-01 10:00:00'),('Europe/Madrid','2020-02-01 10:00:00'),('Europe/London', '2020-02-01 10:00:00'),('America/Sao_Paulo', '2020-02-01 10:00:00')],["OriginTz","Time"])df2=spark.createDataFrame([('Africa/Nairobi', '2020-02-01 10:00:00'),('Asia/Damascus', '2020-02-01 10:00:00'),('Asia/Singapore', '2020-02-01 10:00:00'),('Atlantic/Bermuda','2020-02-01 10:00:00'),('Canada/Mountain','2020-02-01 10:00:00'),('Pacific/Tahiti','2020-02-01 10:00:00')],["OriginTz", "Time"])df.createOrReplaceTempView("test")df2.createOrReplaceTempView("test2")tables = ["test", "test2"]frames = list(range(0,2))def hive_read_func(tables, frames):for table, frame in zip(tables, frame

Hello Team, can you please advise why connection to Kafka fails? Is the issue with keytab? Keytab is working fine on Linux machine.sasl.jaas.config= com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="D:\services.kerberos.keytab" principal="services/rb-hadoop-06.mtg.local@INNOV.LOCAL"; org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClientat org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:540)at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.$anonfun$create$2(PluginsAwareKafkaAdmin.scala:17)at io.conduktor.plugins.PluginsResources$$anon$1.executeWithPlugins(PluginsResources.scala:74)at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.$anonfun$create$1(PluginsAwareKafkaAdmin.scala:17)at scala.util.Try$.apply(Try.scala:210)at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.create(PluginsAwareKafkaAdmin.scala:17)at io.condukt

Is there a limit to the size of an array that can be copied using INSERT INTO SELECT FROM?I have several source Hive EXTERNAL tables, stored as ORC files, that have ARRAY<DOUBLE> columns.  One such table has an array with up to 8192 values.  Another table has an array with exactly 1024 values in all rows.  SELECTing from either table gives the expected results. INSERT INTO another_table SELECT FROM  source_table succeeds for the table with 1024 values, and fails with an "ArrayIndexOutOfBoundsException: 1024" for the table that has up to 8192 values in some rows.  I'm on HDP 3.1.4.0-315 with HIVE 3.1.0 Stack track snippet:Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024at org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.storePrimitiveRowColumn(VectorDeserializeRow.java:568)at org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.storeComplexFieldRowColumn(VectorDeserializeRow.java:778)at org.apache.hadoop.hi

NIFI Version 1.15.3Java Version 8.0.72 2 node cluster 2022-04-21 10:33:08,447 ERROR [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Unable to save flow controller configuration due to: org.apache.nifi.encrypt.EncryptionException: Failed to get Cipher for Algorithm [AES/GCM/NoPadding]org.apache.nifi.encrypt.EncryptionException: Failed to get Cipher for Algorithm [AES/GCM/NoPadding]

Dear Team, kindly advise on why connection from our CDC tools to Kudu is not working? What does KuduWriterException: End of file: unable to send message: Other end of pipe was closed (error 0) mean? at java.lang.Thread.run(Thread.java:750)com.webaction.common.exc.ConnectionException: Couldn't establish connection with Kudu ( RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.m