Q&A

Hello guys,I just wanted to know that what are the issues in hive and spark that mostly faces in production , and how to figure out this.could anyone tell me??

HI all, please suggest how to choose variants of CDP. we have HDP on prem wanted to migrate CDP latest. CDP DC or CDP Private cloude Thanks,kant 

One of the JSON values needs to find the string last element. how to get the particular value in attribute with the last element.{"TOT_NET_AMT" : "55.00","H_OBJECT" : "File","H_GROSS_AMNT" : "55.00,58.00","TOT_TAX_AMT" : "9.55"} expected value - {"TOT_NET_AMT" : "55.00","H_OBJECT" : "File","H_GROSS_AMNT" : "58.00","TOT_TAX_AMT" : "9.55"} Much appreciate it in advance.

Hi Team, Can anyone suggest me I need help in Nifi Cron job which runs Every other Wednesday. For Example the next execution is on 11th AUG (wed) and then it has to execute 25th AUG(wed) and then 8th SEP(WED) and so on. Is there any possible way of getting this Cron Expression, could you please help me on this? I tried scheduling like 0 0/10 13 11/14 * ? * but its like running 11th of every month, which is not accurate one.  Regards,Pradeep

Hi Community Team. I´m configuring Distcp between secure clusters in different kerberos realms.Source= HDP 2.6.5Target= CDP Private Cloud Base 7.1.5 I am setting up, but in the process I have doubts as to how these 2 steps are done, I add the links that show little detail1)First doubt (what is the step by step of this):https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/scaling-namespaces/topics/hdfs-distcp-truststore-properties.html 2)Second doubt (what is the step by step of this):https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/scaling-namespaces/topics/hdfs-distcp-set-hadoop-conf.html thanking the help

Am trying to add new host to cluster, but I keep getting this error message `https://archive.cloudera.com/cm6/6.0.1/allkeys.asc/`, logs show same message without details 

Team , Any idea on the migration process from HDP 2.x version to CDP version . RegardsBharad

While trying to create a case to send it to support, it shows me this message : "Access to page is restrictedAccessing the requested page requires special permissions "  what shall i do ?

set hive.exec.dynamic.partition=true;set hive.exec.dynamic.partition.mode=nostrict;dataset.write.mode("overwrite").InsertInto("table") with dynamic partitions  Tried in spark program and also in SPARK3-SHELLERROR Hive: Exception when loading partition with parameters partPath=hdfs://nameservice/user/hive/warehouse/tmp.db/id_name2/.hive-staging_hive_2020-08-05_14-38-00_715_3629476922121193803-1/-ext-10000/dt=2001, table=id_name2, partSpec={dt=2001}, loadFileType=REPLACE_ALL, listBucketingLevel=0, isAcid=false, resetStatistics=falseorg.apache.hadoop.hive.ql.metadata.HiveException: Directory hdfs://nameservice/user/hive/warehouse/tmp.db/id_name2/dt=2001 could not be cleaned up. at org.apache.hadoop.hive.ql.metadata.Hive.deleteOldPathForReplace(Hive.java:4666) at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:4597) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2132) at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java

Hello boys!Trying to get the free version of the CDH to be able to present a solution in the company, but I can't download it.I'm copying and pasting link tamplate https://igor.ufrn@gmail.com:MyPassword@archive.cloudera.com/cdh6/6.3.2-patch4071/ubuntu1804/apt/ And I get the answer:"403 Forbidden (varnish) the provided credentials were incorrect"What should I do?Thank you very much for the support.

I am trying to install Cloudera in AWS for test purposes and the first obstacle I ran into was when I tried to run the cloudera-manager-installer.bin which I downloaded from https://archive.cloudera.com/p/cm7/7.3.1/cloudera-manager-installer.bin. The problem is that when you run this it fails with "https://archive.cloudera.com/cm7/7.3.1/redhat7/yum/repodata/repomd.xml: [Errno 14]  Error 404 - The requested URL returned error: 404 Not Found" and this seems like a bug because the file is there but in a diff location ie https://archive.cloudera.com/p/cm7/7.3.1/redhat7/yum/repodata/repomd.xml (please note the "p" in working link .  How do we fix this ? Looks like installer is looking in wrong location ? 

Hi Folks,Can anyone tell me how to check how many users are connected to the hive server2?And  along with can we see which user is connected ?

Hello Everyone,How can we renew the kerberos ticket from both MIT as well as AD kerberoswhats the step?   Thanks in AdvanceNilesh

Hi, community. I have NiFi cluster, which consist of 2 nodes. Also I have single node NiFi, that connected with cluster by site-to-site. On the side of cluster NiFi I have a client, that make HTTP GET requests, on the side of single node NiFi I have the server that should response for client requests. So, I wan't to use cluster, single node NiFi and their site-to site connection as a proxy for client and server. I Used Handle HTTP request and response processors at the cluster, and Invoke HTTP at the single node NiFi. But answers that came from single node nifi after Invoke HTTP can go to not same node of cluster at which it initially came from HandleHTTPRequest processor. It results, that another node doesn't know http.context.identifier of flowfile and can't properly send it by handle http response processor. Is it possible to make something to make it work?

Hi experts,We recently changed the Ip address of our ambari in our dev enviornment.The cluster seems to be up and running and working properly, however, ambari is not recognizing which namenode is active and which is passive. Also some of the users are unable to access the ambari hive view.This is the error message when trying to access the hive view via ambari. USER HOME CheckMessage: test01.dmicorp.com:50070: No route to host (Host unreachable) Any help is much appreciated. Thanks,

Hi Community, I am trying to open the Hive weburi and getting below error HTTP ERROR 403Problem accessing /hiveserver2.jsp. Reason: java.lang.IllegalArgumentException  Weburi is configured in system with all required parameters. pls advice.  ThanksAmit

Hello, I need your advice regarding what seems like a strange behavior  in the cluster I'm using. Running Spark (2.4, Cloudera) on Yarn the configuration calls the set up of 10 executors:  spark = (SparkSession .builder.master("yarn") .config("spark.executor.cores", "12") .config("spark.executor.memory", "8G") .config("spark.num.executors", "10") .config("spark.driver.memory", "6G") .config("spark.yarn.am.memoryOverhead", "6G") .config("spark.executor.memoryOverhead", "5G") .config("spark.driver.memoryOverhead", "5G") .config("spark.sql.hive.convertMetastoreOrc", "true") .config("spark.executor.heartbeatInterval", "60s") .config("spark.network.timeout", "600s") .config("spark.driver.maxResultSize", "2g") .config("spark.driver.cores","4") .config("spark.executor.extraClassPath", "-Dhdp.version=current") .config("spark.debug.maxToStringFields", 200) .config("spark.sql.catalogImplementation", "hive") .config("spark.memory.f

One of of my data-node won't start. The error log on {HADOOP_LOG}/hdfs/{datanode}.err shown that the port of dfs.datanode.http.address already in use. We use custom port for  dfs.datanode.http.address. When I run netstat -lpn | grep 1022, no service run on it. While for {datanode}.log file is not updated.  Initializing secure datanode resourcesisSaslEnabled:trueOpened streaming server at /0.0.0.0:1019java.net.BindException: Address already in use /0.0.0.0:1022 at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:461) at sun.nio.ch.Net.bind(Net.java:453) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78) at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:158) at org.a

Our ambari webui is encountering some errors when ambari-server restart. When some users are connected and we restart the ambari-server, it doesn't logout connected users and 401 errors appears This behaviour is very constraining because it generate lot of lines in ambari logs (Thread)How to force ambari to log out all connected users when ambari-server restart ?

Hi experts,As the root user, I am trying to delete a directory in HDFS which was created by root.However, when I try to delete it, it says "Permission denied: user=root, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x"Why does it say permission denied on "/user" when I am trying to delete the directory "/tmp/root/testdirectory" The error message is below. [root@test02 ~]# hdfs dfs -ls /tmp/root/Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=trueJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0Found 2 itemsdrwxrwxrwx - root hdfs 0 2021-08-09 20:35 /tmp/root/testdirectory-rw-r--r-- 3 root hdfs 0 2021-08-10 13:54 /tmp/root/test[root@test02 ~]# hdfs dfs -rmr /tmp/root/testdirectoryPicked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=trueJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0rmr: DEPRECATED: Please use '-rm -r'

Hello All,   A Hive Query just ran for almost 7.5 Hours and I'm trying to understand what has caused such a significantly long runtime.   These are the query stats:     [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED [22;0m[2K----------------------------------------------------------------------------------------------[2KMap 1 .......... container SUCCEEDED 3348 3348 0 0 0 0 Map 4 .......... container SUCCEEDED 27 27 0 0 0 0 Reducer 2 ...... container SUCCEEDED 1009 1009 0 0 0 0 Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0 [2K----------------------------------------------------------------------------------------------[2K[31;1mVERTICES: 04/04 [==========================>>] 100% ELAPSED TIME: 27285.00 s [22;0m[2K--------------------------

I am wondering what difference in IO can be expected for hbase with storage in the cloud VS storage on hdfs.I would expect that when data is retrieved from hdfs, it will be a lot faster than from the cloud (like adls-in my specific case adls gen2=abfs).Is there somewhere where I can test this?Or find a previous study for this?If this is the case, then one would expect that current hbase performance for reading data is a bit less than some years ago when everything was on premise and typically hdfs was used?  Maybe I am missing something obvious here, so any insight is appreciated ! 

Hello Experts, I have a requirement where nifi flows imp log information should be written to MongDB, so that another application can read mongodb and display those interesting logs info on web UI. I am looking for best way to do this.One way I am thinking of is tail nifi log (using tail file processor) and filter for imp log and put into MongoDb using native processor. Please suggest if any other better approaches,thanks in advance. ThanksMahendra

getting error for access to mysql  1. [cloudera@quickstart ~]$ mysql -uroot -pclouderaERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)2. [cloudera@quickstart ~]$ mysql -urootERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)3. [cloudera@quickstart ~]$ mysql -u root -p clouderaEnter password:ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)  in 3 step putting password as cloudera but it not works 

Hello Team, how can we extract XML values from the column below (type string) from Impala (Kudu) Regards, Roshan