Q&A

If I am trying to access Zeppelin interpreter page , I keep getting the error "You don't have permission on this page" . 

Hello,we observe lot of annoying messages in kafka logs like this. 2021-09-01 17:09:47,435 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: unable to return groups for user OU=Dept,O=Company,C=DE,ST=Germany,CN=some_userPartialGroupNameException The user name 'OU=Dept,O=Company,C=DE,ST=Germany,CN=some_user' is not found. id: OU=Dept,O=Company,C=DE,ST=Germany,CN=some_user: no such userid: OU=Dept,O=Company,C=DE,ST=Germany,CN=some_user: no such userat org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:291)at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:215)at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroupsSet(ShellBasedUnixGroupsMapping.java:123)at org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupSet(Groups.java:413)at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:351)at org.apache.hadoop.security.Groups$

HelloI was wondering how one would best implement the case where multiple users have their own python environment with their respective packages installed in a Spark2 interpreter. I have not implemented or tested the two approaches below yet but was wondering if I am on a completely wrong path and there would be a very easy and straightforward solution.   Two ways I could think of would be the following: 1.) For each notebook set the variable zeppelin.python If one would instantiate the interpreter per user you could run the following at the start.   %spark2.confzeppelin.python </your/users/python_env>   This would add some complexity for the users because they have to write down their environment paths but would otherwise be very easy to implement (If it really works without hiccups).    2.) Create a separate Interpreter for every user This approach would be to create an interpreter for every user (%spark2 -> %spark2userName). This would be relatively

Hola,i need stop/start the roles and CM services many time for patching the servers. Is there a solution by one script?Thank you!!

after start the ssl_kafkathe error log was"ssl key store is specified,but key store password is not specified" but I have already sot ssl.keystore.location and ssl.keystore.password.generatorI don't konw  I can do the next step! I need help,thanks

Hi, I have an HDFS cluster with 3 nodes. The cluster holds lots of small files (KB) and I have reached Millions of blocks per node. I have added 4 more new servers to the cluster and started the balancer process but it looks that it does not do much. - The goal is to reduce the Million of blocks per server  In order to balance the small-size files should i change the value of the following parameter to support moving files from 1KB size?Ddfs.balancer.getBlocks.min-block-size=1048

Hello Team, can someone please help me, i am facing out of memory in spark jobs also find below parameters and also find below configurations. 21/10/11 17:22:53 INFO executor.Executor: Finished task 194.0 in stage 79.0 (TID 14855). 11767 bytes result sent to driver21/10/11 17:23:34 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM21/10/11 17:23:34 ERROR executor.Executor: Exception in task 167.0 in stage 79.0 (TID 14825)java.lang.OutOfMemoryError: Java heap spac 21/10/11 17:23:34 INFO storage.DiskBlockManager: Shutdown hook called21/10/11 17:23:34 INFO util.ShutdownHookManager: Shutdown hook called21/10/11 17:23:34 INFO executor.Executor: Not reporting error to driver during JVM shutdown.21/10/11 17:23:34 ERROR util.SparkUncaughtExceptionHandler: [Container in shutdown] Uncaught exception in thread Thread[Executor task lajava.lang.OutOfMemoryError: Java heap spaceat org.apache.spark.sql.catalyst.expressions.UnsafeRow.copy(UnsafeRow.java:502)at org.apache.

Dear team, We are facing the below issue on one of the Solr nodes.  2021-10-17 04:05:57.006 ERROR (qtp1916575798-2477) [c:ranger_audits s:shard1 r:core_node2 x:ranger_audits_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are disabled. at org.apache.solr.update.processor.DistributedZkUpdateProcessor.zkCheck(DistributedZkUpdateProcessor.java:1245) at org.apache.solr.update.processor.DistributedZkUpdateProcessor.setupRequest(DistributedZkUpdateProcessor.java:582) at org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:239) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) at org.apache.solr.update.processor.AddSchemaFieldsUpdateProc

Dears, Greetings!I have the requirement to get the all HDFS filename which are older than N days.I got the all last level directory which are older But requirement is all filenames.Kindly support with hdfs command or script or code for the same.  also suggest on if we query any Hive table and done some sum on one column or join with other table Is this will change the timestamp of underlining hdfs file of respective Hive table. obviously write or update to table will change the timestamp of respective hdfs file. Appreciate swift response

Hi, I have installed Hortonworks Sandbox HDP3.0 I leave all services to start there is an issue with starting Timeline Service V2.0 Reader   2021-09-29 05:47:58,587 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=5385 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server sandbox-hdp.hortonworks.com,17020,1632894462010 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1487) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecu

Hi Team, I have attended the cca175 exam on 4th Oct 2021 and got my score card on 5th Oct 2021 saying I have cleared the exam.But, Its been more than 2 weeks I haven't received my certificate yet.Please help me to get my certificate. Thanks,Shankaranand  

I am a newbie to NiFi and I am trying to pass a JSON object to NiFi. Having failed at multiple attempts, I am trying a very simple flow as below. I have checked the JSON in online validators and have confirmed that it's a valid JSON file. Yet, I am not getting the output I want i.e. Country and Capital in flow file attributes. Please advise where I am going wrong. Thanks in advance. 

Hi, I would like to know which platform I should download to try Cloudera on my desktop (16GB/1TB SSD).  I would prefer a docker solution which would allow me to test the docker on multiple clouds.  

I installed a 2 node hadoop cluster. The master and slave node starts separately but the datanode isn't shown in namenode webUI. The log file for datanode shows the following error : 2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting 2016-02-09 23:30:54,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2016-02-09 23:30:55,977 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)2016-02-09 23:21:15,062 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/172.17.25.5:9000 The hosts

I have a requirement of loading Data from MySQL to Salesforce (using Salesforce BLUK API) using Nifi. For that I am first calling a Salesforce SOAP API to get session_id(accessToken), then I am passing this session_id to another REST API to create a BULK Load job on Salesforce and I am extracting the job_id and adding both the session_id and job_id into the attribute of the flow file.Till now everything was fine, but now I need to extract the data from MySQL Table using QueryDatabaseTable processor which doesn't accepts any upstream link, but I need to use the session_id and job_id to pass it along with the MySQL Data to Salesforce BULK API REST call using InvokeHTTP, so now I have two disconnected flows, one has the job_id and session_id to be passed to Salesforce REST API and the other flow has the Data from MySQL DB, but I am unable to use both of these in the InvokeHTTP. Its a very tricky situation. Is there any way, that I can store the session_id and job_id from the 1st flow

I am trying to use a ListSFTP processor to delete files from an SFTP site older than the most recent  "History" file. In order to do this, I need to continually (process would run daily or weekly) query all the files in the SFTP server to see if a new "History" file was added, and if so to remove all files from the site before this newest "History" file. The problem is that ListSFTP only shows the newest files added so I am unable to do this. I can ClearState on the processor to make it work, but I would like to avoid this being a manual process. So how can I have ListSFTP show all the files every single time I run it? Note: I've seen some posts about automatically clearing the state using a curl request, but this only works on a stopped processor which defeats the purpose of this being an automated process (https://community.cloudera.com/t5/Support-Questions/Nifi-clear-state/td-p/242347)

Hi have multiple process groups for Data Extractions (one processgroup for Oracle, one for MySQL) which are converted to csv and later are fed to a big data processing engine as source files.  I want to implement a condition trigger by reading a file (which tells me Oracle or MySQL), based on the content if this file, I need to trigger the corresponding data extraction ProcessGroup. Note : I dont want to do any action manually on the UI, this has to be an automated solution, which will work based on the file data which says, which ProcessGroup should be triggered (Oracle or MySQL). Both should not run at the same time. How can I implement this ? Any help will be highly appreciated.

Hi Everyone, Wanted to check your opinion on one backup and DR situation. So for one of our Dev Platform, we are still on CDH 5.4 version (with EOD already passed not under technical support scope), all the servers for this platform are going to be physically migrated from one Data Center to another Data Center. These 34 servers are placed in three separate racks, so considering in-built replication and Rack awareness, so technically I believe as long as we loose few servers and even one single Rack, we should still not be facing any data loss. Now considering the worst case scenario that multiple Racks or all servers are damaged during migration, I need to have a backup and DR solution in place. To consider few limitations, we don’t have architectural capacity to spin up separate cluster to copy existing hdfs data there for recovery purpose. Currently we are thinking to backup all servers using Avamar backup tool, all OS data and data disks we can backup. I am going to take

I would like to undo the delete operation performed using Impala on the underlying Kudu table. Is it something supported in Impala/Kudu? Could someone help me with it? 

Hi all, I recently installed Ranger on Cloudera Private Cloud Base 7.1.7. Everything is installed perfectly but the ranger admin UI is not loading. The cluster is kerberized. I am able to curl the admin URL with the --negotiate flag. Is the UI not loading because of spnego authentication? If yes, how do I disable it?We have option to disable that for HDFS and YARN, but not sure about Ranger.Thank you!

Hi,I found this new CVE for Hadoop (CVE-2020-9492).The solution is to upgrade to Hadoop 2.10.1 but HDP 2.6.5 (with Apache Hadoop version 2.6.7) is the latest version of Apache Hadoop 2.Could be possible to fix any CVE into HDP 2.6.5.Thanks in advance.

Running some queries e we get this error:   Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 102.0 failed 1 times, most recent failure: Lost task 11.0 in stage 102.0 (TID 4722, localhost, executor driver): java.io.FileNotFoundException: /tmp/blockmgr-95c88a5a-8d68-4b16-878d-158f40123999/1c/temp_shuffle_f82e49e7-a383-4803-82b3-030425703624 (Too many open files)   We modified limits.conf adding these value :   zeppelin soft nofile 32000zeppelin hard nofile 32000   anyway looking at value of zeppelin process about open file limits the value is 4096   zeppelin@ ~]$ cat /proc/24222/limitsLimit Soft Limit Hard Limit Units Max processes 510371 510371 processesMax open files 4096 4096 filesMax locked memory 65536 65536 bytes [zeppelin@~]$   Is there some configuration file to set somewhere?   Thanks in advance  

Our customer looking to get logs of all jobs ran in a particular cluster .How to gather such information ?can someone help on this.

Dears,   I have a requirement to enable all backups and restoration for all services of CDH and CM and Management services and their meta data db's  and configuration of CDH clod & hot clusters. Kindly support with needful details.

Hi experts,I just wanted to confirm my understanding or help me better understand the yarn local and log dirs.So my understanding is that yarn will download the data locally to a filesystem so that it is more easily accessible when a job is run as well as logs for that particular application. I believe these are temporary files as they will be stored in HDFS after the job completes. [root@test01 log]# ll /hadoop/yarn/total 0drwxr-xr-x. 6 yarn hadoop 78 Oct 13 08:09 localdrwxrwxr-x. 8 yarn hadoop 239 Oct 13 08:09 logCan someone please help confirm my understanding or help me better understand this concept?Also what is usually best practice in regards to mounting these directories onto a local filesystem or onto another hard drive or can I have this directory share a hard drive with one of the datanode directories? Any help is much appreciated. Thanks,