Q&A

I am using HDP. The inode in the following code is a managed hive table. # id zeppelinuid=1017(zeppelin) gid=1003(hadoop) groups=1003(hadoop),1005(zeppelin)# sudo -u zeppelin hadoop fs -ls /warehouse/tablespace/managed/hive/test1ls: Permission denied: user=zeppelin, access=READ_EXECUTE, inode="/warehouse/tablespace/managed/hive/test1":hive:hadoop:drwxrwx--- The user zeppelin is in hadoop group, which has full permisison on the hdfs folder. So why do I get the permission error?

Hi, I am starting a new career in AI as a solution architect, and I need to know what exactly the knowledge needed for such a role, what should I know, which product should I start work on and use and what is the best way to start with Cloudera product.     Kindly I need advice on this matter ASAP. Thanks in advance.

Is cloudera express still an open source/free? please if anyone knows about this help.

HelloI receive an error while enabling kerberos on ambari as below;i have installed krb5-kdc krb5-admin-server and config krb5.conf, kdc.conf and kadm5.acl then created new principle (as attached)Note when i wrote the realm name in the kdc file in uppercase letter i got an error while using kadmin.local master key cannot be fetch, it only works in lowercase letterAlso when i try to restart the krb5 services, it said service can't be found although it is running so i restart the server insteadLast thing when i installed krb5-kdc krb5-admin-server the /var/kerberos folder didn't create automatically and i had to create it manually.Please help me solve this issue, thank you in advanced.KADM5.acl.txtKDC.conf.txtKRB5.conf.txt

Nifi - HandleHttpRequest processor throwing an exception saying "Failed to Initialize the server". Functionality wise it is working fine but I always see the below error message.  Will be able to post the request using curl command to this processor and it is accepting the request without any issue. Please help me to get rid of this error? Thanks. Error message:13:42:37 EST- hostname:port - ERRORHandleHttpRequest[id=123] failed to process session due to failed to initialize the server: org.apache.nifi.processor.exception.ProcessException: Failed to Initialize the server. Configuration:

  since metrics take space on the local disk ,   we want to move to distributed mode , so metrics data will be on HDFS and not on local disk   we start to work according to the link - https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode   we are using ambari cluster 2.6.4 with active/stand by name node     so we start the configuration    first we set in ambari the following     ams-site, Set timeline.metrics.service.operation.mode = distributed   now  we need to Edit ams-hbase-site,   hdfs://<nameservice-id>/user/ams/hbase    the question is - how to find the nameservice-id ?   since we have active/stand by name node   then how to know the - nameservice-id value?            

Hello,    we have a a CDH5.16.2 Cluster with Hue connected to ActiveDirectory.  We can login and access Hive as an example.    But the automatic creation of a user is not working.  The error Message is : [09/Jan/2020 17:57:10 +0100] views ERROR Could not create home directory at login for *****.Traceback (most recent call last):File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/desktop/core/src/desktop/auth/views.py", line 129, in dt_loginensure_home_directory(request.fs, user)File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/apps/useradmin/src/useradmin/views.py", line 811, in ensure_home_directoryfs.do_as_user(username, fs.create_home_dir, home_directory)File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 102, in do_as_userreturn fn(*args, **kwargs)File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 1

Hi All, I have a fresh Ambari eco system installed in my VMware Vsphere, assume its hostname is ambari01.example.com and IP Address is 192.168.1.10, I cloned it to ambari02 and changed IP 192.168.1.11. but unable to start services in ambari02.example, I think all eco system configuration files need to be modified, but not sure what all the files need to be modified. Might be some guys already tried this, can someone assist me. Thanks Niruu

Hi, I'm trying to install Ambari 2.6 on CentOS7 x64. I have HDP 2.6 already installed. These versions are compatible to each other per Horton Support Matrix.   When trying to create a cluster through wizard, the host registration is failing, below is the log error and I've also tried all common troubleshooting I could find online = stop, reset, start server = change hostname = disable firewall = change python code (https://community.cloudera.com/t5/Support-Questions/Ambari-server-host-register-failed/td-p/233413)   None of it seems to help. Looking forward to any help.   06 Jan 2020 22:53:19,011 INFO [Thread-23] AbstractPoolBackedDataSource:212 - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connec

Is it possible to add a second user on hadoop cluster like the spark user?

Hi, I am receiving the flowfile from ConsumerKafka in json form.     Then I am using EvaluateJsonPath to extract the json data once the data matched, i want call RouteOnAttribute to call script, but the expression which i have used  in EvaluateJsonPath   processor looks to be wrong and throwing out unmatched info. Plz help me out to add right expression to extract  the fields  

Hi,   I am adding ranger in Cloudera runtime(CDP) but i am getting below error while adding ranger service 'db_password' not found in /var/run/cloudera-scm-agent/process/1546333400-ranger-RANGER_ADMIN-SetupRangerCommand/install.properties I have created ranger db, ranger username, mysql password password but still i am getting this error

Using CentOS7 Cloudera 5.16 version installations... got below error   Failed to install Cloudera-manager-agent package   Attaching full logs from the CM.

I have recently enabled NameNode HA, ResourceManager HA, and Hive HA on an HDP 2.6 cluster by following the instructions provided here:  https://docs.cloudera.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-operations/content/ch_managing_service_high_availability.html.   Everything continues to work fine if I stop all HDP components on host where the primary NameNode and ResourceManager are running by logging into Ambari, selecting that host, and then selecting the Host Action "Stop All Components".  But as soon as I power down that host, beeline requests become intolerably slow.  For example, a beeline request for "show tables;" takes about 20 minutes to complete.  The request will hang at this point for an extended period of time: Setting property: [silent, false]issuing: !connect jdbc:hive2://nn2:10000/intrepid hdfs [passwd stripped]Connecting to jdbc:hive2://nn2:10000/intrepid (nn2 is the host name where the standby NameNode is running.)   After a long

I just download HDP 3.0.1.  I can log into Ambari as maria_dev, but I can't use putty (or Windows 10 ssh) to log in from a terminal (using Virtual Box, here)   I know the root password is hadoop, but when I try to connect, I get the error message "remote side unexpectedly closed network connection."  I tried: ssh maria_dev@localhost -p 4200   I also tried ssh root@localhost -p 4200 -pwd hadoop   ...same error.  I'm never even given the chance to log in as either user, maria_dev nor root, so I can't even enter the password hadoop to change the root password.  Didn't have this problem in HDP 2.5

Prv account part of AD group is not getting sync'd to Ambari and default access is not working. 2020-01-07T15:18:15.345Z, User(null), RemoteIp(10.176.18.143), Operation(User login), Roles(), Status(Failed), Reason(Authentication required)2020-01-07T15:18:15.346Z, User(null), RemoteIp(10.176.18.143), Operation(User login), Roles(), Status(Failed), Reason(Invalid JWT token) Normal lanid works fine and both are part of same AD group which has admin privileges.   the login is direct and not via knox , another weird thing noticed is prv works fine for knox login

After enabling Ranger Plugin for Hive, I was running a "dfs -ls" in hive/ beeline to list hdfs files. I'm getting the following error :jdbc:hive2://vserver69901.example.com> dfs -ls ;Error: Error while processing statement: Permission denied: user [ambari] does not have privilege for [DFS] command (state=,code=1)Do we need to update/enable  any other properties. hadoop fs -ls works without any issuesThe user has admin access in Ranger. The storage is on Isilon , so the HDFS Ranger plugin cannot be enabled. 2015-11-12 17:07:31,980 INFO  [HiveServer2-Handler-Pool: Thread-46]: operation.Operation (HiveCommandOperation.java:setupSessionIO(69)) - Putting temp output to file /tmp/hive/012f5aa7-fa31-4fb2-8cd5-4f3fe3f3120624919258595801789.pipeout2015-11-12 17:07:31,981 ERROR [HiveServer2-Handler-Pool: Thread-46]: processors.CommandUtil (CommandUtil.java:authorizeCommand(66)) - Error authorizing command [-ls]org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException:

Hi,   I am running Spark jobs on YARN, using HDP 3.1.1.0-78 version.    I have set the Spark Scheduler Mode to FAIR by setting the parameter "spark.scheduler.mode" to FAIR.  The fairscheduler.xml is as follows:   I have also configured my program to use "production" pool.      Upon running the job, it has been observed that although 4 stages are running, only 1 stage run under "production" and rest 3 run under "default" pool.    So, at any point of time, I am able to make sure that only 2 tasks are running in parallel.  If I want to make sure that 3 tasks or more run in parallel, then 2 tasks should run under "production" and rest 2 should run under "default".    Is there any programmatic way to achieve that, by setting configuration parameters?    Any inputs will be really helpful.    Thanks and Regards, Sudhindra 

Hi Team,I am trying to use FairScheduler in HDP 3.0.1. As per https://stackoverflow.com/questions/38208845/switching-from-capacity-scheduler-to-fair-scheduler-on-hortonworks-data-platform it seems to be possible in HDP 3.0. I have changed the schedule claa and set yarn.scheduler.fair.allocation.file in custom yarn-site in ambari (screenshot attached). But when I restart ResourceManager it is failing with below error.2020-01-02 16:21:54,134 INFO fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/etc/hadoop/conf/fair-scheduler.xml2020-01-02 16:21:54,221 INFO monitor.SchedulingMonitor (SchedulingMonitor.java:serviceInit(60)) - Initializing SchedulingMonitor=SchedulingMonitor (ProportionalCapacityPreemptionPolicy)2020-01-02 16:21:54,221 INFO capacity.ProportionalCapacityPreemptionPolicy (ProportionalCapacityPreemptionPolicy.java:init(160)) - Preemption monitor:org.apache.hadoop.yarn.server.resourcemanage

Hello I encountered the error below on nifi after an upgrade, please help to solve it   PutSQL[id=11f8203a-164b-1df4-47b3-137747566249] Failed to process session due to java.sql.SQLException: Cannot create PoolableConnectionFactory (ORA-01034: ORACLE not availableORA-27101: shared memory realm does not existLinux-x86_64 Error: 2: No such file or directory): org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create PoolableConnectionFactory (ORA-01034: ORACLE not availableORA-27101: shared memory realm does not existLinux-x86_64 Error: 2: No such file or directory)  

I built Spark2 with CDH 5.16 and able to submit scala jobs with no issues. Now I am able to launch pyspark2 and when I am trying to run simple job, its throwing the below error? can you please suggest on this? what is the alternate to submit python jobs on spark jobs apart from Jupyter notebook? Pls advise   [I 23:08:33.864 NotebookApp] Adapting to protocol v5.1 for kernel f8d7200b-6718-49f6-86e9-c051fb6d84a6 [Stage 0:>                                                          (0 + 0) / 2]Exception in thread "dispatcher-event-loop-0" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)

I have installed the HDP sandbox HDP-3.0.1.0 (3.0.1.0-187) using Docker and Kerberized the cluster.   I also spinned up an LDAP server and setup User- and Group-sync with Ranger.   Finally I have also set HDFS advanced core-site configuration.   I can run an ldapsearch succesfully.             However hdfs groups doesn't seem to work.   I really have no idea on where to look from here. Any help would be greatly appreciated.    

Hello Community,I want to decommission an entire cluster, so I went looking around for any documentation that mentions any recommended procedure to follow but couldn't find any. Hence, i turned towards the community, if anyone here have decommissioned a kerberized and sentry-enabled cluster and can advice me on any best practices I can follow.I think it is worth mentioning that our cluster is created using Cloudera Altus Director. Thank you,Snehasish

So I have had this issue for a quite long time so far, and it's been really challenging.  So, on one of the servers (secondary namenode) of a 12 node development cluster, the heartbeat is being sent and received. The IDs of each heartbeat request are incremental, which I believe means it's working fine from ambari-agent aspect. I have SNameNode, HiveServer and other services on that server, and whenever I try to start any of them I get this directly from Ambari UI Caught an exception while executing custom service command: <type 'exceptions.KeyError'>: 'Host_Level_Params for cluster_id=2 is missing. Check if server sent it.'; 'Host_Level_Params for cluster_id=2 is missing. Check if server sent it.' I'm not able to start no slave or master services on that server AT ALL.  I don't see anything in the logs to be honest that could point me to a direction.  Few information of the environment:1- Redhat 7.7 servers2- Latest JDK 8 2313- HDP 3.

As the title explains, we have moved our cluster from `taco.net` to `dev.taco.net`, and in the process this has made our Cloudera Manager server unable to start up because it continues to try to connect to database.taco.net instead of database.dev.taco.net. Hosts files are updated on all servers, network interfaces are configured to their new IPs.I would imagine there is something I should update on the Cloudera Manager server, but I'm not sure where to look to update the URL.