Cloudera Community
rvillanueva
user rank
Expert Contributor

Re: Ambari unable to run custom hook for modifying...

by Master Collaborator Master Collaborator in Support Questions
‎10-03-2021 01:14 AM
‎10-03-2021 01:14 AM
I was facing the similar error and got it resolved by added Hadoop users to passwd file.    resource_management.core.exceptions.ExecutionFailed: Execution of 'usermod -G hadoop -g hadoop hive' returned 6. usermod: user 'hive' does not exist in /etc/passwd Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-59009.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-59009.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']   >> File location /etc/passwd >> Adduser hadoop  ... View more

Re: How to run spark-submit in virtualenv for pysp...

by Cloudera Employee in Support Questions
‎12-16-2019 07:29 AM
‎12-16-2019 07:29 AM
@rvillanueva  Please refer article  https://community.cloudera.com/t5/Customer/Unable-to-start-Pyspark-jobs-when-running-with-Python-3/ta-p/272990 ... View more

Re: LDAP/AD users not appearing in Ranger

by Expert Contributor in Support Questions
‎12-11-2019 11:30 AM
1 Kudo
‎12-11-2019 11:30 AM
1 Kudo
From the Ranger email list, this is another bit of information that I found helpful:   ----------   I’ve configured ranger using the following approach to control who must be synced with AD. Only users belonging to groups inside a specific OU will be synced. I’ve created the OU  OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC=domain,DC=com Create a group called R2Users inside that OU. I put all desired sync users as its members. Also, you can put other groups as its member.  And, you can create other groups like R2TEAM as well. Remember to update this property ranger.usersync.ldap.user.searchfilter to include more than one. I’ve configured ranger to sync groups before users.   Here is the configuration. in COMMON CONFIGS Label Property Value LDAP/AD URL ranger.usersync.ldap.url ldap://myacticedirectoryserver.domain.com:389 Bind User ranger.usersync.ldap.binddn CN=LDAP_AD_ACCOUNT,OU=Service Accounts,OU=LCB,OU=Brazil,DC=domain,DC=com Bind User Password ranger.usersync.ldap.ldapbindpassword LDAP_AD_ACCOUNT user’s password Inclemetal Sync ranger.usersync.ldap.deltasync Yes Enable LDAP STARTTLS ranger.usersync.ldap.starttls No GROUP CONFIGS Label Property Value Enable Group Sync ranger.usersync.group.searchenable Yes Group Member Attribute ranger.usersync.group.memberattributename member Group Name Attribute ranger.usersync.group.nameattribute Cn Group Object Class ranger.usersync.group.objectclass Group Group Search Base ranger.usersync.group.searchbase OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC=domain,DC=com Group Search Filter ranger.usersync.group.searchfilter   Enable Group Search First ranger.usersync.group.search.first.enabled Yes Sync Nested Groups is_nested_groupsync_enabled Yes Group Hierarchy Levels ranger.usersync.ldap.grouphierarchylevels 5 USER CONFIGS Label Property Value Username Attribute ranger.usersync.ldap.user.nameatributte sAMAccountName User Object Class ranger.usersync.ldap.objectclass User User Search Base ranger.usersync.ldap.searchbase DC=domain,DC=com User Search Filter ranger.usersync.ldap.user.searchfilter (memberOf=CN=R2Users,OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC=domain,DC=com) User Search Scope ranger.usersync.ldap.user.searchscope Sub User Group Name Attribute ranger.usersync.ldap.groupnameattribute sAMAccountName Group User Map Sync ranger.usersync.group.usermapsyncenabled Yes Enable User Search ranger.usersync.user.searchenabled Yes ADVANCED Ranger Settings Label Property Value Authentication method   ACTIVE_DIRECTORY AD Settings Label Property Value AD Bind Password ranger.ldap.ad.bind.password LDAP_AD_ACCOUNT user’s password Domain Name (Only for AD) anger.ldap.ad.domain DC=domain,DC=com AD Base DN ranger.ldap.ad.base.dn DC=domain,DC=com AD Referreal ranger.ldap.ad.referreal Follow AD User Serach Filter ranger.ldap.ad.user.search (sAMAccountName={0}) Advanced ranger-ugsync-site Label Property Value ranger.usersync.ldap.referral ranger.usersync.ldap.referral Follow ... View more

Re: Difference between installing from Apache vs H...

by Super Guru in Support Questions
‎11-04-2019 10:02 AM
1 Kudo
‎11-04-2019 10:02 AM
1 Kudo
Very good question here.   Let me share some of my thoughts as I have installed ambari both from source and from Hortonworks Repos.  Before I get started you should know that Hortonworks was a major contributor to Ambari Project, as such their documentation is very detailed for how to install Ambari and its components.   In my opinion this is the preferred documentation.      Hortonwork repos are THE public repos for ambari.  Using them is much easier than building from source.   The Ambari Project page at ambari.apache.org is just the project page.  The documentation is specifically for ambari, and not necessarily for "hadoop" and does not include all the screen shots and deeper info you will find in the HortonWorks/Cloudera documentation for the same.   Although the Project Page does not go into much detail, it does have the required artifacts, and enough information to setup nodes and get into the Cluster Install Wizard.   For those organizations which are required to use private repos or to build their own,  the Ambari Project page is very important.         ... View more

Re: HDP Ambari installation throws "org.postgresql...

by New Member in Support Questions
‎09-24-2019 07:50 PM
‎09-24-2019 07:50 PM
OK. tested and successful   ... View more

Re: Where are the spark2 binaries?

by Expert Contributor in Support Questions
‎08-27-2019 03:03 PM
‎08-27-2019 03:03 PM
Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_sample_spark_2_x_applications.html The binaries appear to be in /usr/hdp/current/spark2-client/bin Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client ... View more

Re: Installing HDF management pack, does version n...

by Master Mentor Master Mentor in Support Questions
‎08-26-2019 06:34 PM
‎08-26-2019 06:34 PM
@rvillanueva  HDF and HDP versions can be different in a cluster. They need not to  be exactly same. For example please refer to the https://supportmatrix.hortonworks.com/ Click on "HDP 3.1"  (or click on desired HDF version like HDF 3.4.1.1) and then you will find the compatibility matrix with Ambari + HDF versions. ... View more

Re: sqoop ERROR tool.ImportTool: Import failed: or...

by Expert Contributor in Support Questions
‎07-31-2019 11:28 PM
‎07-31-2019 11:28 PM
Issue was that the --target-dir path included some variables in the start of the path and ended up with the path looking like //some/hdfs/path and the "empty" // was confusing sqoop. ... View more

Re: What are HDFS NFS "access times"?

by Expert Contributor in Support Questions
‎08-14-2019 07:33 PM
‎08-14-2019 07:33 PM
Here is the answer that I was told from discussion on the apache hadoop mailing list: I think access time refers to the POSIX atime attribute for files, the “time of last access” as described here for instance [1]. While HDFS keeps a correct modification time (mtime), which is important, easy and cheap, it only keeps a very low-resolution sense of last access time, which is less important, and expensive to monitor and record, as described here [2] and here [3]. It doesn’t even expose this low-rez atime value in the hadoop fs -stat command; you need to use Java if you want to read it from HDFS apis. However, to have a conforming NFS api, you must present atime, and so the HDFS NFS implementation does. But first you have to configure it on. The documentation says that the default value is 3,600,000 milliseconds (1 hour), but many sites have been advised to turn it off entirely by setting it to zero, to improve HDFS overall performance. See for example here ( [4], section "Don’t let Reads become Writes”). So if your site has turned off atime in HDFS, you will need to turn it back on to fully enable NFS. Alternatively, you can maintain optimum efficiency by mounting NFS with the “noatime” option, as described in the document you reference. I don’t know where the nfs3 daemon log file is, but it is almost certainly on the server node where you’ve configured the NFS service to be served from. Log into it and check under /var/log, eg with find /var/log -name ‘*nfs3*’ -print [1] https://www.unixtutorial.org/atime-ctime-mtime-in-unix-filesystems [2] https://issues.apache.org/jira/browse/HADOOP-1869 [3] https://superuser.com/questions/464290/why-is-cat-not-changing-the-access-time [4] https://community.hortonworks.com/articles/43861/scaling-the-hdfs-namenode-part-4-avoiding-performa.html ... View more
Contact Me
Online
Offline
Last Visited
‎10-31-2020 09:19 PM
Community Statistics
Member Since ‎07-11-2019 08:54 PM
Last Visited ‎10-31-2020 09:19 PM
Posts 102
Kudos received 4