Member since
07-11-2019
102
Posts
4
Kudos Received
9
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
12279 | 12-13-2019 12:03 PM | |
2325 | 12-09-2019 02:42 PM | |
1822 | 11-26-2019 01:21 PM | |
881 | 08-27-2019 03:03 PM | |
1421 | 08-14-2019 07:33 PM |
05-26-2020
02:06 PM
Having problem where HDFS (HDP v3.1.0) is running out of storage space (which is also causing problems with spark jobs hanging in ACCEPTED mode). I assume that there is some configuration where I can have HDFS use more of the storage space already present on the node hosts, but exactly what was not clear from quick googling. Can anyone with more experience help with this? In ambari UI, I see... (from ambari UI) (from NameNode UI). Yet when looking at the overall hosts via ambari UI, there appears to be still a good amount of space left on the cluster hosts (each node excluding the first in this list has a total of 140GB) Not sure what setting are relevant, but here are the general setting in ambari: My interpretation of the "Reserved Space for HDFS" setting is that it shows there should be 13GB reserved for non-DFS (ie. local FS) storage, so does not seem to make sense that HDFS is already running out of space. Am I interpreting this wrongly? Any other HDFS configs that should be shown in this question? I assume that there is some configuration where I can have HDFS use more of the storage space already present on the node hosts, but exactly what was not clear from quick googling. Can anyone with more experience help with this? Also if anyone could LMK if this may be due to other problems I am not seeing?
... View more
Labels:
- Labels:
-
HDFS
-
Hortonworks Data Platform (HDP)
05-13-2020
02:02 AM
Running a HortonWork hadoop cluster (HDP-3.1.0.0) and getting a bunch of Failed on local exception: java.io.IOException: Too many open files errors when running spark jobs that up until this point have worked fine. I have seen many other questions like this where the answer is to increase the ulimit settings for open files and processes (this is also in the HDP docs) (and I'll note that I believe that mine are still at the system default settings), but... My question is: Why is this only happening now when previously the spark jobs have been running fine for months? The spark jobs I have been running have been running fine for months without incident and I have made no recent code changes. Don't know enough about the internals of spark to theorize about why things could be going wrong only now (would be odd to me if open files just build up in the course of running spark, but that seems like what is happening). Just as an example, just this code... .
.
. sparkSession = SparkSession . builder . appName ( "GET_TABLE_COUNT" ). getOrCreate () sparkSession . _jsc . sc (). getExecutorMemoryStatus (). keySet (). size ()
.
.
. now generates errors like... .
.
.
[ 2020 - 05 - 12 19 : 04 : 45 , 810 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 45 INFO Client : Application report for application_1579648183118_19918 ( state : ACCEPTED )
[ 2020 - 05 - 12 19 : 04 : 46 , 813 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 46 INFO Client : Application report for application_1579648183118_19918 ( state : ACCEPTED )
[ 2020 - 05 - 12 19 : 04 : 47 , 816 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 47 INFO Client : Application report for application_1579648183118_19918 ( state : ACCEPTED )
[ 2020 - 05 - 12 19 : 04 : 48 , 818 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 48 INFO Client : Application report for application_1579648183118_19918 ( state : ACCEPTED )
[ 2020 - 05 - 12 19 : 04 : 49 , 820 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 49 INFO Client : Application report for application_1579648183118_19918 ( state : ACCEPTED )
[ 2020 - 05 - 12 19 : 04 : 50 , 822 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 50 INFO Client : Application report for application_1579648183118_19918 ( state : ACCEPTED )
[ 2020 - 05 - 12 19 : 04 : 51 , 828 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 51 INFO Client : Application report for application_1579648183118_19918 ( state : FAILED )
[ 2020 - 05 - 12 19 : 04 : 51 , 829 ] { bash_operator . py : 128 } INFO - 20 / 05 / 12 19 : 04 : 51 INFO Client :
[ 2020 - 05 - 12 19 : 04 : 51 , 829 ] { bash_operator . py : 128 } INFO - client token : N / A [ 2020 - 05 - 12 19 : 04 : 51 , 829 ] { bash_operator . py : 128 } INFO - diagnostics : Application application_1579648183118_19918 failed 2 times due to Error launching appattempt_1579648183118_19918_000002 . Got exception : java . io . IOException : DestHost : destPort hw005 . co . local : 45454 , LocalHost : localPort hw001 . co . local / 172.18 . 4.46 : 0. Failed on local exception : java . io . IOException : Too many open files [ 2020 - 05 - 12 19 : 04 : 51 , 829 ] { bash_operator . py : 128 } INFO - at sun . reflect . GeneratedConstructorAccessor808 . newInstance ( Unknown Source )
[ 2020 - 05 - 12 19 : 04 : 51 , 829 ] { bash_operator . py : 128 } INFO - at sun . reflect . DelegatingConstructorAccessorImpl . newInstance ( DelegatingConstructorAccessorImpl . java : 45 ) My RAM and ulimit setting on the cluster look like... [ root@HW001 ]# clush - ab free - h --------------- HW001 --------------- total used free shared buff / cache available Mem : 31G 9.0G 1.1G 1.7G 21G 19G
Swap : 8.5G 44K 8.5G
--------------- HW002 --------------- total used free shared buff / cache available Mem : 31G 7.3G 5.6G 568M 18G 22G
Swap : 8.5G 308K 8.5G
--------------- HW003 --------------- total used free shared buff / cache available Mem : 31G 6.1G 4.0G 120M 21G 24G
Swap : 8.5G 200K 8.5G
--------------- HW004 --------------- total used free shared buff / cache available Mem : 31G 2.9G 2.8G 120M 25G 27G
Swap : 8.5G 28K 8.5G
--------------- HW005 --------------- total used free shared buff / cache available Mem : 31G 2.9G 4.6G 120M 23G 27G
Swap : 8.5G 20K 8.5G
--------------- airflowetl --------------- total used free shared buff / cache available Mem : 46G 5.3G 13G 2.4G 28G 38G
Swap : 8.5G 124K 8.5G
[ root@HW001 ]#
[ root@HW001 ]#
[ root@HW001 ]#
[ root@HW001 ]# clush - ab ulimit - a --------------- HW [ 001 - 005 ] ( 5 ) --------------- core file size ( blocks , - c ) 0 data seg size ( kbytes , - d ) unlimited
scheduling priority (- e ) 0 file size ( blocks , - f ) unlimited
pending signals (- i ) 127886 max locked memory ( kbytes , - l ) 64 max memory size ( kbytes , - m ) unlimited
open files (- n ) 1024 pipe size ( 512 bytes , - p ) 8 POSIX message queues ( bytes , - q ) 819200 real - time priority (- r ) 0 stack size ( kbytes , - s ) 8192 cpu time ( seconds , - t ) unlimited
max user processes (- u ) 127886 virtual memory ( kbytes , - v ) unlimited
file locks (- x ) unlimited --------------- airflowetl --------------- core file size ( blocks , - c ) 0 data seg size ( kbytes , - d ) unlimited
scheduling priority (- e ) 0 file size ( blocks , - f ) unlimited
pending signals (- i ) 192394 max locked memory ( kbytes , - l ) 64 max memory size ( kbytes , - m ) unlimited
open files (- n ) 1024 pipe size ( 512 bytes , - p ) 8 POSIX message queues ( bytes , - q ) 819200 real - time priority (- r ) 0 stack size ( kbytes , - s ) 8192 cpu time ( seconds , - t ) unlimited
max user processes (- u ) 192394 virtual memory ( kbytes , - v ) unlimited
file locks (- x ) unlimited Don't know much about Hadoop admin, but just looking at the Ambari dashboard, the cluster does not seem to be overly taxed... (though could not actually check the RM web UI, since it just throws a "too many open files" error). Anyone with more spark/hadoop experience know why this would be happening now?
... View more
Labels:
01-31-2020
04:26 PM
What is the difference between NameNode heap usage and ResourceManager heap usage? I am trying to find heavy NameNode heap usage cause. In the ambari dashboard, I see... when running some sqoop jobs. Not sure what is causing the NN usage to be so high here (not a lot of experience with hadoop admin stuff)? Furthermore the sqoop jobs appear to be frozen after 100% completion of the mapreduce task for abnormal amount of time than usual, eg. seeing... [ 2020 - 01 - 31 14 : 00 : 55 , 193 ] INFO mapreduce . JobSubmitter : number of splits : 12
[ 2020 - 01 - 31 14 : 00 : 55 , 402 ] INFO mapreduce . JobSubmitter : Submitting tokens for job : job_1579648183118_1085 [ 2020 - 01 - 31 14 : 00 : 55 , 402 ] INFO mapreduce . JobSubmitter : Executing with tokens : []
[ 2020 - 01 - 31 14 : 00 : 55 , 687 ] INFO conf . Configuration : found resource resource - types . xml at file :/ etc / hadoop / 3.1 . 0.0 - 78 / 0 / resource - types . xml [ 2020 - 01 - 31 14 : 00 : 55 , 784 ] INFO impl . YarnClientImpl : Submitted application application_1579648183118_1085 [ 2020 - 01 - 31 14 : 00 : 55 , 837 ] mapreduce . Job : The url to track the job : http :// hw001 . ucera . local : 8088 / proxy / application_1579648183118_1085 /
[ 2020 - 01 - 31 14 : 00 : 55 , 837 ] mapreduce . Job : Running job : job_1579648183118_1085 [ 2020 - 01 - 31 14 : 01 : 02 , 964 ] mapreduce . Job : Job job_1579648183118_1085 running in uber mode : false [ 2020 - 01 - 31 14 : 01 : 02 , 965 ] mapreduce . Job : map 0 % reduce 0 %
[ 2020 - 01 - 31 14 : 01 : 18 , 178 ] mapreduce . Job : map 8 % reduce 0 %
[ 2020 - 01 - 31 14 : 02 : 21 , 552 ] mapreduce . Job : map 17 % reduce 0 %
[ 2020 - 01 - 31 14 : 04 : 55 , 239 ] mapreduce . Job : map 25 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 36 , 417 ] mapreduce . Job : map 33 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 37 , 424 ] mapreduce . Job : map 42 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 40 , 440 ] mapreduce . Job : map 50 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 41 , 444 ] mapreduce . Job : map 58 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 44 , 455 ] mapreduce . Job : map 67 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 52 , 484 ] mapreduce . Job : map 75 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 56 , 499 ] mapreduce . Job : map 83 % reduce 0 %
[ 2020 - 01 - 31 14 : 05 : 59 , 528 ] mapreduce . Job : map 92 % reduce 0 %
[ 2020 - 01 - 31 14 : 06 : 00 , 534 ] INFO mapreduce . Job : map 100 % reduce 0 %
<... after some time longer than usual ...>
[ 2020 - 01 - 31 14 : 10 : 05 , 446 ] INFO mapreduce . Job : Job job_1579648183118_1085 completed successfully My hadoop version [airflow@airflowetl root]$ hadoop version
Hadoop 3.1.1.3.1.0.0-78
Source code repository git@github.com:hortonworks/hadoop.git -r e4f82af51faec922b4804d0232a637422ec29e64
Compiled by jenkins on 2018-12-06T12:26Z
Compiled with protoc 2.5.0
From source with checksum eab9fa2a6aa38c6362c66d8df75774
This command was run using /usr/hdp/3.1.0.0-78/hadoop/hadoop-common-3.1.1.3.1.0.0-78.jar Anyone with more hadoop experience know what could be going on here? Any debugging advice?
... View more
Labels:
12-24-2019
11:56 AM
[This question was moved here because it was posted 12-24-2019 to a thread previously marked 'Solved' 10-22-2016 03:03 AM —Moderator]
Could you help explain this hadoop.proxyuser.<user>.hosts config a bit more or link to docs explaining exactly what it does / means (ie. I hope it does not imply that root / super-user privileges are conferred or potentially accessible to other users accessing HDFS in any way)? Found the docs here (https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Superusers.html) and not much experience with HDFS admin, but
In other words super is impersonating the user joe
gives me some concern about security. Could anyone help allay these concerns?
... View more
12-24-2019
11:54 AM
@jsensharma Could you help explain the hadoop.proxyuser.<USER>.hosts config a bit more or link to docs explaining exactly what it does / means (ie. I hope it does not imply that root / super-user privileges are conferred to other users accessing HDFS in any way)? Found the docs here (https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Superusers.html) and not much experience with HDFS admin, but In other words super is impersonating the user joe gives me some concern about security. Could anyone help allay these concerns?
... View more
12-24-2019
11:13 AM
[This question was moved here because it was posted 12-24-2019 to multiple threads previously marked 'Solved' as early as 10-22-2016 03:03 AM—Moderator]
Could you explain a bit (or link to docs) on what exactly the hadoop.proxyuser.$.hosts and hadoop.proxyuser.$.groups are (what they mean and why setting * helps here)?
... View more
12-24-2019
11:08 AM
Looking at more existing post here (https://community.cloudera.com/t5/Support-Questions/Unauthorized-connection-for-super-user-root-from-IP-192-168/m-p/193467#M155531) and here (https://community.cloudera.com/t5/Support-Questions/File-View-Error-Unauthorized-connection-for-super-user-root/m-p/145021#M107593) found that also needed to change hadoop.proxyuser.root.hosts=* Setting this in HDFS configs in ambari and restarting the service seems to have fixed the problem. Could anyone help explain this config a bit more or link to docs explaining exactly what it does / means (ie. I hope it does not imply that root / super-user privileges are conferred to other users accessing HDFS in any way)? Found the docs here (https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Superusers.html). Not much experience with HDFS admin, but In other words super is impersonating the user joe gives me some concern about security. Could anyone help allay these concerns?
... View more
12-24-2019
11:06 AM
Having problems uploading or opening files in Ambari (v2.7.3) FileView (HDP v3.1.0). Getting error Unauthorized connection for super-user: root at <my local IP> I configs that I think are relevant here are: hadoop.proxyuser.hdfs.groups=*
hadoop.proxyuser.hdfs.hosts=*
hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=hw001.co.local Have seen similar questions here (https://community.cloudera.com/t5/Support-Questions/Unauthorized-connection-for-super-user/m-p/102118#M65078) but not sure the problems are totally analogous. Anyone know how to fix or get more debugging info?
... View more
Labels:
12-23-2019
04:19 PM
[The following question was moved here after originally being posted 12-23-2019 to this thread which was marked 'Solved' 06-01-2017 12:49 AM —Moderator]
@jsensharma Could you explain a little more about what exactly this
hadoop.proxyuser.root.groups
config is? Any docs describing it more?
... View more
Labels:
- Labels:
-
Apache Hadoop
12-13-2019
12:03 PM
@jsensharma 1. Need to use python3 and would like to continue to do so in the future considering that python2 will stop being maintained in 2020 (I would think others would have a similar desire as well) and am currently adding the option export PYSPARK_PYTHON=/path/to/my/virtualenv/bin/python; spark-submit sparksubmit.test.py as a workaround (else, this may be helpful: https://stackoverflow.com/a/51508990/8236733 or using the --pyfiles option). 2. IDK where that path reference is coming from since "../venv/bin/activate" is just activating a virtualenv and "sparksubmit.test.py" code is just from os import environ
import time
import pprint
import platform
pp = pprint.PrettyPrinter(indent=4)
sparkSession = SparkSession.builder.appName("TEST").getOrCreate()
sparkSession._jsc.sc().setLogLevel("WARN")
print(platform.python_version())
def testfunc(num: int) -> str:
return "type annotations look ok"
print(testfunc(1))
print("\n\nYou are using %d nodes in this session\n\n" % sparkSession._jsc.sc().getExecutorMemoryStatus().keySet().size())
pp.pprint(sparkSession.sparkContext._conf.getAll()) but that blank space in "/usr/hdp//hadoop/lib" is interesting to see, especially since I use export HADOOP_CONF_DIR=/etc/hadoop/conf for the HADOOP_CONF_DIR in the terminal when trying to run the command. Furthermore, looking at my (client node) FS, I don't even see that path... [airflow@airflowetl tests]$ ls -lha /usr/hdp/current/hadoop-
hadoop-client/ hadoop-httpfs
hadoop-hdfs-client/ hadoop-mapreduce-client/
hadoop-hdfs-datanode/ hadoop-mapreduce-historyserver/
hadoop-hdfs-journalnode/ hadoop-yarn-client/
hadoop-hdfs-namenode/ hadoop-yarn-nodemanager/
hadoop-hdfs-nfs3/ hadoop-yarn-registrydns/
hadoop-hdfs-portmap/ hadoop-yarn-resourcemanager/
hadoop-hdfs-secondarynamenode/ hadoop-yarn-timelinereader/
hadoop-hdfs-zkfc/ hadoop-yarn-timelineserver/
[airflow@airflowetl tests]$ ls -lha /usr/hdp/current/hadoop
ls: cannot access /usr/hdp/current/hadoop: No such file or directory (note I am using HDP v3.1.0)
... View more
12-12-2019
03:35 PM
Is there a way to run spark-submit (spark v2.3.2 from HDP 3.1.0) while in a virtualenv? Have situation where have python file that uses python3 (and some specific libs) in a virtualenv (to isolate lib versions from rest of system). I would like to run this file with /bin/spark-submit, but attempting to do so I get...
[ me@myserver tests ] $ source ../ venv / bin / activate ; / bin / spark - submit sparksubmit . test . py
File "/bin/hdp-select" , line 255 print "ERROR: Invalid package - " + name
^
SyntaxError : Missing parentheses in call to 'print' . Did you mean print ( "ERROR: Invalid package - " + name )? ls : cannot access / usr / hdp // hadoop / lib : No such file or directory Exception in thread "main" java . lang . IllegalStateException : hdp . version is not set while running Spark under HDP , please set through HDP_VERSION in spark - env . sh or add a java - opts file in conf with - Dhdp . version = xxx
at org . apache . spark . launcher . Main . main ( Main . java : 118 )
# also tried...
( venv ) [ me@myserver tests ] $ export HADOOP_CONF_DIR =/ etc / hadoop / conf ; spark - submit -- master yarn -- deploy - mode cluster sparksubmit . test . py 19 / 12 / 12 13 : 50 : 20 WARN util . NativeCodeLoader : Unable to load native - hadoop library for your platform ... using builtin - java classes where applicable 19 / 12 / 12 13 : 50 : 20 WARN shortcircuit . DomainSocketFactory : The short - circuit local reads feature cannot be used because libhadoop cannot be loaded .
Exception in thread "main" java . lang . NoClassDefFoundError : com / sun / jersey / api / client / config / ClientConfig at org . apache . hadoop . yarn . client . api . TimelineClient . createTimelineClient ( TimelineClient . java : 55 )
.... at org . apache . spark . deploy . SparkSubmit . main ( SparkSubmit . scala )
Caused by : java . lang . ClassNotFoundException : com . sun . jersey . api . client . config . ClientConfig
Not sure what to make of this or how to proceed further and did not totally understand the error message after googling it.
Anyone with more experience have any further debugging tips for this or fixes?
... View more
Labels:
- Labels:
-
Apache Spark
12-11-2019
11:30 AM
1 Kudo
From the Ranger email list, this is another bit of information that I found helpful: ---------- I’ve configured ranger using the following approach to control who must be synced with AD. Only users belonging to groups inside a specific OU will be synced. I’ve created the OU OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC= domain ,DC=com Create a group called R2Users inside that OU. I put all desired sync users as its members. Also, you can put other groups as its member. And, you can create other groups like R2TEAM as well. Remember to update this property ranger.usersync.ldap.user.searchfilter to include more than one. I’ve configured ranger to sync groups before users. Here is the configuration. in COMMON CONFIGS Label Property Value LDAP/AD URL ranger.usersync.ldap.url ldap:// myacticedirectoryserver . domain.com : 389 Bind User ranger.usersync.ldap.binddn CN=LDAP_AD_ACCOUNT,OU=Service Accounts,OU=LCB,OU=Brazil,DC= domain ,DC=com Bind User Password ranger.usersync.ldap.ldapbindpassword LDAP_AD_ACCOUNT user’s password Inclemetal Sync ranger.usersync.ldap.deltasync Yes Enable LDAP STARTTLS ranger.usersync.ldap.starttls No GROUP CONFIGS Label Property Value Enable Group Sync ranger.usersync.group.searchenable Yes Group Member Attribute ranger.usersync.group.memberattributename member Group Name Attribute ranger.usersync.group.nameattribute Cn Group Object Class ranger.usersync.group.objectclass Group Group Search Base ranger.usersync.group.searchbase OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC= domain ,DC=com Group Search Filter ranger.usersync.group.searchfilter Enable Group Search First ranger.usersync.group.search.first.enabled Yes Sync Nested Groups is_nested_groupsync_enabled Yes Group Hierarchy Levels ranger.usersync.ldap.grouphierarchylevels 5 USER CONFIGS Label Property Value Username Attribute ranger.usersync.ldap.user.nameatributte sAMAccountName User Object Class ranger.usersync.ldap.objectclass User User Search Base ranger.usersync.ldap.searchbase DC= domain ,DC=com User Search Filter ranger.usersync.ldap.user.searchfilter (memberOf=CN=R2Users,OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC= domain ,DC= com ) User Search Scope ranger.usersync.ldap.user.searchscope Sub User Group Name Attribute ranger.usersync.ldap.groupnameattribute sAMAccountName Group User Map Sync ranger.usersync.group.usermapsyncenabled Yes Enable User Search ranger.usersync.user.searchenabled Yes ADVANCED Ranger Settings Label Property Value Authentication method ACTIVE_DIRECTORY AD Settings Label Property Value AD Bind Password ranger.ldap.ad.bind.password LDAP_AD_ACCOUNT user’s password Domain Name (Only for AD) anger.ldap.ad.domain DC= domain ,DC=com AD Base DN ranger.ldap.ad.base.dn DC= domain ,DC=com AD Referreal ranger.ldap.ad.referreal Follow AD User Serach Filter ranger.ldap.ad.user.search (sAMAccountName={0}) Advanced ranger-ugsync-site Label Property Value ranger.usersync.ldap.referral ranger.usersync.ldap.referral Follow
... View more
12-09-2019
02:43 PM
Saving this for later. https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/ambari-authentication-ldap-ad/content/setting_up_ldap_user_authentication.html
... View more
12-09-2019
02:42 PM
1 Kudo
Appears to have been able to sync AD users after changing the bing user path to: CN=hwldap,OU=Users,OU=HortonworksUsers,DC=ucera,DC=local as opposed to using the "uid" entry key. IDK why this would make a difference, but seems to have worked. Would anyone with more AD experience have an idea why (note that when I look at the attributes for this entry in our AD both the CN and UID attributes are present)?
... View more
12-06-2019
05:02 PM
Attempting to set LDAP/AD users for Ranger (v1.2.0) following the docs (https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/configuring-ranger-authe-with-unix-ldap-ad/content/configuring_ranger_authentication_with_unix_ldap_or_ad.html) and this older video (https://www.youtube.com/watch?v=2aZ9GBhCOhA), but when looking at the Ranger Users tab in the Ranger UI, seeing only the original Unix users. In the UI I see... Looking at the usersync logs, near the tail I see: .... 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization started 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldap://172.18.4.42:389, ldapBindDn: UID=hwldap,OU=Users,OU=HortonworksUsers,DC=ucera,DC=local, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: dc=hadoop,dc=apache,dc=org, userSearchBase: [dc=ucera,dc=local], userSearchScope: 2, userObjectClass: user, userSearchFilter: (memberOf=UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local), extendedUserSearchFilter: (&(objectclass=user)(memberOf=UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local)), userNameAttribute: sAMAccountName, userSearchAttributes: [sAMAccountName, memberof], userGroupNameAttributeSet: [memberof], pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled: false, groupSearchBase: [dc=ucera,dc=local], groupSearchScope: 2, groupObjectClass: group, groupSearchFilter: (CN=hwusers), extendedGroupSearchFilter: (&(objectclass=group)(CN=hwusers)(|(cn={0})(cn={1}))), extendedAllGroupsSearchFilter: (&(objectclass=group)(CN=hwusers)), groupMemberAttributeName: cn, groupNameAttribute: UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local, groupSearchAttributes: [UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local, cn], groupUserMapSyncEnabled: true, groupSearchFirstEnabled: false, userSearchEnabled: false, ldapReferral: ignore 06 Dec 2019 14:21:51 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder updateSink started 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Performing user search first 06 Dec 2019 14:21:51 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getUsers() failed with exception: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580]; remaining name 'dc=ucera,dc=local' 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getUsers() user count: 0 06 Dec 2019 14:21:51 INFO UserGroupSync [UnixUserSyncThread] - End: initial load of user/group from source==>sink .... So it seems like Ranger is trying to use AD, encountering an error, and falling back to Unix based users. Did see this article (https://community.cloudera.com/t5/Community-Articles/Ranger-Ldap-Integration/ta-p/245494), but already have the cluster nodes linked to AD via SSSD, so would think the LDAP/AD sync should already be configured on the nodes and that Ranger should be able to use AD once the configs where entered. Any idea what is going on here? Any further debugging tips or information (very unfamiliar with AD/LDAP admin stuff)?
... View more
Labels:
- Labels:
-
Apache Ranger
11-26-2019
05:24 PM
Looking at the HDP docs for using AD or LDAP for Ranger in Ambari (https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-ranger-authe-with-unix-ldap-ad/content/configuring_ranger_authentication_with_unix_ldap_or_ad.html), AD and LDAP appear to be treated as separate things (ie. are two different options):
Yet, when looking at the UI in my actual HDP ( 3.1.0.0-78 ) installation, it is shown as a single option "AD/LDAP" (which to me makes more sense, since LDAP is just a protocol to communicate with AD (not much experience with AD but that is what is seems to me)).
Can anyone with more LDAP/AD experience explain the difference? Anyone know the difference between using LDAP vs AD for Ranger integration?
... View more
Labels:
11-26-2019
01:21 PM
After just giving in and trying to manually create the hive user myself, I see [root@airflowetl ~]# useradd -g hadoop -s /bin/bash hive useradd: user 'hive' already exists
[root@airflowetl ~]# cat /etc/passwd | grep hive
[root@airflowetl ~]# id hive uid=379022825(hive) gid=379000513(domain users) groups=379000513(domain users) The fact that this existing user's uid looks like this and is not in the /etc/passwd file made me think that there is some existing Active Directory user (which this client node syncs with via installed SSSD) that already has the name hive. Checking our AD users, this turned out to be true. Temporarily stopping the SSSD service to stop sync with AD (service sssd stop) (since, not sure if you can get a server to ignore AD syncs on an individual user basis) before rerunning the client host add in Ambari fixed the problem for me.
... View more
11-26-2019
12:14 PM
Adding some log printing lines near the offending final line in the error trace, ie. File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 147, in get_supported_packages, I print the code and stdout: 2 ambari - python - wrap : can 't open file ' / usr / bin / hdp - select ': [Errno 2] No such file or directory So what the heck? It wants hdp-select to already be there, but ambari add-host UI complains if I manually install that binary myself beforehand. When I do manually install it (using the same repo file as in the rest of the existing cluster nodes) all I see is... 0
Packages : accumulo - client
accumulo - gc
accumulo - master
accumulo - monitor
accumulo - tablet
accumulo - tracer
atlas - client
atlas - server
beacon
beacon - client
beacon - server
druid - broker
druid - coordinator
druid - historical
druid - middlemanager
druid - overlord
druid - router
druid - superset
falcon - client
falcon - server
flume - server
hadoop - client
hadoop - hdfs - client
hadoop - hdfs - datanode
hadoop - hdfs - journalnode
hadoop - hdfs - namenode
hadoop - hdfs - nfs3
hadoop - hdfs - portmap
hadoop - hdfs - secondarynamenode
hadoop - hdfs - zkfc
hadoop - httpfs
hadoop - mapreduce - client
hadoop - mapreduce - historyserver
hadoop - yarn - client
hadoop - yarn - nodemanager
hadoop - yarn - registrydns
hadoop - yarn - resourcemanager
hadoop - yarn - timelinereader
hadoop - yarn - timelineserver
hbase - client
hbase - master
hbase - regionserver
hive - client
hive - metastore
hive - server2
hive - server2 - hive
hive - server2 - hive2
hive - webhcat
hive_warehouse_connector
kafka - broker
knox - server
livy - client
livy - server
livy2 - client
livy2 - server
mahout - client
oozie - client
oozie - server
phoenix - client
phoenix - server
pig - client
ranger - admin
ranger - kms
ranger - tagsync
ranger - usersync
shc
slider - client
spark - atlas - connector
spark - client
spark - historyserver
spark - schema - registry
spark - thriftserver
spark2 - client
spark2 - historyserver
spark2 - thriftserver
spark_llap
sqoop - client
sqoop - server
storm - client
storm - nimbus
storm - slider - client
storm - supervisor
superset
tez - client
zeppelin - server
zookeeper - client
zookeeper - server Aliases : accumulo - server
all
client
hadoop - hdfs - server
hadoop - mapreduce - server
hadoop - yarn - server
hive - server
Command failed after 1 tries
... View more
11-25-2019
03:31 PM
Attempting to add a client node to cluster via Ambari (v2.7.3.0) (HDP 3.1.0.0-78) and seeing odd error
stderr :
Traceback ( most recent call last ):
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py" , line 38 , in < module >
BeforeAnyHook (). execute ()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 352 , in execute
method ( env )
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py" , line 31 , in hook
setup_users ()
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py" , line 51 , in setup_users
fetch_nonlocal_groups = params . fetch_nonlocal_groups ,
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py" , line 166 , in __init__
self . env . run ()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py" , line 160 , in run
self . run_action ( resource , action )
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py" , line 124 , in run_action
provider_action ()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/accounts.py" , line 90 , in action_create
shell . checked_call ( command , sudo = True )
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 72 , in inner
result = function ( command , ** kwargs )
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 102 , in checked_call
tries = tries , try_sleep = try_sleep , timeout_kill_strategy = timeout_kill_strategy , returns = returns )
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 150 , in _call_wrapper
result = _call ( command , ** kwargs_copy )
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 314 , in _call
raise ExecutionFailed ( err_msg , code , out , err ) resource_management . core . exceptions . ExecutionFailed : Execution of 'usermod -G hadoop -g hadoop hive' returned 6. usermod : user 'hive' does not exist in / etc / passwd Error : Error : Unable to run the custom hook script [ '/usr/bin/python' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py' , 'ANY' , '/var/lib/ambari-agent/data/command-632.json' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY' , '/var/lib/ambari-agent/data/structured-out-632.json' , 'INFO' , '/var/lib/ambari-agent/tmp' , 'PROTOCOL_TLSv1_2' , '' ] 2019 - 11 - 25 13 : 07 : 58 , 000 - Reporting component version failed Traceback ( most recent call last ):
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 363 , in execute
self . save_component_version_to_structured_out ( self . command_name )
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 223 , in save_component_version_to_structured_out
stack_select_package_name = stack_select . get_package_name ()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 109 , in get_package_name
package = get_packages ( PACKAGE_SCOPE_STACK_SELECT , service_name , component_name )
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 223 , in get_packages
supported_packages = get_supported_packages ()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 147 , in get_supported_packages
raise Fail ( "Unable to query for supported packages using {0}" . format ( stack_selector_path ))
Fail : Unable to query for supported packages using / usr / bin / hdp - select
stdout :
2019 - 11 - 25 13 : 07 : 57 , 644 - Stack Feature Version Info : Cluster Stack = 3.1 , Command Stack = None , Command Version = None -> 3.1
2019 - 11 - 25 13 : 07 : 57 , 651 - Using hadoop conf dir : / usr / hdp / current / hadoop - client / conf 2019 - 11 - 25 13 : 07 : 57 , 652 - Group [ 'livy' ] {}
2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'spark' ] {}
2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'ranger' ] {}
2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'hdfs' ] {}
2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'zeppelin' ] {}
2019 - 11 - 25 13 : 07 : 57 , 655 - Group [ 'hadoop' ] {}
2019 - 11 - 25 13 : 07 : 57 , 655 - Group [ 'users' ] {}
2019 - 11 - 25 13 : 07 : 57 , 656 - User [ 'yarn-ats' ] { 'gid' : 'hadoop' , 'fetch_nonlocal_groups' : True , 'groups' : [ 'hadoop' ], 'uid' : None }
2019 - 11 - 25 13 : 07 : 57 , 658 - User [ 'hive' ] { 'gid' : 'hadoop' , 'fetch_nonlocal_groups' : True , 'groups' : [ 'hadoop' ], 'uid' : None }
2019 - 11 - 25 13 : 07 : 57 , 659 - Modifying user hive Error : Error : Unable to run the custom hook script [ '/usr/bin/python' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py' , 'ANY' , '/var/lib/ambari-agent/data/command-632.json' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY' , '/var/lib/ambari-agent/data/structured-out-632.json' , 'INFO' , '/var/lib/ambari-agent/tmp' , 'PROTOCOL_TLSv1_2' , '' ]
2019 - 11 - 25 13 : 07 : 57 , 971 - The repository with version 3.1 . 0.0 - 78 for this command has been marked as resolved . It will be used to report the version of the component which was installed 2019 - 11 - 25 13 : 07 : 58 , 000 - Reporting component version failed Traceback ( most recent call last ):
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 363 , in execute
self . save_component_version_to_structured_out ( self . command_name )
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 223 , in save_component_version_to_structured_out
stack_select_package_name = stack_select . get_package_name ()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 109 , in get_package_name
package = get_packages ( PACKAGE_SCOPE_STACK_SELECT , service_name , component_name )
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 223 , in get_packages
supported_packages = get_supported_packages ()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 147 , in get_supported_packages
raise Fail ( "Unable to query for supported packages using {0}" . format ( stack_selector_path ))
Fail : Unable to query for supported packages using / usr / bin / hdp - select
Command failed after 1 tries
The problem appears to be
resource_management . core . exceptions . ExecutionFailed : Execution of 'usermod -G hadoop -g hadoop hive' returned 6. usermod : user 'hive' does not exist in / etc / passwd
caused by
2019 - 11 - 25 13 : 07 : 57 , 659 - Modifying user hive Error : Error : Unable to run the custom hook script [ '/usr/bin/python' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py' , 'ANY' , '/var/lib/ambari-agent/data/command-632.json' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY' , '/var/lib/ambari-agent/data/structured-out-632.json' , 'INFO' , '/var/lib/ambari-agent/tmp' , 'PROTOCOL_TLSv1_2' , '' ]
Though, when running
[ root@HW001 . ssh ]# / usr / bin / hdp - select versions 3.1 . 0.0 - 78
from the ambari server node, I can see the command runs.
Looking at what the hook script is trying to run/access, I see
[ root@client001 ~]# ls - lha / var / lib / ambari - agent / cache / stack - hooks / before - ANY / scripts / hook . py - rw - r -- r -- 1 root root 1.2K Nov 25 10 : 51 / var / lib / ambari - agent / cache / stack - hooks / before - ANY / scripts / hook . py [ root@client001 ~]# ls - lha / var / lib / ambari - agent / data / command - 632.json
- rw ------- 1 root root 545K Nov 25 13 : 07 / var / lib / ambari - agent / data / command - 632.json
[ root@client001 ~]# ls - lha / var / lib / ambari - agent / cache / stack - hooks / before - ANY
total 0 drwxr - xr - x 4 root root 34 Nov 25 10 : 51 . drwxr - xr - x 8 root root 147 Nov 25 10 : 51 .. drwxr - xr - x 2 root root 34 Nov 25 10 : 51 files
drwxr - xr - x 2 root root 188 Nov 25 10 : 51 scripts [ root@client001 ~]# ls - lha / var / lib / ambari - agent / data / structured - out - 632.json ls : cannot access / var / lib / ambari - agent / data / structured - out - 632.json : No such file or directory [ root@client001 ~]# ls - lha / var / lib / ambari - agent / tmp
total 96K drwxrwxrwt 3 root root 4.0K Nov 25 13 : 06 . drwxr - xr - x 10 root root 267 Nov 25 10 : 50 . . drwxr - xr - x 6 root root 4.0K Nov 25 13 : 06 ambari_commons - rwx ------ 1 root root 1.4K Nov 25 13 : 06 ambari - sudo . sh - rwxr - xr - x 1 root root 1.6K Nov 25 13 : 06 create - python - wrap . sh - rwxr - xr - x 1 root root 1.6K Nov 25 10 : 50 os_check_type1574715018 . py - rwxr - xr - x 1 root root 1.6K Nov 25 11 : 12 os_check_type1574716360 . py - rwxr - xr - x 1 root root 1.6K Nov 25 11 : 29 os_check_type1574717391 . py - rwxr - xr - x 1 root root 1.6K Nov 25 13 : 06 os_check_type1574723161 . py - rwxr - xr - x 1 root root 16K Nov 25 10 : 50 setupAgent1574715020 . py - rwxr - xr - x 1 root root 16K Nov 25 11 : 12 setupAgent1574716361 . py - rwxr - xr - x 1 root root 16K Nov 25 11 : 29 setupAgent1574717392 . py - rwxr - xr - x 1 root root 16K Nov 25 13 : 06 setupAgent1574723163 . py
notice there is ls: cannot access /var/lib/ambari-agent/data/structured-out-632.json: No such file or directory. Not sure if this is normal, though.
Anyone know what could be causing this or any debugging hints from this point?
... View more
Labels:
- Labels:
-
Apache Ambari
11-04-2019
02:12 PM
Trying to set up LDAP/AD integration for Ranger on Ambari, but after switching Sync Source to LDAP/AD, when looking at the Configs > Advanced > LDAP Settings tab in the Ambari UI, seeing template values and am not sure if they are actually correct or not. Eg. if the LADP url is correct or needs to be changed.
Is there any where to see what these templated values actually are?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Ranger
11-01-2019
03:03 PM
Is there a difference between installing ambari via the apache docs vs Hortonworks docs ? I assume that the end result is exactly the same since Hortonworks labels the distribution in the docs as "Apache" and the repo they instruct to add uses apache.org as the package URL: [root@HW001 ~]# yum info ambari-server Installed Packages Name : ambari-server Arch : x86_64 Version : 2.7.3.0 Release : 139 Size : 418 M Repo : installed From repo : ambari-2.7.3.0 Summary : Ambari Server URL : http://www.apache.org License : (c) Apache Software Foundation Description : Maven Recipe: RPM Package. However, the installation instructions that the ambari project site links to are different and only involve building from source via maven (and seem to nowhere mention installation options via package manager), so gives me pause as to whether these are exactly the same. Could anyone with more experience here explain this a bit more to me? Is the underlying code any different when getting the same ambari version from Hortonworks vs building from source via the apache docs? And the reason for asking was due to the differences in installation method, yet having both marketed as "Apache".
... View more
Labels:
- Labels:
-
Apache Ambari
09-27-2019
02:21 PM
Running installation of HDP 3.1.0 and attempting to run the default Zeppelin examples (here the R notebook). When running any of the paragraphs in the notebook, see the error like...
java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.1.3.1.0.0-78 at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
The configs are set to the initial default Ambari-given configs for Zeppelin, so not sure why would be seeing this. Any debugging advice for this?
... View more
Labels:
- Labels:
-
Apache Zeppelin
09-24-2019
05:59 PM
Had only done sudo -u postgres /usr/bin/pg_ctl -D $PGDATA reload from https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/configuring_postgresql_for_ranger.html So I think restarting the service helped (honestly, did many other things so hard to tell which did the trick). For other finding this, a hint that the service should have been restarted could have been found in the docs here: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/install-postgres.html
... View more
09-20-2019
03:20 PM
Attempting to install HDP 3.1.0 via Ambari 2.7.3 using an existing postgresql DB after following the docs here (https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/configuring_postgresql_for_ranger.html) and entering the commands below:
----------
dbname=hive
postgres=postgres
user=hive
passwd=hive
echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres
echo "alter user $postgres superuser;" | sudo -u $postgres psql -U postgres
dbname=oozie
postgres=postgres
user=oozie
passwd=oozie
echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres
echo "alter user $postgres superuser;" | sudo -u $postgres psql -U postgres
dbname=ranger
postgres=postgres
user=rangeradmin
passwd=ranger
echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres
echo "alter user $user superuser;" | sudo -u $postgres psql -U postgres
dbname=rangerkms
postgres=postgres
user=rangerkms
passwd=ranger
echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres
echo "alter user $user superuser;" | sudo -u $postgres psql -U postgres
dbname=superset
postgres=postgres
user=superset
passwd=superset
echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres
echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres
echo "alter user $user superuser;" | sudo -u $postgres psql -U postgres
----------
This was done based on the docs here: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/administering-ambari/content/amb_using_hive_with_postgresql.html
However, when doing the connection tests in the Ambari installation phase that checks the databases for the services to be installed, getting the error
Error injecting constructor, java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "<some host>", user "<some service user>", database "<some service user>", SSL off
for the hive DB and I assume it would be the same for the druid and superset DBs as well if Ambari had provided a "test connection" button for those.
My question is: what is the problem here? The docs don't seem to indicate that anything more should be done (unlike with the docs for installing ranger), so what should be done?
Currently, my though is to do something like was done for ranger:
[root@HW001 ~]# echo "local all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid trust" >> /var/lib/pgsql/data/pg_hba.conf [root@HW001 ~]# echo "host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid 0.0.0.0/0 trust" >> /var/lib/pgsql/data/pg_hba.conf [root@HW001 ~]# echo "host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid ::/0 trust" >> /var/lib/pgsql/data/pg_hba.conf [root@HW001 ~]# cat /var/lib/pgsql/data/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only local all postgres peer # IPv4 local connections: host all postgres 127.0.0.1/32 ident # IPv6 local connections: host all postgres ::1/128 ident # Allow replication connections from localhost, by a user with the # replication privilege. #local replication postgres peer #host replication postgres 127.0.0.1/32 ident #host replication postgres ::1/128 ident
local all ambari,mapred md5 host all ambari,mapred 0.0.0.0/0 md5 host all ambari,mapred ::/0 md5 local all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid trust host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid 0.0.0.0/0 trust host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid ::/0 trust
but not sure if there is something else I'm missing here or some other things that I should be seeing that I am not. Is this is the correct thing to do or something else?
... View more
- Tags:
- hdp-3.1.0
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
-
Apache Ranger
09-18-2019
05:13 PM
1 Kudo
Am currently evaluating HDP using version 3.1.4 and it appears that this version is not compatible with any form of HDF either whole-platform or the mpack. Looking at the compatibility matrix, this is what I see: I also saw this github issue that makes it seem like the latest version of HDP can't even use the HDF mpack to use NiFi: https://github.com/hortonworks/ansible-hortonworks/issues/166 Can anyone confirm these issues? Is there any workaround? What is the expected use case for a version of HDP that does not integrate with the data intake HDF platform?
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
08-27-2019
03:03 PM
Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_sample_spark_2_x_applications.html The binaries appear to be in /usr/hdp/current/spark2-client/bin Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client
... View more
08-27-2019
02:58 PM
Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg.
(venv) ➜ ~ spark
zsh: spark: command not found...
zsh: command not found: spark
(venv) ➜ ~ spark2
zsh: spark2: command not found...
zsh: command not found: spark2
Checking the filesystem, nothing seems to be related directly to any spark binaries:
(venv) ➜ ~ find / -name spark 2>&1 | grep -v "Permission denied"
/home/spark
/var/lib/smartsense/hst-agent/resources/collection-scripts/spark
/var/log/spark
/var/spool/mail/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark
/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark
/usr/hdp/3.1.0.0-78/oozie/share/lib/spark
Anyone know where the spark binaries are for any given nodes?
... View more
Labels:
08-26-2019
06:54 PM
Problem appears to be related to How to properly change uid for HDP / ambari-created user? and the fact that having a user exist on a node and have a hdfs://user/<username> directory with correct permissions (as I was lead to believe from a Hortonworks forum post) is not sufficient to be acknowledges as "existing" on the cluster. Running the hadoop jar command for a different user (in this case, the Ambari-created hdfs user) that exists on all cluster nodes (even though Ambari created this user having different uids across nodes (IDK if this is a problem)) and has a hdfs://user/hdfs dir, found that the h2o jar ran as expected. Will look into this a bit more before posting as an answer. I think basically will need to look for a bit more clarification as to when HDP considers a user to "exist" on a cluster.
... View more
08-26-2019
06:29 PM
Looking at the docs for installing NiFi on HDP 3.1 via management pack and looking at list of repository locations here for HDF 3.4. Was wondering if the HDF management pack version needs to be the same as the HDP version for correct installation (else how to tell if the version are compatible with each other)?
... View more
Labels:
08-15-2019
02:40 AM
@Geoffrey Shelton Okot What about if the cluster is not using kerberos (eg. hadoop.security.authentication=local)?
... View more