Is there a way to run   spark-submit   (spark v2.3.2 from HDP 3.1.0) while in a virtualenv? Have situation where have python file that uses python3 (and some specific libs) in a virtualenv (to isolate lib versions from rest of system). I would like to run this file with   /bin/spark-submit, but attempting to do so I get... [ me@myserver tests ] $ source ../ venv / bin / activate ; / bin / spark - submit sparksubmit . test . py File "/bin/hdp-select" , line 255 print "ERROR: Invalid package - " + name ^ SyntaxError : Missing parentheses in call to 'print' . Did you mean print ( "ERROR: Invalid package - " + name )? ls : cannot access / usr / hdp // hadoop / lib : No such file or directory Exception in thread "main" java . lang . IllegalStateException : hdp . version is not set while running Spark under HDP , please set through HDP_VERSION in spark - env . sh or add a java - opts file in conf with - Dhdp . version = xxx at org . apache . spark . launcher . Main . main ( Main . java : 118 ) # also tried... ( venv ) [ me@myserver tests ] $ export HADOOP_CONF_DIR =/ etc / hadoop / conf ; spark - submit -- master yarn -- deploy - mode cluster sparksubmit . test . py 19 / 12 / 12 13 : 50 : 20 WARN util . NativeCodeLoader : Unable to load native - hadoop library for your platform ... using builtin - java classes where applicable 19 / 12 / 12 13 : 50 : 20 WARN shortcircuit . DomainSocketFactory : The short - circuit local reads feature cannot be used because libhadoop cannot be loaded . Exception in thread "main" java . lang . NoClassDefFoundError : com / sun / jersey / api / client / config / ClientConfig at org . apache . hadoop . yarn . client . api . TimelineClient . createTimelineClient ( TimelineClient . java : 55 ) .... at org . apache . spark . deploy . SparkSubmit . main ( SparkSubmit . scala ) Caused by : java . lang . ClassNotFoundException : com . sun . jersey . api . client . config . ClientConfig Not sure what to make of this or how to proceed further and did not totally understand the error message after googling it. Anyone with more experience have any further debugging tips for this or fixes? ... View more

Attempting to set LDAP/AD users for Ranger (v1.2.0) following the docs (https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/configuring-ranger-authe-with-unix-ldap-ad/content/configuring_ranger_authentication_with_unix_ldap_or_ad.html) and this older video (https://www.youtube.com/watch?v=2aZ9GBhCOhA), but when looking at the Ranger Users tab in the Ranger UI, seeing only the original Unix users. In the UI I see...     Looking at the usersync logs, near the tail I see:   .... 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization started 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldap://172.18.4.42:389, ldapBindDn: UID=hwldap,OU=Users,OU=HortonworksUsers,DC=ucera,DC=local, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: dc=hadoop,dc=apache,dc=org, userSearchBase: [dc=ucera,dc=local], userSearchScope: 2, userObjectClass: user, userSearchFilter: (memberOf=UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local), extendedUserSearchFilter: (&(objectclass=user)(memberOf=UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local)), userNameAttribute: sAMAccountName, userSearchAttributes: [sAMAccountName, memberof], userGroupNameAttributeSet: [memberof], pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled: false, groupSearchBase: [dc=ucera,dc=local], groupSearchScope: 2, groupObjectClass: group, groupSearchFilter: (CN=hwusers), extendedGroupSearchFilter: (&(objectclass=group)(CN=hwusers)(|(cn={0})(cn={1}))), extendedAllGroupsSearchFilter: (&(objectclass=group)(CN=hwusers)), groupMemberAttributeName: cn, groupNameAttribute: UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local, groupSearchAttributes: [UID=hwusers,OU=groups,OU=HortonworksUsers,DC=ucera,DC=local, cn], groupUserMapSyncEnabled: true, groupSearchFirstEnabled: false, userSearchEnabled: false, ldapReferral: ignore 06 Dec 2019 14:21:51 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder updateSink started 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Performing user search first 06 Dec 2019 14:21:51 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getUsers() failed with exception: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580]; remaining name 'dc=ucera,dc=local' 06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getUsers() user count: 0 06 Dec 2019 14:21:51 INFO UserGroupSync [UnixUserSyncThread] - End: initial load of user/group from source==>sink ....   So it seems like Ranger is trying to use AD, encountering an error, and falling back to Unix based users.   Did see this article (https://community.cloudera.com/t5/Community-Articles/Ranger-Ldap-Integration/ta-p/245494), but already have the cluster nodes linked to AD via SSSD, so would think the LDAP/AD sync should already be configured on the nodes and that Ranger should be able to use AD once the configs where entered.   Any idea what is going on here? Any further debugging tips or information (very unfamiliar with AD/LDAP admin stuff)? ... View more

Attempting to add a client node to cluster via Ambari (v2.7.3.0) (HDP 3.1.0.0-78) and seeing odd error stderr : Traceback ( most recent call last ): File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py" , line 38 , in < module > BeforeAnyHook (). execute () File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 352 , in execute method ( env ) File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py" , line 31 , in hook setup_users () File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py" , line 51 , in setup_users fetch_nonlocal_groups = params . fetch_nonlocal_groups , File "/usr/lib/ambari-agent/lib/resource_management/core/base.py" , line 166 , in __init__ self . env . run () File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py" , line 160 , in run self . run_action ( resource , action ) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py" , line 124 , in run_action provider_action () File "/usr/lib/ambari-agent/lib/resource_management/core/providers/accounts.py" , line 90 , in action_create shell . checked_call ( command , sudo = True ) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 72 , in inner result = function ( command , ** kwargs ) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 102 , in checked_call tries = tries , try_sleep = try_sleep , timeout_kill_strategy = timeout_kill_strategy , returns = returns ) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 150 , in _call_wrapper result = _call ( command , ** kwargs_copy ) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py" , line 314 , in _call raise ExecutionFailed ( err_msg , code , out , err ) resource_management . core . exceptions . ExecutionFailed : Execution of 'usermod -G hadoop -g hadoop hive' returned 6. usermod : user 'hive' does not exist in / etc / passwd Error : Error : Unable to run the custom hook script [ '/usr/bin/python' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py' , 'ANY' , '/var/lib/ambari-agent/data/command-632.json' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY' , '/var/lib/ambari-agent/data/structured-out-632.json' , 'INFO' , '/var/lib/ambari-agent/tmp' , 'PROTOCOL_TLSv1_2' , '' ] 2019 - 11 - 25 13 : 07 : 58 , 000 - Reporting component version failed Traceback ( most recent call last ): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 363 , in execute self . save_component_version_to_structured_out ( self . command_name ) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 223 , in save_component_version_to_structured_out stack_select_package_name = stack_select . get_package_name () File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 109 , in get_package_name package = get_packages ( PACKAGE_SCOPE_STACK_SELECT , service_name , component_name ) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 223 , in get_packages supported_packages = get_supported_packages () File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 147 , in get_supported_packages raise Fail ( "Unable to query for supported packages using {0}" . format ( stack_selector_path )) Fail : Unable to query for supported packages using / usr / bin / hdp - select stdout : 2019 - 11 - 25 13 : 07 : 57 , 644 - Stack Feature Version Info : Cluster Stack = 3.1 , Command Stack = None , Command Version = None -> 3.1 2019 - 11 - 25 13 : 07 : 57 , 651 - Using hadoop conf dir : / usr / hdp / current / hadoop - client / conf 2019 - 11 - 25 13 : 07 : 57 , 652 - Group [ 'livy' ] {} 2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'spark' ] {} 2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'ranger' ] {} 2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'hdfs' ] {} 2019 - 11 - 25 13 : 07 : 57 , 654 - Group [ 'zeppelin' ] {} 2019 - 11 - 25 13 : 07 : 57 , 655 - Group [ 'hadoop' ] {} 2019 - 11 - 25 13 : 07 : 57 , 655 - Group [ 'users' ] {} 2019 - 11 - 25 13 : 07 : 57 , 656 - User [ 'yarn-ats' ] { 'gid' : 'hadoop' , 'fetch_nonlocal_groups' : True , 'groups' : [ 'hadoop' ], 'uid' : None } 2019 - 11 - 25 13 : 07 : 57 , 658 - User [ 'hive' ] { 'gid' : 'hadoop' , 'fetch_nonlocal_groups' : True , 'groups' : [ 'hadoop' ], 'uid' : None } 2019 - 11 - 25 13 : 07 : 57 , 659 - Modifying user hive Error : Error : Unable to run the custom hook script [ '/usr/bin/python' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py' , 'ANY' , '/var/lib/ambari-agent/data/command-632.json' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY' , '/var/lib/ambari-agent/data/structured-out-632.json' , 'INFO' , '/var/lib/ambari-agent/tmp' , 'PROTOCOL_TLSv1_2' , '' ] 2019 - 11 - 25 13 : 07 : 57 , 971 - The repository with version 3.1 . 0.0 - 78 for this command has been marked as resolved . It will be used to report the version of the component which was installed 2019 - 11 - 25 13 : 07 : 58 , 000 - Reporting component version failed Traceback ( most recent call last ): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 363 , in execute self . save_component_version_to_structured_out ( self . command_name ) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py" , line 223 , in save_component_version_to_structured_out stack_select_package_name = stack_select . get_package_name () File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 109 , in get_package_name package = get_packages ( PACKAGE_SCOPE_STACK_SELECT , service_name , component_name ) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 223 , in get_packages supported_packages = get_supported_packages () File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py" , line 147 , in get_supported_packages raise Fail ( "Unable to query for supported packages using {0}" . format ( stack_selector_path )) Fail : Unable to query for supported packages using / usr / bin / hdp - select Command failed after 1 tries The problem appears to be resource_management . core . exceptions . ExecutionFailed : Execution of 'usermod -G hadoop -g hadoop hive' returned 6. usermod : user 'hive' does not exist in / etc / passwd caused by 2019 - 11 - 25 13 : 07 : 57 , 659 - Modifying user hive Error : Error : Unable to run the custom hook script [ '/usr/bin/python' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py' , 'ANY' , '/var/lib/ambari-agent/data/command-632.json' , '/var/lib/ambari-agent/cache/stack-hooks/before-ANY' , '/var/lib/ambari-agent/data/structured-out-632.json' , 'INFO' , '/var/lib/ambari-agent/tmp' , 'PROTOCOL_TLSv1_2' , '' ]   Though, when running [ root@HW001 . ssh ]# / usr / bin / hdp - select versions 3.1 . 0.0 - 78 from the ambari server node, I can see the command runs.   Looking at what the hook script is trying to run/access, I see [ root@client001 ~]# ls - lha / var / lib / ambari - agent / cache / stack - hooks / before - ANY / scripts / hook . py - rw - r -- r -- 1 root root 1.2K Nov 25 10 : 51 / var / lib / ambari - agent / cache / stack - hooks / before - ANY / scripts / hook . py [ root@client001 ~]# ls - lha / var / lib / ambari - agent / data / command - 632.json - rw ------- 1 root root 545K Nov 25 13 : 07 / var / lib / ambari - agent / data / command - 632.json [ root@client001 ~]# ls - lha / var / lib / ambari - agent / cache / stack - hooks / before - ANY total 0 drwxr - xr - x 4 root root 34 Nov 25 10 : 51 . drwxr - xr - x 8 root root 147 Nov 25 10 : 51 .. drwxr - xr - x 2 root root 34 Nov 25 10 : 51 files drwxr - xr - x 2 root root 188 Nov 25 10 : 51 scripts [ root@client001 ~]# ls - lha / var / lib / ambari - agent / data / structured - out - 632.json ls : cannot access / var / lib / ambari - agent / data / structured - out - 632.json : No such file or directory [ root@client001 ~]# ls - lha / var / lib / ambari - agent / tmp total 96K drwxrwxrwt 3 root root 4.0K Nov 25 13 : 06 . drwxr - xr - x 10 root root 267 Nov 25 10 : 50 . . drwxr - xr - x 6 root root 4.0K Nov 25 13 : 06 ambari_commons - rwx ------ 1 root root 1.4K Nov 25 13 : 06 ambari - sudo . sh - rwxr - xr - x 1 root root 1.6K Nov 25 13 : 06 create - python - wrap . sh - rwxr - xr - x 1 root root 1.6K Nov 25 10 : 50 os_check_type1574715018 . py - rwxr - xr - x 1 root root 1.6K Nov 25 11 : 12 os_check_type1574716360 . py - rwxr - xr - x 1 root root 1.6K Nov 25 11 : 29 os_check_type1574717391 . py - rwxr - xr - x 1 root root 1.6K Nov 25 13 : 06 os_check_type1574723161 . py - rwxr - xr - x 1 root root 16K Nov 25 10 : 50 setupAgent1574715020 . py - rwxr - xr - x 1 root root 16K Nov 25 11 : 12 setupAgent1574716361 . py - rwxr - xr - x 1 root root 16K Nov 25 11 : 29 setupAgent1574717392 . py - rwxr - xr - x 1 root root 16K Nov 25 13 : 06 setupAgent1574723163 . py notice there is   ls: cannot access /var/lib/ambari-agent/data/structured-out-632.json: No such file or directory. Not sure if this is normal, though.   Anyone know what could be causing this or any debugging hints from this point? ... View more

Is there a difference between installing ambari via the  apache docs  vs  Hortonworks docs ?   I assume that the end result is exactly the same since Hortonworks labels the distribution in the docs as "Apache" and the repo they instruct to add uses   apache.org   as the package URL: [root@HW001 ~]# yum info ambari-server Installed Packages Name        : ambari-server Arch        : x86_64 Version     : 2.7.3.0 Release     : 139 Size        : 418 M Repo        : installed From repo   : ambari-2.7.3.0 Summary     : Ambari Server URL         :  http://www.apache.org License     : (c) Apache Software Foundation Description : Maven Recipe: RPM Package.   However, the installation instructions that the ambari project site links to are different and only involve building from source via maven (and seem to nowhere mention installation options via package manager), so gives me pause as to whether these are exactly the same.   Could anyone with more experience here explain this a bit more to me?  Is the underlying code any different when getting the same ambari version from Hortonworks vs building from source via the apache docs? And the reason for asking was due to the differences in installation method, yet having both marketed as "Apache". ... View more

Attempting to install HDP 3.1.0 via Ambari 2.7.3 using an existing postgresql DB after following the docs here (https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/configuring_postgresql_for_ranger.html) and entering the commands below:   ---------- dbname=hive postgres=postgres user=hive passwd=hive echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter user $postgres superuser;" | sudo -u $postgres psql -U postgres     dbname=oozie postgres=postgres user=oozie passwd=oozie echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter user $postgres superuser;" | sudo -u $postgres psql -U postgres   dbname=ranger postgres=postgres user=rangeradmin passwd=ranger echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter user $user superuser;" | sudo -u $postgres psql -U postgres     dbname=rangerkms postgres=postgres user=rangerkms passwd=ranger echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter user $user superuser;" | sudo -u $postgres psql -U postgres   dbname=superset postgres=postgres user=superset passwd=superset echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter user $user superuser;" | sudo -u $postgres psql -U postgres ----------   This was done based on the docs here: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/administering-ambari/content/amb_using_hive_with_postgresql.html   However, when doing the connection tests in the Ambari installation phase that checks the databases for the services to be installed, getting the error   Error injecting constructor, java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "<some host>", user "<some service user>", database "<some service user>", SSL off   for the hive DB and I assume it would be the same for the druid and superset DBs as well if Ambari had provided a "test connection" button for those.   My question is: what is the problem here? The docs don't seem to indicate that anything more should be done (unlike with the docs for installing ranger), so what should be done?   Currently, my though is to do something like was done for ranger:   [root@HW001 ~]# echo "local all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid trust" >> /var/lib/pgsql/data/pg_hba.conf [root@HW001 ~]# echo "host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid 0.0.0.0/0 trust" >> /var/lib/pgsql/data/pg_hba.conf [root@HW001 ~]# echo "host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid ::/0 trust" >> /var/lib/pgsql/data/pg_hba.conf [root@HW001 ~]# cat /var/lib/pgsql/data/pg_hba.conf  # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all postgres peer # IPv4 local connections: host all postgres 127.0.0.1/32 ident # IPv6 local connections: host all postgres ::1/128 ident # Allow replication connections from localhost, by a user with the # replication privilege. #local replication postgres peer #host replication postgres 127.0.0.1/32 ident #host replication postgres ::1/128 ident local all ambari,mapred md5 host all ambari,mapred 0.0.0.0/0 md5 host all ambari,mapred ::/0 md5 local all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid trust host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid 0.0.0.0/0 trust host all postgres,rangeradmin,rangerlogger,hive,oozie,ranger,rangerkms,superset,druid ::/0 trust   but not sure if there is something else I'm missing here or some other things that I should be seeing that I am not. Is this is the correct thing to do or something else? ... View more

Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg.    (venv) ➜ ~ spark zsh: spark: command not found... zsh: command not found: spark (venv) ➜ ~ spark2 zsh: spark2: command not found... zsh: command not found: spark2   Checking the filesystem, nothing seems to be related directly to any spark binaries:   (venv) ➜ ~ find / -name spark 2>&1 | grep -v "Permission denied" /home/spark /var/lib/smartsense/hst-agent/resources/collection-scripts/spark /var/log/spark /var/spool/mail/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark /tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark /usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark /usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark /usr/hdp/3.1.0.0-78/oozie/share/lib/spark   Anyone know where the spark binaries are for any given nodes? ... View more

Looking at the docs for installing NiFi on HDP 3.1 via management pack and looking at list of repository locations here for HDF 3.4. Was wondering if the HDF management pack version needs to be the same as the HDP version for correct installation (else how to tell if the version are compatible with each other)? ... View more