This article outlines the steps needed to setup ODBC access to Hive via Apache Knox from a Linux workstation. While there seems to be some "reasonable" documentation other there on setting up this access from Windows, it took quite some time to figure out how to do this from Linux. For this example, I am installing the Hortonworks Linux ODBC driver on CentOS Linux release 7.2.1511 then verifying it works using both the isql ODBC command line tool and PyODBC. Installation Steps 1) Install the UnixODBC Dependencies sudo yum install unixODBC 2) Download and Install Hortonworks ODBC Linux Driver I developed this guide using the below version of the driver. Note that the following URL is for the CentOS 7 version of the driver. If you are running an older version of the OS, then visit https://hortonworks.com/downloads/ and use the CentOS 6 driver. Also note that the below URL will likely change as new versions of the driver are released so it is a good idea to check the Hortonworks download page for the latest version when you run this procedure. I can tell you that a college used this procedure on RHEL 6.4 and used the CentOS 6 driver and it worked fine. wget http://public-repo-1.hortonworks.com/HDP/hive-odbc/2.1.5.1006/centos7/hive-odbc-native-2.1.5.1006-1.el7.x86_64.rpm sudo yum install hive-odbc-native-2.1.5.1006-1.el7.x86_64.rpm 3) Create $HOME/.odbc.ini You don't necessarily need this file. For example, you could build up a DSN string in code in a Python program that you use to connect to Hive using PyODBC. But, you can also just specify a DSN name that PyODBC then maps to an entry in your $HOME/.odbc.ini file. So, either way, here are the settings that I used to build up a minimal $HOME/.odbc.ini used in this guide. Simple copy the following values into $HOME/.odbc.ini. If you want to know the details of what each field means and its optional values, you can find it all at the end of the ODBC installation document. Note that the following sample entries have some fields that you MUST change that are environment and user specific. Namely the host, trustStorePassword, and ClientCert fields. All other fields should be left as they are. [ODBC Data Sources] # The below key 'Knox' can be anything but it MUST MATCH the key name used # in the section below. The right side is just descriptive and can be anything Knox=DSN for access to the production cluster using Beeline via Knox [Knox] # Optional description of this DSN Description=Hortonworks Knox QA DSN # Where the actual ODBC driver library is loaded from. This is the default # location where Hortonworks installs the driver Driver=/usr/lib/hive/lib/native/Linux-amd64-64/libhortonworkshiveodbc64.so # The hostname of the server running the Knox gateway Host=api01.quasar.local # The port used to connect to the Knox gateway port=8443 # The same httppath value used in a typical beeline connection string that # connects via Knox. For our cluster, it is always "quasar/jupstats/hive" HttpPath=quasar/jupstats/hive # The database you want to be on once beeline connects to Hive. This is basically # like executing the HQL "use my_database" command after the connection is made. schema=default # Must be 0 because we want to connect directly to Hiveserver2 inside the cluster. ServiceDiscoveryMode=0 # Must be 2 because we are using hiveserver2 inside the cluster. HiveServerType=2 # This indicates we need to provide a username and password in the connect string AuthMech=3 # This indicates the transport mode for Hive is HTTP, which is required for Knox ThriftTransport=2 # Indicates the connection to Knox uses SSL. Note that our Hive configuration has # hive.server2.use.SSL=false but, the connection to hiverserver2 is via Knox and, # to connect to Knox, you must use SSL. So, the ODBC driver is first connecting # to Knox hence we need SSL=1 here. If you were using ODBC to directly connect # to hiveserver2 (not via Knox), then, for the configuration we have in our cluster, # this value would need to be 0 SSL=1 # But NOT two-way SSL meaning no client side authentication - only server-side TwoWaySSL=0 # These two are the same values used in the beeline connect string. The password is # the password you used to add the Knox certificate to your local user key store. # The client certifificate path is just the path to that key store trustStorePassword=<your-trust-store-password> ClientCert=/home/mpetronic/keystore/keystore.jks # Must be 1 to support using self-signed certificates. Since we generated and # self-signed certificate in our cluster, we need this. AllowSelfSignedServerCert=1 # This prevents ODBC from doing query preprocessing which is not required since we are # running against a Hive-aware end point UseNativeQuery=1 Regarding the use of the UseNativeQuery=1 option, the ODBC docs say: This option specifies whether the driver uses native HiveQL queries, or converts them into an equivalent form in HiveQL. Enabled (1): The driver does not transform the queries emitted by an application, and executes HiveQL queries directly. Disabled (0): The driver transforms the queries emitted by an application and converts them into an equivalent form in HiveQL. Note: If the application is Hive-aware and already emits HiveQL, then enable this option to avoid the extra overhead of query transformation. Hive logs the queries that are run in various places like log files and in the Tez View. Admins need to be able to read these to help debug query execution problems. When UseNativeQuery=1 is set, the logged queries look like what you would normally expect - this: select count(*) from my_db.some_table where year=2016 and month=10 and day=2 But, if UseNativeQuery=0 is used, then the same query will end up looking like this: SELECT COUNT(1) AS `C_455850525f31` FROM `my_db`.`some_table` `C_6a7570315f73746174735f63726f5f6361706163697479` WHERE ((`C_6a7570315f73746174735f63726f5f6361706163697479`.`day` = 2) AND (`C_6a7570315f73746174735f63726f5f6361706163697479`.`year` = 2016) AND (`C_6a7570315f73746174735f63726f5f6361706163697479`.`month` = 10)) If you had to debug queries in logs, which would you prefer? Right! ALWAYS set UseNativeQuery=1. Plus, per the docs, this also improves performance by reducing the overhead of unnecessary query transformation. 4) Test your Connection via Command Line When you installed the unixODBC packages, it came with "isql", an interactive SQL CLI. Run the following command to see if you can connect to Hive via ODBC via Knox using the DSN you just created above. You should get some valid results from your query: [mpetronic@mpws Downloads]$ echo "show databases" | isql -v -d, Knox <your-Linux-username> <your-Linux-password> +---------------------------------------+ | Connected! | | | | sql-statement | | help [tablename] | | quit | | | +---------------------------------------+ SQL> show databases default test_db another_db 5) Test your Connection via PyODBC The following is a simple Python app that uses PyODBC to access the same database using the same DSN. Copy the below sample code to your workstation, read the comments in the script and do what they say to do to be able to run the program. Then run it to verify you can successfully connect and execute a query via Knox. It will print the query results to stdout. #!/usr/bin/env python import pyodbc import argparse ''' Demonstrates the bare bones code needed to access Hive via Beeline via Knox. For this sample to work, you must provide either the full ODBC configuration via a connection string or reference an odbc.ini file that already has a DSN defined for the connection to Knox you are trying to make. To go the odbc.ini route, do the following: 1. Create .odbc.ini in your $HOME directory. Note this is a hidden file and that is what the ODBC driver expects by default 2. Populate $HOME/.odbc.ini with the following values. These are the minimum values needed for the file to work with Knox. I am not going to explain all the values in detail. If you want to know what each one means, look them up in this doc: https://hortonworks.com/wp-content/uploads/2016/03/Hortonworks-Hive-ODBC-Driver-User-Guide.pdf. Also, change the host, trustStorePassword, and ClientCert values as needed for your setup. [ODBC Data Sources] # This key 'Knox' can be anything but it MUST MATCH the key name used in the # section below. The right side is just descriptive and can be anything Knox=DNS for access to Hive via Knox [Knox] # Optional description of this DSN Description=Hortonworks Knox QA DSN # Where the actually ODBC driver library is loaded from. This is the default # location where Hortonworks installs the driver Driver=/usr/lib/hive/lib/native/Linux-amd64-64/libhortonworkshiveodbc64.so # The hostname of the sever running the Knox gateway Host=api01.qa # The port used to connect to the Knox gateway port=8443 # The same Http path value used in a typical beeline connect string that # connects to Knox. For Quasar, it is always "quasar/jupstats/hive" HttpPath=quasar/jupstats/hive # The default database you want to be one once beeline connects to Hive schema=jup1_stats # Must be zero as we want to connect directly to Hiveserver2 inside the cluster. ServiceDiscoveryMode=0 # Must be 2 because we are using hiveserver2 which is what beeline uses as well HiveServerType=2 # This indicates we need to provide a username and passwork in the connect string AuthMech=3 # This indicates the transport mode for Hive is HTTP, which is required for Knox ThriftTransport=2 # Indicates the connection to Knox uses SSL SSL=1 # But NOT two-way SSL meaning no client side authentication - only server-side TwoWaySSL=0 # These two are the same values used in the beeline connect string. This is the # password you used when you added the Knox certificate to your own trust store trustStorePassword=<your-trust-store-password> ClientCert=/home/mpetronic/keystore/keystore.jks # Must be 1 to support the self-signed certificate mode we use for Knox AllowSelfSignedServerCert=1 3. Install dependencies for this script to run. First, install the unixODBC-devel package (via yum, if you can or download the RPM and install that way). This is needed for header files that are used build pyodbc. Then install PyODBC using "pip install pyodbc==3.0.10". Note that the pyodbc installation needs to build some source code so you will also need the gcc tools chain installed. 4. Run this script and provide the require command line options to test your connection To instead, NOT require the $HOME/.odbc.ini file, you can build a DSN string directly in code. Below is an example of how you can build a connection string to directly pass into the pyodbc.connect() method. Using this approach, you DO NOT need to have a .odbc.ini file present. This gives you full programmatic control of the process if you wish to use it this way. This simply builds a string like "A=1;B=2;C=3" containing all the exact same parameters that could be defined in the .odbc.ini file. ''' CONNECTION_STRING_EXAMPLE = ';'.join(''' Description=Hortonworks Knox DSN Driver=/usr/lib/hive/lib/native/Linux-amd64-64/libhortonworkshiveodbc64.so Host=api01 port=8443 HttpPath=quasar/jupstats/hive schema=default ServiceDiscoveryMode=0 HiveServerType=2 AuthMech=3 ThriftTransport=2 SSL=1 TwoWaySSL=0 trustStorePassword=<your-trust-store-password> ClientCert=/home/mpetronic/keystore/keystore.jks AllowSelfSignedServerCert=1 uid=<your-linux-username> pwd=<your-linux-password> '''.splitlines()) #=============================================================================== def parse_args(): parser = argparse.ArgumentParser() parser.add_argument('--user', '-u', type=str, required=True, help='User name used to connect to Hive with beeline via Knox.') parser.add_argument('--password', '-p', type=str, required=True, help='Password used to connect to Hive with beeline via Knox.') parser.add_argument('--dsn', '-d', type=str, required=True, help='The DSN name from $HOME/.odbc.ini to use for this connection.') return parser.parse_args() #=============================================================================== def execute_query(sql, dsn, user, password): try: connect_string = 'DSN=%(dsn)s;uid=%(user)s;pwd=%(password)s' % locals() conn = pyodbc.connect(connect_string, autocommit=True) cursor = conn.cursor() cursor.execute(sql) for row in cursor: print row except Exception as e: print e finally: cursor.close() conn.close() ########################################################### # MAIN ########################################################### if __name__ == '__main__': args = parse_args() execute_query('show databases', args.dsn, args.user, args.password) ... View more

I am trying to get ODBC via Knox working on CentOS 7 using this Hortonworks ODBC driver (which support said is the right one for my Linux OS version): hive-odbc-native-2.1.4.1004-1.el7.x86_64.rpm After installing that, I am able to use this driver to connect directly to Hive using a DSN string like this in odbc.ini: [testdsn] Description=Test Driver=/usr/lib/hive/lib/native/Linux-amd64-64/libhortonworkshiveodbc64.so Host=sn01.qa.quasar.local:2181 Schema=default ServiceDiscoveryMode=1 ZKNamespace=/hiveserver2 HiveServerType=2 AuthMech=3 ThriftTransport=2 Port=10001 HttpPath=cliservice And I can test access like this and all works well. I can also use this via pyodbc and that works too. [mpetronic@mpws ~]$ echo "show tables" | isql testdsn mpetronic anypwd +---------------------------------------+ | Connected! | | | | sql-statement | | help [tablename] | | quit | | | +---------------------------------------+ SQL> show tables +---------------------------------------+ | tab_name | +---------------------------------------+ | test | +---------------------------------------+ SQLRowCount returns -1 1 rows fetched Now, I need to make this work via Knox. I have read everything I could possibly find on setting up Hive access via ODBC via Knox. Most postings are about Windows. I need help with Linux configuration. There does not seem to be any clear documentation on how to setup this up on Linux. Can anyone provide the actual, specific odbc.ini settings or DSN you are using to connect to Hive via Knox from a Linux client over ODBC? Here is the connect string that I can use to directly connect to Hive via Knox without ODBC. Just cannot figure out how to make it work via ODBC. beeline -u "jdbc:hive2://api01.qa:8443/;ssl=true;sslTrustStore=/home/mpetronic/keystore/keystore.jks;trustStorePassword=password;transportMode=http;httpPath=quasar/jupstats/hive" -n mpetronic -w ~/pw.txt ... View more

We have lots of partitioned tables and need to write queries that have partition clauses that include year, month, day values that are not as simple as: where year=2016 and month=8 and day between 7 and 14 Often, they require non contiguous ranges of days over different months and years, etc. So, I am trying to come up with a way to help users craft those partitions clauses more easily and programatically (to the greatest extent possible). I don't want users to have to write wrapper scripts around queries or have to write queries into files first as they can do something like this: beeline 'connect-string' --hivevar part_string=$(./make_part.py 2015-12-25 2016-01-07) -f some.hql Where make_part.py might be a Python script that takes two dates and forms the full partition clause string that can then be simply referenced in some.hql, like this, which I know would work: select * from table where ${hivevar:part_string}; What I would like to do is something "functionally" like this but from within beeline so users can work more interactively in the beeline shell. For example, I wish you could do this from within beeline: set hivevar:part_string=!sh ./make_part.py 2015-12-25 2016-01-07 And have the output of the !sh command become the value of the hive variable. That does not work, of course. So, I was wondering, is it possible to create a UDF that could be used in a WHERE clause that could return the partition clause string which would get evaluated by Hive properly and work as expected - meaning, proper partition pruning. Something like this: select * from table where udf_make_part("2015-12-25", "2016-01-07"); Where udf_make_part would do the same thing as make_part.py - take some date arguments and return some generated partition string. I've not worked with UDFs so far but just wondering if they could be used in this context in the WHERE clause. Or, does anyone have another useful approach for dealing with long, complicated partition clauses? ... View more

I'm setting up a tiny pseudo-distributed cluster for testing. I have only one datanode. I works and I can load and query data in Hive. Great. Then I changed the HDFS replication factor from 3 to 1, and also changed the max replication factor to 1, too. I restarted all HDFS, YARN, and MR processes (that's all that Ambari indicated that needed restarted). Now, when I run a Hive query, I see this in the hadoop-hdfs-namenode-log 2016-05-19 15:26:03,322 INFO ipc.Server (Server.java:run(2172)) - IPC Server handler 103 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 172.19.64.3:57450 Call#27745 Retry#0java.io.IOException: file /tmp/hive/mpetronic/_tez_session_dir/5e16aba9-d9d3-4138-afda-58c1d0027e4d/hive-hcatalog-core.jar on client 172.19.64.3.Requested replication 3 exceeds maximum 1 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.verifyReplication(BlockManager.java:988) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2374) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:688) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145) And my query fails with this... 0: jdbc:hive2://mpws:10000/default> select count(device_id) from vsat_modc_batch where device_id like 'DSS%' limit 10; INFO : Tez session hasn't been created yet. Opening session ERROR : Failed to execute tez graph. java.io.IOException: Previous writer likely failed to write hdfs://mpws:8020/tmp/hive/mpetronic/_tez_session_dir/9216a3de-e9fc-4940-a4b3-21e49bcca4b5/hive-hcatalog-core.jar. Failing because I am unlikely to write too. at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:982) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:862) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:805) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:233) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:158) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1) So, it seems that something did not get the memo about the reduced replication factor. Did I miss some additional configuration needed to run at replication factor = 1? I reconfigured back to 3 and everything works again. I could not find anything indicating I need to configure something in the hive or tez clients that is related to replication. ... View more

HDP 2.3.4/Ambari 2.2.1.1 I am having some trouble understanding how the various Ranger users are used and where to configure the passwords. What I believe to be true (based on some other posts on Ranger in this forum), is: Ranger admin user - I understand that the password for this has to be the same in Ranger Admin UI and under Ranger Advanced config for the field "admin_password" in the Advanced ranger-env configuration Ranger amb_ranger_admin user - I understand that the password for this has to be the same in Ranger Admin UI and under Ranger Advanced config for the field "Ranger Admin username for Ambari" in the Advanced Ranger Admin configuration Ranger rangerusersync user - This one seems to be my problem. It appears, based on the error trace below, that the password for this user needs to be configured somewhere but is not correct and I cannot figure out where I need to make changes. Note that I have Ranger configured for Unix sync mode. Ranger admin and usersync processes are running on host api01 and my two HA name nodes are running on host nn01 and nn02. So, what I am looking for is two things: An explanation of how these three users are used to make Ranger operate Should the usersync process be running on one of the name nodes instead of a different host? I believe that I only need to configure users and groups on the two name nodes and that other services, like Hive, rely on the equivalent of "hdfs groups <user>" to obtain group information and that that information comes from the Linux OS configured user/group information on the namenode. Since I am running usersync on a non-NN host, I am thinking this might be wrong. Some help in solving the error related to rangerusersync, namely this one (full traces below): 03 May 2016 06:04:41 INFO PasswordValidator [Thread-8] - Response [FAILED: [rangerusersync] does not exists.] for user: rangerusersync 03 May 2016 06:04:09 INFO UnixAuthenticationService [main] - Starting User Sync Service! 03 May 2016 06:04:09 INFO UnixAuthenticationService [main] - Enabling Unix Auth Service! 03 May 2016 06:04:09 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder 03 May 2016 06:04:10 WARN NativeCodeLoader [main] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 03 May 2016 06:04:10 INFO UnixAuthenticationService [main] - Enabling Protocol: [SSLv2Hello] 03 May 2016 06:04:10 INFO UnixAuthenticationService [main] - Enabling Protocol: [TLSv1] 03 May 2016 06:04:10 INFO UnixAuthenticationService [main] - Enabling Protocol: [TLSv1.1] 03 May 2016 06:04:10 INFO UnixAuthenticationService [main] - Enabling Protocol: [TLSv1.2] 03 May 2016 06:04:41 INFO PasswordValidator [Thread-8] - Response [FAILED: [rangerusersync] does not exists.] for user: rangerusersync 03 May 2016 06:04:41 ERROR UserGroupSync [UnixUserSyncThread] - Failed to initialize UserGroup source/sink. Will retry after 60000 milliseconds. Error details: com.sun.jersey.api.client.UniformInterfaceException: GET http://api01.qa.quasar.local:6080/service/xusers/groups/?pageSize=1000&startIndex=0 returned a response status of 401 Unauthorized at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:507) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildGroupList(PolicyMgrUserGroupBuilder.java:346) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildUserGroupInfo(PolicyMgrUserGroupBuilder.java:156) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.init(PolicyMgrUserGroupBuilder.java:152) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:51) at java.lang.Thread.run(Thread.java:745) Thanks in advance... ... View more

I an doing a clean install of HDP 2.3.4.7 using Ambari 2.2.1.1. By default, the HDP repo URL does not point to the latest version of HDP right now, which is 2.3.4.7. I conferred with @Neeraj Sabharwal to determine if it is feasible to change that URL to 2.3.4.7 and directly do the clean install of that latest version to avoid the need to install the earlier release of 2.3.4 then immediately have to upgrade. He said yes, so that's what I did. I encountered errors in the final step of the install at the "Install, Start, and Test" step where all the services are installed on the nodes. I was seeing errors from the scripts trying obtain the HDP versions so I was wondering if my URL tweak was actually causing a problem. In the end it was not. Here is a sample error trace: stderr: <script id="metamorph-39638-start" type="text/x-placeholder"></script>Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 37, in <module> AfterInstallHook().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 31, in hook setup_hdp_symlinks() File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py", line 44, in setup_hdp_symlinks hdp_select.select_all(version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hdp_select.py", line 122, in select_all Execute(command, only_if = only_if_command) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^2.3 | tail -1`' returned 1. Traceback (most recent call last): File "/usr/bin/hdp-select", line 378, in <module> printVersions() File "/usr/bin/hdp-select", line 235, in printVersions result[tuple(map(int, versionRegex.split(f)))] = f ValueError: invalid literal for int() with base 10: 'hadoop' ERROR: set command takes 2 parameters, instead of 1 usage: hdp-select [-h] [<command>] [<package>] [<version>] Set the selected version of HDP. positional arguments: <command> One of set, status, versions, or packages <package> the package name to set <version> the HDP version to set optional arguments: -h, --help show this help message and exit -r, --rpm-mode if true checks if there is symlink exists and creates the symlink if it doesn't Commands: set : set the package to a specified version status : show the version of the package versions : show the currently installed versions packages : show the individual package names<script id="metamorph-39638-end" type="text/x-placeholder"></script> Being a Python guy, I dug into /usr/bin/hdp-select and found where the error was occurring. There are two issues: This function is not very bullet proof and prone to errors The reason for the error is an unexpected directory exists in /usr/hdp where this function is looking to parse installed versions The function in question is: # Print the installed packages def printVersions(): result = {} for f in os.listdir(root): if f not in [".", "..", "current", "share", "lost+found"]: result[tuple(map(int, versionRegex.split(f)))] = f keys = result.keys() keys.sort() for k in keys: print result[k] The problem is that if any but the excluded directories are found on the scan, this function will raise an exception where it tries to map the regex output to an int(), where the output is a non-numeric string - like the value "hadoop". On some of my nodes, the /usr/hdp directory looks like this (which does NOT result in this function throwing an exception): [nn01.qa] out: drwxr-xr-x. 13 root root 4096 Apr 23 07:09 2.3.4.7-4 [nn01.qa] out: drwxr-xr-x. 2 root root 4096 Apr 23 16:32 current [nn01.qa] out: drwx------. 2 root root 16384 Apr 19 16:09 lost+found But others look like this (note presence of unexpected hadoop directory that was causing the exception): [nn02.qa] out: drwxr-xr-x. 13 root root 4096 Apr 23 07:09 2.3.4.7-4 [nn02.qa] out: drwxr-xr-x. 2 root root 4096 Apr 23 16:32 current [nn02.qa] out: drwxr-xr-x. 4 root root 4096 Apr 23 16:12 hadoop [nn02.qa] out: drwx------. 2 root root 16384 Apr 19 16:09 lost+found I fixed the problem by changing that function code as follows (adding exception handling), copying this update to all my nodes, retrying the install. It completed successfully after that. # Print the installed packages def printVersions(): result = {} for f in os.listdir(root): if f not in [".", "..", "current", "share", "lost+found"]: try: result[tuple(map(int, versionRegex.split(f)))] = f except: pass keys = result.keys() keys.sort() for k in keys: print result[k] I don't recommend this code as a final fix. Instead of a black list condition, I would get rid of that code an use a new regex filter to verify that each directory looked like "^\d+\.\d+.*" or something like that which requires it to at least look like some form of the expected version format of 2.3.4.7-n. Then it does not matter how many unexpected entries you find - you will just gracefully skip over them. One interesting point about this bug... The first time I ran the "Install, Start, and Test" step in the Ambari GUI, all 8 nodes failed, one with a red failure and the others orange and stopped due to warnings. I guess when one fails, the others seem to abort as well with warnings. Just for the heck of it, I then clicked the "Retry" button and one node completed (turned blue) but another failed and the rest also stopped with warnings. I though that was weird so I keep retrying over and over. I got to the point where 4 of the 8 turned blue but then the rest would never complete no matter how many times I retried. When I looked at the contents of all the nodes that made it to blue, they did NOT have "hadoop" subdir in /usr/hdp so that is why the function above did not raise an exception. I seems like, at some stage of the install, /usr/hdp/hadoop gets created when certain applications need to be installed there and, if you have to retry after that, you are dead. However, after my fixed code was deployed and I retried, all nodes completely installed and EVERY node now has a /usr/hdp/hadoop directory. @Ryan Chapin ... View more

Just installed Ranger on 2.4.0 to start experimenting with it. Basically got it working and was starting to play around with making policy changes and seeing how they impact access to prove it works as I anticipate. By default, it created three services names nadcluster_hive, nadcluster_hdfs, and nadcluster_yarn because my cluster name is nadcluster, I suppose. So, I decided to rename these to be more reflective of the service. I renamed nadcluster_hive to jupstats_hive. I started to see errors in the hiveserver2 log like below. Seems the hiveserver2 ranger plugin is still trying to download policies data from a URL like: http://vmwhaddev01:6080/service/plugins/policies/download/nadcluster_hive When it should be trying against the newly named service like: http://vmwhaddev01:6080/service/plugins/policies/download/jupstats_hive And, in fact, I manually hit the REST end point from a browser with a URL like this and did get a proper response: http://vmwhaddev01:6080/service/plugins/policies/download/jupstats_hive?lastKnownVersion=4&pluginId=hiveServer2@vmwhaddev01-jupstats_hive Here's the error in the log: 2016-04-22 04:38:17,290 ERROR [Thread-9]: client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(81)) - Error getting policies. request=http://vmwhaddev01:6080/service/plugins/policies/download/nadcluster_hive?lastKnownVersion=4&pluginId=hiveServer2@vmwhaddev01-nadcluster_hive, response={"httpStatusCode":400,"statusCode":1,"msgDesc":"Serivce:nadcluster_hive not found","messageList":[{"name":"DATA_NOT_FOUND","rbKey":"xa.error.data_not_found","message":"Data not found"}]}, serviceName=nadcluster_hive 2016-04-22 04:38:17,290 ERROR [Thread-9]: util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(228)) - PolicyRefresher(serviceName=nadcluster_hive): failed to refresh policies. Will continue to use last known version of policies (4) java.lang.Exception: Serivce:nadcluster_hive not found at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:83) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:205) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:175) at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:154) And here is when it started working after I renamed the service back to what it was looking for: 2016-04-22 04:38:47,375 INFO [Thread-9]: util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(218)) - PolicyRefresher(serviceName=nadcluster_hive): found updated version. lastKnownVersion=4; newVersion=18 Questions: What has to be restarted when you change something in a Ranger policy via that Ranger GUI? I restarted Ranger admin and usersync and hiveserver2, but the wrong service name was still being used in the plugin URL? Where is the hiveserver2 Ranger plugin learning this URL in the first place [UPDATE] Ok, I spent some more time playing around. I figured out that the policy configuration is located here (noting the fact that the service name "nadcluster_hive" is in path and filename: /etc/ranger/nadcluster_hive/policycache/hiveServer2_nadcluster_hive.json I performed some testing. With the service name in Ranger UI set to "nadcluster_hive", I made various changes to one of the policies, like adding a new user, enabling/disabling table or column permissions, etc. I tailed the json file above and, every time I made a change and saved, within 30 seconds, I would see the json file be rewritten with the updates. Cool. That seems right. Next, I renamed the service to nadcluster_hive_1 and repeated the tests. The json file never once changed - as expected because the wrong REST URL is being used. But, I would have expected that maybe a brand new json file with the new service name would have appeared with a path like: /etc/ranger/nadcluster_hive_1/policycache/hiveServer2_nadcluster_hive_1.json But, it never did. So, is this expected behavior or a bug? ... View more