Member since
02-29-2016
108
Posts
213
Kudos Received
14
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1027 | 08-18-2017 02:09 PM | |
2182 | 06-16-2017 08:04 PM | |
1649 | 01-20-2017 03:36 AM | |
4209 | 01-04-2017 03:06 AM | |
1927 | 12-09-2016 08:27 PM |
08-18-2017
02:26 PM
2 Kudos
I did created a truststore for queue manager view. But I believe although the truststore is located on Ambari server, by importing Ambari HTTPS cert to the store it is actually used by Ambari views to connect to Ambari HTTPS server. It is not really for other client like SAM.
... View more
08-18-2017
02:09 PM
2 Kudos
I am able to make it work by adding Ambari cert to SAM node's Java truststore at JAVA_HOME/jre/lib/security/cacerts Since the document did not mentioned it, I wonder if this is the standard practice or there is a truststore for SAM itself.
... View more
08-17-2017
07:47 PM
2 Kudos
Try to create a service pool in SAM from a secured cluster with secured Ambari server. Ambari HTTPS is done using self generated certificates. The webUI could be access with following url https://[AMBARI_HOST]:8443/#/main/dashboard/metrics When putting the following url in creating service pool WebUI in SAM https://[AMBARI_HOST]:8443/api/v1/clusters/[AMBARI_CLUSTER_NAME] javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Is there a truststore location for SAM or should I just import the cert to Java truststore?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Certification
-
Security
06-16-2017
08:04 PM
5 Kudos
Found the reason. I was following an older version which has the schema registered with name postfix ":v", it apparently is not needed any more. So after register the schema again with name as the kafka topic, everything works fine.
... View more
06-16-2017
06:28 PM
3 Kudos
Following the instruction under link below to build my first SAM applicaiton https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_getting-started-with-stream-analytics/content/ch_building-end-to-end-app.html I kept getting error with the kafka source added with error message as "Schema not found" While looking at the SAM log, I got 404 error INFO [18:14:55.230] [dw-95 - GET /api/v1/schemas/truck_events_avro] c.h.s.s.s.SchemaResource - Received path: [truck_events_avro]
ERROR [18:14:55.269] [dw-95 - GET /api/v1/schemas/truck_events_avro] c.h.s.s.s.SchemaResource - Got exception: [RuntimeException] / message [javax.ws.rs.NotFoundException: HTTP 404 Not Found] / related resource location: [com.hortonworks.streamline.streams.service.SchemaResource.getKafkaSourceSchema](SchemaResource.java:131) Where on Registry log, it seems that the schema SAM try to get does not exist, the request did get through but no schema exist with that schema key INFO [18:14:55.260] [dw-19350 - GET /api/v1/schemaregistry/schemas/truck_events_avro/versions/latest] c.h.r.s.w.SchemaRegistryResource - No schemas found with schemakey: [truck_events_avro]
I did create all the schema in SchemaRegistry following the steps in the guide and I can see all the schema from the dash board on http://REGISTRY_SERVER:7788/ However, when try to get the schema with API from http://REGISTRY_SERVER:7788/api/v1/schemaregistry/schemas/truck_events_avro/versions/latest i got the same error like in registry.log {"responseCode":1101,"responseMessage":"Entity with id [truck_events_avro] not found."}
Any suggestions on how to fix the probelm?
... View more
Labels:
- Labels:
-
Schema Registry
05-18-2017
12:42 PM
1 Kudo
@dvillarreal Problem fixed with you suggestion. This looks like a defect for Ambari/Atlas, will this get addressed in next minor release? I know not many people use ldap, but still it is quite annoying to escape all the commas, and it is not needed anywhere else with similar setting.
... View more
05-18-2017
12:42 PM
3 Kudos
I stand up a test cluster with HDP2.5.3 and use a self created openLDAP server for credential. Cluster is kerberized with MIT KDC. I could log into ranger with LDAP ids ok, but atlas always returns me a error. log says "LDAP: error code 49 - Invalid Credentials". ldapserach returns the ids fine on atlas node What this error message means, is the ldap could not find the id or the user/pwd is wrong? here is my config in atlas atlas.authentication.method.ldap.type: LDAP atlas.authentication.method.ldap.url: ldap://qwang-kdc-ldap.field.hortonworks.com:389 atlas.authentication.method.ldap.userDNpattern: uid={0},ou=Users,dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.groupSearchBase: ou=Groups,dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.groupSearchFilter: (cn={0}) atlas.authentication.method.ldap.groupRoleAttribute: cn atlas.authentication.method.ldap.base.dn: dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.bind.dn: cn=admin,dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.bind.password: password atlas.authentication.method.ldap.referral: ignore atlas.authentication.method.ldap.user.searchfilter: (uid={0}) atlas.authentication.method.ldap.default.role: ROLE_USER And ldapsearch works fine ldapsearch -W -H ldap://qwang-kdc-ldap.field.hortonworks.com:389 -D "cn=admin,dc=field,dc=hortonworks,dc=com" -b "ou=Groups,dc=field,dc=hortonworks,dc=com" "(cn=sales)"
ldapsearch -W -H ldap://qwang-kdc-ldap.field.hortonworks.com:389 -D "cn=admin,dc=field,dc=hortonworks,dc=com" -b "dc=field,dc=hortonworks,dc=com" "(uid=sales1)"
What else could I look into for the problem?
... View more
Labels:
- Labels:
-
Apache Atlas
04-21-2017
12:18 PM
17 Kudos
With the release of HDP 2.6 comes the Hive ACID Merge feature. This is long sought after feature for people who want to batch update table content in ACID manner. This tutorial will walk you through step-by-step how to use this new feature with some really simple dataset. The example is done on HDP 2.6.0.3 installation. First, ACID in Hive need to be turned on in Ambari UI Then we will create 2 tables, one as the target of merge and one as the source of merge. Please note that the target table must be bucketed, set as transaction enabled and stored in orc format. CREATE DATABASE merge_data;
CREATE TABLE merge_data.transactions(
ID int,
TranValue string,
last_update_user string)
PARTITIONED BY (tran_date string)
CLUSTERED BY (ID) into 5 buckets
STORED AS ORC TBLPROPERTIES ('transactional'='true');
CREATE TABLE merge_data.merge_source(
ID int,
TranValue string,
tran_date string)
STORED AS ORC;
Then we will populate the target and source table with some data. INSERT INTO merge_data.transactions PARTITION (tran_date) VALUES
(1, 'value_01', 'creation', '20170410'),
(2, 'value_02', 'creation', '20170410'),
(3, 'value_03', 'creation', '20170410'),
(4, 'value_04', 'creation', '20170410'),
(5, 'value_05', 'creation', '20170413'),
(6, 'value_06', 'creation', '20170413'),
(7, 'value_07', 'creation', '20170413'),
(8, 'value_08', 'creation', '20170413'),
(9, 'value_09', 'creation', '20170413'),
(10, 'value_10','creation', '20170413');
INSERT INTO merge_data.merge_source VALUES
(1, 'value_01', '20170410'),
(4, NULL, '20170410'),
(7, 'value_77777', '20170413'),
(8, NULL, '20170413'),
(8, 'value_08', '20170415'),
(11, 'value_11', '20170415');
When we examine the 2 tables, we would expect after the merge, row 1 should be left untouched, row 4 should be deleted (we imply a business rule here: NULL value indicates deletion), row 7 will be update and row 11 will be insertion for new value. The more interesting use case is with row 8, where it involves move the row from one partition to another. Merge currently does not support change of partition value on the fly. This need to happen as delete in the old partition and insert in the new partition. In real world use case, you need to construct the source table base on this critiria. Then we will create the merge statement as the following. Please note that not all 3 WHEN of the merge statements need to exist, it is fine to have only 2 or even 1 of the WHEN statement. We labeled the data with different last_update_user. For more details on Hive Merge, please refer to Hive document MERGE INTO merge_data.transactions AS T
USING merge_data.merge_source AS S
ON T.ID = S.ID and T.tran_date = S.tran_date
WHEN MATCHED AND (T.TranValue != S.TranValue AND S.TranValue IS NOT NULL) THEN UPDATE SET TranValue = S.TranValue, last_update_user = 'merge_update'
WHEN MATCHED AND S.TranValue IS NULL THEN DELETE
WHEN NOT MATCHED THEN INSERT VALUES (S.ID, S.TranValue, 'merge_insert', S.tran_date);
As part of the update clause, the set value statement should not contain the target table decorator "T.", otherwise you will get SQL compile error. Once the merge finished, re-examining the data shows the data is merged just as expected row 1 wasn't changed ; row 4 was deleted; row 7 was updated and row 11 was inserted. And row 8, as we see, got moved to a new partition. SELECT * FROM merge_data.transactions order by ID;
+----+-----------------------+------------------------------+-----------------------+
| id | transactions.tranvalue| transactions.last_update_user| transactions.tran_date|
+----+-----------------------+------------------------------+-----------------------+
| 1 | value_01 | creation | 20170410 |
| 2 | value_02 | creation | 20170410 |
| 3 | value_03 | creation | 20170410 |
| 5 | value_05 | creation | 20170413 |
| 6 | value_06 | creation | 20170413 |
| 7 | value_77777 | merge_update | 20170413 |
| 8 | value_08 | merge_insert | 20170415 |
| 9 | value_09 | creation | 20170413 |
| 10 | value_10 | creation | 20170413 |
| 11 | value_11 | merge_insert | 20170415 |
+----+-----------------------+------------------------------+-----------------------+
This simple example provides instructions on how to use ACID merge with HDP 2.6 or later. Of course the real world use case would be much more complicated comparing to this oversimplified example, but they all follow the same principles. And the beauty behind this is, you don't need to create a ETL process to accomplish this any more.
... View more
- Find more articles tagged with:
- Data Processing
- hdp-2.6.0
- Hive
- How-ToTutorial
- merge
02-03-2017
07:11 PM
Michal, my problem was related to KMS proxy user and addressed by adding livy as proxy user. My cluster was secured with Kerberos and with Ranger and KMS. I am not sure if your is similar. You may want to start a new thread with more details.
... View more
02-03-2017
06:53 PM
2 Kudos
So here are the values that work in my environment nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.pattern.kerb=^(.*?)@(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.value.kerb=$1
Also in Ranger the Nifi nodes need to be added as internal user and create policy for them to access proxy, flow and data
... View more
02-03-2017
05:32 PM
3 Kudos
I have a test environment with OpenLDAP and MIT KDC as backend directory services. I tried to use it to test Nifi authorization through Ranger and running into an issue where the user name seems to not matching correctly. Here is my setup HDF 2.1.1.0, Nifi 1.1.0 and Ranger 0.6.2 Cluster installed with all HDF components except Storm and kafka Cluster Kerberized with MIT KDC Credentials in OpenLDAP Ranger sync with OpenLDAP Ranger Nifi policy created for a user with all permissions. I could get to Nifi login page and login with the credentials from OpenLDAP, but then it complains about not have enough access Looking at the audit log, the user name get logged in Ranger is hadoopadmin@FIELD.HORTONWORKS.COM rather than hadoopadmin, it seems the KDC principal name get used here I haven't setup identity mapping and the values are empty now. What values should I use to get the username mapped correctly? Thanks,
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Ranger
01-20-2017
03:33 AM
@Sergey Soldatov Add it in ambari at the end of "Advanced zeppelin-env" -> "zeppelin_env_content" worked perfectly.
... View more
01-19-2017
03:06 AM
2 Kudos
@Sergey Soldatov It works after following your steps. It would be nice to figure out how to do step 3 in ambari if possible, always worry some later ambari update would wipe the change out.
thanks a lot for your help!
... View more
01-18-2017
11:09 PM
2 Kudos
@dvillarreal I read your comments on https://community.hortonworks.com/articles/38348/ranger-is-not-allowing-access-to-knox-resources-wh.html If you look at my topology, it contains the group section just like in your post. The only difference I could think of is that I use open-ldap as the directory server rather than AD. And the values for object class and attributes are different than AD. <param>
<name>main.ldapRealm.authorizationEnabled</name>
<value>true</value>
</param>
<param>
<name>main.ldapRealm.groupSearchBase</name>
<value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
<name>main.ldapRealm.groupObjectClass</name>
<value>posixgroup</value>
</param>
<param>
<name>main.ldapRealm.groupIdAttribute</name>
<value>cn</value>
</param>
... View more
01-18-2017
10:59 PM
@Sergey Soldatov Are you running into the same problem on a secured cluster or non-secured one?
... View more
01-18-2017
10:58 PM
2 Kudos
@Josh Elser HBase is up and running fine. I can create table in both hbase shell and sqlline.py hbase(main):001:0> create 'my_table1', {NAME =>'cf1'}, {NAME =>'cf2'}
0 row(s) in 2.9560 seconds
=> Hbase::Table - my_table1
hbase(main):002:0> put 'my_table1', 'rowkey01', 'cf1:c1', 'test value'
0 row(s) in 0.6850 seconds
hbase(main):003:0> get 'my_table1', 'rowkey01'
COLUMN CELL
cf1:c1 timestamp=1484780190152, value=test value
1 row(s) in 0.0780 seconds
... View more
01-18-2017
08:28 PM
@Josh Elser HBase is running fine. I cold pull the table list and look into "atlas_titan" tables content [root@qwang-hdp5 logs]# hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.1.2.2.5.3.0-37, rcb8c969d1089f1a34e9df11b6eeb96e69bcf878d, Tue Nov 29 18:48:22 UTC 2016
hbase(main):001:0> list
TABLE
ATLAS_ENTITY_AUDIT_EVENTS
SYSTEM.CATALOG
SYSTEM.FUNCTION
SYSTEM.SEQUENCE
SYSTEM.STATS
atlas_titan
driver_dangerous_event
my_table
8 row(s) in 0.5710 seconds
hbase(main):003:0> scan 'atlas_titan'
... View more
01-18-2017
08:23 PM
@lmccay As you mentioned, the log does indicate the group search is no returning the right group 17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|
17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|Groups: []
How do I config knox to do group lookup? Don't see anything in my topology related to group lookup <topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<!-- changes for AD/user sync -->
<param>
<name>main.ldapContextFactory</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>
<!-- main.ldapRealm.contextFactory needs to be placed before other main.ldapRealm.contextFactory* entries -->
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
<!-- AD url -->
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://qwang-kdc-ldap.field.hortonworks.com:389</value>
</param>
<!-- system user -->
<param>
<name>main.ldapRealm.contextFactory.systemUsername</name>
<value>cn=admin,dc=field,dc=hortonworks,dc=com</value>
</param>
<!-- pass in the password using the alias created earlier -->
<param>
<name>main.ldapRealm.contextFactory.systemPassword</name>
<value>password</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
<!-- AD groups of users to allow -->
<param>
<name>main.ldapRealm.searchBase</name>
<value>ou=Users,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
<name>main.ldapRealm.userObjectClass</name>
<value>person</value>
</param>
<param>
<name>main.ldapRealm.userSearchAttributeName</name>
<value>uid</value>
</param>
<!-- changes needed for group sync-->
<param>
<name>main.ldapRealm.authorizationEnabled</name>
<value>true</value>
</param>
<param>
<name>main.ldapRealm.groupSearchBase</name>
<value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
<name>main.ldapRealm.groupObjectClass</name>
<value>posixgroup</value>
</param>
<param>
<name>main.ldapRealm.groupIdAttribute</name>
<value>cn</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url>
</service>
<service>
<role>JOBTRACKER</role>
<url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
</service>
<service>
<role>WEBHDFS</role>
<url>http://{{namenode_host}}:{{namenode_http_port}}/webhdfs</url>
</service>
<service>
<role>WEBHCAT</role>
<url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
</service>
<service>
<role>OOZIE</role>
<url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://{{hbase_master_host}}:{{hbase_master_port}}</url>
</service>
<service>
<role>HIVE</role>
<url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://{{rm_host}}:{{rm_port}}/ws</url>
</service>
</topology>
... View more
01-18-2017
03:58 PM
2 Kudos
HDP 2.5 secured cluster with Zeppelin and HBase installed. I could connect to Phoenix using the sqlline.py utility using the following parameter. /usr/hdp/current/phoenix-client/bin/sqlline.py <zk1>,<zk2>,<zk3>:2181:/hbase-secure:hbase@DOMAIN.COM:/etc/security/keytabs/hbase.headless.keytab
However when I try to use similar parameter for jdbc conn string in phoenix, I got null
org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=1, exceptions:
Wed Jan 18 15:13:11 UTC 2017, RpcRetryingCaller{globalStartTime=1484752390908, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Broken pipe
class org.apache.zeppelin.interpreter.InterpreterException
The jdbc interpreter settings for Phoenix are Do I need to create phoenixuser principal and use that keytab instead? and what is phoenix.user and phoenix.password?
... View more
Labels:
- Labels:
-
Apache Phoenix
-
Apache Zeppelin
01-18-2017
03:38 PM
2 Kudos
HDP 2.5 secured cluster with Knox installed as gateway. Ranger policy is create for default topology and WEBHDFS service. However when the policy only contains only hr group which hr1 user is part of, I got error from gateway request curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 Forbidden</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /gateway/default/webhdfs/v1/hr/exempt. Reason:
<pre> Forbidden</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
When user hr1 is included in the policy, it works fine curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS
{"FileStatuses":{"FileStatus":[{"accessTime":1483632050751,"blockSize":134217728,"childrenNum":0,"fileId":152421,"group":"hr","length":23,"modificationTime":1483632051087,"owner":"hdfs","pathSuffix":"testfile","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"}]}} This happens with other knox services as well like hive through knox. The group policy does work fine on other Ranger policies not for knox, like HDFS and Hive. Only the policies for Knox seems to have this particular problem. And in HDFS custom core-site, I have hadoop.proxyuser.knox.hosts=*
hadoop.proxyuser.knox.groups=*
... View more
Labels:
- Labels:
-
Apache Knox
-
Apache Ranger
01-05-2017
08:01 PM
4 Kudos
Found the answer in another HCC post https://community.hortonworks.com/questions/16887/beeline-connect-via-knox-ssl-issue.html The truststore and password of truststore need to be included in the hive conn string. beeline -u "jdbc:hive2://knoxnode:8443/;ssl=true;sslTrustStore=/root/myLocalTrustStore.jks;trustStorePassword=<password>;transportMode=http;httpPath=gateway/default/hive" -n hadoopadmin -p <password> The step is like the following. First get the certification from Knox server, then add the cert to a key file that will be used by the client, keep the keypassword handy, that is the password for the connection string. You don't have to add the cert in the java key file. openssl s_client -connect ${knoxserver}:8443 <<<'' | openssl x509 -out /tmp/knox.crt
keytool -import -keystore myLocalTrustStore.jks -file knox.crt
... View more
01-05-2017
07:19 PM
3 Kudos
@dvillarreal So I used SSLPoke to connect to knox server and it works fine [root@qwang-hdp2 sslpoke]# java -Djavax.net.ssl.trustStore=/usr/jdk64/jdk1.8.0_77/jre/lib/security/cacert SSLPoke qwang-hdp5.field.hortonworks.com 8443
Successfully connected
I guess the question now is what truststore beeline is using. Where can I get that information?
... View more
01-05-2017
05:03 PM
2 Kudos
@dvillarreal I followed your steps, the java version is the same as you mentioned [root@qwang-hdp2 ~]# ps -ef | grep beeline
root 32310 26098 39 16:58 pts/0 00:00:05 /usr/jdk64/jdk1.8.0_77/bin/java -Xmx1024m -Dhdp.version=2.5.3.0-37 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.5.3.0-37 -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.5.3.0-37/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.5.3.0-37/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.5.3.0-37/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m -Djava.util.logging.config.file=/usr/hdp/2.5.3.0-37/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.5.3.0-37/hive/lib/hive-beeline-1.2.1000.2.5.3.0-37.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://qwang-hdp5.field.hortonworks.com:8443/;ssl=true;transportMode=http;httpPath=gateway/default/hive -n hadoopadmin -p password
root 32542 10102 0 16:58 pts/1 00:00:00 grep --color=auto beeline
then import the certificate to /usr/jdk64/jdk1.8.0_77/jre/lib/security/cacert keytool -import -trustcacerts -keystore /usr/jdk64/jdk1.8.0_77/jre/lib/security/cacert -storepass changeit -noprompt -alias knox -file /tmp/knox.crt
But I still get the same error starting beeline beeline -u "jdbc:hive2://knoxnode:8443/;ssl=true;transportMode=http;httpPath=gateway/default/hive" -n hadoopadmin -p password
17/01/05 16:58:46 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
You mentioned restart beeline client, I am not sure what exactly that means.
... View more
01-05-2017
03:31 PM
2 Kudos
@Vipin Rathor Just included the beeline error in original message. I tailed /var/log/hive/hiveserver2.log and did not see any error with beeline command pop errors. From the beeline error, it seems to be related to ssl, that was why I included the cmd I import cert on both nodes.
... View more
01-05-2017
03:16 PM
1 Kudo
the intention is to access hive through knox, not directly.
... View more
01-05-2017
03:03 AM
3 Kudos
I am following the security lab and getting to lab 8 for knox. https://github.com/HortonworksUniversity/Security_Labs#lab-8 It was all fine for WEBHDFS steps but I got some error in HIVE step when I use openssl to imprt the self-signed cert, i got some error. not sure if that was the cause knoxserver=$(hostname -f)
openssl s_client -connect ${knoxserver}:8443 <<<'' | openssl x509 -out /tmp/knox.crt
depth=0 C = US, ST = Test, L = Test, O = Hadoop, OU = Test, CN = qwang-hdp5.field.hortonworks.com
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = Test, L = Test, O = Hadoop, OU = Test, CN = qwang-hdp5.field.hortonworks.com
verify return:1
DONE
On beeline node, the cert was imported fine with the following command keytool -import -trustcacerts -keystore /etc/pki/java/cacerts -storepass changeit -noprompt -alias knox -file /tmp/knox.crt But when I use the following cmd to access hive, i get error looks like the user was not granted access in Ranger, but the user is already include in "all - topology, service" policy beeline -u "jdbc:hive2://knoxnode:8443/;ssl=true;transportMode=http;httpPath=gateway/default/hive" -n hadoopadmin -p password Error from beeline 17/01/05 15:22:19 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at org.apache.hive.service.cli.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:154)
at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:146)
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:552)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:170)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:146)
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:211)
at org.apache.hive.beeline.Commands.connect(Commands.java:1190)
at org.apache.hive.beeline.Commands.connect(Commands.java:1086)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:990)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:715)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:777)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:491)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:474)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:395)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:354)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
... 30 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
... 51 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
... 57 more
Error: Could not establish connection to jdbc:hive2://qwang-hdp5.field.hortonworks.com:8443/;ssl=true;transportMode=http;httpPath=gateway/default/hive: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (state=08S01,code=0)
Looking at the knox log, it complains about error writting to log file 2017-01-05 03:01:24,993 ERROR provider.BaseAuditHandler (BaseAuditHandler.java:logError(329)) - Error writing to log file.
org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2160)
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1443)
at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:273)
at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:44)
at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:159)
at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:156)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:170)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:880)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:828)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:758)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy48.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy49.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)
... 17 more
2017-01-05 03:01:24,995 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(710)) - Error sending logs to consumer. provider=knox.async.multi_dest.batch, consumer=knox.async.multi_dest.batch.hdfs
I couldn't make sense of the error, what does it really mean?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Knox
01-04-2017
03:06 AM
3 Kudos
Finally found the resolution: because I have Ranger KMS installed on this cluster, livy user also need to be added in proxy user for ranger KMS Add following in Ambari => Ranger KMS => custom core site hadoop.kms.proxyuser.livy.hosts=*
hadoop.kms.proxyuser.livy.users=*
... View more
01-04-2017
02:57 AM
2 Kudos
@Bikas really appreciate your detailed explanation I have the following in spark config. Do I need anything else for livy impersonation? livy.impersonation.enabled=true Also inside Ambari, HDFS=> custom core-site, I have hadoop.proxyuser.zeppelin.hosts=*
hadoop.proxyuser.zeppelin.groups=*
hadoop.proxyuser.livy.hosts=*
hadoop.proxyuser.livy.groups=*
Also if I kinit as the same user in console, I could start spark-shell fine [root@qwang-hdp1 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hadoopadmin@FIELD.HORTONWORKS.COM
Valid starting Expires Service principal
01/04/2017 02:41:42 01/05/2017 02:41:42 krbtgt/FIELD.HORTONWORKS.COM@FIELD.HORTONWORKS.COM
[root@qwang-hdp1 ~]# spark-shell
I also noticed in the exception, the authentication actually failed for Ranger KMS if you look at the url. I did install Ranger KMS on the cluster but I did not enable it for any HDFS folder yet. Do I need to add livy user somewhere in KMS?
... View more
01-03-2017
06:30 PM
2 Kudos
Still getting the same error after setting livy.spark.master=yarn-cluster And the log in /var/log/spark/ doesn't have much information 17/01/03 18:16:54 INFO SecurityManager: Changing view acls to: spark
17/01/03 18:16:54 INFO SecurityManager: Changing modify acls to: spark
17/01/03 18:16:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
17/01/03 18:16:54 INFO FsHistoryProvider: Replaying log path: hdfs://qwang-hdp0.field.hortonworks.com:8020/spark-history/local-1483414292027
17/01/03 18:16:54 INFO SecurityManager: Changing acls enabled to: false
17/01/03 18:16:54 INFO SecurityManager: Changing admin acls to:
17/01/03 18:16:54 INFO SecurityManager: Changing view acls to: hadoopadmin
Also I am seeing the following error in livy log, I use hadoopadmin user to log into Zeppelin. What authentication caused this failure? INFO: Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://qwang-hdp4.field.hortonworks.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&doAs=hadoopadmin&renewer=rm%2Fqwang-hdp1.field.hortonworks.com%40FIELD.HORTONWORKS.COM&user.name=livy, status: 403, message: Forbidden
... View more