Member since
02-29-2016
108
Posts
213
Kudos Received
14
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1914 | 08-18-2017 02:09 PM | |
3304 | 06-16-2017 08:04 PM | |
3080 | 01-20-2017 03:36 AM | |
8347 | 01-04-2017 03:06 AM | |
4095 | 12-09-2016 08:27 PM |
08-18-2017
02:26 PM
2 Kudos
I did created a truststore for queue manager view. But I believe although the truststore is located on Ambari server, by importing Ambari HTTPS cert to the store it is actually used by Ambari views to connect to Ambari HTTPS server. It is not really for other client like SAM.
... View more
08-18-2017
02:09 PM
2 Kudos
I am able to make it work by adding Ambari cert to SAM node's Java truststore at JAVA_HOME/jre/lib/security/cacerts Since the document did not mentioned it, I wonder if this is the standard practice or there is a truststore for SAM itself.
... View more
08-17-2017
07:47 PM
2 Kudos
Try to create a service pool in SAM from a secured cluster with secured Ambari server. Ambari HTTPS is done using self generated certificates. The webUI could be access with following url https://[AMBARI_HOST]:8443/#/main/dashboard/metrics When putting the following url in creating service pool WebUI in SAM https://[AMBARI_HOST]:8443/api/v1/clusters/[AMBARI_CLUSTER_NAME] javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Is there a truststore location for SAM or should I just import the cert to Java truststore?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Certification
-
Security
06-16-2017
08:04 PM
5 Kudos
Found the reason. I was following an older version which has the schema registered with name postfix ":v", it apparently is not needed any more. So after register the schema again with name as the kafka topic, everything works fine.
... View more
06-16-2017
06:28 PM
3 Kudos
Following the instruction under link below to build my first SAM applicaiton https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_getting-started-with-stream-analytics/content/ch_building-end-to-end-app.html I kept getting error with the kafka source added with error message as "Schema not found" While looking at the SAM log, I got 404 error INFO [18:14:55.230] [dw-95 - GET /api/v1/schemas/truck_events_avro] c.h.s.s.s.SchemaResource - Received path: [truck_events_avro]
ERROR [18:14:55.269] [dw-95 - GET /api/v1/schemas/truck_events_avro] c.h.s.s.s.SchemaResource - Got exception: [RuntimeException] / message [javax.ws.rs.NotFoundException: HTTP 404 Not Found] / related resource location: [com.hortonworks.streamline.streams.service.SchemaResource.getKafkaSourceSchema](SchemaResource.java:131) Where on Registry log, it seems that the schema SAM try to get does not exist, the request did get through but no schema exist with that schema key INFO [18:14:55.260] [dw-19350 - GET /api/v1/schemaregistry/schemas/truck_events_avro/versions/latest] c.h.r.s.w.SchemaRegistryResource - No schemas found with schemakey: [truck_events_avro]
I did create all the schema in SchemaRegistry following the steps in the guide and I can see all the schema from the dash board on http://REGISTRY_SERVER:7788/ However, when try to get the schema with API from http://REGISTRY_SERVER:7788/api/v1/schemaregistry/schemas/truck_events_avro/versions/latest i got the same error like in registry.log {"responseCode":1101,"responseMessage":"Entity with id [truck_events_avro] not found."}
Any suggestions on how to fix the probelm?
... View more
Labels:
- Labels:
-
Schema Registry
05-18-2017
12:42 PM
1 Kudo
@dvillarreal Problem fixed with you suggestion. This looks like a defect for Ambari/Atlas, will this get addressed in next minor release? I know not many people use ldap, but still it is quite annoying to escape all the commas, and it is not needed anywhere else with similar setting.
... View more
05-18-2017
12:42 PM
3 Kudos
I stand up a test cluster with HDP2.5.3 and use a self created openLDAP server for credential. Cluster is kerberized with MIT KDC. I could log into ranger with LDAP ids ok, but atlas always returns me a error. log says "LDAP: error code 49 - Invalid Credentials". ldapserach returns the ids fine on atlas node What this error message means, is the ldap could not find the id or the user/pwd is wrong? here is my config in atlas atlas.authentication.method.ldap.type: LDAP atlas.authentication.method.ldap.url: ldap://qwang-kdc-ldap.field.hortonworks.com:389 atlas.authentication.method.ldap.userDNpattern: uid={0},ou=Users,dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.groupSearchBase: ou=Groups,dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.groupSearchFilter: (cn={0}) atlas.authentication.method.ldap.groupRoleAttribute: cn atlas.authentication.method.ldap.base.dn: dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.bind.dn: cn=admin,dc=field,dc=hortonworks,dc=com atlas.authentication.method.ldap.bind.password: password atlas.authentication.method.ldap.referral: ignore atlas.authentication.method.ldap.user.searchfilter: (uid={0}) atlas.authentication.method.ldap.default.role: ROLE_USER And ldapsearch works fine ldapsearch -W -H ldap://qwang-kdc-ldap.field.hortonworks.com:389 -D "cn=admin,dc=field,dc=hortonworks,dc=com" -b "ou=Groups,dc=field,dc=hortonworks,dc=com" "(cn=sales)"
ldapsearch -W -H ldap://qwang-kdc-ldap.field.hortonworks.com:389 -D "cn=admin,dc=field,dc=hortonworks,dc=com" -b "dc=field,dc=hortonworks,dc=com" "(uid=sales1)"
What else could I look into for the problem?
... View more
Labels:
- Labels:
-
Apache Atlas
04-21-2017
12:18 PM
17 Kudos
With the release of HDP 2.6 comes the Hive ACID Merge feature. This is long sought after feature for people who want to batch update table content in ACID manner. This tutorial will walk you through step-by-step how to use this new feature with some really simple dataset. The example is done on HDP 2.6.0.3 installation. First, ACID in Hive need to be turned on in Ambari UI Then we will create 2 tables, one as the target of merge and one as the source of merge. Please note that the target table must be bucketed, set as transaction enabled and stored in orc format. CREATE DATABASE merge_data;
CREATE TABLE merge_data.transactions(
ID int,
TranValue string,
last_update_user string)
PARTITIONED BY (tran_date string)
CLUSTERED BY (ID) into 5 buckets
STORED AS ORC TBLPROPERTIES ('transactional'='true');
CREATE TABLE merge_data.merge_source(
ID int,
TranValue string,
tran_date string)
STORED AS ORC;
Then we will populate the target and source table with some data. INSERT INTO merge_data.transactions PARTITION (tran_date) VALUES
(1, 'value_01', 'creation', '20170410'),
(2, 'value_02', 'creation', '20170410'),
(3, 'value_03', 'creation', '20170410'),
(4, 'value_04', 'creation', '20170410'),
(5, 'value_05', 'creation', '20170413'),
(6, 'value_06', 'creation', '20170413'),
(7, 'value_07', 'creation', '20170413'),
(8, 'value_08', 'creation', '20170413'),
(9, 'value_09', 'creation', '20170413'),
(10, 'value_10','creation', '20170413');
INSERT INTO merge_data.merge_source VALUES
(1, 'value_01', '20170410'),
(4, NULL, '20170410'),
(7, 'value_77777', '20170413'),
(8, NULL, '20170413'),
(8, 'value_08', '20170415'),
(11, 'value_11', '20170415');
When we examine the 2 tables, we would expect after the merge, row 1 should be left untouched, row 4 should be deleted (we imply a business rule here: NULL value indicates deletion), row 7 will be update and row 11 will be insertion for new value. The more interesting use case is with row 8, where it involves move the row from one partition to another. Merge currently does not support change of partition value on the fly. This need to happen as delete in the old partition and insert in the new partition. In real world use case, you need to construct the source table base on this critiria. Then we will create the merge statement as the following. Please note that not all 3 WHEN of the merge statements need to exist, it is fine to have only 2 or even 1 of the WHEN statement. We labeled the data with different last_update_user. For more details on Hive Merge, please refer to Hive document MERGE INTO merge_data.transactions AS T
USING merge_data.merge_source AS S
ON T.ID = S.ID and T.tran_date = S.tran_date
WHEN MATCHED AND (T.TranValue != S.TranValue AND S.TranValue IS NOT NULL) THEN UPDATE SET TranValue = S.TranValue, last_update_user = 'merge_update'
WHEN MATCHED AND S.TranValue IS NULL THEN DELETE
WHEN NOT MATCHED THEN INSERT VALUES (S.ID, S.TranValue, 'merge_insert', S.tran_date);
As part of the update clause, the set value statement should not contain the target table decorator "T.", otherwise you will get SQL compile error. Once the merge finished, re-examining the data shows the data is merged just as expected row 1 wasn't changed ; row 4 was deleted; row 7 was updated and row 11 was inserted. And row 8, as we see, got moved to a new partition. SELECT * FROM merge_data.transactions order by ID;
+----+-----------------------+------------------------------+-----------------------+
| id | transactions.tranvalue| transactions.last_update_user| transactions.tran_date|
+----+-----------------------+------------------------------+-----------------------+
| 1 | value_01 | creation | 20170410 |
| 2 | value_02 | creation | 20170410 |
| 3 | value_03 | creation | 20170410 |
| 5 | value_05 | creation | 20170413 |
| 6 | value_06 | creation | 20170413 |
| 7 | value_77777 | merge_update | 20170413 |
| 8 | value_08 | merge_insert | 20170415 |
| 9 | value_09 | creation | 20170413 |
| 10 | value_10 | creation | 20170413 |
| 11 | value_11 | merge_insert | 20170415 |
+----+-----------------------+------------------------------+-----------------------+
This simple example provides instructions on how to use ACID merge with HDP 2.6 or later. Of course the real world use case would be much more complicated comparing to this oversimplified example, but they all follow the same principles. And the beauty behind this is, you don't need to create a ETL process to accomplish this any more.
... View more
02-03-2017
07:11 PM
Michal, my problem was related to KMS proxy user and addressed by adding livy as proxy user. My cluster was secured with Kerberos and with Ranger and KMS. I am not sure if your is similar. You may want to start a new thread with more details.
... View more
02-03-2017
06:53 PM
2 Kudos
So here are the values that work in my environment nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.pattern.kerb=^(.*?)@(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.value.kerb=$1
Also in Ranger the Nifi nodes need to be added as internal user and create policy for them to access proxy, flow and data
... View more