Member since
07-30-2019
181
Posts
205
Kudos Received
51
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4957 | 10-19-2017 09:11 PM | |
1589 | 12-27-2016 06:46 PM | |
1235 | 09-01-2016 08:08 PM | |
1174 | 08-29-2016 04:40 PM | |
3009 | 08-24-2016 02:26 PM |
09-13-2022
12:50 PM
@emaxwell A few suggested additions to this excellent article: 1. The keystore password is not required to list the keystore via the keytool command. When prompted for a password, you can just hit "enter" without entering a password. The password would be needed if you want to export the actual privateCertEntry from the keystore which is not necessary in this procedure. 2. The ldap-provider configured in the login-identity-providers.xml file has two options for how to identify the authenticated ldap user: <property name="Identity Strategy">USE_DN</property> Above will use the full Distinguished Name (DN) as you mention in the article. <property name="Identity Strategy">USE_USERNAME</property> Above will use the string entered by user at the login window as the user identity. 3. Identity mapping properties in the nifi.properties file have three properties: nifi.security.identity.mapping.pattern.<unique string>=<Java regex>
nifi.security.identity.mapping.value.<unique string>=<Resulting string when java regex matches, Java Regex capture groups can be used here>
nifi.security.identity.mapping.transform.<unique string>=<NONE, LOWER, UPPER> The <unique string> can be anything but must be same exact string for all three properties. Also patterns are checked in alphanumeric order against the identity strings. First matching pattern has its value and transform applied. So you want to make sure the more complex patterns come first to avoid having wrong pattern's value and transform being applied. So a pattern like "^(.*)$" should definitely come last. The "transform" property allows you to apply an all uppercase, all lowercase, or no transform to the resulting value string. Thanks, Matt
... View more
08-17-2022
12:48 PM
You are using an array, this is why you got null values. You have 2 options: Change the expressions to $[0].ARRIVAL_DATE and $[0].BOOK_ID, and it will work Read the file and split the array using the SplitJson processor - set JsonPath expression to: $, and connect the SplitJson processor to your EvaluateJsonPath processor and it should work.
... View more
05-05-2022
04:36 PM
As a general statement this is not right by any means. LDAP provides secure and encrypted authentication (encrypted user password and SSL/TLS communication) , together with user/group management. It's only the Hadoop stack does not support this and the two only autentication methods implemented for all the CDP components are the dummy simple auth (described above) and the Kerberos authentication (used in combination with PAM or LDAP for user/group mappings). As an example, nothing less than Knox (the security gateway to HDP or CDP) implements full authenticacion using only LDAP (with TLS), and it only relies on Kerberos to authenticate a single service/proxy user to communicate with the rest of the cluster.
... View more
02-17-2020
06:29 PM
RULE:[1:$1@$0](^.*@AD\.HORTONWORKS\.COM$)s/^(.*)@AD\.HORTONWORKS\.COM$/$1/g I am not quiet sure if the brackets arountd (.*) are really necessary.
... View more
01-07-2020
11:56 PM
Hello @saikrishna_tara @bbende @emaxwell . Thanks for the solution and it worked well for me. I am new to Nifi and got the same problem statement as @saikrishna_tara. I am able to make it till merge content, i can see my files are in parent flow files. but in parent name of files are uuid of the flow files and not the actual name of the file which is processed. I need to put all the parent flow file's actual name via email processor to the outside world. Please let me know in case more details are required. Regards Nitin
... View more
10-04-2019
02:46 PM
Hello, I'm looking your answer 3 years later because I'm in a similar situation :). In my company (telco) we're planning using 2 hot clusters with dual ingest because our RTO is demanding and we're looking for mechanisms to monitor and keep in sync both clusters. We ingest data in real-time with kafka + spark streaming, loading to HDFS and consuming with Hive/Impala. I'm thinking about a first approach making simple counts with Hive/Impala tables on both clusters each hour/half hour and comparing. If something is missing in one of the clusters, we will have to "manually" re-ingest the missing data (or copy it with cloudera BDR from one cluster to the other) and re-process enriched data. I'm wondering if have you dealt with similar scenarios or suggestions you may have. Thanks in advance!
... View more
10-19-2017
09:11 PM
1 Kudo
@dsun During the upgrade process, a component is supposed to be restarted after the hdp-select command has been run so it will pick up the new binaries. However, the component needs to shut down and start up after the hdp-select command has been run. That way it will report to Ambari that it's version has changed and what it's current state is. In the event that you get stuck (as you did) during the upgrade you can unwind the versioning with a process like this: Make all pieces of the component are running Run `hdp-select set` command on all nodes in the cluster to set the new version. Make sure you get all of the pieces for the component (e.g. hadoop-hdfs-namenode, hadoop-ndfs-journalnode, etc.) Restart all processes for the component Verify that the O/S processes are running with the proper version of jar files Lather, rinse, and repeat for all components in the cluster Once you have successfully gotten everything restarted with the proper bits, you should be able to manually finalize the upgrade with the following command to the Ambari Server: ambari-server set-current --cluster=<custername> --version-display-name=HDP-2.6.2.0 If you get an error that components are not upgraded, you can check the components and hosts again. If everything seems ok, then you may need to tweak a table in the database. I ran into this when Atlas did not properly report the upgraded version to Ambari. NOTE: THIS SHOULD BE DONE WITH THE GUIDANCE OF HORTONWORKS SUPPORT ONLY ambari=> SELECT h.host_name, hcs.service_name, hcs.component_name, hcs.version FROM hostcomponentstate hcs JOIN hosts h ON hcs.host_id = h.host_id ORDER BY hcs.version, hcs.service_name, hcs.component_name, h.host_name;
host_name | service_name | component_name | version
----------------------------------+----------------+-------------------------+-------------
scregione1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm0.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm2.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionw0.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionw1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm0.field.hortonworks.com | ATLAS | ATLAS_SERVER | 2.6.1.0-129
scregionm1.field.hortonworks.com | DRUID | DRUID_BROKER | 2.6.2.0-205
scregionm1.field.hortonworks.com | DRUID | DRUID_COORDINATOR | 2.6.2.0-205
scregionw0.field.hortonworks.com | DRUID | DRUID_HISTORICAL | 2.6.2.0-205
scregionw1.field.hortonworks.com | DRUID | DRUID_HISTORICAL | 2.6.2.0-205
scregionw0.field.hortonworks.com | DRUID | DRUID_MIDDLEMANAGER | 2.6.2.0-205
scregionw1.field.hortonworks.com | DRUID | DRUID_MIDDLEMANAGER | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_OVERLORD | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_ROUTER | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_SUPERSET | 2.6.2.0-205
scregione1.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
scregionm0.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
scregionm1.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
. . . After verifying that you have, indeed, upgraded the components, a simple update command will set the proper version for the erroneous components and allow you to finalize the upgrade: ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_CLIENT';
UPDATE 6
ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_SERVER';
UPDATE 1
After cycling the Ambari Server, you should be able to finalize: [root@hostname ~]# ambari-server set-current --cluster=<cluster> --version-display-name=HDP-2.6.2.0
Using python /usr/bin/python
Setting current version...
Enter Ambari Admin login: <username>
Enter Ambari Admin password:
Current version successfully updated to HDP-2.6.2.0
Ambari Server 'set-current' completed successfully.
... View more
06-12-2017
01:00 PM
what is the password for the above command?
... View more
09-06-2017
07:33 AM
Hi, you are right, the only solution is to change locale to english. the code implementation to support multi-language is complex.
... View more
08-26-2016
06:52 PM
1 Kudo
What came up in a customer setting was the suggestion to not even use multitenancy. Instead, set up one shared realm so users can authenticate on all clusters. Then, assign access privileges using Ranger. In older HDP versions this would create the problem of sharing the service principals and technical user accounts across clusters. I believe this has largely been solved in HDP 2.3.
... View more