About eric_walk

eric_walk · ‎08-10-2016

I'd check what the default AMI the cloudbreak uses is, spawn an EC2 instance from that, apply your customization, create a new AMI and then instruct cloudbreak to use your new private AMI. This way you don't have to worry about what cloudbreak needs, you're starting from cloudbreak's default config and customizing forward from there.

eric_walk · ‎08-01-2016

To resolve this permanently you might also want to add /usr/hdp/<version>/hbase/lib/hbase-client-1.1.2.2.4.0.0-169.jar to templeton.libjars or /usr/hdp/<version>/hbase/lib to templeton.hive.extra.files in webhcat-site.xml There are a few other places that you might make the adjustment depending on your runtime needs. I think, though, that the hive view in ambari uses the templeton endpoint.

eric_walk · ‎08-01-2016

Question 1: I believe that in this case you do not need the hostname in the principal (i.e. an SPN is not required). This principal is the service account that you are using to authenticate to Solr. If you wanted to do something like SPNEGO and forward the end-user's KRB Ticket to Solr, that'd be a different configuration that would require that you create a SPN. Question 2: You don't need to do this with KTPass since you don't need a SPN, I'd recommend using the ktab.exe bundled with Java, it'll be more reliable for what you're trying to accomplish. http://docs.oracle.com/javase/7/docs/technotes/tools/windows/ktab.html

eric_walk · ‎07-26-2016

@ARUN Cloudbreak is the tool that will allow you to automate this using Ambari Blueprint.

eric_walk · ‎07-06-2016

As the failure occurs while trying to add KDC Entries to your LDAP, you should make sure your LDAP is setup correctly. Are there any errors in the OpenLDAP server logs? That may be revealing...

eric_walk · ‎07-05-2016

Try this: https://community.hortonworks.com/content/kbentry/30653/openldap-setup.html

eric_walk · ‎07-05-2016

That sounds right given the error message.

eric_walk · ‎07-01-2016

LDAP 49 errors can mean a lot of different things. Can you please check the logs from the LDAP server for the specific error cause? This is very likely a bad password error... Atlassian has a good explanation of the different error codes: https://confluence.atlassian.com/kb/common-user-management-errors-820119309.html

eric_walk · ‎07-01-2016

@revan wabale Per discussions at Hadoop Summit, we should be expecting an alpha with Hive 2.x. So, I wouldn't expect this to be production ready for some time yet.

eric_walk · ‎06-23-2016

To explain why you use different commands for the two nodes: The idea here is that you don't want both nodes to attempt the rollback and risk becoming inconsistent. In a perfect world you could probably do the rollback on both, but the world isn't perfect. The best way to guarantee consistency is to have one node do the rollback and then rebootstrap the second node to the first (i.e. overwrite node 2's state information with that from the rolled back node 1).

eric_walk · ‎06-21-2016

@Sowmya Ramesh, @Margus Roo It looks like the CURRENT_USER object is thread local. I know that the Hive hook will run in a separate thread by design, perhaps there's an issue there? Perhaps it's a problem because Kerberos is not on? I'd venture that most use-cases for Atlas involve kerberized clusters... That said, I'm not familiar with the code details, so I could be reaching here.

eric_walk · ‎06-21-2016

AD is most definitely the easiest answer, unless you're morally opposed to it ;). You get integrated LDAP and KRB with nice user management tools. IPA does have some nice ootb features, though, around self service, etc.

eric_walk · ‎06-21-2016

I generally recommend letting DNS handle this. The latest versions of the KRB client will default to resolving the KDC from SRV records in the DNS for the realm. This should be configured by default if you use Microsoft Active Directory (or AWS Simple AD). If you want it explicit in your krb5.conf file, you can use DNS round robin with the A/AAA/CNAME and reference that name in krb5.conf. Further, you could have multiple "kdc" entries for a realm in krb5.conf and a master_kdc entry which is only used when there are certain kinds of issues. You can always manage the krb5.conf from Ambari inside the Kerberos component configs.

eric_walk · ‎06-21-2016

@Margus Roo Is Kerberos enabled for your cluster?

eric_walk · ‎06-20-2016

@Shishir Saxena In approach a, lack of involvement from enterprise security teams is not a positive thing. When dealing with a large enterprise, security is paramount and we should never be recommending that HDP administrators be permitted to manage authentication systems. Separation of duties is a core security principle and should not be taken lightly.

eric_walk · ‎06-20-2016

For a general enterprise scenario I'd recommend approaches b and c. Depending on the security administrators, they will agree to one of these. In general it is preferable to reduce the number of sources of identity within an organization to allow for easily managed, secure control. I would very strongly advise against the stand-alone KDC approach in any real production environment.

eric_walk · ‎06-09-2016

@Davide Isoardi, Can you please send the complete log from solr at the time if the error, including krb5 debug? That error message is generic and the logs you sent don't show it happening. The message means that either curl isn't sending a ticket, or solr can't interpret it or verify it.

eric_walk · ‎06-08-2016

Make sure the JVM that Solr is using has the JCE Unlimited Strength Keys Policy installed Check the Solr or ZK logs and send any relevant output as there might be more detail there Try adding this JVM parameter to the Solr/ZK JVMs to increase debug output: -Dsun.security.krb5.debug=true

eric_walk · ‎05-11-2016

@Chris McGuire, that is probably the case, I'm not very familiar with the way spark is configured. I do know that, generally speaking, unless you explicitly say insert or use hive streaming you don't have deltas and don't need to worry about compaction. The partition append merging is a whole different story...

eric_walk · ‎05-11-2016

@mbalakrishnan, do you think it might be missing it because they are originating from spark, not map, mapred or tez?

eric_walk · ‎05-11-2016

@Chris McGuire, I'm not sure you're using the Hive Streaming API, then. I'm not sure how Spark Streaming is setup to write out to hive, so it could be behaving correctly.

eric_walk · ‎05-11-2016

Hi @Chris McGuire, Can you please provide an "hdfs dfs -ls -R <table-folder>" Compaction only operates on tables with delta directories. I suspect that the method you're using (SaveMode.Append) is just appending to the existing partition (or adding a new partition) and not actually creating deltas. Best, Eric

eric_walk · ‎01-25-2016

I would generally say that if you're talking SSO for ODBC on Windows, the easiest option client side is Kerberos. As long as the user is logged into his workstation as an AD user that has rights on the Hive tables, the Kerberos ticket the OS gets at login will be sufficient for authentication to Hive over the SASL transport. You just need to make sure you've kerberized your cluster using one of your Active Directory Domain Controllers as the KDC (http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Ambari_Security_Guide/content/ch_configuring_amb_hdp_for_kerberos.html). At that point just configure the ODBC driver for kerberos. Again, the only client side requirement is that the machine is joined to the domain and the user logs in with an authorized account.

eric_walk · ‎01-11-2016

A few things to double check: 1. Is there any message on the Hiveserver2 that correlates to the Beeline error? 2. Can you double check that you have the Unlimited Strength Key JCE Policy installed correctly and that alternatives is pointing at the copy of java that has this poilcy (I'm noticing that your KRB Ticket only supports AES256) 3. Set system property sun.security.krb5.debug to true on your JVM and see if you can get any details from the debug logs.

eric_walk · ‎12-31-2015

In the short run you can always look at the metastore database (assuming you're using the db txn manager) and try to clear them manually from the tables there.

eric_walk · ‎12-22-2015

@Darpan Patel I would double check those host names and that the ports are open.

eric_walk · ‎12-22-2015

@Darpan Patel Haven't tried setting that up in a NameNodeHA environment yet, but it seems that it is trying to resolve the reference to the NN Service Name in DNS and failing. As for the Hive error, I'd suggest stopping ambari-server, doing a kdestroy for the user as which ambari-server runs and a kinit as the ambari-server user before starting it again.

eric_walk · ‎12-21-2015

No worries, I hope some of these things have been fixed since I went through this back in September (#4 should be resolved in Ambari 2.1.2). The Kdestroy/Kinit thing was definitely strange, never did work out why that was needed.

eric_walk · ‎12-21-2015

So I had a bunch of trouble with these, here are some of the things to note: When creating the view in Ambari don't use the "Local Ambari Managed Cluster" option, always use the custom when you have a kerberized cluster. Definitely read the instructions carefully (i.e. this one: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_ambari_views_guide/content/section_pig_view_kerberos_config.html) per @Neeraj Sabharwal. Stop Ambari Server, do a kdestroy for the user ambari-server run as, do a kinit for the ambari user using it's proper keytab as the ambari linux user, then start ambari-server again. Do this procedure each time you restart Ambari Server. For the pig view, there was a known issue where you needed to add: ,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar to your templeton.libjars for WebHCat (https://issues.apache.org/jira/browse/AMBARI-13096). Check your Ambari version...

eric_walk · ‎12-18-2015

If you wish to reference a file in S3 from a pig script you might do something like this: set fs.s3n.awsSecretAccessKey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx'; set fs.s3n.awsAccessKeyId 'xxxxxxxxxxxxxxxxxxxxx'; A = load 's3n://<bucket>/<path-to-file>' USING TextLoader; If you're on HDP 2.2.6, you'll likely see this error: Error: java.io.Exception, no filesystem for scheme: s3n The following steps resolve this issue: In core-site.xml add: <property> <name>fs.s3n.impl</name> <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value> <description>The FileSystem for s3n: (Native S3) uris.</description> </property> Then add to the MR2 and/or TEZ class path(s): /usr/hdp/${hdp.version}/hadoop-mapreduce/* These configs ensure 2 things: That the worker YARN containers spawned by pig have access to the hadoop-aws.jar file That the worker YARN containers know which class implements the file system type identified by "s3n://" References: Apache Mailing Lists Topic from Legacy Hortonworks Forums

Online	Offline
Last Visited	‎09-06-2016 06:29 PM

Date Registered	‎12-08-2015 06:44 PM
Last Visited	‎09-06-2016 06:29 PM
Total Messages Posted	34
Total Kudos Received	19

Re: HDP on AWS

Re: Kerberos KDC secondary

Re: Hive compactions on External table

Re: How to Bootstrap a Custom AMI on AWS for Cloud...

Re: org.apache.hive.service.cli.HiveSQLException: ...

Re: solrj + kerbero

Re: HDP on AWS

Re: KDC - OpenLdap integration

Re: KDC - OpenLdap integration

Re: issue with openldap/kerberos

Re: issue with openldap/kerberos

Re: Is Hbase ready to consider as metastore for hi...

Re: What's the difference of -bootstrapstandby and...

Re: atlas falcon hook - Atlas hook failed (FalconH...

Re: Looking for an automated integration of HDP/Am...

Re: Kerberos KDC secondary

Re: atlas falcon hook - Atlas hook failed (FalconH...

Re: Choosing Kerberos approach for Hadoop cluster ...

Re: Keberos Implementation Approach: is it recomme...

Re: Use solr with kerberos

Re: Use solr with kerberos

Re: Hive compactions on External table

Re: Hive compactions on External table

Re: Hive compactions on External table

Re: Hive compactions on External table

Re: Windows authentication (Active Directory) to H...

Re: beeline and kerberos

Re: Hive Acid: How to kill Locks & transaction ?

Re: Configuring ambari views on Kerberized Cluster

Re: Configuring ambari views on Kerberized Cluster

Re: Configuring ambari views on Kerberized Cluster

Re: Configuring ambari views on Kerberized Cluster

HDP 2.2 Configuration required for S3