Member since
12-08-2015
34
Posts
19
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
534 | 07-26-2016 06:52 PM | |
660 | 06-21-2016 06:45 PM | |
3349 | 05-11-2016 06:11 PM |
07-26-2016
06:52 PM
1 Kudo
@ARUN Cloudbreak is the tool that will allow you to automate this using Ambari Blueprint.
... View more
07-01-2016
07:04 PM
LDAP 49 errors can mean a lot of different things. Can you please check the logs from the LDAP server for the specific error cause? This is very likely a bad password error... Atlassian has a good explanation of the different error codes: https://confluence.atlassian.com/kb/common-user-management-errors-820119309.html
... View more
06-23-2016
07:13 PM
2 Kudos
To explain why you use different commands for the two nodes: The idea here is that you don't want both nodes to attempt the rollback and risk becoming inconsistent. In a perfect world you could probably do the rollback on both, but the world isn't perfect. The best way to guarantee consistency is to have one node do the rollback and then rebootstrap the second node to the first (i.e. overwrite node 2's state information with that from the rolled back node 1).
... View more
06-21-2016
06:53 PM
AD is most definitely the easiest answer, unless you're morally opposed to it ;). You get integrated LDAP and KRB with nice user management tools. IPA does have some nice ootb features, though, around self service, etc.
... View more
06-21-2016
06:45 PM
I generally recommend letting DNS handle this. The latest versions of the KRB client will default to resolving the KDC from SRV records in the DNS for the realm. This should be configured by default if you use Microsoft Active Directory (or AWS Simple AD). If you want it explicit in your krb5.conf file, you can use DNS round robin with the A/AAA/CNAME and reference that name in krb5.conf. Further, you could have multiple "kdc" entries for a realm in krb5.conf and a master_kdc entry which is only used when there are certain kinds of issues. You can always manage the krb5.conf from Ambari inside the Kerberos component configs.
... View more
06-20-2016
01:06 PM
@Shishir Saxena In approach a, lack of involvement from enterprise security teams is not a positive thing. When dealing with a large enterprise, security is paramount and we should never be recommending that HDP administrators be permitted to manage authentication systems. Separation of duties is a core security principle and should not be taken lightly.
... View more
06-20-2016
01:03 PM
For a general enterprise scenario I'd recommend approaches b and c. Depending on the security administrators, they will agree to one of these. In general it is preferable to reduce the number of sources of identity within an organization to allow for easily managed, secure control. I would very strongly advise against the stand-alone KDC approach in any real production environment.
... View more
05-11-2016
07:29 PM
@Chris McGuire, that is probably the case, I'm not very familiar with the way spark is configured. I do know that, generally speaking, unless you explicitly say insert or use hive streaming you don't have deltas and don't need to worry about compaction. The partition append merging is a whole different story...
... View more
05-11-2016
07:09 PM
@mbalakrishnan, do you think it might be missing it because they are originating from spark, not map, mapred or tez?
... View more
05-11-2016
07:09 PM
@Chris McGuire, I'm not sure you're using the Hive Streaming API, then. I'm not sure how Spark Streaming is setup to write out to hive, so it could be behaving correctly.
... View more
05-11-2016
06:11 PM
2 Kudos
Hi @Chris McGuire, Can you please provide an "hdfs dfs -ls -R <table-folder>" Compaction only operates on tables with delta directories. I suspect that the method you're using (SaveMode.Append) is just appending to the existing partition (or adding a new partition) and not actually creating deltas. Best, Eric
... View more
01-25-2016
02:58 PM
4 Kudos
I would generally say that if you're talking SSO for ODBC on Windows, the easiest option client side is Kerberos. As long as the user is logged into his workstation as an AD user that has rights on the Hive tables, the Kerberos ticket the OS gets at login will be sufficient for authentication to Hive over the SASL transport. You just need to make sure you've kerberized your cluster using one of your Active Directory Domain Controllers as the KDC (http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Ambari_Security_Guide/content/ch_configuring_amb_hdp_for_kerberos.html). At that point just configure the ODBC driver for kerberos. Again, the only client side requirement is that the machine is joined to the domain and the user logs in with an authorized account.
... View more
01-11-2016
03:30 PM
1 Kudo
A few things to double check: 1. Is there any message on the Hiveserver2 that correlates to the Beeline error?
2. Can you double check that you have the Unlimited Strength Key JCE Policy installed correctly and that alternatives is pointing at the copy of java that has this poilcy (I'm noticing that your KRB Ticket only supports AES256)
3. Set system property sun.security.krb5.debug to true on your JVM and see if you can get any details from the debug logs.
... View more
12-31-2015
06:12 PM
In the short run you can always look at the metastore database (assuming you're using the db txn manager) and try to clear them manually from the tables there.
... View more
12-22-2015
10:28 PM
@Darpan Patel I would double check those host names and that the ports are open.
... View more
12-22-2015
02:23 PM
@Darpan Patel Haven't tried setting that up in a NameNodeHA environment yet, but it seems that it is trying to resolve the reference to the NN Service Name in DNS and failing. As for the Hive error, I'd suggest stopping ambari-server, doing a kdestroy for the user as which ambari-server runs and a kinit as the ambari-server user before starting it again.
... View more
12-21-2015
07:01 PM
1 Kudo
No worries, I hope some of these things have been fixed since I went through this back in September (#4 should be resolved in Ambari 2.1.2). The Kdestroy/Kinit thing was definitely strange, never did work out why that was needed.
... View more
12-21-2015
06:40 PM
1 Kudo
So I had a bunch of trouble with these, here are some of the things to note:
When creating the view in Ambari don't use the "Local Ambari Managed Cluster" option, always use the custom when you have a kerberized cluster. Definitely read the instructions carefully (i.e. this one: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_ambari_views_guide/content/section_pig_view_kerberos_config.html) per @Neeraj Sabharwal. Stop Ambari Server, do a kdestroy for the user ambari-server run as, do a kinit for the ambari user using it's proper keytab as the ambari linux user, then start ambari-server again. Do this procedure each time you restart Ambari Server. For the pig view, there was a known issue where you needed to add: ,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar to your templeton.libjars for WebHCat (https://issues.apache.org/jira/browse/AMBARI-13096). Check your Ambari version...
... View more
12-18-2015
08:16 PM
3 Kudos
If you wish to reference a file in S3 from a pig script you might do something like this: set fs.s3n.awsSecretAccessKey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
set fs.s3n.awsAccessKeyId 'xxxxxxxxxxxxxxxxxxxxx';
A = load 's3n://<bucket>/<path-to-file>' USING TextLoader; If you're on HDP 2.2.6, you'll likely see this error: Error: java.io.Exception, no filesystem for scheme: s3n
The following steps resolve this issue: In core-site.xml add: <property>
<name>fs.s3n.impl</name>
<value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
<description>The FileSystem for s3n: (Native S3) uris.</description>
</property> Then add to the MR2 and/or TEZ class path(s): /usr/hdp/${hdp.version}/hadoop-mapreduce/* These configs ensure 2 things: That the worker YARN containers spawned by pig have access to the hadoop-aws.jar file That the worker YARN containers know which class implements the file system type identified by "s3n://" References:
Apache Mailing Lists Topic from Legacy Hortonworks Forums
... View more
- Find more articles tagged with:
- Cloud & Operations
- error
- hdp-2.2.6
- Issue Resolution
- issue-resolution
- Pig
- S3
12-17-2015
09:13 PM
The root user's umask is 0027
... View more
12-17-2015
09:09 PM
1 Kudo
Yeah, I've thought about that, I don't like it... it's one more thing that's hacked together manually and has to be tracked and maintained. I'd much prefer a fix to Ambari's config...
... View more
12-15-2015
07:46 PM
1 Kudo
This is HDP 2.2.6, Ambari 2.1.2.1
... View more
12-15-2015
04:32 PM
1 Kudo
Ambari sets up set-hdfs-plugin-env.sh with root:hadoop 700 permissions when it restarts HDFS, this causes ranger integration to break as the hdfs user cannot execute this script when the namenode starts. I can fix the problem if I restart the namenode manually, but that means to deploy config changes I need to restart from Ambari, then correct permissions and restart manually. How
... View more
Labels: