Member since
09-21-2015
85
Posts
73
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
495 | 04-21-2016 12:22 PM | |
1848 | 03-12-2016 02:19 PM | |
347 | 10-29-2015 07:50 PM | |
399 | 10-02-2015 04:21 PM | |
1426 | 09-29-2015 03:08 PM |
10-03-2018
03:18 PM
This would result in the keystore pass being stored in plain-text. Is there a more secure method of storing the keystore pass?
... View more
09-07-2018
09:29 AM
Pravin - Did you resolve the issue? According to this JIRA, it's been fixed since Ranger 0.5.3. If not we should raise a new bug. https://issues.apache.org/jira/browse/RANGER-746
... View more
05-16-2018
07:47 AM
Is this fixed in Ambari 2.6.2 and Ambari 2.7 (for the CentOS7 and related versions)?
... View more
02-24-2018
11:04 AM
Logsearch stores logs in the collection "hadoop_logs". What does it store in "audit_logs" and is there an overlap with Ranger? I don't see any documentation for logsearch explaining what is in each collection. And what "solr_audit_logs_use_ranger" means in the logsearch config.
... View more
02-23-2018
09:53 AM
What is the difference between these Solr collections? Ranger: 'ranger_audits' LogSearch: 'audit_logs'
... View more
Labels:
01-02-2018
03:45 PM
How would this be set when you have multiple solr hosts? <strname="solr.hdfs.security.kerberos.principal">infra-solr/<hostname>@EXAMPLE.COM</str>
... View more
11-09-2017
11:06 AM
How do we force an update/reconciliation of Atlas tags into Ranger? rangertagsync sources changes from Kafka. If the Kafka topic is lost or corrupted, how do we ensure Ranger matches what is in Atlas?
... View more
Labels:
10-16-2017
08:22 AM
Curl shows the same.
... View more
10-15-2017
10:21 AM
We are unable to make queries to collections on Ambari Infra Solr. This same request works on other Ambari 2.5 clusters.
# curl -g --negotiate -u : "http://hostname:8886/solr/ranger_audits/query?debug=query&q=*:*"<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /solr/ranger_audits/select. Reason:
<pre> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
Here is the krb5 debug log showing the duplicate key after setting SOLR_OPTS="$SOLR_OPTS -Dsun.security.krb5.debug=true . >>> KrbApReq: authenticate succeed.
Krb5Context setting peerSeqNumber to: 907174024
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 512754846
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Looking for keys for: HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Added key: 17version: 2
Found unsupported keytype (1) for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Found unsupported keytype (3) for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Added key: 18version: 2
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
MemoryCache: add 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM to infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM|HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
MemoryCache: Existing AuthList:
#3: 1508065826/137763/263ACEC1894287E10DE785337DF032E1/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM
#2: 1508065868/906511/338AF89A3C5C5E73950D89CD559EBEFD/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM
#1: 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM
But administrative requests work:
# sudo curl -g --negotiate -u : "http://hostname:8886/solr/admin/collections?action=LIST"
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><arr name="collections"><str>fulltext_index</str><str>ranger_audits</str><str>edge_index</str><str>vertex_index</str></arr>
</response>
... View more
Labels:
08-31-2017
12:39 PM
This is expected if looking at the "standby" namenode in an HA NameNode environment. Only the active NameNode will have an accurate count. If the standby becomes active it's count will be correct. Hadoop >2.8 will report accurately. Read more here: https://issues.apache.org/jira/browse/HDFS-9396
... View more
06-28-2017
04:57 PM
Answering my own question. Credit to @lmccay. RTFM:
http://knox.apache.org/books/knox-0-9-0/user-guide.html#SwitchCase+Identity+Assertion+Provider
http://knox.apache.org/books/knox-0-12-0/user-guide.html#SwitchCase+Identity+Assertion+Provider
... View more
06-28-2017
04:24 PM
Are there options to lower the case of users & groups within Knox? (In my case is with ldapRealm).
Throughout HDP we often lower the case of users/groups. Such as in Ranger where these are typically set:
ranger.usersync.ldap.username.caseconversion=local
ranger.usersync.ldap.groupname.caseconversion=local This results in Knox polices in Ranger not functioning. For example:
- Group in Knox (from AD): HDP_access
- Group in HDFS & Ranger: hdp_access
... View more
Labels:
06-19-2017
08:06 AM
Why do the HDP 2.6 docs say to remove Galera and install TokuDB when using MariaDB? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/meet-min-system-requirements.html
... View more
05-24-2017
11:16 AM
Which HA (High Availability) mechanisms does HDP (Ambari, Hive, Ranger, Oozie) support for the database which the services are backed by (MySQL, MariaDB & PostgreSQL)? I've heard that MySQL HA & PostgreSQL HA are not supported, but not seen anything official. What about Galera, Percona XtraDB Cluster, MySQL MHA, ... and/or proxy mechanisms such as ProxySQL?
... View more
05-23-2017
11:15 AM
Are there any problems with using a 'bind' mount for any part of HDP?
Such as /usr/hdp, HDFS disks, YARN disks, ...
... View more
05-23-2017
11:10 AM
The docs say "MariaDB 10" but RHEL7 comes with "MariaDB 5".
... View more
05-19-2017
04:12 PM
6 Kudos
Note: HDP 2.5 is used, but can be updated for HDP 2.6 and beyond. Boot an Amazon Linux instance with at least 16GB of RAM Execute the following. It will take a while
## install docker
sudo yum update -y
sudo yum install -y docker
## fix docker for importing the large sandbox image
sed -i.backup 's/\(^OPTIONS=.*\)"$/\1 --storage-opt=dm.basesize=20G"/' /etc/sysconfig/docker
## start docker
sudo service docker start
## confirm docker is working
sudo usermod -a -G docker ec2-user
docker info
## download docker image
curl -O http://hortonassets.s3.amazonaws.com/2.5/HDP_2.5_docker.tar.gz
## load docker image
docker load -i HDP_2.5_docker.tar.gz
## confirm image is available
docker images
## get sandbox docker startup script
curl -O https://raw.githubusercontent.com/hortonworks/tutorials/hdp-2.5/tutorials/hortonworks/hortonworks-sandbox-hdp2.5-guide/start_sandbox.sh
## start sandbox
bash start_sandbox.sh
## configure to start at boot
echo "bash /root/start_sandbox.sh" >> /etc/rc.local
## Print the URL for accessing the Sandbox
echo -e "##\nAccess the Sandbox at:\nhttp://$(curl -sS4 icanhazip.com):8888\n##"
... View more
- Find more articles tagged with:
- amazon
- aws
- cloud
- docker
- How-ToTutorial
- Sandbox
- Sandbox & Learning
Labels:
01-23-2017
09:39 PM
1 Kudo
Try point to the library provided by the system package as described in the docs? https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-reference/content/using_hive_with_mysql.html
... View more
12-21-2016
09:39 AM
@Matt Foley - Does HDFS have home references such as ~ or ${HOME} ?
... View more
12-13-2016
09:04 AM
That's not true. While the input/output will use the objectstorage (i.e. s3a://) if specified, many are things will still land locally. Such as hadoop tmp, yarn local-dirs, ...
... View more
12-12-2016
11:17 AM
How much storage is required and for what purposes? Those that come to mind:
HDFS intermediate yarn.app.*.am.staging-dir yarn.nodemanager.resource.local-dirs hadoop.tmp.dir
... View more
Labels:
11-24-2016
05:06 PM
Samet - Yes, the concatenated method is just something I've seen implemented. I don't know which directory system and how it was configured.
... View more
11-23-2016
07:12 PM
That AMI is not owned or maintained by Hortonworks. Also it appears this AMI was created in 2014 so is very dated: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;search=ami-02ae336a;sort=name
... View more
11-23-2016
05:31 PM
I've updated the answer with this detail:
For SSO, Knox supports SAML 2.0. Any SAML provider should work, including RSA SecurID. Hue appears to support SAML for SSO (see answer for link)
... View more
11-21-2016
07:24 PM
1 Kudo
Samet - Using Knox would be ideal. It could be approached in 2 ways:
a) SSO thanks to Knox support for it.
With this users continue to auth as they always have. Less integration points are needed. No additional authentication points. Less to maintain. Less to break b) Alternatively, configure your directory system against 2FA, such that LDAP logins use 2-factor.
Example: Instead of a standard password, they enter "password + pin from 2f device". With this Knox could still be the single point, but you could support any other services that have LDAP support. This also means nothing special/extra is needed in Hadoop. Users use the system as they use other systems. Apache Knox covers most of the services you mentioned:
SSO:
SAML 2.0 for SSO (as of HDP 2.5). Nearly all SSO systems support SAML. Web UIs which Knox provides SSO for (as of HDP 2.5):
Ambari Ranger Services which Knox provides (as of HDP 2.5):
YARN WebHDFS WebHCat/Templeton Oozie HBase (Stargate REST API) Hive (via WebHCat) Hive (via JDBC) Ambari API Ranger API For direct access to the cluster, use bastion hosts:
Typically SSH or Remote Desktop. Authentication to those systems is configured for SSO
Many options here. Typically SSSD, Centrify, ... These systems can automatically get the kerberos token on behalf of the user. Then the user would use services as usual. An alternative is to use VPN or SSH-tunnel with 2-factor to gain network access to the cluster. Then the user would need a kerberos token, but the 2-factor level of access is provided at the network layer. Hue:
Appears to support SAML for SSO. Could use the LDAP method mentioned earlier. Keep in mind that Hue is not an Apache-community project. It's not maintained by Hortonworks or the open community. Documentation:
Knox SSO (HDP 2.5) Knox supported Hadoop Services (HDP 2.5) Hue support for SAML --
Sean Roberts
@seano
... View more
11-01-2016
01:51 PM
1 Kudo
To work-around this you could configure your web service to proxy the HBase requests. For example, in Apache HTTP: /etc/httpd/conf.d/hbase-rest.conf ProxyPass /hbase/ http://localhost:8081/
ProxyPassReverse /hbase/ https://localhost:8081/
... View more
08-02-2016
12:18 PM
@Xiaoyu Yao - Can you update your original answer with that detail and delete this separate answer? I can accept it then.
... View more
08-01-2016
06:59 PM
@Xiaoyu Yao - Are you surethat is for the network? It appears to be tied to this setting which says "io" in it:
`dfs.datanode.slow.io.warning.threshold.ms` https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
... View more
08-01-2016
06:53 PM
@Xiaoyu Yao - Thanks. Do you know if that settings is adjustable and do you know a good reference for why that number was chosen?
... View more