Member since
07-30-2019
53
Posts
136
Kudos Received
16
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 8938 | 01-30-2017 05:05 PM | |
| 4776 | 01-13-2017 03:46 PM | |
| 2350 | 01-09-2017 05:36 PM | |
| 1434 | 01-09-2017 05:29 PM | |
| 1104 | 10-07-2016 03:34 PM |
10-02-2015
10:15 PM
3 Kudos
I've just installed and Kerberized my cluster: Ambari 2.1.1 CentOS 7 IPA 4 for LDAP and Kerberos (IPA Clients Configured across the cluster hosts) Oracle JDK 1.7.0_79 (with JCE) HDP 2.3.0 The cluster comes up just fine and all the services seem to be happy talking to each other. So I'm pretty convinced that all the keytabs are configured correctly. From any node on the cluster, after getting a valid ticket (kinit) and trying to do a basic hdfs command, I get (kerberos debug enabled): ONLY HAPPENS FROM IPA Clients. Other host client access works fine (read on). -sh-4.2$ klist
Ticket cache: KEYRING:persistent:100035:krb_ccache_T7mkWNw
Default principal: dstreev@HDP.LOCAL
Valid starting Expires Service principal
10/02/2015 09:17:07 10/03/2015 09:17:04 krbtgt/HDP.LOCAL@HDP.LOCAL
-sh-4.2$ hdfs dfs -ls .
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
>>>KinitOptions cache name is /tmp/krb5cc_100035
15/10/02 18:07:48 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/10/02 18:07:48 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/10/02 18:07:48 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over m2.hdp.local/10.0.0.161:8020 after 1 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "m3.hdp.local/10.0.0.162"; destination host is: "m2.hdp.local":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1431)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:685)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:648)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:735)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
at org.apache.hadoop.ipc.Client.call(Client.java:1397)
... 28 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
... 31 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 40 more
15/10/02 18:07:48 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
(it retries 15-20 times before quitting, which happens really fast. If I try to access the cluster from a host that is NOT part of the IPA hosts (from my Mac, as an example). I do NOT get this error, and I can interact with the cluster. ➜ conf klist
Credentials cache: API:D44F3F89-A095-40A5-AA7C-BD06698AA606
Principal: dstreev@HDP.LOCAL
Issued Expires Principal
Oct 2 17:52:13 2015 Oct 3 17:52:00 2015 krbtgt/HDP.LOCAL@HDP.LOCAL
Oct 2 18:06:53 2015 Oct 3 17:52:00 2015 host/m3.hdp.local@HDP.LOCAL
➜ conf hdfs dfs -ls /
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
15/10/02 18:10:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>KinitOptions cache name is /tmp/krb5cc_501
>> Acquire default native Credentials
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23.
>>> Obtained TGT from LSA: Credentials:
client=dstreev@HDP.LOCAL
server=krbtgt/HDP.LOCAL@HDP.LOCAL
authTime=20151002215213Z
startTime=20151002215213Z
endTime=20151003215200Z
renewTill=20151009215200Z
flags=FORWARDABLE;RENEWABLE;INITIAL;PRE-AUTHENT
EType (skey)=18
(tkt key)=18
15/10/02 18:10:59 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Found ticket for dstreev@HDP.LOCAL to go to krbtgt/HDP.LOCAL@HDP.LOCAL expiring on Sat Oct 03 17:52:00 EDT 2015
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for dstreev@HDP.LOCAL to go to krbtgt/HDP.LOCAL@HDP.LOCAL expiring on Sat Oct 03 17:52:00 EDT 2015
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KdcAccessibility: reset
>>> KrbKdcReq send: kdc=m3.hdp.local UDP:88, timeout=30000, number of retries =3, #bytes=654
>>> KDCCommunication: kdc=m3.hdp.local UDP:88, timeout=30000,Attempt =1, #bytes=654
>>> KrbKdcReq send: #bytes read=637
>>> KdcAccessibility: remove m3.hdp.local
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 227177742
Created InitSecContextToken:
0000: 01 00 6E 82 02 3C 30 82 02 38 A0 03 02 01 05 A1 ..n..<0..8......
0010: 03 02 01 0E A2 07 03 05 00 20 00 00 00 A3 82 01 ......... ......
...
0230: 99 AC EE FB DF 86 B5 2A 19 CB A1 0B 8A 8E F7 9B .......*........
0240: 81 08 ..
Entered Krb5Context.initSecContext with state=STATE_IN_PROCESS
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting peerSeqNumber to: 40282898
Krb5Context.unwrap: token=[05 04 01 ff 00 0c 00 00 00 00 00 00 02 66 ab 12 01 01 00 00 8e 14 7a df 34 d7 c5 3d 5d d1 ce b5 ]
Krb5Context.unwrap: data=[01 01 00 00 ]
Krb5Context.wrap: data=[01 01 00 00 ]
Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 0d 8a 75 0e 01 01 00 00 9c a5 73 25 59 0f b5 64 24 f0 a8 78 ]
Found 8 items
drwxrwxrwx - yarn hadoop 0 2015-09-28 15:55 /app-logs
drwxr-xr-x - hdfs hdfs 0 2015-09-28 15:57 /apps
drwxr-xr-x - hdfs hdfs 0 2015-09-28 15:53 /hdp
drwxr-xr-x - mapred hdfs 0 2015-09-28 15:53 /mapred
drwxrwxrwx - mapred hadoop 0 2015-09-28 15:54 /mr-history
drwxr-xr-x - hdfs hdfs 0 2015-09-28 19:20 /ranger
drwxrwxrwx - hdfs hdfs 0 2015-09-29 13:09 /tmp
drwxr-xr-x - hdfs hdfs 0 2015-10-02 17:51 /user
➜ conf
Since I can get to the cluster and interact with it, from a host that hasn't been configured by the IPA client. I'm pretty sure that my IPA environment is tweaked. Any idea where to look in IPA to fix this for the hosts that are part of the IPA environment?
... View more
10-02-2015
09:35 PM
The KMS is also backed by a DB. Make sure kms db user/host combination has been granted permissions in MySQL (if that's what you're using).
... View more
10-02-2015
09:29 PM
Check the permissions (on the Oozie server) of the files in directory below. When updates are made to Hive, these files are updated. This is fairly new with Oozie, these files are automatically part of the path, so you don't need to add "hive" related site files to your Oozie workflows when doing Hive Actions. /etc/oozie/conf/action-conf/hive
... View more
10-02-2015
09:20 PM
2 Kudos
Example Maven Section: <repositories>
<repository>
<releases>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
<checksumPolicy>warn</checksumPolicy>
</releases>
<snapshots>
<enabled>false</enabled>
<updatePolicy>never</updatePolicy>
<checksumPolicy>fail</checksumPolicy>
</snapshots>
<id>HDPReleases</id>
<name>HDP Releases</name>
<url>http://repo.hortonworks.com/content/repositories/releases/</url>
<layout>default</layout>
</repository>
...
... View more
10-02-2015
09:15 PM
1 Kudo
I think you've set the value in the wrong place. Try adding the value to your job.properties file and try again. In this case, you embedded it in an action, when I believe it's meant to be a job level variable.
... View more
10-02-2015
09:08 PM
8 Kudos
Goal
Create and Setup IPA Keytabs for HDP Tested on CentOS 7 with IPA 4, HDP 2.3.0 and Ambari 2.1.1 Assumptions
Ambari does NOT currently (2.1.x) support the automatic generation of keytabs against IPA IPA Server is already installed and IPA clients have been installed/configured on all HDP cluster nodes If you are using Accumulo, you need to create the user 'tracer' in IPA. A keytab for this user will be requested for Accumulo Tracer. We'll use Ambari's 'resources' directory to simplify the distribution of scripts and keys. Enable kerberos using wizard In Ambari, start security wizard by clicking Admin -> Kerberos and click Enable Kerberos. Then select "Manage Kerberos principals and key tabs manually" option Enter your realm (Optional, Read on before doing this...) Remove clustername from smoke/hdfs principals to remove the -${cluster_name} references to look like below
smoke user principal: ${cluster-env/smokeuser}@${realm} HDFS user principal: ${hadoop-env/hdfs_user}@${realm} HBase user principal: ${hbase-env/hbase_user}@${realm} Spark user principal: ${spark-env/spark_user}@${realm} Accumulo user principal: ${accumulo-env/accumulo_user}@${realm} Accumulo Tracer User: tracer@${realm} If you don't remove the "cluster-name" from above, Ambari will generate/use principal names that are specific to your cluster. This could be very important if you are supporting multiple clusters with the same IPA implementation. On next page download csv file but DO NOT click Next yet!
If you are deploying storm, the storm user maybe missing from the storm USER row. If you see something like the below: storm@HORTONWORKS.COM,USER,,/etc
replace the ,, with ,storm, storm@HORTONWORKS.COM,USER,storm,/etc
Copy above csv to the Ambari Server and place it in the /var/lib/ambari-server/resources directory, making sure to remove the header and any empty lines at the end. vi kerberos.csv
From any IPA Client Node Create principals and service accounts using csv file. ## authenticate
kinit admin
AMBARI_HOST=<ambari_host>
# Get the kerberos.csv file from Ambari
wget http://${AMBARI_HOST}:8080/resources/kerberos.csv -O /tmp/kerberos.csv
# Create IPA service entries.
awk -F"," '/SERVICE/ {print "ipa service-add --force "$3}' /tmp/kerberos.csv | sort -u > ipa-add-spn.sh
sh ipa-add-spn.sh
# Create IPA User accounts
awk -F"," '/USER/ {print "ipa user-add "$5" --first="$5" --last=Hadoop --shell=/sbin/nologin"}' /tmp/kerberos.csv | sort | uniq > ipa-add-upn.sh
sh ipa-add-upn.sh
On Ambari node authenticate and create the keytabs for the headless user accounts and initialize the service keytabs. ## authenticate
sudo echo '<kerberos_password>' | kinit --password-file=STDIN admin
## or (IPA 4)
sudo echo '<kerberos_password>' | kinit -X password-file=STDIN admin
Should be run as root (adjust for your Ambari Host/Port). AMBARI_HOST_PORT=<ambari_host>
wget http://${AMBARI_HOST_PORT}/resources/kerberos.csv -O /tmp/kerberos.csv
ipa_server=$(grep server /etc/ipa/default.conf | awk -F= '{print $2}')
if [ "${ipa_server}X" == "X" ]; then
ipa_server=$(grep host /etc/ipa/default.conf | awk -F= '{print $2}')
fi
if [ -d /etc/security/keytabs ]; then
mv -f /etc/security/keytabs /etc/security/keytabs.`date +%Y%m%d%H%M%S`
fi
mkdir -p /etc/security/keytabs
chown root:hadoop /etc/security/keytabs/
if [ ! -d /var/lib/ambari-server/resources/etc/security/keytabs ]; then
mkdir -p /var/lib/ambari-server/resources/etc/security/keytabs
fi
grep USER /tmp/kerberos.csv | awk -F"," '{print "ipa-getkeytab -s '${ipa_server}' -p "$3" -k "$6";chown "$7":"$9,$6";chmod "$11,$6}' | sort -u > gen_keytabs.sh
# Copy the 'user' keytabs to the Ambari Resources directory for distribution.
echo "cp -f /etc/security/keytabs/*.* /var/lib/ambari-server/resources/etc/security/keytabs/" >> gen_keytabs.sh
# ReGenerate Keytabs for all the required Service Account, EXCEPT for the HTTP service account on the IPA Server host.
grep SERVICE /tmp/kerberos.csv | awk -F"," '{print "ipa-getkeytab -s '${ipa_server}' -p "$3" -k "$6";chown "$7":"$9,$6";chmod "$11,$6}' | sort -u | grep -v HTTP\/${ipa_server} >> gen_keytabs.sh
# Allow the 'admins' group to retrieve the keytabs.
grep SERVICE /tmp/kerberos.csv | awk -F"," '{print "ipa service-allow-retrieve-keytab "$3" --group=admins"}' | sort -u >> gen_keytabs.sh
bash ./gen_keytabs.sh
# Now remove the keytabs, they'll be replaced by the distribution phase.
mv -f /etc/security/keytabs /etc/security/genedkeytabs.`date +%Y%m%d%H%M%S`
mkdir /etc/security/keytabs
chown root:hadoop /etc/security/keytabs
Build a distribution script used to create host specific keytabs (adjust for your Ambari Host/Port). vi retrieve_keytabs.sh
# Set the location of Ambari
AMBARI_HOST_PORT=<ambari_host>
# Retrieve the kerberos.csv file from the wizard
wget http://${AMBARI_HOST_PORT}/resources/kerberos.csv -O /tmp/kerberos.csv
ipa_server=$(grep server /etc/ipa/default.conf | awk -F= '{print $2}')
if [ "${ipa_server}X" == "X" ]; then
ipa_server=$(grep host /etc/ipa/default.conf | awk -F= '{print $2}')
fi
if [ ! -d /etc/security/keytabs ]; then
mkdir -p /etc/security/keytabs
fi
chown root:hadoop /etc/security/keytabs/
# Retrieve WITHOUT recreating the existing keytabs for each account.
grep USER /tmp/kerberos.csv | awk -F"," '/'$(hostname -f)'/ {print "wget http://'$( echo ${AMBARI_HOST_PORT})'/resources"$6" -O "$6";chown "$7":"$9,$6";chmod "$11,$6}' | sort -u > get_host_keytabs.sh
grep SERVICE /tmp/kerberos.csv | awk -F"," '/'$(hostname -f)'/ {print "ipa-getkeytab -s '$(echo $ipa_server)' -r -p "$3" -k "$6";chown "$7":"$9,$6";chmod "$11,$6}' | sort -u >> get_host_keytabs.sh
bash ./get_host_keytabs.sh
Copy file to the Ambari Servers resource directory for distribution. scp retrieve_keytabs.sh root@<ambari_host>:/var/lib/ambari-server/resources
On the Each HDP node via 'pdsh' authenticate and create the keytabs # Should be logging in as 'root'
pdsh -g <host_group> -l root
> ## authenticate as the KDC Admin
> echo '<kerberos_password>' | kinit --password-file=STDIN admin
> # or (for IPA 4)
> echo '<kerberos_password>' | kinit -X password-file=STDIN admin
> wget http://<ambari_host>:8080/resources/retrieve_keytabs.sh -O /tmp/retrieve_keytabs.sh
> bash /tmp/retrieve_keytabs.sh
> ## Verify kinit works before proceeding (should not give errors)
> # Service Account Check (replace REALM with yours)
> sudo -u hdfs kinit -kt /etc/security/keytabs/nn.service.keytab nn/$(hostname -f)@HORTONWORKS.COM
> # Headless Check (check on multiple hosts)
> sudo -u ambari-qa kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa@HORTONWORKS.COM
> sudo -u hdfs kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@HORTONWORKS.COM
WARNING: IPA UI maybe broken after above procedure When the process to build keytabs for services is run on the same host that IPA lives on, it will invalidate the keytab used by Apache HTTPD to authenticate. I've added a step that should eliminate the "re"-creation of the key tab, but just incase.. Replace /etc/httpd/conf/ipa.keytab with /etc/security/keytabs/spnego.service.keytab cd /etc/httpd/conf
mv ipa.keytab ipa.keytab.orig
cp /etc/security/keytabs/spnego.service.keytab ipa.keytab
chown apache:apache ipa.keytab
service httpd restart
Remove the headless.keytabs.tgz file from /var/lib/ambari-server/resources on the Ambari-Server. Press next on security wizard and proceed to stop services Proceed with the next steps of wizard by clicking Next Once completed, click Complete and now the cluster is kerborized Using your Kerberized cluster Try to run commands without authenticating to kerberos. $ hadoop fs -ls /
15/07/15 14:32:05 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
$ curl -u someuser -skL "http://$(hostname -f):50070/webhdfs/v1/user/?op=LISTSTATUS"
<title>Error 401 Authentication required</title>
Get a token ## for the current user
sudo su - gooduser
kinit
## for any other user
kinit someuser
Use the cluster Hadoop Commands $ hadoop fs -ls /
Found 8 items
[...]
WebHDFS ## note the addition of `--negotiate -u : `
curl -skL --negotiate -u : "http://$(hostname -f):50070/webhdfs/v1/user/?op=LISTSTATUS"
Hive (using Beeline or another Hive JDBC client)
Hive in Binary mode (the default) beeline -u "jdbc:hive2://localhost:10000/default;principal=hive/$(hostname -f)@HORTONWORKS.COM"
Hive in HTTP mode ## note the update to use HTTP and the need to provide the kerberos principal.
beeline -u "jdbc:hive2://localhost:10001/default;transportMode=http;httpPath=cliservice;principal=HTTP/$(hostname -f)@HORTONWORKS.COM"
Thank you to @abajwa@hortonworks.com (Ali Bajwa) for his original workshop this is intended to extend. https://github.com/abajwa-hw/security-workshops
... View more
10-02-2015
08:51 PM
5 Kudos
The Phoenix documentation here leaves out a few pieces in order to make a successful connection to HBase, through the Phoenix Driver. They assume that the connection is from the 'localhost'. May work great, but that's unlikely in the real world. Required Jars phoenix-client.jar hbase-client.jar (not mentioned in the Phoenix Docs) URL Details The full adbc connection URL syntax for phoenix is: Basic Connections
jdbc:phoneix[:zk_quorum][:zk_port][:zk_hbase_path] The "zk_quorum" is a comma separated list of the ZooKeeper Servers. The "zk_port" is the ZooKeeper port. The "zk_hbase_path" is the path used by Hbase to stop information about the instance. On a 'non' kerberized cluster the default zk_hbase_path for HDP is '/hbase-unsecure'. Sample JDBC URL
jdbc:phoenix:m1.hdp.local,m2.hdp.local,d1.hdp.local: 2181 :/hbase-unsecure
For Kerberos Cluster Connections
jdbc:phoneix[:zk_quorum][:zk_port][:zk_hbase_path][:headless_keytab_file:principal]
Sample JDBC URL
jdbc:phoenix:m1.hdp.local,m2.hdp.local,d1.hdp.local: 2181 :/hbase-secure:/Users/dstreev/keytabs/myuser.headless.keytab:dstreev @HDP .LOCAL They say that the above items in the url beyond 'phoenix' are optional if the clusters 'hbase-site.xml' file is in the path. I found that when I presented a copy of the 'hbase-site.xml' from the cluster and left off the optional elements, I got errors referencing 'Failed to create local dir'. When I connected successfully from DBVisualizer to HBase with the Phoenix JDBC Driver, it took about 10-20 seconds to connect.
... View more
Labels:
10-02-2015
08:48 PM
7 Kudos
httpfs is needed to support a centralized WebHDFS interface to an HA enable NN Cluster. This can be used by Hue or any other WebHDFS enabled client that needs to use a cluster configured with a High-Availability Namenode. The installation is a piece of cake: yum install hadoop-httpfs But that's were the fun ends!!! Configuring is a whole other thing. It's not hard, if you know the right buttons to push. Unfortunately, the buttons and directions for doing this can be quite aloof. The httpfs service is a tomcat application that relies on having the Hadoop libraries and configuration available, so it can resolve your HDP installation. When you do the installation (above), a few items are installed. /usr/hdp/2.2.x.x-x/hadoop-httpfs
/etc/hadoop-httpfs/conf
/etc/hadoop-httpfs/tomcat-deployment Configuring - The Short Version Set the version for current with hdp-select set hadoop-httpfs 2.2.x.x-x From this point on, many of our changes are designed to "fix" the "hardcoded" implementations in the deployed scripts. Adjust the /usr/hdp/current/hadoop-httpfs/sbin/httpfs.sh script #!/bin/bash
# Autodetect JAVA_HOME if not defined
if [ -e /usr/libexec/bigtop-detect-javahome ]; then
. /usr/libexec/bigtop-detect-javahome
elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
. /usr/lib/bigtop-utils/bigtop-detect-javahome
fi
### Added to assist with locating the right configuration directory
export HTTPFS_CONFIG=/etc/hadoop-httpfs/conf
### Remove the original HARD CODED Version reference... I mean, really???
export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client}
export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec
exec /usr/hdp/current/hadoop-httpfs/sbin/httpfs.sh.distro "$@" Now let's create a few symlinks to connect the pieces together cd /usr/hdp/current/hadoop-httpfs
ln -s /etc/hadoop-httpfs/tomcat-deployment/conf conf
ln -s ../hadoop/libexec libexec
Like all the other Hadoop components, httpfs follows use *-env.sh files to control the startup environment. Above, in the httpfs.sh script we set the location of the configuration directory. That is used to find and load the httpfs-env.sh file we'll modified below. # Add these to control and set the Catalina directories for starting and finding the httpfs application
export CATALINA_BASE=/usr/hdp/current/hadoop-httpfs
export HTTPFS_CATALINA_HOME=/etc/hadoop-httpfs/tomcat-deployment
# Set a log directory that matches your standards
export HTTPFS_LOG=/var/log/hadoop/httpfs
# Set a tmp directory for httpfs to store interim files
export HTTPFS_TEMP=/tmp/httpfs That's it!! Now run it! cd /usr/hdp/current/hadoop-httpfs/sbin
./httpfs.sh start
# To Stop
./httpfs.sh stop Try it out!! http://m1.hdp.local:14000/webhdfs/v1/user?user.name=hdfs&op=LISTSTATUS Obviously, changing out with your target host. The default port is 14000. If you want to change that, add the following to: export HTTPFS_HTTP_PORT=<new_port> Want to Add httpfs as a Service (auto-start)? The HDP installation puts a set of init.d files in the specific versions directory. cd /usr/hdp/<hdp.version>/etc/rc.d/init.d Create a symlink to this in /etc/init.d ln -s /usr/hdp/<hdp.version>/etc/rc.d/init.d/hadoop-httpfs /etc/init.d/hadoop-httpfs Then set up the service to run on restart # As Root User
chkconfig --add hadoop-httpfs # Start Service
service hadoop-httpfs start
# Stop Service
service hadoop-httpfs stop This method will run the service as the 'httpfs' user. Ensure that the 'httpfs' user has permissions to write to the log directory (/var/log/hadoop/httpfs if you followed these directions). A Little More Detail Proxies are fun, aren't they? We'll they'll affect you here as well. The directions here mention these proxy settings in core-site.xml. <property>
<name>hadoop.proxyuser.httpfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.httpfs.hosts</name>
<value>*</value>
</property> This means that httpfs.sh must be run as the httpfs user, in order to work. If you want to run the service with another user, adjust the proxy settings above.
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
... View more
Labels:
09-29-2015
06:12 PM
1 Kudo
With distcp, you'll be able to encrypt a large set of data faster. Don't forget to add the distcp command options: --skipcrccheck --update
... View more
09-29-2015
01:16 AM
1 Kudo
Is this an upgraded cluster? From which version? Was Ranger installed before the upgrade? xasecure-audit.xml indicates an older version of Ranger. I just check a new install of Ranger on HDP 2.3 and those files are ranger-hdfs-*.xml. Maybe there's an old configuration file in the /etc/hadoop/conf directory. Can you list all the files in the conf directory?
... View more