Support Questions
Find answers, ask questions, and share your expertise

Geomesa in secured cluster with kerberos

I am using HDP 2.6.5 10 node cluster with 6 data nodes and secured with kerberos. Recently I installed Geomesahbase 1.3.5 in cluster. I created kerberos principal and keytabs for that node. I can able ingest csv data from local file system to geomesahbase data store but I am not able to ingest from HDFS directory. I am getting kerberos ticket issue.

Does kerberos support geomesahbase?

Is there anything i have to do?

21 REPLIES 21

Mentor

@santhosh kumar rathode

Can you share the error log. The keytabs you create should is valid cluster-wide but without attaching the error it won't be possible to help out.

I have geomesahbase in edge node.

I am running as hbaseGeomesa user:

I created same user in all the nodes.

So i created principal hbaseGeomesa/<edgenode_IP>.realm and keytabs.

2019-02-13 17:48:25,814 INFO [main] tools.user: Tracking available at http://<hostname>:8088/proxy/application_1549479078340_0328/ [============================================================] 100% complete 0 ingested 0 failed in 00:01:26 2019-02-13 17:49:46,718 ERROR [main] tools.user: Job failed with state FAILED due to: Task failed task_1549479078340_0328_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 2019-02-13 17:49:46,721 INFO [main] tools.user: Distributed ingestion complete in 00:01:26 2019-02-13 17:49:46,722 INFO [main] tools.user: Ingested 0 features with no failures. 2019-02-13 17:49:46,723 INFO [Thread-7] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2019-02-13 17:49:46,723 INFO [Thread-7] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x168e345e329263c 2019-02-13 17:49:46,724 INFO [Thread-7] zookeeper.ZooKeeper: Session: 0x168e345e329263c closed 2019-02-13 17:49:46,724 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down

	2019-02-13 11:49:09,058 FATAL [IPC Server handler 3 on 45782] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1549479078340_0328_m_000000_0 - exited : org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Could not set up IO Streams to fqdn:16000

	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1560)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1580)

	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1731)

	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isMasterRunning(ConnectionManager.java:946)

	at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3255)

	at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.checkClusterAvailability(HBaseDataStoreFactory.scala:99)

	at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.org$locationtech$geomesa$hbase$data$HBaseDataStoreFactory$$globalConnection$lzycompute(HBaseDataStoreFactory.scala:44)

	at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.org$locationtech$geomesa$hbase$data$HBaseDataStoreFactory$$globalConnection(HBaseDataStoreFactory.scala:41)

	at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory$$anonfun$2.apply(HBaseDataStoreFactory.scala:61)

	at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory$$anonfun$2.apply(HBaseDataStoreFactory.scala:61)

	at scala.Option.getOrElse(Option.scala:121)

	at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:61)

	at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:36)

	at org.geotools.data.DataAccessFinder.getDataStore(DataAccessFinder.java:130)

	at org.geotools.data.DataStoreFinder.getDataStore(DataStoreFinder.java:89)

	at org.locationtech.geomesa.jobs.mapreduce.GeoMesaRecordWriter.<init>(GeoMesaOutputFormat.scala:83)

	at org.locationtech.geomesa.jobs.mapreduce.GeoMesaOutputFormat.getRecordWriter(GeoMesaOutputFormat.scala:60)

	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:647)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:422)

	p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545}
span.Apple-tab-span {white-space:pre}

	Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:422)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745)

	... 33 more

	Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:422)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)

	... 33 more

	Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)

Mentor

@santhosh kumar rathode

I can see the error is related to Kerberos "Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]"

To validate you have no valid ticket as user ...... run

$ klist 

Expected output

klist: No credentials cache found (filename: /tmp/krb5cc_1013) 

That confirms you have no valid ticket

# Solution

To resolve the problem you will need a valid Kerberos ticket please follow the steps below assuming your keytab is named hbaseGeomesa and is located in your home directory /home/{user}

$ klist -kt /home/{user}/hbaseGeomesa.keytab 

This will give you the principal attached to the above keytab assuming its also hbaseGeomesa.keytab

Keytab name: FILE:/home/{user}/hbaseGeomesa.keytab 
KVNO         Timestamp       Principal 
---- ----------------- -------------------------------------------------------- 
1 02/13/19 22:25:31 hbaseGeomesa/<edgenode@{REALM} 
1 02/13/19 22:25:31 hbaseGeomesa/<edgenode@{REALM} 
1 02/13/19 22:25:31 hbaseGeomesa/<edgenode@{REALM} 
1 02/13/19 22:25:31 hbaseGeomesa/<edgenode@{REALM} 
1 02/13/19 22:25:31 hbaseGeomesa/<edgenode@{REALM} 

With the above output as user hbaseGeomesa ouput grab a Kerberos ticket with the below step by concatenating

kinit -kt $keytab $ principal see below example

$ kinit -kt /home/{user}/hbaseGeomesa.keytab hbaseGeomesa/<edgenode@{REALM} 

Now you should see an output similar to this

$ klist Ticket cache: FILE:/tmp/krb5cc_1013 
Default principal: hbaseGeomesa/<edgenode@{REALM} 
Valid          starting       Expires                 Service principal 
02/13/2019     22:50:16       02/14/2019 21:50:16      krbtgt/<edgenode@{REALM} 

The best solution is to ingest from your own hdfs home , to create that you need to switch to the hdfs user and complete the below steps

# create your home directory

$ hdfs dfs -mkdir /user/hbaseGeomesa 

# change the correct permissions

$ hdfs dfs -chown hbaseGeomesa:hdfs /user/hbaseGeomesa 

Copy the files you want to ingest to hdfs from local

$ hdfs dfs -copyFromLocal /path_to-your_file/files_to_ingest /user/hbaseGeomesa 

Now you can launch your ingesting from HDFS directory !!! The user running the ingestion should have at least read permission else the ingestion will fail.

HTH

@Geoffrey Shelton Okot

As you explained:

I have user in hdfs and with RWCA permission, i copied csv file to /user/hbaseGeomsa/file_.csv

I did kinit -kt /home/{user}/hbaseGeomesa.keytab hbaseGeomesa/<edgenode@{REALM}

I have ownership of user --> hbaseGeomesa:hdfs /user/hbaseGeomesa

As i told you before i can ingest csv data from local to geomesahbase datastore(it works fine)

	.bin/geomesa-hbase ingest -c geo-csv -s /home/hbaseGeomesa/geo.sft -C /home/hbaseGeomesa/geo.convert /home/hbaseGeomesa/geo.csv

when i am trying to ingest from hdfs://clustename/user/hbaseGeomesa/file_.csv , i am getting above error.

I am using below command to run:

	.bin/geomesa-hbase ingest -c geo-csv -s /home/hbaseGeomesa/geo.sft -C /home/hbaseGeomesa/geo.convert hdfs://clustername/user/hbaseGeomesa/*

Mentor

@santhosh kumar rathode

When you are ingesting from hdfs the first input should be the hdfs directory I was expecting something like

.bin/geomesa-hbase ingest  -c /user/hbaseGeomesa/geo-csv .............

Where the csv is located in hdfs

Without hdfs://clustername/ location it won't search from hdfs.

If i give /user/hbaseGeomesa/geo-csv ............. it searching for directory in local and says no file exist.

New Contributor

When ingesting a local file, GeoMesa runs the ingest process locally. When ingesting from HDFS, GeoMesa launches a map/reduce job to ingest. It seems like the tasks are not able to authenticate with kerberos. Would you need to create a ticket on each task node?

,

When ingesting from a local file, the GeoMesa ingest will run locally on the current machine. When ingesting from HDFS, it will launch a map/reduce job. It seems like the remote tasks are not able to authenticate with kerberos. Would you need to create a ticket on each node in the cluster?

@Emilio Lahr-Vivaz and @Geoffrey Shelton Okot

Yes Emilio. when i am trying to run from HDFS its launched map/reduce job to ingest. right now I created principal for edge node as hbaseGeomesa/<edgenode_fqdn>.realm and keytab as hbase.geomesa.keytab.

Do i need to create principal for all the nodes?

ex: hbaseGeomesa/<node1_fqdn>.realm

hbaseGeomesa/<node2_fqdn>.realm .........so on ?

or just i need to create hbaseGeomesa/<edgenode_fqdn>.realm keytab in all the nodes.


another question is :

right now i added hbaseGeomesa/<edgenode_fqdn>.realm and hbase.geomesa.keytab in hbase-site.xml.

is that fine or do i need add all the node principal in hbase-site.xml.