Member since
11-01-2013
11
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 14940 | 05-05-2014 10:57 AM |
05-05-2014
10:57 AM
Hello Clint, I followed the directions in an article told by Ryan Geno (Cloudera Employee) and it seems to have fixed the issue - http://prodlife.wordpress.com/2013/11/22/using-oozie-in-kerberized-cluster/ - go to “Oozie service->Configuration->Oozie Server(default)->Advanced-> Oozie Server Configuration Safety Valve for oozie-site.xml” and add: 1 2 3 4 <property> <name>oozie.credentials.credentialclasses</name> <value>hcat=org.apache.oozie.action.hadoop.HCatCredentials</value> </property> After I restarted the Oozie Service and ran the ResourceClipper Job – it completed…. Thanks Murthy
... View more
04-25-2014
11:30 AM
Hello Clint, Yes we are using CDH 4.6... I have copied the hbase-site.xml to /user/oozie/share/lib/hbase folder in HDFS. I got the error - i have restarted the cluster also... Here are some lines from the syslog - 2014-04-25 13:19:41,310 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection 2014-04-25 13:19:41,331 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 42965@ldxhdfsw4.dx.deere.com 2014-04-25 13:19:41,337 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2014-04-25 13:19:41,339 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused Murthy
... View more
04-21-2014
08:04 AM
Here is the information from the syslog - the syslog shows hundreds of the following errors until the job dies: 2014-04-21 08:44:26,676 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2014-04-21 08:44:26,676 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075) 2014-04-21 08:44:26,686 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid 2014-04-21 08:44:26,686 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1...
... View more
04-21-2014
07:26 AM
Hello, We are trying to execute a MapReduce job to write to HBASE using OOZIE Workflow. We have implemented Kerberos in our Cluster and that is when these issues started.... The Job is failing after 10 minutes with an error - java.lang.RuntimeException: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for supertemp,,99999999999999 after 10 tries. We had a similar issue with beeline not able to run HBASE Commands and the solution was - I think the issue is that HBase does not allow hive to impersonate users. So you'll need to setup hive as a proxy user in HBase. Can you try the following: - Go to the HDFS service configuration in CM. - Go to Service-Wide->Advanced and add the following to "Cluster-wide Configuration Safety Valve for core-site.xml": <property> <name>hadoop.proxyuser.hive.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hive.groups</name> <value>*</value> </property> I have added similar entry for OOZIE - HDFS Service-Wide Advanced properties and add the following to Cluster-wide ConfigurationSafety Valve for core-site.xml <property> <name>hadoop.proxyuser.oozie.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.oozie.groups</name> <value>*</value> </property> Here is what i have in the logs - Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.MapReduceMain], main() threw exception, org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for supertemp,,99999999999999 after 10 tries. java.lang.RuntimeException: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for supertemp,,99999999999999 after 10 tries. at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1010) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:974) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:974) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:948) at org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:97) at org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:57) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for supertemp,,99999999999999 after 10 tries. at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:980) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:885) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:987) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:889) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:846) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:270) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:210) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:169) at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:201) ... 26 more Oozie Launcher failed, finishing Hadoop job gracefully
... View more
Labels:
04-10-2014
09:45 AM
Hello We are trying to create a HBASE table in beeline and getting an error - We have implemented Kerberos security... This is how we are logging in to beeline - beeline> !connect jdbc:hive2://<Servername>:10000/default;principal=hive/<Kerberos Realm> org.apache.hive.jdbc.HiveDriver scan complete in 3ms Connecting to jdbc:hive2://<Servername>:10000/default;principal=hive/<kerberos Realm> Enter password for jdbc:hive2://<Servername>:10000/default;principal=hive/<kerberos Realm>: Connected to: Apache Hive (version 0.10.0) Driver: Hive (version 0.10.0-cdh4.6.0) Transaction isolation: TRANSACTION_REPEATABLE_READ Here is the command we are running to create the HBASE Table - 0: jdbc:hive2://<Servername>:10000/> CREATE EXTERNAL TABLE hbase_supercells_01(key string, ssurgo map<string,binary>, counties map<string,binary>, mpe map<string,binary>) . . . . . . . . . . . . . . . . . . . . . . .> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' . . . . . . . . . . . . . . . . . . . . . . .> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,ssurgo:,counties:,mpe:") . . . . . . . . . . . . . . . . . . . . . . .> TBLPROPERTIES ("hbase.table.name" = "supercells.01"); Error – 2014-04-03 16:01:06,707 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: getMaster attempt 4 of 14 failed; retrying after sleep of 2008 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive/<kerberos Realm> is not allowed to impersonate mv66708 We have updated the property in Cloudera Manager --> Services --> Hive1 -->HiveServer2(Default) HiveServer2 Enable Impersonation - this is checked We have updated the property in Cloudera Manager --> Services --> Hive1 -->HiveServer2 --> Advanced HiveServer2 Configuration Safety Valve for hive-site.xml <property> <name>hive.server2.authentication</name> <value>KERBEROS</value> </property> <property> <name>hive.aux.jars.path</name> <value>file:///opt/cloudera/parcels/CDH/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///opt/cloudera/parcels/CDH/lib/hbase/hbase.jar,file:///opt/cloudera/parcels/CDH/lib/zookeeper/zookeeper.jar,file:///opt/cloudera/parcels/CDH/lib/hive/lib/guava-11.0.2.jar,file:///opt/cloudera/parcels/CDH/lib/hadoop/lib/hive-serdes-1.0-SNAPSHOT.jar,file:///opt/cloudera/parcels/CDH/lib/hadoop/lib/TwitterUtil.jar</value> </property> <property> <name>hbase.security.authentication</name> <value>kerberos</value> </property> <property> <name>mapred.input.pathFilter.class</name> <value>com.twitter.util.FileFilterExcludeTmpFiles</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>ldxhdfsm3.dx.deere.com,ldxhdfsm2.dx.deere.com,ldxhdfsm1.dx.deere.com</value> </property> <property> <name>hbase.master.kerberos.principal</name> <value>hbase/_HOST@HADOOP.DEV.DEERE.COM</value> </property> <property> <name>hbase.regionserver.kerberos.principal</name> <value>hbase/_HOST@HADOOP.DEV.DEERE.COM</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property>
... View more