Member since
05-19-2016
93
Posts
17
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5277 | 01-30-2017 07:34 AM | |
3639 | 09-14-2016 10:31 AM |
06-07-2017
04:44 AM
2 Kudos
I am experience java programmer and want to shift my career in Data Science and Machine Learning . Which language needs to be learn - R or Python ?
... View more
05-08-2017
11:21 AM
I am trying to load data from RDBMS to kafka topic. I am using kafka jdbc connector provided by Confluent. My questions are 1. If I have millions of data in the database, How can I proceed with multi treading option ? 2. Can we use multiple broker in Kafka connect 3. How can we implement security in this offload from RDBMS to Kafka topic? 4. During data offload lets say my server goes down . How it will behave after kafka server restart? Please help. This is urgent.
... View more
Labels:
- Labels:
-
Apache Kafka
02-08-2017
09:20 AM
@Josh Elser I am using HDInsight cluster in cloud. is it possible to raise firewall request for cloud host/port from local machine?
... View more
02-08-2017
09:19 AM
@Artem Ervits I am using HDInsight cluster in cloud. is it possible to raise firewall request for cloud host/port from local machine?
... View more
02-08-2017
09:18 AM
@Josh Elser I am using HDInsight cluster in cloud. is it possible to raise firewall request for cloud host/port?
... View more
02-08-2017
06:59 AM
This issue is resolved after setting proper configuration in hive.
... View more
02-08-2017
06:18 AM
I am trying to execute below query from java api CREATE TABLE hive_aps (key struct<f1:string, f2:string>, value string) ROW FORMAT DELIMITED COLLECTION ITEMS TERMINATED BY '~' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping"=":key,f:c1") TBLPROPERTIES ("hbase.table.name" = "hbase_aps", "hbase.mapred.output.outputtable" = "hbase_aps") And getting below error java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:405)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:415)
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:214)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:690)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:683)
at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:159)
at com.sun.proxy.$Proxy14.createTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:757)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4322)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:314)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at com.bigframe.perl.api.HiveHbaseIntegrationHDInsight.main(HiveHbaseIntegrationHDInsight.java:72) Above query is running fine from hive CLI and it is creating table in hive as well as in HBase. I am getting this issue only from java api. Also I am able to create simple hive table from java . If I try hive and hbase together , getting above issue. Please help. This is urgent.
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Hive
02-06-2017
12:53 PM
@Artem Ervits Thanks for your quick reply . How can I check this ? using telnet ? I am running below command from client machine telnet zookeepernode0 2181 I am getting below error telnet: Unable to connect to remote host: Connection refused Please suggest.
... View more
02-06-2017
12:21 PM
I am getting below error from from client machine (outside Hadoop cluster) when trying to access Hbase from java api Mon Feb 06 15:40:51 IST 2017, RpcRetryingCaller{globalStartTime=1486374714593, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397)
at com.bigframe.hbase.api.HDInsightHbase.main(HDInsightHbase.java:43)
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704)
at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
... 4 more
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:906)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524)
... 8 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895)
... 11 more MY code snapshot is below Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "zookeepernode0,zookeepernode1,zookeepernode2");
config.set("hbase.zookeeper.property.clientPort", "2181");
config.set("hbase.cluster.distributed", "true");
config.set("zookeeper.znode.parent","/hbase-unsecure");
HBaseAdmin admin = new HBaseAdmin(config);
TableName[] tm = admin.listTableNames();
for(int i = 0; i<tm.length;i++){
System.out.println(tm[i].getNameAsString());
} Same code is running fine from inside hadoop cluster . Should I raise firewall request for hbase.zookeeper.quorum nodes from client machine ? Please help. This is urgent.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
01-30-2017
07:34 AM
This issue has been fixed after setting below properties in kms-site.xml. hadoop.kms.proxyuser.hive.users=*
hadoop.kms.proxyuser.hive.hosts =* Now I am not getting any authentication error.
... View more