I am experience java programmer and want to shift in Data Science. I would like to apply machine learning algorithm on very very large data set (few TB to PB). Which language is preferable to use - Scala / Python / R ? ... View more

Is there any certification for HBase ? How to prepare it? ... View more

I am trying to load data from RDBMS to kafka topic. I am using kafka jdbc connector provided by Confluent. My questions are 1. If I have millions of data in the database, How can I proceed with multi treading option ? 2. Can we use multiple broker in Kafka connect 3. How can we implement security in this offload from RDBMS to Kafka topic? 4. During data offload lets say my server goes down . How it will behave after kafka server restart? Please help. This is urgent. ... View more

I am planning for Spark certification . Could you please let me know which one is best for career perspective for Spark developer 1. Databricks Certified Developer (From Databricks) 2. HDP CERTIFIED APACHE SPARK DEVELOPER (From Hortonworks) 3. CCA Spark and Hadoop Developer (From Cloudera) Please suggest. ... View more

I would like to grant permission to HBase table using java API. is there any java api available for HBase 1.1.2 ? ... View more

I am trying to connect hive non exist database using jdbc url . It does not throw any error and it is connecting with default database which should not be. This issue is occurring for hive-jdbc-1.2.1. if we use version hive-jdbc-0.12.0-cdh5.1.2 then we are getting proper exception showing Database does not exist. I am using HDP 2.4.2 and Hive 1.2.1.2.4 . I do not want to use cloudera jar. My code snapshot is given below where "acb" is unknown database. Class.forName("org.apache.hive.jdbc.HiveDriver"); conn = DriverManager.getConnection("jdbc:hive2://*****.***.***:10000/acb"); Is it bug for hive jdbc driver ? Please help to fix this issue asap. ... View more

I want to offload data from Teradata to HDFS in HDC for AWS using sqoop. . Which teradata connector we can use for data offload? Please advice urgent basis. ... View more

I want to connect HBase in HDinsight using java api. My client machine is located at outside of HDInsight cluster. I am getting below error Mon Feb 06 15:40:51 IST 2017, RpcRetryingCaller{globalStartTime=1486374714593, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at com.bigframe.hbase.api.HDInsightHbase.main(HDInsightHbase.java:43) Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ... 4 more Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:906) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ... 8 more Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) ... 11 more MY code snapshot is below Configuration config = HBaseConfiguration.create(); config.set("hbase.zookeeper.quorum", "zookeepernode0,zookeepernode1,zookeepernode2"); config.set("hbase.zookeeper.property.clientPort", "2181"); config.set("hbase.cluster.distributed", "true"); config.set("zookeeper.znode.parent","/hbase-unsecure"); HBaseAdmin admin = new HBaseAdmin(config); TableName[] tm = admin.listTableNames(); for(int i = 0; i<tm.length;i++){ System.out.println(tm[i].getNameAsString()); } Same code is running from inside HDInsight cluster . So I do not have any issue in java code. Please provide suggestion for running this code from outside the cluster. As it is cloud platform, I do not have idea regarding firewall ip/port. This is urgent. Please help. ... View more

I am getting below error from from client machine (outside Hadoop cluster) when trying to access Hbase from java api Mon Feb 06 15:40:51 IST 2017, RpcRetryingCaller{globalStartTime=1486374714593, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at com.bigframe.hbase.api.HDInsightHbase.main(HDInsightHbase.java:43) Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ... 4 more Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:906) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ... 8 more Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) ... 11 more MY code snapshot is below Configuration config = HBaseConfiguration.create(); config.set("hbase.zookeeper.quorum", "zookeepernode0,zookeepernode1,zookeepernode2"); config.set("hbase.zookeeper.property.clientPort", "2181"); config.set("hbase.cluster.distributed", "true"); config.set("zookeeper.znode.parent","/hbase-unsecure"); HBaseAdmin admin = new HBaseAdmin(config); TableName[] tm = admin.listTableNames(); for(int i = 0; i<tm.length;i++){ System.out.println(tm[i].getNameAsString()); } Same code is running fine from inside hadoop cluster . Should I raise firewall request for hbase.zookeeper.quorum nodes from client machine ? Please help. This is urgent. ... View more

I am trying to execute below query in kerberised cluster with Hive principal and keytab from java api. This query will create table in Hive and as well as in HBase also. CREATE TABLE hive_aps (key struct<f1:string, f2:string>, value string) ROW FORMAT DELIMITED COLLECTION ITEMS TERMINATED BY '~' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (\"hbase.columns.mapping\"=\":key,f:c1\") TBLPROPERTIES (\"hbase.table.name\" = \"hbase_aps\", \"hbase.mapred.output.outputtable\" = \"hbase_aps\") And getting below error java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=hive, scope=default, params=[namespace=default,table=default:hbase_aps,family=f],action=CREATE) at org.apache.hadoop.hbase.security.access.AccessController.requireNamespacePermission(AccessController.java:622) at org.apache.hadoop.hbase.security.access.AccessController.preCreateTable(AccessController.java:991) at org.apache.hadoop.hbase.master.MasterCoprocessorHost$11.call(MasterCoprocessorHost.java:216) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1140) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preCreateTable(MasterCoprocessorHost.java:212) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1533) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:226) at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:240) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:140) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4083) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:723) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:644) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:577) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:217) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:670) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:663) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156) at com.sun.proxy.$Proxy9.createTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:717) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4271) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:311) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1728) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1485) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1121) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:419) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:400) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy20.executeStatement(Unknown Source) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:263) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException): org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=hive, scope=default, params=[namespace=default,table=default:hbase_aps,family=f],action=CREATE) at org.apache.hadoop.hbase.security.access.AccessController.requireNamespacePermission(AccessController.java:622) at org.apache.hadoop.hbase.security.access.AccessController.preCreateTable(AccessController.java:991) at org.apache.hadoop.hbase.master.MasterCoprocessorHost$11.call(MasterCoprocessorHost.java:216) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1140) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preCreateTable(MasterCoprocessorHost.java:212) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1533) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1226) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.createTable(MasterProtos.java:58320) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$4.createTable(ConnectionManager.java:1821) at org.apache.hadoop.hbase.client.HBaseAdmin$5.call(HBaseAdmin.java:728) at org.apache.hadoop.hbase.client.HBaseAdmin$5.call(HBaseAdmin.java:724) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ... 50 more ) at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:167) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:155) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:210) at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:333) at com.bigframe.hive.api.HiveClient_Kerberos.doCreateHiveTable(HiveClient_Kerberos.java:114) at com.bigframe.hive.api.HiveClient_Kerberos.main(HiveClient_Kerberos.java:220) This query is able to create table in Hive but failing for HBase due to permission issue. Because Hive principal does not have permission for create table in HBase. What should I do? Please help. This is urgent. ... View more