baig_mujeeb
baig_mujeeb
Explorer
My Accepted Solutions
Show All

Exception submitting spark job in HDP 3.0

by Explorer baig_mujeeb in Support Questions
‎07-31-2019 03:16 PM
‎07-31-2019 03:16 PM
Hi, I am getting the following exception on one node when submitting a spark job, it is working fine if submitted from any other node. Any help will be highly appreciated. 19/07/31 15:56:19 INFO Executor: Starting executor ID driver on host localhost 19/07/31 15:56:19 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35492. 19/07/31 15:56:19 INFO NettyBlockTransferService: Server created on hdata5.dom.local:35492 19/07/31 15:56:19 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 19/07/31 15:56:19 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hdata5.dom.local, 35492, None) 19/07/31 15:56:19 INFO BlockManagerMasterEndpoint: Registering block manager hdata5.dom.local:35492 with 366.3 MB RAM, BlockManagerId(driver, hdata5.dom.local, 35492, None) 19/07/31 15:56:19 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hdata5.dom.local, 35492, None) 19/07/31 15:56:19 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, hdata5.dom.local, 35492, None) Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/Class;)Lorg/apache/hadoop/io/retry/RetryPolicy;         at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:318)         at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:235)         at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139)         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510)         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)         at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)         at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:468)         at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1897)         at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:74)         at org.apache.spark.SparkContext.<init>(SparkContext.scala:520)         at com.dom.pipeline.spark.MergeDom$.main(MergeDom.scala:100)         at com.dom.pipeline.spark.MergeDom.main(MergeDom.scala)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 19/07/31 15:56:20 INFO DiskBlockManager: Shutdown hook called 19/07/31 15:56:20 INFO ShutdownHookManager: Shutdown hook called 19/07/31 15:56:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-9dc84539-36a1-40b3-b991-ccf28473cea1 ... View more
Labels:

Re: Unable to start HBase master after enabling ke...

by Explorer baig_mujeeb in Support Questions
‎11-15-2018 12:35 PM
‎11-15-2018 12:35 PM
Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!. ... View more

Re: Zookeeper - HBase master node is missing after...

by Explorer baig_mujeeb in Support Questions
‎11-15-2018 12:16 PM
‎11-15-2018 12:16 PM
Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!. ... View more

Zookeeper - HBase master node is missing after ker...

by Explorer baig_mujeeb in Support Questions
‎11-15-2018 10:07 AM
‎11-15-2018 10:07 AM
Hi, After upgrading to 3.0.1 and enabling kerberos, Hbase master is failing to start due to missing 'master' znode. It is leading to Atlas start failure. atlas TABLE Took 8.2656 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master 2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus ATLAS_ENTITY_AUDIT_EVENTS atlas TABLE Took 8.2337 secondsjava exception In zookeeper 'master' znode is present is hbase-unsecure but not in secure-hbase [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table] [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table] How do I fix this? Is this a known issue in 3.0.1? Thanks in advance. ... View more
Labels:

Re: Unable to start HBase master after enabling ke...

by Explorer baig_mujeeb in Support Questions
‎11-14-2018 05:29 PM
‎11-14-2018 05:29 PM
Some more logs atlas TABLE Took 8.2656 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master 2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus ATLAS_ENTITY_AUDIT_EVENTS atlas TABLE Took 8.2337 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master ... View more

Re: Unable to start HBase master after enabling ke...

by Explorer baig_mujeeb in Support Questions
‎11-14-2018 04:31 PM
‎11-14-2018 04:31 PM
2018-11-14 16:15:03,009 WARN [master/hdata4:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-11-14 16:15:03,009 INFO [master/hdata4:16000] assignment.AssignmentManager: Stopping assignment manager 2018-11-14 16:15:03,032 WARN [master/hdata4:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher 2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.ProcedureExecutor: Stopping 2018-11-14 16:15:03,036 INFO [master/hdata4:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store, isAbort=false 2018-11-14 16:15:03,070 ERROR [master/hdata4:16000] wal.WALProcedureStore: Unable to close the stream org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Client (=DFSClient_NONMAPREDUCE_-826562846_1) is not the lease owner (=DFSClient_NONMAPREDUCE_-1195801889_1: /apps/hbase/data/MasterProcWALs/pv2-00000000000000000011.log (inode 10368947) Holder DFSClient_NONMAPREDUCE_-826562846_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2837) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:685) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:671) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2858) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:928) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:607) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy18.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:550) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy20.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy20.complete(Unknown Source) 2018-11-14 16:15:03,071 INFO [master/hdata4:16000] hbase.ChoreService: Chore service for: master/hdata4:16000.splitLogManager. had [] on shutdown 2018-11-14 16:15:03,071 INFO [master/hdata4:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2018-11-14 16:15:03,071 ERROR [master/hdata4:16000] access.TableAuthManager: Something wrong with the TableAuthManager reference counting: org.apache.hadoop.hbase.security.access.TableAuthManager@7e83992 whose count is null Zookeeper znode listing for unsecure and secure. master znode is missing from hbase-secure [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table] [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table] ... View more

Reinstalling HBase service

by Explorer baig_mujeeb in Support Questions
‎11-14-2018 01:24 PM
‎11-14-2018 01:24 PM
Hi I am still getting this problem https://community.hortonworks.com/questions/226373/unable-to-start-hbase-master-after-enabling-kerber.html so deciding to re-install HBase as we don't have any data in it. Wondering what side effects it will have Is this something can be done safely? ... View more

Unable to start HBase master after enabling kerber...

by Explorer baig_mujeeb in Support Questions
‎11-12-2018 02:20 PM
‎11-12-2018 02:20 PM
After enablling kerberos HBase master failing to restart. On investigation found that in zookeeper /hbase-secure/master node is missing. Zookeeper shows following Acl [zk: localhost:2181(CONNECTED) 1] getAcl /hbase-secure 'world,'anyone : r 'sasl,'hbase : cdrwa 'sasl,'hbase : cdrwa Tried all the threads eg. https://community.hortonworks.com/content/supportkb/151088/how-to-force-remove-znode-with-stale-acl.html https://community.hortonworks.com/articles/82405/how-to-remove-acl-protected-zk-node.html Any ideas? ... View more
Labels:

Re: hive server2 interactive logs

by Explorer baig_mujeeb in Support Questions
‎11-08-2018 11:53 AM
‎11-08-2018 11:53 AM
Many Thanks Sandeep. Yes it is working now. I would like to add more details for fix: Update hive-interactive-env template and hive-env template Hive config properties from Ambari UI. ... View more

hive server2 interactive logs

by Explorer baig_mujeeb in Support Questions
‎11-07-2018 03:15 PM
‎11-07-2018 03:15 PM
Hi Our Hive Server2 Interactive is stopping after around 10 min after restart. I couldn't find hiveserver2Interactive.log in /var/log/hive, hive-server2-interactive.err is showing following: /usr/hdp/current/hive-server2/conf_llap//hive-env.sh: line 43: [: !=: unary operator expected 2018-11-07 14:40:33: Starting HiveServer2 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID = c4e8edc5-ed5e-4f96-ba70-00779ac20dc9 Hive Session ID = b5ad883d-4a65-45f7-a6dd-1ea3797724ea Any help will be appreciated. Thanks ... View more
Labels:

disable spnego

by Explorer baig_mujeeb in Support Questions
‎11-06-2018 05:38 PM
‎11-06-2018 05:38 PM
Hi, What is the correct way to disable SPNego completely for all UI links? ... View more

Data Analytics Studio

by Explorer baig_mujeeb in Support Questions
‎11-06-2018 11:19 AM
‎11-06-2018 11:19 AM
I read about it (Data Analytics Studio) in some threads and want to know more about it (trial/download etc.), is it open source? Thanks ... View more

Re: hdp 3.0.1 upgrade from 2.6.3 pre-requisite

by Explorer baig_mujeeb in Support Questions
‎11-05-2018 04:11 PM
‎11-05-2018 04:11 PM
Since we did not have ACID tables, it worked without pre-upgrade tool. ... View more

Re: keytab regeneration stage failing - upgrading ...

by Explorer baig_mujeeb in Support Questions
‎11-05-2018 04:09 PM
‎11-05-2018 04:09 PM
After some investigation foend that KDC Administrator Credentials to the Ambari Credential Store was not properly set. Then set it from Ambari and it started to work ... View more

keytab regeneration stage failing - upgrading to H...

by Explorer baig_mujeeb in Support Questions
‎11-05-2018 11:27 AM
‎11-05-2018 11:27 AM
When upgrading to 3.0.1 from 2.6.5 from Ambari I am getting following error message An internal system exception occurred: Unable to build keytab regeneration stage Any idea? Where should I look for exception error? Tried restarting KDC but of no help. ... View more

hdp 3.0.1 upgrade from 2.6.3 pre-requisite

by Explorer baig_mujeeb in Support Questions
‎11-04-2018 02:26 PM
‎11-04-2018 02:26 PM
The pre upgrade tool is reporting error requires Hive 2.x. Actual: 1.2.1000 I have Hive Interactive enabled and confirms to this thread https://community.hortonworks.com/questions/55051/no-hive-21-in-hdp-25.html I don't have ACID tables in Hive. Is it OK if I go ahead with version upgrade from Ambari without running pre-upgrade tool? Will it upgrade Hive? Kindly let me know ... View more

Unable to enable hive Interactive Query from Ambar...

by Explorer baig_mujeeb in Support Questions
‎10-24-2018 04:03 PM
‎10-24-2018 04:03 PM
Hi, In HDP 2.6 I am trying to enable Hive Interactive Query, after selecting the host it does not pass through the Kerberos admin authentication screen. It says 'Admin session expiration error' it keeps showing this screen even after entering correct admin login details. I have restarted kerberos client, ambari server. Kerberos service check screen says "Loading...." for sometimes and then disappears. Kerberos kinit is working fine across the cluster. Running out of solutions!!! ... View more

Hive 2.1 in hdp 2.5

by Explorer baig_mujeeb in Support Questions
‎10-24-2018 11:44 AM
‎10-24-2018 11:44 AM
Hi, I am in the process of upgrading hdp from 2.6 to 3.0.1, but got stuck with hive version 1.2. Getting following error from PreUpgradetool 18/10/23 17:55:19 ERROR acid.PreUpgradeTool: PreUpgradeTool failed java.lang.IllegalStateException: preUpgrade requires Hive 2.x. Actual: 1.2.1000 In ambari both YARN pre-emption and Hive Interactive are greyed so unable to change them. My beeline command is as follows and it works fine and loads hive 1.2.1000 beeline -u "jdbc:hive2://data1.dev.local:10000/default;" How can I upgrade to hive 2.0+? At the moment I don't have ACID tables so do I still need to go through preUpgrade tool? will version upgrade not automatically upgrade from 1.2 to 3.0? Regards MB ... View more
Labels:

Re: Spark2 missing keytab file for HDP 2.7

by Explorer baig_mujeeb in Support Questions
‎08-16-2018 10:31 AM
‎08-16-2018 10:31 AM
Thanks for quick reply. I tried but still spark2 service principle is missing. I even checked it with kadmin. Wondering that Spark2 clients are running fine on the respective nodes, but issue is with Spark2 History Server and Spark2 Thrift Server. I can see in Spark2 config that spark.history.kerberos.keytab = /etc/security/keytabs/spark2.headless.keytab Looks like a bug somewhere! ... View more

Spark2 missing keytab file for HDP 2.7

by Explorer baig_mujeeb in Support Questions
‎08-15-2018 04:09 PM
‎08-15-2018 04:09 PM
Hi, Recently upgraded to HDP 2.7 and added Spark2 via Ambari on kerberised cluster. It is failing to start Spark2 history server because of missing Spark2 headless keytab file. What is the right way to resolve it. ... View more
Labels:

Re: HDP 2.6 cluster OS upgrade - Ubuntu

by Explorer baig_mujeeb in Support Questions
‎07-26-2018 03:05 PM
‎07-26-2018 03:05 PM
@Sandeep Nemuri Thanks for swift reply. Is it OK if I also upgrade the underlying OS of Ambari host first followed by Ambari upgrade OR do the OS upgrade separately after Ambari is upgraded to 2.6.2.2? ... View more

Re: HDP 2.6 cluster OS upgrade - Ubuntu

by Explorer baig_mujeeb in Support Questions
‎07-26-2018 02:43 PM
‎07-26-2018 02:43 PM
Thanks Felix, almost there, although we have a small cluster but In-Place & Restore options suits us (set up with best practice in mind -:). Also a silly question: Where in Ambari UI is Ambari "Recover Host" functionality? We are using Ambari 2.5.0.3 Thanks MB ... View more

HDP 2.6 cluster OS upgrade - Ubuntu

by Explorer baig_mujeeb in Support Questions
‎07-26-2018 12:30 PM
‎07-26-2018 12:30 PM
Hi, I have a 4 node dev cluster which was recently upgraded to 2.6.0.3. It is running on Ubuntu 14. I want to upgrade the underlying OS to 16.04.4. What are the right steps to achieve this? I am thinking on the following lines: 1. Pick one server at a time 2. Switch on maintenance mode and stop all services 3. run 'do-release-upgrade' command to do release upgrade 4. start all the services on the server Once all the servers are done, perform the version upgrade from Ambari UI for Ubuntu 16. Is this the right approach? Kindly let me know. Thanks in advance as usual. Regards ... View more

Re: Storm - Supervisor and Nimbus dropping immedia...

by Explorer baig_mujeeb in Support Questions
‎07-26-2018 11:23 AM
‎07-26-2018 11:23 AM
Following jar files should be present gmetric4j-1.0.3.jar jmxetric-1.0.4.jar oncrpc-1.0.7.jar Since I did the version upgrade from 2.5 to 2.6 I copied them from /usr/hdp/2.5.3.0-37/storm/contrib/storm-jmxetric/lib folder Hope this helps ... View more

Re: Upgrading the cluster - OS, HDP, resources

by Explorer baig_mujeeb in Support Questions
‎06-07-2018 09:33 AM
‎06-07-2018 09:33 AM
Thanks Gaurav, it was helpful. ... View more

Upgrading the cluster - OS, HDP, resources

by Explorer baig_mujeeb in Support Questions
‎06-03-2018 12:56 PM
‎06-03-2018 12:56 PM
Hi, We have a development/test cluster with 4 nodes each having 12 cores vcpu, 2 TB SDD and 32GB RAM previously used for cassandra. All running on Ubuntu 14. It is configured as follows 1 NN 1 SNN 4 DN Jobs are taking longer due to limited resources, so we want to upgrade the HDP version from 2.5 to 2.6 and OS to Ubuntu 16.04 and add more resources. We are planning to upgrade as follows, 1. Increase RAM to 128 GB on all nodes 2. Double the storage by adding additional SAS disks 3. Add 2 more additional Data Nodes with 10 cores, 128 GB Questions I got is 1. Is it a good practice to use all of them as DN, or we should separate the NN 2. Since it was borrowed off Cassandra, is it ok if we keep all nodes of same specs including 2 new nodes 3. There is no harm in mixed storage options ie. adding extra storage in SAS wrt existing SSD 4. With the proposed new cluster, upgrading to HDP 3.0 shouldn't be a issue, I guess Thanks in advance MB ... View more
Labels:

Sqoop export error

by Explorer baig_mujeeb in Support Questions
‎04-05-2018 04:10 PM
‎04-05-2018 04:10 PM
Hi, Getting error half way through during sqoop export. Is there a setting which can enable logging the error record. Many Thanks MB ... View more

Re: Hive update a column from another table

by Explorer baig_mujeeb in Support Questions
‎02-13-2018 12:25 PM
‎02-13-2018 12:25 PM
Thanks Eugene for swift reply. We are running on HDP 2.5 and MERGE is available in 2.6. What options do I have? Thanks MB ... View more

Hive update a column from another table

by Explorer baig_mujeeb in Support Questions
‎02-12-2018 05:32 PM
‎02-12-2018 05:32 PM
Hi, I have a clustered bucketted table, with ACID enabled Does hive supports updates from another table? This is something I am trying to achieve, updating a column values from another table UPDATE transaction_bak SET fl_t_order = tt.t_order FROM trans_temp tt WHERE transaction_bak.id = tt.id; I tried with left join, but no success, getting ParseException line 2:0 missing SET at 'left' UPDATE transaction_bak LEFT OUTER JOIN trans_temp tt SET fl_t_order = tt.t_order WHERE transaction_bak.id = tt.id; Any idea ... View more
Labels:

Hive table update from another table

by Explorer baig_mujeeb in Support Questions
‎02-12-2018 05:13 PM
‎02-12-2018 05:13 PM
Hi, I have a clustered bucketted table, with ACID enabled Does hive supports updates from another table? This is something I am trying to achieve, updating a column values from another table UPDATE transaction_bak SET fl_t_order = tt.t_order FROM trans_temp tt WHERE transaction_bak.id = tt.id; I tried with left join, but no success, getting ParseException line 2:0 missing SET at 'left' UPDATE transaction_bak LEFT OUTER JOIN trans_temp tt SET fl_t_order = tt.t_order WHERE transaction_bak.id = tt.id; Any idea ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎07-31-2019 03:22 PM
Public Statistics
Date Registered ‎11-13-2017 04:35 PM
Last Visited ‎07-31-2019 03:22 PM
Total Messages Posted 49