Even after passing the executor memory and driver memory as 1 Gb still it is getting maximum heap size rounded to 512 MB   spark-shell --conf spark.executor.memory=1g --conf spark.driver.memory=1g Warning: Maximum heap size rounded up to 512 MB   Anything that is missing? ... View more

Hi All, I am using  Postgress for Hive metastore How can we configure postgress HA with Hive with true HA on the Hive Metastore without changing the connection string manually? ... View more

Hi All We have our hive database on Postgress and wanted to migrate it to MySQL, Is there any easy way to do it?   Thanks ... View more

I am getting info messages like the one below while reading and writing files in hdfs. can anyone explain the meaning of this message, will it cause any issues? I have 3 Namenode(open-source Hadoop) in HA.   INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:108) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2094) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1550) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3342) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1208) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:1042) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over hostname/ip:8020 after 1 failover attempts. Trying to failover after sleeping for 1195ms.   Thanks   ... View more

We have an MYSQL HA cluster, In case of failover, for the secondary master to take the place of the failed master, the hive metastore connection string configuration would need to be changed to point to the new master, and the services restarted. is it possible to have true HA on the Hive Metastore without changing the connection string manually ?   ... View more

Is there any other way to migrate the Hortonworks HDP3  cluster to the Apache Hadoop 3.3.2  other than the distcp? is it possible to upgrade by updating the jar and pointing the older metadata to the newer stack of Hadoop?   Thanks ... View more

Hi everyone. I am getting stale alerts every hr, if the last point of contact between Data nodes and Namenode is more than 30 s we get these alerts. I am not able to find the root cause of this slowness, I have 32 cores system, but when this alert is generated in htop hdfs usage is more but not all cores are 100% utilized. DataNode Health Summary DataNode Health: [Live=5, Stale=1, Dead=0] Please suggest changes required to resolve this. ... View more