Member since
04-19-2018
21
Posts
0
Kudos Received
0
Solutions
08-12-2019
01:19 PM
added livy service to hdp cluster.its running. but its not using any port. not listening on port 8999. (hdp version:hdp 2.6.0)
... View more
Labels:
07-18-2019
10:08 AM
developer needs acess to hbase tables using rest api. which things need to do on ambari cluster to access hbase tables using rest api.
... View more
Labels:
07-07-2019
06:33 AM
In hdp cluster one of the host is down permanently. What are the prerequisites to remove that host from cluster.
... View more
Labels:
07-05-2019
06:15 AM
can we pass config parameter while running spark query instead of doing changes from ambari. how to do?
... View more
Labels:
07-05-2019
06:12 AM
while running phoenix query on 370 M records,getting timeout exception. Phoenix query timeout=1 min how to resolve this and can we pass config parameter at runtime instead of doing config changes from ambari.
... View more
Labels:
06-20-2019
06:57 AM
in ambari there is alert regarding 1 missing block from 6 days. but when i check using " hdfs fsck / " its showing there is not any missing block. is this bug in ambari ?
... View more
Labels:
06-12-2019
06:09 AM
in spark2-history ,long running application generating logs with 30GB+ size. how to control spark2-history size for each application .
... View more
Labels:
06-10-2019
08:05 AM
ambari-metrics collector running but in log its showing that no live collector to collect data. hbase-ams-master.log: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. ambari-metrics-collector.log: 2019-06-10 13:29:52,653 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=15, retries=35, started=629189 ms ago, cancelled=false, msg=java.io.IOException: Table Namespace Manager not fully initialized, try again later at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2753) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2976) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1746) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:491) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:59956) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167) ambari alert: metrics-collector process alert connection refused to ambari-metrics collector:6188
... View more
Labels:
06-06-2019
09:45 AM
i did logrotate using linux,but i am not getting where current logs are getting dumped.in that todays logs are also in compressed format till 3:00 am.after that there are no logs. everyday hdfs audit logs generated aroun 7 gb . I need keep 7 days log and compress logs older than a day. which config changes i need to do.
... View more
Labels:
06-04-2019
06:36 AM
while starting oozie service getting below error: oozie server status:Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie http://master1:11000/oozie -status' returned 255. Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 1 sec. Retry count = 1 Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. oozie server web ui:Connection failed to master1:11000/oozie/?user.name=oozie (<urlopen error [Errno 111] Connection refused>) oozie log: Jun 03, 2019 8:34:40 PM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-11000 Jun 03, 2019 8:34:41 PM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Jun 03, 2019 8:34:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/oozie] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Jun 03, 2019 8:34:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/oozie] appears to have started a thread named [FileWatchdog] but has failed to stop it. This is very likely to create a memory leak.
... View more
Labels:
05-08-2019
10:06 PM
when last 50 reducers are remaining then reducers are failing and getting killed: --------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 ......... RUNNING 1215 1128 0 87 5 0
Reducer 2 ..... RUNNING 1009 983 0 26 10 5
--------------------------------------------------------------------------------
VERTICES: 00/02 [========================>>--] 94% ELAPSED TIME: 2417.20 s
--------------------------------------------------------------------------------
Status: Failed
Vertex re-running, vertexName=Map 1, vertexId=vertex_1557224797954_0084_1_00
Vertex re-running, vertexName=Map 1, vertexId=vertex_1557224797954_0084_1_00
Vertex re-running, vertexName=Map 1, vertexId=vertex_1557224797954_0084_1_00
Vertex failed, vertexName=Reducer 2, vertexId=vertex_1557224797954_0084_1_01, diagnostics=[Task failed, taskId=task_1557224797954_0084_1_01_000470, diagnostics=[TaskAttempt 0 failed, info=[Container container_e380_1557224797954_0084_01_000125 finished with diagnostics set to [Container failed, exitCode=-100. Container released on a *lost* node]], TaskAttempt 1 failed, info=[Error: exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in DiskToDiskMerger [Map_1]
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:357)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:334)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:261)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.tez.runtime.library.common.sort.impl.IFileOutputStream.write(IFileOutputStream.java:120)
at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:153)
at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
at org.apache.hadoop.io.compress.BlockCompressorStream.write(BlockCompressorStream.java:100)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.writeValue(IFile.java:402)
at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.append(IFile.java:393)
at org.apache.tez.runtime.library.common.sort.impl.TezMerger.writeFile(TezMerger.java:207)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$OnDiskMerger.merge(MergeManager.java:863)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
... View more
Labels:
05-03-2019
03:51 PM
i am also getting this error.we are processing 500 gb data and nodemanager local-dir size is 100 gb.at the time of job execution it was 91 % full.1100 mappers and 1000 reducers are there .mapper job completed ,some reducer job failed and killed.Please help me to solve that. error will be as: Status: Failed Vertex re-running, vertexName=Map 1, vertexId=vertex_1556753562511_0049_1_00 Vertex failed, vertexName=Reducer 2, vertexId=vertex_1556753562511_0049_1_01, diagnostics=[Task failed, taskId=task_1556753562511_0049_1_01_000470, diagnostics=[TaskAttempt 0 failed, info=[Error: exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in DiskToDiskMerger [Map_1] at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:357) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:334) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1556753562511_0049_1_01_000470_0_10014_src_1187_spill_-1 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$OnDiskMerger.merge(MergeManager.java:841) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89) , errorMessage=Shuffle Runner Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in DiskToDiskMerger [Map_1] at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:357) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:334) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
... View more
04-27-2019
03:25 PM
ambari metrics collector getting down frequentally with error: WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
... View more
Labels:
04-27-2019
03:23 PM
Running mapreduce job on 450 gb data with replication factor 1.But jobs are failing again and again. not mapred.loc.dir in mapred config. How to resolve this Error:Status: Failed
Vertex re-running, vertexName=Map 1, vertexId=vertex_1555964902314_0036_1_00
Vertex failed, vertexName=Reducer 2, vertexId=vertex_1555964902314_0036_1_01, diagnostics=[Task failed, taskId=task_1555964902314_0036_1_01_000470, diagnostics=[TaskAttempt 0 failed, info=[Error: exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in DiskToDiskMerger [Map_1]
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:357)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:334)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1555964902314_0036_1_01_000470_0_10008_src_1186_spill_-1
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$OnDiskMerger.merge(MergeManager.java:841)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
, errorMessage=Shuffle Runner Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in DiskToDiskMerger [Map_1]
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:357)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:334)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1555964902314_0036_1_01_000470_0_10008_src_1186_spill_-1
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$OnDiskMerger.merge(MergeManager.java:841)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
], TaskAttempt 1 failed, info=[Container container_e352_1555964902314_0036_02_000213 finished with diagnostics set to [Container failed, exitCode=-100. Container released on a *lost* node]], TaskAttempt 2 failed, info=[Error: exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in DiskToDiskMerger [Map_1]
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:357)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:334)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1555964902314_0036_1_01_000470_2_10002_src_1164_spill_-1
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$OnDiskMerger.merge(MergeManager.java:841)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
, errorMessage=Shuffle Runner Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in DiskToDiskMerger [Map_1]
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:357)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:334)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1555964902314_0036_1_01_000470_2_10002_src_1164_spill_-1
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$OnDiskMerger.merge(MergeManager.java:841)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
... View more
Labels:
04-16-2019
03:54 PM
i did following changes,but hdfs-audit logs are not rotating, hdfs.audit.logger=INFO,console log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger} log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false #log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAAUDIT=org.apache.log4j.RollingFileAppender log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.DRFAAUDIT.MaxFileSize=100MB log4j.appender.DRFAAUDIT.MaxBackupIndex=5
... View more
04-12-2019
11:04 AM
Ambari metricss collector getting down with following error and we are not using hbase: WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
... View more
Labels:
04-10-2019
06:54 AM
on one of the datanode jobs are failing .stopping nodemanager is helpful or not. Please help.
... View more
Labels:
04-04-2019
01:21 PM
Getting this error at the time of job submission,please help me regarding whats the issue: 19/04/02 21:39:09 WARN DFSClient: DataStreamer Exception
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[10.0.1.7:50010,DS-a5960a3c-6e53-45a7-a692-5374b4d0646b,DISK], DatanodeInfoWithStorage[10.0.1.6:50010,DS-0c1ff599-7179-4566-baea-e26249589633,DISK]], original=[DatanodeInfoWithStorage[10.0.1.7:50010,DS-a5960a3c-6e53-45a7-a692-5374b4d0646b,DISK], DatanodeInfoWithStorage[10.0.1.6:50010,DS-0c1ff599-7179-4566-baea-e26249589633,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1059)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1122)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1280)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1005)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:512)
19/04/02 21:39:22 INFO RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2006)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1404)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewLease(FSNamesystem.java:4785)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewLease(NameNodeRpcServer.java:1117)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:675)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over 10.0.1.5:8020. Trying to failover immediately.
19/04/02 21:39:22 INFO RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2006)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1404)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewLease(FSNamesystem.java:4785)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewLease(NameNodeRpcServer.java:1117)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:675)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) …while invoking ClientNamenodeProtocolTranslatorPB.renewLease over 10.0.1.17:8020 after 1 failover
... View more
Labels:
04-03-2019
04:46 PM
We have to load 1 month data from local file system to hdfs. to load 1 days data its taking 30 min,to load 1 month data its taking 15 hrs. so how to improve speed of loading data from local file system to hdfs.
... View more
Labels:
04-03-2019
04:44 PM
I restarted all services but this alert is still there. What is the reason?
... View more
Labels:
04-19-2018
02:25 AM
If kdc in kerberos fails what to do as an hadoop administrator
... View more
Labels: