STARTUP_MSG: java = 1.8.0_112 ************************************************************/ 18/03/13 09:49:20 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 18/03/13 09:49:20 INFO namenode.NameNode: createNameNode [] 18/03/13 09:49:21 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 18/03/13 09:49:21 INFO timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 18/03/13 09:49:21 INFO timeline.HadoopTimelineMetricsSink: Identified hostname = rjio-prodcluster-namenode01, serviceName = namenode SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 18/03/13 09:49:21 INFO timeline.HadoopTimelineMetricsSink: No suitable collector found. 18/03/13 09:49:21 INFO timeline.HadoopTimelineMetricsSink: RPC port properties configured: {8020=client} 18/03/13 09:49:21 INFO impl.MetricsSinkAdapter: Sink timeline started 18/03/13 09:49:21 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 18/03/13 09:49:21 INFO impl.MetricsSystemImpl: NameNode metrics system started 18/03/13 09:49:21 INFO namenode.NameNode: fs.defaultFS is hdfs://cfms-prod-1 18/03/13 09:49:21 INFO namenode.NameNode: Clients are to use cfms-prod-1 to access this namenode/service. 18/03/13 09:49:22 INFO security.UserGroupInformation: Login successful for user nn/rjio-prodcluster-namenode01@RJIO.CFMS.UAT using keytab file /etc/security/keytabs/nn.service.keytab 18/03/13 09:49:22 INFO hdfs.DFSUtil: Starting web server as: HTTP/rjio-prodcluster-namenode01@RJIO.CFMS.UAT 18/03/13 09:49:22 INFO hdfs.DFSUtil: Starting Web-server for hdfs at: http://rjio-ProdCluster-namenode01:50070 18/03/13 09:49:22 INFO util.JvmPauseMonitor: Starting JVM pause monitor 18/03/13 09:49:22 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 18/03/13 09:49:22 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 18/03/13 09:49:22 INFO http.HttpRequestLog: Http request log for http.requests.namenode is not defined 18/03/13 09:49:22 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 18/03/13 09:49:22 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs 18/03/13 09:49:22 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 18/03/13 09:49:22 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 18/03/13 09:49:22 INFO security.HttpCrossOriginFilterInitializer: CORS filter not enabled. Please set hadoop.http.cross-origin.enabled to 'true' to enable it 18/03/13 09:49:22 INFO http.HttpServer2: Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter) 18/03/13 09:49:22 INFO http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/* 18/03/13 09:49:22 INFO http.HttpServer2: Adding Kerberos (SPNEGO) filter to getDelegationToken 18/03/13 09:49:22 INFO http.HttpServer2: Adding Kerberos (SPNEGO) filter to renewDelegationToken 18/03/13 09:49:22 INFO http.HttpServer2: Adding Kerberos (SPNEGO) filter to cancelDelegationToken 18/03/13 09:49:22 INFO http.HttpServer2: Adding Kerberos (SPNEGO) filter to fsck 18/03/13 09:49:22 INFO http.HttpServer2: Adding Kerberos (SPNEGO) filter to imagetransfer 18/03/13 09:49:22 INFO http.HttpServer2: Jetty bound to port 50070 18/03/13 09:49:22 INFO mortbay.log: jetty-6.1.26.hwx 18/03/13 09:49:23 INFO server.KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/rjio-prodcluster-namenode01@RJIO.CFMS.UAT 18/03/13 09:49:23 INFO server.KerberosAuthenticationHandler: Map server: rjio-prodcluster-namenode01 to principal: [HTTP/rjio-prodcluster-namenode01@RJIO.CFMS.UAT], added = true 18/03/13 09:49:23 INFO server.KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/rjio-prodcluster-namenode01@RJIO.CFMS.UAT 18/03/13 09:49:23 INFO server.KerberosAuthenticationHandler: Map server: rjio-prodcluster-namenode01 to principal: [HTTP/rjio-prodcluster-namenode01@RJIO.CFMS.UAT], added = true 18/03/13 09:49:23 INFO mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@rjio-ProdCluster-namenode01:50070 18/03/13 09:49:23 WARN common.Util: Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Util: Path /opt/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Util: Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Util: Path /opt/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Util: Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Util: Path /opt/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Util: Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Util: Path /opt/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 18/03/13 09:49:23 WARN common.Storage: set restore failed storage to true 18/03/13 09:49:23 INFO namenode.FSEditLog: Edit logging is async:false 18/03/13 09:49:23 INFO namenode.FSNamesystem: No KeyProvider found. 18/03/13 09:49:23 INFO namenode.FSNamesystem: Enabling async auditlog 18/03/13 09:49:23 INFO namenode.FSNamesystem: fsLock is fair:false 18/03/13 09:49:23 INFO blockmanagement.HeartbeatManager: Setting heartbeat recheck interval to 30000 since dfs.namenode.stale.datanode.interval is less than dfs.namenode.heartbeat.recheck-interval 18/03/13 09:49:23 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 18/03/13 09:49:23 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 18/03/13 09:49:23 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 18/03/13 09:49:23 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:01:00:00.000 18/03/13 09:49:23 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Mar 13 10:49:23 18/03/13 09:49:23 INFO util.GSet: Computing capacity for map BlocksMap 18/03/13 09:49:23 INFO util.GSet: VM type = 64-bit 18/03/13 09:49:23 INFO util.GSet: 2.0% max memory 1011.3 MB = 20.2 MB 18/03/13 09:49:23 INFO util.GSet: capacity = 2^21 = 2097152 entries 18/03/13 09:49:23 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=true 18/03/13 09:49:23 INFO blockmanagement.BlockManager: dfs.block.access.key.update.interval=600 min(s), dfs.block.access.token.lifetime=600 min(s), dfs.encrypt.data.transfer.algorithm=null 18/03/13 09:49:23 INFO blockmanagement.BlockManager: defaultReplication = 3 18/03/13 09:49:23 INFO blockmanagement.BlockManager: maxReplication = 50 18/03/13 09:49:23 INFO blockmanagement.BlockManager: minReplication = 1 18/03/13 09:49:23 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 18/03/13 09:49:23 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 18/03/13 09:49:23 INFO blockmanagement.BlockManager: encryptDataTransfer = false 18/03/13 09:49:23 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 18/03/13 09:49:23 INFO namenode.FSNamesystem: fsOwner = nn/rjio-prodcluster-namenode01@RJIO.CFMS.UAT (auth:KERBEROS) 18/03/13 09:49:23 INFO namenode.FSNamesystem: supergroup = hdfs 18/03/13 09:49:23 INFO namenode.FSNamesystem: isPermissionEnabled = true 18/03/13 09:49:23 INFO namenode.FSNamesystem: Determined nameservice ID: cfms-prod-1 18/03/13 09:49:23 INFO namenode.FSNamesystem: HA Enabled: true 18/03/13 09:49:23 INFO namenode.FSNamesystem: Append Enabled: true 18/03/13 09:49:23 INFO util.GSet: Computing capacity for map INodeMap 18/03/13 09:49:23 INFO util.GSet: VM type = 64-bit 18/03/13 09:49:23 INFO util.GSet: 1.0% max memory 1011.3 MB = 10.1 MB 18/03/13 09:49:23 INFO util.GSet: capacity = 2^20 = 1048576 entries 18/03/13 09:49:23 INFO namenode.FSDirectory: ACLs enabled? false 18/03/13 09:49:23 INFO namenode.FSDirectory: XAttrs enabled? true 18/03/13 09:49:23 INFO namenode.FSDirectory: Maximum size of an xattr: 16384 18/03/13 09:49:23 INFO namenode.NameNode: Caching file names occuring more than 10 times 18/03/13 09:49:23 INFO util.GSet: Computing capacity for map cachedBlocks 18/03/13 09:49:23 INFO util.GSet: VM type = 64-bit 18/03/13 09:49:23 INFO util.GSet: 0.25% max memory 1011.3 MB = 2.5 MB 18/03/13 09:49:23 INFO util.GSet: capacity = 2^18 = 262144 entries 18/03/13 09:49:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9900000095367432 18/03/13 09:49:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 18/03/13 09:49:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 18/03/13 09:49:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 18/03/13 09:49:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 18/03/13 09:49:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 18/03/13 09:49:23 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 18/03/13 09:49:23 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 18/03/13 09:49:23 INFO util.GSet: Computing capacity for map NameNodeRetryCache 18/03/13 09:49:23 INFO util.GSet: VM type = 64-bit 18/03/13 09:49:23 INFO util.GSet: 0.029999999329447746% max memory 1011.3 MB = 310.7 KB 18/03/13 09:49:23 INFO util.GSet: capacity = 2^15 = 32768 entries 18/03/13 09:49:23 INFO common.Storage: Lock on /hadoop/hdfs/namenode/in_use.lock acquired by nodename 28936@rjio-ProdCluster-namenode01 18/03/13 09:49:23 INFO common.Storage: Lock on /opt/hadoop/hdfs/namenode/in_use.lock acquired by nodename 28936@rjio-ProdCluster-namenode01 18/03/13 09:49:24 INFO namenode.FSImage: Planning to load image: FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000000000367341, cpktTxId=0000000000000367341) 18/03/13 09:49:24 INFO namenode.FSImageFormatPBINode: Loading 164 INodes. 18/03/13 09:49:24 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 0 seconds. 18/03/13 09:49:24 INFO namenode.FSImage: Loaded image for txid 367341 from /hadoop/hdfs/namenode/current/fsimage_0000000000000367341 18/03/13 09:49:24 INFO namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@33c2bd expecting start txid #367342 18/03/13 09:49:24 INFO namenode.FSImage: Start loading edits file http://rjio-prodcluster-namenode02:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1, http://rjio-prodcluster-datanode01:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1 18/03/13 09:49:24 INFO namenode.RedundantEditLogInputStream: Fast-forwarding stream 'http://rjio-prodcluster-namenode02:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1, http://rjio-prodcluster-datanode01:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1' to transaction ID 367342 18/03/13 09:49:24 INFO namenode.RedundantEditLogInputStream: Fast-forwarding stream 'http://rjio-prodcluster-namenode02:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1' to transaction ID 367342 18/03/13 09:49:24 ERROR namenode.EditLogInputStream: caught exception initializing http://rjio-prodcluster-namenode02:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1 java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://rjio-prodcluster-namenode02:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1&user.name=nn/rjio-prodcluster-namenode01@RJIO.CFMS.UAT, status: 403, message: org.apache.hadoop.security.authentication.client.AuthenticationException at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:509) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:503) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1073) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:723) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://rjio-prodcluster-namenode02:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1&user.name=nn/rjio-prodcluster-namenode01@RJIO.CFMS.UAT, status: 403, message: org.apache.hadoop.security.authentication.client.AuthenticationException at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:281) at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:190) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:471) ... 30 more 18/03/13 09:49:24 ERROR namenode.RedundantEditLogInputStream: Got error reading edit log input stream http://rjio-prodcluster-namenode02:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1; failing over to edit log http://rjio-prodcluster-datanode01:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1 org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 367341; expected file to go up to 367415 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:197) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1073) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:723) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) 18/03/13 09:49:24 INFO namenode.RedundantEditLogInputStream: Fast-forwarding stream 'http://rjio-prodcluster-datanode01:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1' to transaction ID 367342 18/03/13 09:49:24 ERROR namenode.EditLogInputStream: caught exception initializing http://rjio-prodcluster-datanode01:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1 java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://rjio-prodcluster-datanode01:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1&user.name=nn/rjio-prodcluster-namenode01@RJIO.CFMS.UAT, status: 403, message: org.apache.hadoop.security.authentication.client.AuthenticationException at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:509) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:503) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1073) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:723) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://rjio-prodcluster-datanode01:8480/getJournal?jid=cfms-prod-1&segmentTxId=367342&storageInfo=-63%3A1640349139%3A0%3ACID-80aba391-1f24-4596-a94f-702397691bd1&user.name=nn/rjio-prodcluster-namenode01@RJIO.CFMS.UAT, status: 403, message: org.apache.hadoop.security.authentication.client.AuthenticationException at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:281) at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:190) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:471) ... 30 more 18/03/13 09:49:24 ERROR namenode.FSImage: Error replaying edit log at offset 0. Expected transaction ID was 367342 org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 367341; expected file to go up to 367415 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:197) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1073) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:723) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) 18/03/13 09:49:24 INFO namenode.FSNamesystem: FSNamesystem write lock held for 1323 ms via java.lang.Thread.getStackTrace(Thread.java:1556) org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:945) org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1686) org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1101) org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:723) org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Number of suppressed write-lock reports: 0 Longest write-lock held interval: 1323 18/03/13 09:49:24 WARN namenode.FSNamesystem: Encountered exception loading fsimage org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 367342 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1073) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:723) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 367341; expected file to go up to 367415 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:197) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190) ... 12 more 18/03/13 09:49:24 INFO mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@rjio-ProdCluster-namenode01:50070 18/03/13 09:49:24 WARN http.HttpServer2: HttpServer Acceptor: isRunning is false. Rechecking. 18/03/13 09:49:24 WARN http.HttpServer2: HttpServer Acceptor: isRunning is false 18/03/13 09:49:25 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system... 18/03/13 09:49:25 INFO impl.MetricsSinkAdapter: timeline thread interrupted. 18/03/13 09:49:25 INFO impl.MetricsSystemImpl: NameNode metrics system stopped. 18/03/13 09:49:25 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete. 18/03/13 09:49:25 ERROR namenode.NameNode: Failed to start namenode. org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 367342 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1073) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:723) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 367341; expected file to go up to 367415 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:197) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190) ... 12 more 18/03/13 09:49:25 INFO util.ExitUtil: Exiting with status 1 18/03/13 09:49:25 INFO timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 18/03/13 09:49:25 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at rjio-ProdCluster-namenode01/10.10.1.46 ************************************************************/