About sipocootap2

sipocootap2 · ‎07-26-2021

@Shelton Thanks for the detailed steps, Shelton. To make sure if I understood properly: When there are snapshottable directories at hdfs://user/foo/1/1-2 hdfs://user/foo/3/3-2 I expected something like: hdfs dfs -disallowSnapshot -subDirsIncluded -recursively hdfs://user/foo without the need to know where the snapshots are, but you mean that there isn't such simple command, right? Instead, I shouild list all the snapshottable directories and delete each snapshots under the directories.

sipocootap2 · ‎07-26-2021

There are user directories under hdfs:///user/ When a user `foo` retires, I need to delete a root directory for the user: hdfs:///user/foo However, it occasionally fails because of the snapshottable directories under the user root directory. Checking all the subdirectories whether it's snapshottable, disallowing snapshottable subdirectories , and then deleting the user's root directory doesn't seem to be the best way. (Or if this is the best way, I cannot come up with a simple code..) Is there a command to disallow snapshots for all the subdirectories? How can I effectively delete a directory that might have snapshots?

sipocootap2 · ‎06-13-2021

2021-06-09 17:00:54,088 WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(210)) - Transport-level exception trying to monitor health of NameNode at NAMENODE/NAMENODE:PORT java.net.SocketTimeoutException: Call From NAMENODE/NAMENODE to NAMENODE:PORT failed on socket timeout exception: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/NAMENODE:PORT2 remote=NAMENODE/NAMENODE:PORT]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout 2021-06-09 17:00:54,090 INFO ha.HealthMonitor (HealthMonitor.java:enterState(248)) - Entering state SERVICE_NOT_RESPONDING 2021-06-09 17:00:54,090 INFO ha.ZKFailoverController (ZKFailoverController.java:setLastHealthState(893)) - Local service NameNode at NAMENODE/NAMENODE:PORT entered state: SERVICE_NOT_RESPONDING 2021-06-09 17:00:54,191 WARN tools.DFSZKFailoverController (DFSZKFailoverController.java:getLocalNNThreadDump(249)) - Can't get local NN thread dump due to Server returned HTTP response code: 401 for URL: https://NAMENODE:PORT3/stacks 2021-06-09 17:00:54,191 INFO ha.ZKFailoverController (ZKFailoverController.java:recheckElectability(809)) - Quitting master election for NameNode at NAMENODE/NAMENODE:PORT and marking that fencing is necessary 2021-06-09 17:00:54,191 INFO ha.ActiveStandbyElector (ActiveStandbyElector.java:quitElection(412)) - Yielding from election 2021-06-09 17:00:54,192 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x178072757b716f6 closed 2021-06-09 17:00:54,192 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:isStaleClient(1124)) - Ignoring stale result from old client with sessionId 0x1234567 2021-06-09 17:00:54,192 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down I have two namenodes with HA. Suddenly a failover occured, and the above log was found in the previous active namenode. I have no idea why SocketTimeoutException was raised while doing doHealthChecks. Also, regarding "java.net.SocketTimeoutException: Call From NAMENODE/NAMENODE to NAMENODE:PORT failed on socket timeout exception: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/NAMENODE:PORT2 remote=NAMENODE/NAMENODE:PORT]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout" log, when I look for PORT2 in the namenode, that port doesn't seem to be used. Any comments appreciated.

sipocootap2 · ‎06-03-2021

Hi @kingpin. Thanks for the reply. Is there an official document that "/getimage" is deprecated? Cause I couldn't find it. When I run "hdfs dfsadmin -fetchImage" command, it also calls the same curl command. I could get fsimage with hdfs command, but I want to figure out the reason why curl fails.

sipocootap2 · ‎05-31-2021

What if there are multiple namespaces with multiple namenodes? `hdfs dfsadmin -fetchImage` command only reads from the default namespace.

sipocootap2 · ‎05-31-2021

When there are multiple namespaces, <hdfs dfsadmin -fetchImage> command seems to fetch a fsImage from the default namespace. What I need is to get fsImages from every namespaces. How can I pass the target namespace when using dfsadmin command?

sipocootap2 · ‎05-30-2021

After decommissioning, recommission the nodemanager and then start it again.

sipocootap2 · ‎04-18-2021

I authenticated with hadoop-admin user. hdfs dfsadmin -fetchImage ./output command works well. However, when I try to do this with curl command, curl https://{namenode_address}:9470/imagetransfer?getimage=1&txid=latest <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Authentication required</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /imagetransfer. Reason: <pre> Authentication required</pre></p> </body> </html> Which authentication is needed additionally?

sipocootap2 · ‎04-18-2021

I can get fsimage with hdfs dfsadmin -fetchImage ./fetched_image This command seems to get fsimage from the below url. https://{namenode-1}:9470/imagetransfer?getimage=1&txid=latest I have multiple namenodes, but dfsadmin fetches only from the first namenode. So, I want to fetch with curl command, inserting a certain namenode address into {namenode}. How can I achieve this? Just running curl https://{namenode-1}:9470/imagetransfer?getimage=1&txid=latest prints error even though kerberos ticket is the same as dfsadmin: <p>Problem accessing /imagetransfer. Reason: <pre> Authentication required</pre></p>

sipocootap2 · ‎03-03-2021

I solved this referring to https://storm.apache.org/releases/current/Running-topologies-on-a-production-cluster.html. After editting pom.xml, I ran ` mvn assembly:assembly`. Then storm jar ./WordCount-1.0-SNAPSHOT.jar com.microsoft.example.WordCountTopology WordCountTopology

Online	Offline
Last Visited	‎09-06-2022 09:16 AM

Date Registered	‎11-10-2020 07:32 AM
Last Visited	‎09-06-2022 09:16 AM
Total Messages Posted	21
Total Kudos Received	1

Re: Is there a way to delete an HDFS directory whi...

Is there a way to delete an HDFS directory which m...

ha.HealthMonitor (HealthMonitor.java:doHealthCheck...

Re: How can I get fsimage with curl command?

Re: How to read fsimage

How can I fetch a fsImage from a certain namenode/...

Re: Nodemanager service is failing on new host of ...

How can I get fsimage with curl command?

How can I get fsimage with curl command?

Re: How can I submit an Apache Storm topology to a...