About hawkphil

hawkphil · ‎12-14-2015

I found out my YARN deployment got messed up because I didn't add more NodeManager in after I added new hosts manually. Ops!

hawkphil · ‎12-09-2015

Hi all I installed Cloudera 5.5 and Spark YARN. I uploaded a small file as below: Then I ran pyspark as hdfs user and did a simple exercise but it got stuck at Stage 0 as screenshot: It never returned anything. Can someone point me out a way to troubleshoot and fix this?

hawkphil · ‎12-07-2015

It is pointing to /dfs/nn and there is ./current folder in there I think it's corrupted. I replaced with /dfs/snn/current and it's still not working Then I use Cloudera Manager to redeploy the Namenode / Secondary Namenode to some other VMs and it seems to work .. for now. Strange.

hawkphil · ‎12-05-2015

I just want to see if it's possible to use public cloud to handle the spike. The data are already in the internal lab so moving them outside would cause some inconvenience, especially when I don't want to deal with that process of upload and then ingest into AWS Latency is a big deal but I am wondering if there is any case that it would make sense. I am just thinking out loud here. For example, what if there are scenarios where the cluster sustaining a long spike for "Tier 1" processes. But I want to be able to add more computation for other "Tier 2" so it makes sense to add new nodes instead of waiting.

hawkphil · ‎12-04-2015

Let say I already have a cluster internally in my lab. Is it possible to add more hosts but from AWS to act as DataNode?

hawkphil · ‎12-02-2015

Just a gentle nudge... Any suggestion please?

hawkphil · ‎11-30-2015

I am on the latest version, whichever I downaloded last week v5.5 and I installed in vSphere My lab got an outage so the entire cluster went off disruptively. That was the main reason causing all weird thing. I didn't have too much valuable data in there yet

hawkphil · ‎11-29-2015

I have Cluster Express with 1 master role, 1 management role and 6 worker roles. My Namenode and Master showed not running and I cannot restart the Namenode. Error below: 2015-11-29 19:27:32,758 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join java.io.IOException: Failed to load an FSImage file! at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:657) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:275) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:880) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:639) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:500) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:556) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:721) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:705) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1355) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1421) I ssh'ed into the master role VM and tried to fix it manually: hdfs namenode -recover But it is still the same. Let me know where I should troubleshoot next. Thanks

Online	Offline
Last Visited	‎12-17-2015 04:08 PM

Date Registered	‎11-29-2015 04:22 PM
Last Visited	‎12-17-2015 04:08 PM
Total Messages Posted	8

Re: Pyspark stuck at Stage 0

Pyspark stuck at Stage 0

Re: Cannot restart namenode: Failed to load an FSI...

Re: Is it possible to have hybrid deployment with ...

Is it possible to have hybrid deployment with AWS?

Re: Cannot restart namenode: Failed to load an FSI...

Re: Cannot restart namenode: Failed to load an FSI...

Cannot restart namenode: Failed to load an FSImage...