The only option is to "Disable HA" on the MR service, so I disabled it (during the disable process on the step 2 of "removing the jobtracker", I got the error "Could not find config to delete", but it still looks like it was disabled... at least the Actions menu shows the Enable option now). So now when re-enabling I get the error below for the FailoverControllers. The mapreduce HA service shows as "Enabled" on the Actions menu, and the Jobtrackers start successfully.
I may just work on it more on Monday. Thank you for your patience and assistance and have a good weekend.
Unable to start failover controller. Parent znode does not exist.
Run with -formatZK flag to initialize ZooKeeper.
Did you originally enable MapReduce JobTracker HA via the CM wizard?
In general, CM doesn't delete any data when running commands, so when you disable HA it may leave data in data dirs or in ZK and you may need to manually clean those up if you try to re-enable HA.
I started over from scratch. During the install wizard I chose "Cloudera with Real TIme", and assigned ZK to 3 different nodes (both namenodes plus one), and I have one Primary NN and one Sec NN - each as a JobTracker role. I'm at the point now in the wizard that attempts to start the cluster services.
ZooKeeper Service initialize: Success
Start ZK service: Success
Format HDFS: Success
Start HDFS: Success
Create HDFS /tmp: Success
When it tries to start the MapReduce service, I get the following error:
My options are to "Retry" or "Modify...."
... so at this point to even get past this I have to open CM in a 2nd tab... -> Services -> mapreduce1 -> Instances -> Add -> Assign failOverControllers to nn1 and nn2
Now I click "Retry", and I get this error:
So now I flip over the 2nd CM tab and assign each JobTracker instance the failOverController role, assign each JT a logical Nameservice (logicaljt1), then start both JobTrackers.
That at least gets me through the installer process, but the FailOverControllers still fail to start with the same error: "Unable to start failover controller. Parent znode does not exist. Run with -formatZK flag to initialize ZooKeeper." ... but I assume that is because manually assigned that role, for lack of a better option.
I'm just not sure what I am missing during the install wizard, and if it weren't for your help, I would throw in the towel for while... so thanks.
Sorry your screenshot images did not come through for some reason. I'm pointing that out to the community manager to check on.
Anyway- in the cloudera manager UI, for the zookeeper service there is an "Actions" button, under that should be an "Initialize" option. What happens when you run that?
Install (install basic services)
Then you have "workflow wizards" that take you through setup of the NN and JT HA... you dont need to select them on initial install... in case you were trying to define them at initial install. Allow the basic role assignments to lay the basic services down, and then you can go back and for HDFS and MapReduce, use the CM manager wizards to set up JT HA and NN HA.
Hi Todd and Darren, your suggestions seemed to work. During install I chose only one namenode (primary and sec on same node), and chose "Core Hadoop" services to install. All went well except for the Ooozie install kept giving me fits about the mysql jdbc connector, but next time I will do a "Custom" install and not choose Oozie.
After the wizard install was complete I was able to enable HA on mapreduce and hdfs successfully! Question: where's the best place to run the HTTPFS role? I assigned HTTPFS to both namenodes, but maybe I should only added it to one? Other than the DN's and NN's, I have two other servers that I could have assigned it to:
Server01 contains: Hue, CM, Beeswax, Monitoring, Event Server.
Server02 contains: Hive, Gateway, Oozie, JournalNode and ZK
I still need to install Impala, but I think I'm running out of resources in Dev!
Assigning HTTPFS as you've done will be fine as it's a relatively light/passive role compared to most others. Great to hear you're in a good place!
In production, I was planning to use a central MySQL database server for all of the services versus having little database islands all over the place (derby, SQLite, Postgres, etc). It seems the logical thing to do, but wanted to get your take on it. Are there any major caveats to this approach?
This is fine for small clusters, but as your cluster grows you'll want to start using separate databases, and you may want to move master roles to different hosts. The Activity Monitoring database for CM can get very high load, as can the Hive Metastore Server database. It all depends on your use-cases and cluster size though.
If you haven't already, it's also good to look through the recommended configuraiton settings for MySQL in CM documentation:
Do you mean separate databases as in, use each role's native/embedded database, or do you mean separate as in multiple MySQL servers? I don't want to run in to goofy upgrade issues in the future if I were to use MySQL for all roles.