Reply
Contributor
Posts: 30
Registered: ‎10-25-2013

Re: Problems with ZooKeeper

The only option is to "Disable HA" on the MR service, so I disabled it (during the disable process on the step 2 of "removing the jobtracker", I got the error "Could not find config to delete", but it still looks like it was disabled... at least the Actions menu shows the Enable option now).  So now when re-enabling I get the error below for the FailoverControllers.  The mapreduce HA service shows as "Enabled" on the Actions menu, and the Jobtrackers start successfully.

 

I may just work on it more on Monday.  Thank you for your patience and assistance and have a good weekend.

 

Unable to start failover controller. Parent znode does not exist.
Run with -formatZK flag to initialize ZooKeeper.
Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: Problems with ZooKeeper

Did you originally enable MapReduce JobTracker HA via the CM wizard?

 

In general, CM doesn't delete any data when running commands, so when you disable HA it may leave data in data dirs or in ZK and you may need to manually clean those up if you try to re-enable HA.

Highlighted
Contributor
Posts: 30
Registered: ‎10-25-2013

Re: Problems with ZooKeeper

I started over from scratch.  During the install wizard I chose "Cloudera with Real TIme", and assigned ZK to 3 different nodes (both namenodes plus one), and I have one Primary NN and one Sec NN - each as a JobTracker role.  I'm at the point now in the wizard that attempts to start the cluster services.

 

ZooKeeper Service initialize: Success

Start ZK service: Success

Format HDFS: Success

Start HDFS: Success

Create HDFS /tmp: Success

When it tries to start the MapReduce service, I get the following error:

 

mr-error.jpg

 

 

My options are to "Retry" or "Modify...."

... so at this point to even get past this I have to open CM in a 2nd tab...  -> Services -> mapreduce1 -> Instances -> Add -> Assign failOverControllers to nn1 and nn2

 

Now I click "Retry", and I get this error:

 

mr-error2.jpg

 

So now I flip over the 2nd CM tab and assign each JobTracker instance the failOverController role, assign each JT a logical Nameservice (logicaljt1), then start both JobTrackers.

 

That at least gets me through the installer process, but the FailOverControllers still fail to start with the same error:  "Unable to start failover controller. Parent znode does not exist. Run with -formatZK flag to initialize ZooKeeper." ... but I assume that is because manually assigned that role, for lack of a better option.

 

I'm just not sure what I am missing during the install wizard, and if it weren't for your help, I would throw in the towel for while... so thanks.

Cloudera Employee
Posts: 228
Registered: ‎09-23-2013

Re: Problems with ZooKeeper

Sorry your screenshot images did not come through for some reason. I'm pointing that out to the community manager to check on. 

 

Anyway- in the cloudera manager UI, for the zookeeper service there is an "Actions" button, under that should be an "Initialize" option.  What happens when you run that?

 

Todd

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: Problems with ZooKeeper

Hi Mike,

 

I think the problem is you tried to create two jobtrackers in the wizard. You should only pick one. We are already planning to enhance this to make sure you can't pick two by accident.

 

After you've set up everything in simple mode, go to MapReduce and go through the JobTracker High Availability wizard there.

 

Thanks,

Darren

Cloudera Employee
Posts: 228
Registered: ‎09-23-2013

Re: Problems with ZooKeeper

Also

 

Install (install basic services)

 

Then you have "workflow wizards" that take you through setup of the NN and JT HA... you dont need to select them on initial install... in case you were trying to define them at initial install.  Allow the basic role assignments to lay the basic services down, and then you can go back and for HDFS and MapReduce, use the CM manager wizards to set up JT HA and NN HA.

 


Todd

Contributor
Posts: 30
Registered: ‎10-25-2013

Re: Problems with ZooKeeper

Hi Todd and Darren, your suggestions seemed to work.  During install I chose only one namenode (primary and sec on same node), and chose "Core Hadoop" services to install.  All went well except for the Ooozie install kept giving me fits about the mysql jdbc connector, but next time I will do a "Custom" install and not choose Oozie.

 

After the wizard install was complete I was able to enable HA on mapreduce and hdfs successfully!   Question: where's the best place to run the HTTPFS role?  I assigned HTTPFS to both namenodes, but maybe I should only added it to one?  Other than the DN's and NN's, I have two other servers that I could have assigned it to:

 

Server01 contains: Hue, CM, Beeswax, Monitoring, Event Server.

Server02 contains: Hive, Gateway, Oozie, JournalNode and ZK

 

I still need to install Impala, but I think I'm running out of resources in Dev!

Cloudera Employee
Posts: 79
Registered: ‎08-29-2013

Re: Problems with ZooKeeper

Hi Mike,

 

Assigning HTTPFS as you've done will be fine as it's a relatively light/passive role compared to most others. Great to hear you're in a good place!

Contributor
Posts: 30
Registered: ‎10-25-2013

Re: Problems with ZooKeeper

In production, I was planning to use a central MySQL database server for all of the services versus having little database islands all over the place (derby, SQLite, Postgres, etc).  It seems the logical thing to do, but wanted to get your take on it.  Are there any major caveats to this approach?

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: Problems with ZooKeeper

This is fine for small clusters, but as your cluster grows you'll want to start using separate databases, and you may want to move master roles to different hosts. The Activity Monitoring database for CM can get very high load, as can the Hive Metastore Server database. It all depends on your use-cases and cluster size though.

 

If you haven't already, it's also good to look through the recommended configuraiton settings for MySQL in CM documentation:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.