CM failure after initial CD reboot - OS Type UNKNOWN

[#1]

I'm running CD created via AWS CloudFormation templates to spawn custom AWS Centos 6.5 AMI instances (w/ required cloud-init config).

However, after CD initially sets up CM and reboots the box, it comes back with a UNKNOWN Operating System Type error (below)

As I said, I'm running a stock AWS CentOS AMI w/ cloud-init, SELinux off, iptables off.  How/where else could the OS type be queried where it'd come up UNKNOWN? Do I need to place this information somewhere else or install more baseline packages?  This is how the box identifies itself currently. 

[root@ip- ~]# lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.5 (Final)
Release:        6.5
Codename:       Final

[root@ip- ~]# cat /etc/issue
CentOS release 6.5 (Final)
Kernel \r on an \m

[root@ip- ~]# cat /etc/*-release
CentOS release 6.5 (Final)
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)

The error from application.log

[2015-02-10 16:01:40] WARN  [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.2.232 was automatically accepted
[2015-02-10 16:01:40] INFO  [io-thread-1] - ssh:10.0.2.232: Filesystem      Size  Used Avail Use% Mounted on
[2015-02-10 16:01:40] INFO  [io-thread-1] - ssh:10.0.2.232: /dev/xvda1       50G  1.1G   48G   3% /
[2015-02-10 16:01:40] INFO  [io-thread-1] - ssh:10.0.2.232: devtmpfs        3.7G   60K  3.7G   1% /dev
[2015-02-10 16:01:40] INFO  [io-thread-1] - ssh:10.0.2.232: tmpfs           3.7G     0  3.7G   0% /dev/shm
[2015-02-10 16:01:40] INFO  [io-thread-1] - ssh:10.0.2.232: /dev/xvdb        30G   45M   30G   1% /data0
[2015-02-10 16:01:40] INFO  [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: << None{}
[2015-02-10 16:01:40] INFO  [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentC                                                                             ontext{environment=Environment{name='C5-Simple-AWS Environment', provider=Instance ...
[2015-02-10 16:01:40] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
java.lang.UnsupportedOperationException: Operating system type not supported: UNKNOWN
        at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.futureInstallClouderaManagerRepositories(Bootstrap                                                                             ClouderaManager.java:131) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
        at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:82) ~[launchpad-                                                                             bootstrap-1.0.1.jar!/:1.0.1]
        at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:44) ~[launchpad-                                                                             bootstrap-1.0.1.jar!/:1.0.1]
        at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-dat                                                                             abase-1.0.1.jar!/:1.0.1]
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.                                                                             6.jar!/:na]
        at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.j                                                                             ava:213) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-databa                                                                             se-1.0.1.jar!/:1.0.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_34]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_34]
        at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_34]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_34]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_34]
        at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_34]

Thanks for any help.

Mike

[#2 · Accepted solution]

Mike,

The error log points to Cloudera Director not knowing the operating system of the Cloudera Manager instance.

Cloudera Director determines the operating system by sshing to and checking /etc/issue on the Cloudera Manager instance. Can you check /etc/issue on the Cloudera Manager instance? What ami are you using for your instance templates?

David

[#3]

Could you try again using Red Hat 6.4 or Cent OS 6.4 and see if the same
issue occurs? Also like David asked, please tell us which AMI you used and
in which AWS region this is on.

[#4]

Thanks to both of you for your replies.

You put me on the right path, David.  It turned out my Custom AMI string in my aws.conf was swapped for another AMI that was not CentOS.

So....no issues on the CD side, just something we need to fix in our process here.

Thank you for your responses, it's very much appreciated to know if I have a problem, I can ask and get quick help. 

If anyone is reading, for my Custom CentOS 6.5 AMI, I did end up installing the lsb_release package just to be on the safe side for this OS type query.  And be sure to use root as your SSH user in your aws.conf, not ec2-user as SSH will fail.

Mike

[#5]

Just change the versioning number from kernel or update to RedHat 6.6.  6.6 works perfectly.  Down grading isn't a good option in my opinion.

It's simply needs a version check which you can do by just upgrading the kernel version.

Easiest way instead of doing a complete reinstall or downgrade which isn't a good option.

Download kernel from kernel.org

# cd /usr/src/kernels

# wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.18.8.tar.gz

# tar zxvf linux-3.18.8.tar.gz

# cd linux-3.18.8

# make -j32;make modules -j32;make modules_install -j32;make install

# edit /boot/grub/grub.cfg

Change to 0 which is the current kernel version and that's it, reboot and you're set.

[#6]

P.S. Don't forget to create or move your current .config from previous kernel or create new .config.

# cd /usr/src/kernels/linux-3.18.8

# cp ../linux-old-kernel-version/.config .

# make oldconfig -j32;make modules -j32;make modules_install -j32;make install

# Answer all question using just the default. Just enter through the kernel questions.