I found the problem by digging down the the actual daemon that got started and running it without the --daemon option. The issue was that I had underscores in my hostnames. Once I removed them the problem went away.
As a note, if cloudera-scm-agent doesn't work with underscores in the hostnames, this is something that should be caught earlier in the manager, for example when entering the hostnames for installation.