Error: Could not find or load main class org.apache.ambari.server.controller.AmbariServer

[#1]

OS:- Centos7, Centos 6

I was trying to generate code-coverage report from ambari using cobertura 2.1.1. The steps to do that involves the following:-

1. Stop ambari.

2.Instrument the target jars.

sudo su - -c "/tmp/coverage-tmp/cobertura-2.1.1/cobertura-instrument.sh --datafile /tmp/coverage-tmp/cobertura-2.1.1/cobertura-data.ser --includeClasses org.apache.ambari.* --excludeClasses net.sourceforge.cobertura.* --excludeClasses .*[tT][eE][sS][tT].* --excludeClasses .*shaded.* --excludeClasses .*org.apache.ambari.server.controller.* --auxClasspath /usr/lib/ambari-server/jna-4.1.0.jar:/usr/lib/ambari-server/activation-1.1.jar:/usr/lib/ambari-server/guava-16.0.jar:/usr/lib/ambari-server/oro-2.0.8.jar:/usr/lib/ambari-server/ambari-metrics-common-2.5.2.0.95.jar:/usr/lib/ambari-server/guice-persist-3.0.jar:/usr/lib/ambari-server/ambari-server-2.5.2.0.95.jar:/usr/lib/ambari-server/guice-servlet-3.0.jar:/usr/lib/ambari-server/ambari-views-2.5.2.0.95.jar:/usr/lib/ambari-server/jetty-6.1.26.jar:/usr/lib/ambari-server/ant-1.6.5.jar:/usr/lib/ambari-server/h2-1.4.193.jar:/usr/lib/ambari-server/ant-launcher-1.7.1.jar:/usr/lib/ambari-server/jetty-io-8.1.19.v20160209.jar:/usr/lib/ambari-server/antlr-2.7.7.jar:/usr/lib/ambari-server/jline-2.11.jar:/usr/lib/ambari-server/aopalliance-1.0.jar:/usr/lib/ambari-server/guice-3.0.jar:/usr/lib/ambari-server/mimepull-1.9.3.jar:/usr/lib/ambari-server/apache-log4j-extras-1.2.17.jar:/usr/lib/ambari-server/hadoop-annotations-2.6.0.jar:/usr/lib/ambari-server/apacheds-i18n-2.0.0-M19.jar:/usr/lib/ambari-server/gson-2.2.2.jar:/usr/lib/ambari-server/mina-core-2.0.9.jar:/usr/lib/ambari-server/apacheds-kerberos-codec-2.0.0-M19.jar:/usr/lib/ambari-server/hadoop-auth-2.7.2.jar:/usr/lib/ambari-server/api-asn1-api-1.0.0-M26.jar:/usr/lib/ambari-server/hadoop-common-2.7.2.jar:/usr/lib/ambari-server/api-asn1-ber-1.0.0-M26.jar:/usr/lib/ambari-server/jackson-xc-1.9.9.jar:/usr/lib/ambari-server/api-i18n-1.0.0-M26.jar:/usr/lib/ambari-server/java-xmlbuilder-0.4.jar:/usr/lib/ambari-server/api-ldap-model-1.0.0-M26.jar:/usr/lib/ambari-server/javax.inject-1.jar:/usr/lib/ambari-server/api-util-1.0.0-M26.jar:/usr/lib/ambari-server/jsch-0.1.42.jar:/usr/lib/ambari-server/asm-3.3.1.jar:/usr/lib/ambari-server/json-smart-1.1.1.jar:/usr/lib/ambari-server/avro-1.7.4.jar:/usr/lib/ambari-server/jsr305-1.3.9.jar:/usr/lib/ambari-server/c3p0-0.9.5.2.jar:/usr/lib/ambari-server/jsr311-api-1.1.1.jar:/usr/lib/ambari-server/cglib-2.2.2.jar:/usr/lib/ambari-server/kerberos-client-2.0.0-M19.jar:/usr/lib/ambari-server/cloning-1.9.2.jar:/usr/lib/ambari-server/javax.json-1.0.4.jar:/usr/lib/ambari-server/commonj.sdo-2.1.1.jar:/usr/lib/ambari-server/javax.persistence-2.1.1.jar:/usr/lib/ambari-server/commons-beanutils-1.7.0.jar:/usr/lib/ambari-server/guice-assistedinject-3.0.jar:/usr/lib/ambari-server/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-server/javax.servlet-3.0.0.v201112011016.jar:/usr/lib/ambari-server/commons-cli-1.3.1.jar:/usr/lib/ambari-server/javax.servlet-api-3.1.0.jar:/usr/lib/ambari-server/commons-codec-1.8.jar:/usr/lib/ambari-server/jackson-jaxrs-1.9.9.jar:/usr/lib/ambari-server/commons-collections-3.2.1.jar:/usr/lib/ambari-server/jaxb-api-2.2.2.jar:/usr/lib/ambari-server/commons-compress-1.5.jar:/usr/lib/ambari-server/guice-multibindings-3.0.jar:/usr/lib/ambari-server/commons-configuration-1.6.jar:/usr/lib/ambari-server/jul-to-slf4j-1.7.2.jar:/usr/lib/ambari-server/commons-csv-1.1.jar:/usr/lib/ambari-server/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-server/commons-digester-1.8.jar:/usr/lib/ambari-server/jcip-annotations-1.0.jar:/usr/lib/ambari-server/commons-httpclient-3.1.jar:/usr/lib/ambari-server/commons-io-2.1.jar:/usr/lib/ambari-server/log4j-1.2.17.jar:/usr/lib/ambari-server/commons-lang-2.5.jar:/usr/lib/ambari-server/jdk.tools-1.7.jar:/usr/lib/ambari-server/commons-logging-1.1.1.jar:/usr/lib/ambari-server/jersey-client-1.19.jar:/usr/lib/ambari-server/commons-math3-3.1.1.jar:/usr/lib/ambari-server/jersey-core-1.19.jar:/usr/lib/ambari-server/commons-net-1.4.1.jar:/usr/lib/ambari-server/jersey-guice-1.19.jar:/usr/lib/ambari-server/curator-client-2.7.1.jar:/usr/lib/ambari-server/jersey-json-1.19.jar:/usr/lib/ambari-server/curator-framework-2.7.1.jar:/usr/lib/ambari-server/jersey-multipart-1.19.jar:/usr/lib/ambari-server/curator-recipes-2.7.1.jar:/usr/lib/ambari-server/libpam4j-1.8.jar:/usr/lib/ambari-server/ecj-3.5.1.jar:/usr/lib/ambari-server/jersey-server-1.19.jar:/usr/lib/ambari-server/eclipselink-2.6.2.jar:/usr/lib/ambari-server/mailapi-1.5.2.jar:/usr/lib/ambari-server/ehcache-2.10.0.jar:/usr/lib/ambari-server/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-server/metrics-jvm-3.1.0.jar:/usr/lib/ambari-server/httpclient-4.2.5.jar:/usr/lib/ambari-server/metrics-core-3.1.0.jar:/usr/lib/ambari-server/httpcore-4.2.4.jar:/usr/lib/ambari-server/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-server/jackson-annotations-2.1.4.jar:/usr/lib/ambari-server/jersey-servlet-1.19.jar:/usr/lib/ambari-server/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-server/netty-3.7.0.Final.jar:/usr/lib/ambari-server/jets3t-0.9.0.jar:/usr/lib/ambari-server/jetty-client-8.1.19.v20160209.jar:/usr/lib/ambari-server/stax-api-1.0-2.jar:/usr/lib/ambari-server/jetty-continuation-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-http-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-security-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-server-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-servlet-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-servlets-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-util-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-webapp-8.1.19.v20160209.jar:/usr/lib/ambari-server/jetty-xml-8.1.19.v20160209.jar:/usr/lib/ambari-server/jsp-2.1-glassfish-2.1.v20100127.jar:/usr/lib/ambari-server/validation-api-1.1.0.Final.jar:/usr/lib/ambari-server/jsp-api-2.1-glassfish-2.1.v20100127.jar:/usr/lib/ambari-server/mchange-commons-java-0.2.11.jar:/usr/lib/ambari-server/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-server/objenesis-2.1.jar:/usr/lib/ambari-server/objenesis-tck-1.2.jar:/usr/lib/ambari-server/paranamer-2.3.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar:/usr/lib/ambari-server/protobuf-java-2.5.0.jar:/usr/lib/ambari-server/quartz-2.2.1.jar:/usr/lib/ambari-server/quartz-jobs-2.2.1.jar:/usr/lib/ambari-server/slf4j-api-1.7.2.jar:/usr/lib/ambari-server/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-server/smtp-1.5.2.jar:/usr/lib/ambari-server/snappy-java-1.0.4.1.jar:/usr/lib/ambari-server/snmp4j-1.10.1.jar:/usr/lib/ambari-server/spring-aop-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-asm-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-beans-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-context-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-core-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-expression-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-jdbc-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-ldap-core-2.0.4.RELEASE.jar:/usr/lib/ambari-server/velocity-1.7.jar:/usr/lib/ambari-server/spring-security-config-3.1.2.RELEASE.jar:/usr/lib/ambari-server/xmlenc-0.52.jar:/usr/lib/ambari-server/spring-security-core-3.1.2.RELEASE.jar:/usr/lib/ambari-server/cobertura-2.1.1.jar:/usr/lib/ambari-server/spring-security-kerberos-core-1.0.1.RELEASE.jar:/usr/lib/ambari-server/spring-web-3.0.7.RELEASE.jar:/usr/lib/ambari-server/spring-security-kerberos-web-1.0.1.RELEASE.jar:/usr/lib/ambari-server/xz-1.2.jar:/usr/lib/ambari-server/spring-security-ldap-4.0.4.RELEASE.jar:/usr/lib/ambari-server/zookeeper-3.4.6.jar:/usr/lib/ambari-server/spring-security-web-3.1.2.RELEASE.jar:/usr/lib/ambari-server/spring-tx-3.0.7.RELEASE.jar:/usr/lib/oracle/11.2/client64/lib/ojdbc5.jar:/usr/lib/oracle/11.2/client64/lib/ojdbc6.jar:/usr/lib/oracle/11.2/client64/lib/xstreams.jar /usr/lib/ambari-server/ambari-metrics-common-2.5.2.0.95.jar /usr/lib/ambari-server/ambari-server-2.5.2.0.95.jar /usr/lib/ambari-server/ambari-views-2.5.2.0.95.jar" root

The target jars are :-

1. /usr/lib/ambari-server/ambari-metrics-common-2.5.2.0.95.jar

2./usr/lib/ambari-server/ambari-server-2.5.2.0.95.jar

3./usr/lib/ambari-server/ambari-views-2.5.2.0.95.jar

The instrumentation completes with out giving any error:-

2017-07-02 21:16:24,491|INFO|MainThread|machine.py:159 - run()||GUID=a3384dea-adb2-47cc-9c0b-8460a0582a3c|This is MOTD message, added for testing in qe infra
2017-07-02 21:16:24,949|INFO|MainThread|machine.py:159 - run()||GUID=a3384dea-adb2-47cc-9c0b-8460a0582a3c|Cobertura 2.1.1 - GNU GPL License (NO WARRANTY) - See COPYRIGHT file
2017-07-02 21:16:42,679|INFO|MainThread|machine.py:159 - run()||GUID=a3384dea-adb2-47cc-9c0b-8460a0582a3c|[INFO] Cobertura: Coverage data file /tmp/coverage-tmp/cobertura-2.1.1/cobertura-data.ser either does not exist or is not readable.  Creating a new data file.
2017-07-02 21:16:44,442|INFO|MainThread|machine.py:159 - run()||GUID=a3384dea-adb2-47cc-9c0b-8460a0582a3c|[INFO] Cobertura: Saved information on 2089 classes.
2017-07-02 21:16:44,916|INFO|MainThread|machine.py:159 - run()||GUID=a3384dea-adb2-47cc-9c0b-8460a0582a3c|[INFO] Cobertura: Loaded information on 2089 classes.
2017-07-02 21:16:46,618|INFO|MainThread|machine.py:159 - run()||GUID=a3384dea-adb2-47cc-9c0b-8460a0582a3c|[INFO] Cobertura: Saved information on 2089 classes.

As soon as I start ambari-server, it failed:-

2017-07-02 21:16:46,716|INFO|MainThread|machine.py:145 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|RUNNING: sudo su - -c "ambari-server start" root
2017-07-02 21:16:46,771|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|######## Hortonworks #############
2017-07-02 21:16:46,772|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|This is MOTD message, added for testing in qe infra
2017-07-02 21:16:46,794|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Using python  /usr/bin/python
2017-07-02 21:16:46,794|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Starting ambari-server
2017-07-02 21:16:47,831|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Ambari Server running with administrator privileges.
2017-07-02 21:16:47,832|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Organizing resource files at /var/lib/ambari-server/resources...
2017-07-02 21:16:47,832|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Ambari database consistency check started...
2017-07-02 21:16:47,832|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Server PID at: /var/run/ambari-server/ambari-server.pid
2017-07-02 21:16:47,832|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Server out at: /var/log/ambari-server/ambari-server.out
2017-07-02 21:16:47,832|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Server log at: /var/log/ambari-server/ambari-server.log
2017-07-02 21:16:53,880|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|Waiting for server start.........Unable to determine server PID. Retrying...
2017-07-02 21:16:59,917|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|......Unable to determine server PID. Retrying...
2017-07-02 21:17:05,958|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|......Unable to determine server PID. Retrying...
2017-07-02 21:17:05,961|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|ERROR: Exiting with exit code -1.
2017-07-02 21:17:05,961|INFO|MainThread|machine.py:159 - run()||GUID=a282a54e-6b04-4612-a53f-ac7023a29608|REASON: Ambari Server java process died with exitcode 1. Check /var/log/ambari-server/ambari-server.out for more information.

And in the /var/log/ambari-server/ambari-server.out:-

hence I checked for this class in ambari-server.jar

Question:- This has only happened in centos7 and centos6 os. Ambari-server was fine in Debian os. Is ambari-server jar is getting corrupted by instrumentation command. How do I further debug on this?

[#2 · Accepted solution]

@Pradarttana Panda

The Jar permission is correct ? I mean the user who is running the instrument job is having permission to read:

/usr/lib/ambari-server/ambari-server-2.5.2.0.95.jar

.

[#3]

Creating a new cluster, I will verify and update in this thread