Metron REST stopped right after start

[#1]

I am trying to start Metron on cenos7. However, every time I start Metron REST, it just stops after several seconds.

This is my error info in /var/log/metron/metron-rest.log:

18/05/21 09:35:15 WARN embedded.AnnotationConfigEmbeddedWebApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'searchController': Unsatisfied dependency expressed through field 'searchService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'searchServiceImpl' defined in URL [jar:file:/usr/metron/0.4.1/lib/metron-rest-0.4.1.jar!/org/apache/metron/rest/service/impl/SearchServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexDao' defined in class path resource [org/apache/metron/rest/config/IndexConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.metron.indexing.dao.IndexDao]: Factory method 'indexDao' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/shaded/org/apache/commons/logging/LogFactory
18/05/21 09:35:15 INFO zookeeper.ZooKeeper: Session: 0x263747fa8d72ff0 closed
18/05/21 09:35:15 INFO zookeeper.ClientCnxn: EventThread shut down
18/05/21 09:35:15 WARN boot.SpringApplication: Error handling failed (Error creating bean with name 'delegatingApplicationListener' defined in class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration': Initialization of bean failed; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'org.springframework.context.annotation.ConfigurationClassPostProcessor.importRegistry' is defined)
18/05/21 09:35:15 ERROR boot.SpringApplication: Application startup failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'searchController': Unsatisfied dependency expressed through field 'searchService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'searchServiceImpl' defined in URL [jar:file:/usr/metron/0.4.1/lib/metron-rest-0.4.1.jar!/org/apache/metron/rest/service/impl/SearchServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexDao' defined in class path resource [org/apache/metron/rest/config/IndexConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.metron.indexing.dao.IndexDao]: Factory method 'indexDao' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/shaded/org/apache/commons/logging/LogFactory
 at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:569)
 at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
 at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1219)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
 at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541)
 at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
 at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
 at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
 at org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:27)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'searchServiceImpl' defined in URL [jar:file:/usr/metron/0.4.1/lib/metron-rest-0.4.1.jar!/org/apache/metron/rest/service/impl/SearchServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexDao' defined in class path resource [org/apache/metron/rest/config/IndexConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.metron.indexing.dao.IndexDao]: Factory method 'indexDao' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/shaded/org/apache/commons/logging/LogFactory
 at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
 at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1148)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1051)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
 at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:207)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
 at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:566)
 ... 19 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexDao' defined in class path resource [org/apache/metron/rest/config/IndexConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.metron.indexing.dao.IndexDao]: Factory method 'indexDao' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/shaded/org/apache/commons/logging/LogFactory
 at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1128)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1023)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
 at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:207)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
 at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
 at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
 ... 32 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.metron.indexing.dao.IndexDao]: Factory method 'indexDao' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/shaded/org/apache/commons/logging/LogFactory
 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
 at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
 ... 45 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/shaded/org/apache/commons/logging/LogFactory
 at org.apache.hadoop.hbase.HBaseConfiguration.<clinit>(HBaseConfiguration.java:40)
 at org.apache.metron.indexing.dao.HBaseDao.init(HBaseDao.java:81)
 at org.apache.metron.indexing.dao.IndexDaoFactory.create(IndexDaoFactory.java:38)
 at org.apache.metron.rest.config.IndexConfig.indexDao(IndexConfig.java:72)
 at org.apache.metron.rest.config.IndexConfig$$EnhancerBySpringCGLIB$$bbe5d891.CGLIB$indexDao$0(<generated>)
 at org.apache.metron.rest.config.IndexConfig$$EnhancerBySpringCGLIB$$bbe5d891$$FastClassBySpringCGLIB$$aa005056.invoke(<generated>)
 at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
 at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
 at org.apache.metron.rest.config.IndexConfig$$EnhancerBySpringCGLIB$$bbe5d891.indexDao(<generated>)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
 ... 46 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.shaded.org.apache.commons.logging.LogFactory
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 ... 60 more

And here is my REST configuration on Ambari:

Could anyone help me please?

[#2]

Hello @Xubin Chen,

This could be due to bad Solr jar used in Metron Rest service. (Since it is not clear in question which Metron version you are using) This can be fixed by either reinstalling Metro Rest bits with the latest ones from repo OR by replacing the Solr jar with the good copy from the latest Metron repo. (I'd prefer the first option).

Hope this helps!

[#3]

Vipin Rathor Thanks for help! My Metron version is 0.4.1, but what should I do to restall Metron Rest bits? I mean, what commands should I use?

[#4]

Hello @Xubin Chen,

To reinstall Metron Rest RPMs, you can use "yum reinstall <pkg-name>" command. To find the right package name for Metron Rest, you can run "rpm -qa | grep metron" command and look for Rest package in the output.

Hope this helps!

[#5]

Hello @Vipin Rathor,

Thanks for your patient guidence! I executed "yum reinstall <metron-rest pkg-name>" and it was reinstalled successfully, but Metron REST still failed after I started it and the error info didn't change either.

[#6]

Can you please share log files (/var/log/metron/metron-rest.log)

[#7]

Thanks for the response! I have already pasted the content above.

[#8]

Hello @Xubin Chen,

I see that you are on Metron 0.4.1, can you please try installing Metron 0.4.2 (available with HCP 1.4.2)? Also, which Hortonworks CyberSecurity Platform version you are using currently? Thanks.