Install NiFi on HDP 2.5?

[#1]

Hi, i am following the online tutorials and i am currently trying to install NiFi but it keeps loading at "Assign master components to hosts you want to run them on." without moving past this . I havent added any nodes to my cluster yet, is this the issue?

[#2]

@Roger Young

That's now looks a bit tricky with raspberry pis in the picture.

1. Your sandbox should be configure to access the public repo unless you have downlaode the Ambari 2.x,HDP 2.x,HDP-UTILS 1.x etc
2. Have the same version of OS running on Raspberry as the Sandbox
3. Do the basic preparatory configuration for HDP installation.
4. Important is the network setup between the participating nodes otherwise you wont succeed in your installation.
First time someone sets HDP on raspberry. 

Reference

[#3]

OK thank you for your time

[#4]

Hi @Geoffrey Shelton Okot. I decided to add a CentOS vm as a node to my cluster instead of the pi. I have configured all the network and prereq steps, installed ambari server on the sandbox and manually installed the ambari agent on the centos vm. I can ssh no problem but it keeps failing at the "Comfirm Hosts" in Ambari with the following error.

Registering with the server...
Registration with the server failed.

Is there any way of knowing what exactly it is failing on as i have been trying this for a couple of days now? 

Thanks

[#5]

@Roger Young

Registration failures normally should be recorded in these 2 logs.

Can you upload these 2 logs.

Ambari Server logs are found at

/var/log/ambari-server/ambari-server.log 

Ambari Agent logs are found at

/var/log/ambari-agent/ambari-agent.log.

[#6]

Hi, thank for the reply, this is a snippet of the agent log file

WARNING 2016-11-02 19:09:34,388 NetUtil.py:89 - Failed to connect to https://192.168.1.90:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 19:09:34,389 NetUtil.py:112 - Server at https://192.168.1.90:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 19:09:44,389 NetUtil.py:60 - Connecting to https://192.168.1.90:8440/ca
WARNING 2016-11-02 19:09:48,592 NetUtil.py:89 - Failed to connect to https://192.168.1.90:8440/ca due to [Errno 4] Interrupted system call  
WARNING 2016-11-02 19:09:48,593 NetUtil.py:112 - Server at https://192.168.1.90:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:09:40,015 main.py:71 - loglevel=logging.INFO
INFO 2016-11-02 22:09:40,017 main.py:71 - loglevel=logging.INFO
INFO 2016-11-02 22:09:40,023 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-11-02 22:09:40,027 DataCleaner.py:120 - Data cleanup started
INFO 2016-11-02 22:09:40,027 DataCleaner.py:122 - Data cleanup finished
INFO 2016-11-02 22:09:40,054 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2016-11-02 22:09:40,062 main.py:289 - Connecting to Ambari server at https://sandbox.hortonworks.com:8440 (192.168.1.90)
INFO 2016-11-02 22:09:40,062 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:09:40,064 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:09:40,064 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:09:50,065 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:09:50,066 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:09:50,067 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:00,067 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:00,068 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:00,068 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:10,069 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:10,071 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:10,071 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:20,072 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:20,073 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:20,073 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:30,074 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:30,075 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:30,075 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:40,076 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:40,077 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:40,077 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:50,078 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:50,079 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:50,079 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:11:00,080 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...



WARNING 2016-11-02 22:15:10,130 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:15:10,130 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:15:20,131 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca

WARNING 2016-11-02 22:16:00,145 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:16:00,146 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:16:10,147 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:16:10,148 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:16:10,148 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...

[#7]

this is a snippet from earlier server file

02 Nov 2016 18:26:34,122  INFO [main] FilterInvocationSecurityMetadataSourceParser:134 - Creating access control expression attribute 'isAuthenticated()' for /**
02 Nov 2016 18:26:34,142  INFO [main] HttpSecurityBeanDefinitionParser:264 - Checking sorted filter chain: [Root bean: class [org.springframework.security.web.context.SecurityContextPersistenceFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 300, Root bean: class [org.springframework.security.web.authentication.www.BasicAuthenticationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1200, <ambariAuthorizationFilter>, order = 1201, Root bean: class [org.springframework.security.web.savedrequest.RequestCacheAwareFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1300, Root bean: class [org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1400, Root bean: class [org.springframework.security.web.authentication.AnonymousAuthenticationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1700, Root bean: class [org.springframework.security.web.session.SessionManagementFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1800, Root bean: class [org.springframework.security.web.access.ExceptionTranslationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1900, <org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0>, order = 2000]
02 Nov 2016 18:26:34,161  INFO [main] DefaultListableBeanFactory:557 - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@475eb4fd: defining beans [org.springframework.security.filterChains,org.springframework.security.filterChainProxy,org.springframework.security.web.PortMapperImpl#0,org.springframework.security.web.PortResolverImpl#0,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#0,org.springframework.security.authentication.ProviderManager#0,org.springframework.security.web.context.HttpSessionSecurityContextRepository#0,org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy#0,org.springframework.security.web.savedrequest.HttpSessionRequestCache#0,org.springframework.security.web.access.expression.DefaultWebSecurityExpressionHandler#0,org.springframework.security.access.vote.AffirmativeBased#0,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#0,org.springframework.security.authentication.AnonymousAuthenticationProvider#0,org.springframework.security.userDetailsServiceFactory,org.springframework.security.web.DefaultSecurityFilterChain#0,org.springframework.security.authentication.dao.DaoAuthenticationProvider#0,org.springframework.security.authentication.DefaultAuthenticationEventPublisher#0,org.springframework.security.authenticationManager,ambariEntryPoint]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@440d2d64
02 Nov 2016 18:26:34,415  INFO [main] DefaultSecurityFilterChain:28 - Creating filter chain: org.springframework.security.web.util.AnyRequestMatcher@1, [org.springframework.security.web.context.SecurityContextPersistenceFilter@811d8d6, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@7f57a7a4, org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter@6dcc7696, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@45ac973, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@28b7646, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@40239b34, org.springframework.security.web.session.SessionManagementFilter@4e553d60, org.springframework.security.web.access.ExceptionTranslationFilter@1c966488, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@45f6181a]
02 Nov 2016 18:26:34,532  INFO [main] Configuration:855 - Web App DIR test /usr/lib/ambari-server/web
02 Nov 2016 18:26:34,550  INFO [main] CertificateManager:68 - Initialization of root certificate
02 Nov 2016 18:26:34,551  INFO [main] CertificateManager:70 - Certificate exists:true

[#8]

my sandbox is centos7 and the node is centos6, hopefully that doesnt make a difference. I followed this link

https://community.hortonworks.com/storage/attachments/4023-network-and-prereq-setup.pdf

and made the necessary change for centos7. I also gave the sandbox and node static ip addresses but that seems ok as i can ssh without password just fine.

[#9]

This is snippet from the server

02 Nov 2016 22:14:41,980  WARN [main] AbstractLifeCycle:204 - FAILED org.eclipse.jetty.server.Server@33ef393a: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:437)
        at sun.nio.ch.Net.bind(Net.java:429)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:497)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:715)
02 Nov 2016 22:14:41,980 ERROR [main] AmbariServer:518 - Could not bind to server port - instance may already be running. Terminating this instance.
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:437)
        at sun.nio.ch.Net.bind(Net.java:429)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:497)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:715)
02 Nov 2016 22:14:41,980 ERROR [main] AmbariServer:717 - Failed to run the Ambari Server
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:437)
        at sun.nio.ch.Net.bind(Net.java:429)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:497)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:715)
02 Nov 2016 22:14:41,983  INFO [main] ContextHandler:843 - stopped o.e.j.s.ServletContextHandler{/,file:/usr/lib/ambari-server/web/}
02 Nov 2016 22:14:41,986  INFO [main] ELContextCleaner:63 - javax.el.BeanELResolver purged
02 Nov 2016 22:14:41,986  INFO [main] ContextHandler:843 - stopped o.e.j.w.WebAppContext{/views/ADMIN_VIEW/2.1.0/INSTANCE,file:/var/lib/ambari-server/resources/views/work/ADMIN_VIEW%7B2.1.0%7D/},/var/lib/ambari-server/resources/views/work/ADMIN_VIEW{2.1.0}

[#10]

make sure the server is actually stopped, something else is running on that port

do a ps -ef and see if it's still running as a zombie and kill -9 it

If you can hard reboot the server that would help.

[#11]

Hi @Timothy Spann, i stopped the server and tried again. It still failed. Because the server is stopped i cant see what the issue is in in the log.The ambari-agent log produces the same response as before. Connection refused