question Re: Install NiFi on HDP 2.5? in Archives of Support Questions (Read Only)

Install NiFi on HDP 2.5?

r_young — Fri, 28 Oct 2016 00:38:29 GMT

Hi, i am following the online tutorials and i am currently trying to install NiFi but it keeps loading at "Assign master components to hosts you want to run them on." without moving past this . I havent added any nodes to my cluster yet, is this the issue?

Re: Install NiFi on HDP 2.5?

slachterman — Fri, 28 Oct 2016 00:45:22 GMT

Hi @Roger Young, yes, you do need to add nodes to your HDF cluster before assigning components. Please note that NiFi is part of the HDF stack, not HDP, and is managed by a separate Ambari instance.

Re: Install NiFi on HDP 2.5?

r_young — Fri, 28 Oct 2016 00:49:43 GMT

ok thanks for the info.Thanks. When installing nodes through ambari, do i need to manually install anything on the slaves first or does ambari wizard install everything for me?

Re: Install NiFi on HDP 2.5?

Shelton — Fri, 28 Oct 2016 00:50:09 GMT

@Roger Young

HDF 2.0 cannot be installed on an existing HDP cluster ! It is not supported for installing on an Ambari instance with a deployed HDP cluster. HDF 2.0 has it's own Ambari and can be used to create a HDF cluster.

See the below Link

Re: Install NiFi on HDP 2.5?

slachterman — Fri, 28 Oct 2016 00:56:54 GMT

You can choose to use the Ambari wizard to install the Ambari agents on all nodes (via passwordless SSH). Otherwise, you can install ambari-agent manually on each node.

There are pre-requisites you'll need to ensure you have taken care of, as noted in the docs linked by @Geoffrey Shelton Okot, including openssl, ntp, curl, etc. Please the System Requirements and Prepare the Environment sections in http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_ambari-installation/content/ch_getting_ready.html.

Re: Install NiFi on HDP 2.5?

r_young — Fri, 28 Oct 2016 01:10:29 GMT

ok thank you.

Re: Install NiFi on HDP 2.5?

r_young — Fri, 28 Oct 2016 01:17:46 GMT

Hi, thanks for the reply. I am new and still a little confused. So does that mean this tutorial wont work?

http://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/ Or am i missing something?

Re: Install NiFi on HDP 2.5?

Shelton — Fri, 28 Oct 2016 01:39:45 GMT

@Roger Young

The tutorial will definitely work but you need to successfully install your standalone or HDF 2.0 Cluster you dont need to add nodes to your cluster if its a single node.

Question: Are you deploying a single node or cluster ? Remember the HDF 2.0 installation follows the same preparatory steps like the HDP 2.x upto the Ambari Database install and there after you run the management PACK process which ensure that when you start your Ambari you ONLY have the HDF 2.0 repository available !!!

Re: Install NiFi on HDP 2.5?

TimothySpann — Fri, 28 Oct 2016 01:41:41 GMT

You install HDF 2.0 on one cluster; HDP 2.5 on another.

For a sandbox, download nifi.apache.org/downloads NIFI and you can run that as a standalone Java application on HDP sandbox. Just set the port to 8090

Re: Install NiFi on HDP 2.5?

r_young — Fri, 28 Oct 2016 01:54:00 GMT

I started by creating my own cluster for testing purposes with a laptop as master and three raspberry pis acting as nodes. But when I downloaded the sandbox i realized it would be better to format the three raspberry pis and add them as nodes to the sandbox as it has alot of services built in. I havent added them yet, i was planning on doing all the online tutorials first

Re: Install NiFi on HDP 2.5?

Shelton — Fri, 28 Oct 2016 02:23:24 GMT

@Roger Young

That's now looks a bit tricky with raspberry pis in the picture.

1. Your sandbox should be configure to access the public repo unless you have downlaode the Ambari 2.x,HDP 2.x,HDP-UTILS 1.x etc
2. Have the same version of OS running on Raspberry as the Sandbox
3. Do the basic preparatory configuration for HDP installation.
4. Important is the network setup between the participating nodes otherwise you wont succeed in your installation.
First time someone sets HDP on raspberry.

Reference

Re: Install NiFi on HDP 2.5?

r_young — Fri, 28 Oct 2016 02:43:32 GMT

OK thank you for your time

Re: Install NiFi on HDP 2.5?

r_young — Thu, 03 Nov 2016 01:36:15 GMT

Hi @Geoffrey Shelton Okot. I decided to add a CentOS vm as a node to my cluster instead of the pi. I have configured all the network and prereq steps, installed ambari server on the sandbox and manually installed the ambari agent on the centos vm. I can ssh no problem but it keeps failing at the "Comfirm Hosts" in Ambari with the following error.

Registering with the server...
Registration with the server failed.

Is there any way of knowing what exactly it is failing on as i have been trying this for a couple of days now? 

Thanks

Re: Install NiFi on HDP 2.5?

Shelton — Thu, 03 Nov 2016 04:18:15 GMT

@Roger Young

Registration failures normally should be recorded in these 2 logs.

Can you upload these 2 logs.

Ambari Server logs are found at

/var/log/ambari-server/ambari-server.log

Ambari Agent logs are found at

/var/log/ambari-agent/ambari-agent.log.

Re: Install NiFi on HDP 2.5?

r_young — Thu, 03 Nov 2016 05:23:37 GMT

Hi, thank for the reply, this is a snippet of the agent log file

WARNING 2016-11-02 19:09:34,388 NetUtil.py:89 - Failed to connect to https://192.168.1.90:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 19:09:34,389 NetUtil.py:112 - Server at https://192.168.1.90:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 19:09:44,389 NetUtil.py:60 - Connecting to https://192.168.1.90:8440/ca
WARNING 2016-11-02 19:09:48,592 NetUtil.py:89 - Failed to connect to https://192.168.1.90:8440/ca due to [Errno 4] Interrupted system call  
WARNING 2016-11-02 19:09:48,593 NetUtil.py:112 - Server at https://192.168.1.90:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:09:40,015 main.py:71 - loglevel=logging.INFO
INFO 2016-11-02 22:09:40,017 main.py:71 - loglevel=logging.INFO
INFO 2016-11-02 22:09:40,023 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-11-02 22:09:40,027 DataCleaner.py:120 - Data cleanup started
INFO 2016-11-02 22:09:40,027 DataCleaner.py:122 - Data cleanup finished
INFO 2016-11-02 22:09:40,054 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2016-11-02 22:09:40,062 main.py:289 - Connecting to Ambari server at https://sandbox.hortonworks.com:8440 (192.168.1.90)
INFO 2016-11-02 22:09:40,062 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:09:40,064 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:09:40,064 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:09:50,065 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:09:50,066 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:09:50,067 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:00,067 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:00,068 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:00,068 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:10,069 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:10,071 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:10,071 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:20,072 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:20,073 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:20,073 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:30,074 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:30,075 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:30,075 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:40,076 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:40,077 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:40,077 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:10:50,078 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:10:50,079 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:10:50,079 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:11:00,080 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...



WARNING 2016-11-02 22:15:10,130 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:15:10,130 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:15:20,131 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca

WARNING 2016-11-02 22:16:00,145 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:16:00,146 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-11-02 22:16:10,147 NetUtil.py:60 - Connecting to https://sandbox.hortonworks.com:8440/ca
WARNING 2016-11-02 22:16:10,148 NetUtil.py:89 - Failed to connect to https://sandbox.hortonworks.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-11-02 22:16:10,148 NetUtil.py:112 - Server at https://sandbox.hortonworks.com:8440 is not reachable, sleeping for 10 seconds...

Re: Install NiFi on HDP 2.5?

r_young — Thu, 03 Nov 2016 05:59:59 GMT

this is a snippet from earlier server file


02 Nov 2016 18:26:34,122  INFO [main] FilterInvocationSecurityMetadataSourceParser:134 - Creating access control expression attribute 'isAuthenticated()' for /**
02 Nov 2016 18:26:34,142  INFO [main] HttpSecurityBeanDefinitionParser:264 - Checking sorted filter chain: [Root bean: class [org.springframework.security.web.context.SecurityContextPersistenceFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 300, Root bean: class [org.springframework.security.web.authentication.www.BasicAuthenticationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1200, <ambariAuthorizationFilter>, order = 1201, Root bean: class [org.springframework.security.web.savedrequest.RequestCacheAwareFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1300, Root bean: class [org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1400, Root bean: class [org.springframework.security.web.authentication.AnonymousAuthenticationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1700, Root bean: class [org.springframework.security.web.session.SessionManagementFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1800, Root bean: class [org.springframework.security.web.access.ExceptionTranslationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1900, <org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0>, order = 2000]
02 Nov 2016 18:26:34,161  INFO [main] DefaultListableBeanFactory:557 - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@475eb4fd: defining beans [org.springframework.security.filterChains,org.springframework.security.filterChainProxy,org.springframework.security.web.PortMapperImpl#0,org.springframework.security.web.PortResolverImpl#0,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#0,org.springframework.security.authentication.ProviderManager#0,org.springframework.security.web.context.HttpSessionSecurityContextRepository#0,org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy#0,org.springframework.security.web.savedrequest.HttpSessionRequestCache#0,org.springframework.security.web.access.expression.DefaultWebSecurityExpressionHandler#0,org.springframework.security.access.vote.AffirmativeBased#0,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#0,org.springframework.security.authentication.AnonymousAuthenticationProvider#0,org.springframework.security.userDetailsServiceFactory,org.springframework.security.web.DefaultSecurityFilterChain#0,org.springframework.security.authentication.dao.DaoAuthenticationProvider#0,org.springframework.security.authentication.DefaultAuthenticationEventPublisher#0,org.springframework.security.authenticationManager,ambariEntryPoint]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@440d2d64
02 Nov 2016 18:26:34,415  INFO [main] DefaultSecurityFilterChain:28 - Creating filter chain: org.springframework.security.web.util.AnyRequestMatcher@1, [org.springframework.security.web.context.SecurityContextPersistenceFilter@811d8d6, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@7f57a7a4, org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter@6dcc7696, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@45ac973, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@28b7646, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@40239b34, org.springframework.security.web.session.SessionManagementFilter@4e553d60, org.springframework.security.web.access.ExceptionTranslationFilter@1c966488, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@45f6181a]
02 Nov 2016 18:26:34,532  INFO [main] Configuration:855 - Web App DIR test /usr/lib/ambari-server/web
02 Nov 2016 18:26:34,550  INFO [main] CertificateManager:68 - Initialization of root certificate
02 Nov 2016 18:26:34,551  INFO [main] CertificateManager:70 - Certificate exists:true

Re: Install NiFi on HDP 2.5?

r_young — Thu, 03 Nov 2016 06:09:38 GMT

my sandbox is centos7 and the node is centos6, hopefully that doesnt make a difference. I followed this link

https://community.hortonworks.com/storage/attachments/4023-network-and-prereq-setup.pdf

and made the necessary change for centos7. I also gave the sandbox and node static ip addresses but that seems ok as i can ssh without password just fine.

Re: Install NiFi on HDP 2.5?

r_young — Thu, 03 Nov 2016 07:06:35 GMT

This is snippet from the server

02 Nov 2016 22:14:41,980  WARN [main] AbstractLifeCycle:204 - FAILED org.eclipse.jetty.server.Server@33ef393a: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:437)
        at sun.nio.ch.Net.bind(Net.java:429)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:497)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:715)
02 Nov 2016 22:14:41,980 ERROR [main] AmbariServer:518 - Could not bind to server port - instance may already be running. Terminating this instance.
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:437)
        at sun.nio.ch.Net.bind(Net.java:429)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:497)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:715)
02 Nov 2016 22:14:41,980 ERROR [main] AmbariServer:717 - Failed to run the Ambari Server
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:437)
        at sun.nio.ch.Net.bind(Net.java:429)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:497)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:715)
02 Nov 2016 22:14:41,983  INFO [main] ContextHandler:843 - stopped o.e.j.s.ServletContextHandler{/,file:/usr/lib/ambari-server/web/}
02 Nov 2016 22:14:41,986  INFO [main] ELContextCleaner:63 - javax.el.BeanELResolver purged
02 Nov 2016 22:14:41,986  INFO [main] ContextHandler:843 - stopped o.e.j.w.WebAppContext{/views/ADMIN_VIEW/2.1.0/INSTANCE,file:/var/lib/ambari-server/resources/views/work/ADMIN_VIEW%7B2.1.0%7D/},/var/lib/ambari-server/resources/views/work/ADMIN_VIEW{2.1.0}

Re: Install NiFi on HDP 2.5?

TimothySpann — Thu, 03 Nov 2016 07:19:32 GMT

make sure the server is actually stopped, something else is running on that port

do a ps -ef and see if it's still running as a zombie and kill -9 it

If you can hard reboot the server that would help.

Re: Install NiFi on HDP 2.5?

r_young — Thu, 03 Nov 2016 18:11:33 GMT

Hi @Timothy Spann, i stopped the server and tried again. It still failed. Because the server is stopped i cant see what the issue is in in the log.The ambari-agent log produces the same response as before. Connection refused