Nifi 2.0-M2 Support with flow xml gz

[#1]

Hi Team,

I am upgrading my old nifi 1.14 to 2.0 M2 version. But i want to support of both flow xml and flow json. for this reason what should my approach to migrate it as soon as possible.. So my nifi can work either with xml or with json as per my requirement.

Thank you for your time. 

[#2]

@Trilok Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @joseomjr  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

@Trilok 

The older flow.xml.gz format was deprecated as of Apache NiFi 1.16 in favor of the newer flow.json.gz format.   NiFi 1.16+ will only load the flow.xml.gz if the flow.json.gz does not already exist during startup.  Upon successful startup, NiFi will generate the flow.json.gz.  The NiFi 1.16+ version will still generate both the flow.xml.gz and flow.json.gz formats with every change made on the UI. 

With the major release of Apache NiFi 2.x, the deprecated flow.xml.gz format was removed.
There is no option in NiFi 2.0 to support the older flow.xml.gz format.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Thanks @MattWho for the suggestion, i am trying to upgrade from 1.27.0 to 2.0.0M3, and as you suggetsed for deprecation of flow xml file, i updated my nifi.properties accordingly and make it from 

nifi.flow.configuration.file=./conf/flow/flow.xml.gz to 

nifi.flow.configuration.file=./conf/flow/flow.json.gz

Is my understanding correct?
I am still getting nifi pod not coming back and aplication server getting shutdown in few minutes.

Your suggestion would really be appreciated

[#5]

@Shampy 

Apache NiFi does NOT support the older flow.xml.gz format. It can only load the newer flow.json.gz format.  Your Apache NiFi 1.27 should be producing both the flow.xml.gz and flow.json.gz flow storage formats.  You'll need to use the flow.json.gz format in your NiFi 2.x installation.

Apache NiFi 1.x introduced the newer flow.json.gz flow storage format in Apache NiFi 1.16 and newer.  In those newer versions of Apache NiFi 1.16+ will generate the newer flow.json.gz format and still maintain the older flow.xml.gz format.   This positions you for upgrading to Apache NiFi 2.x.  You'll now have the flow.json.gz needed to load in your 2.x version.

The proper path to Apache NiFi 2.x is to first to upgrade to the latest Apache NiFi 1.x release.  Before upgrading to Apache NiFi 2.x version, you should review the release notes between your current version to the version you plan to upgrade to.  This allow you to see if you are using and components that have been removed or if any breaking changes impact your dataflows:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes/#ReleaseNotes-Version2.1.0

NOTE: 
In Apache NiFi 1.x versions that support both the flow.xml.gz and the newer flow.json.gz format, the flow.xml.gz format file will be ignored on startup if a flow.json.gz exists. 

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#6]

thanks @MattWho

You are right i updated the properties file to use flow.json.gz instead of flow.xml.gz

On deploying i got some another weird error which is quite much not clear to me to look after, it would be very grateful if you can suggest something on this. 

What i am getting from below error that it could be because of flow.json.gz might be corrupted or so. So for that i cleared out the old flow json file and did a quick restart for nifi.

But getting same error now as below (pod logs context pasted below)

org.apache.nifi.controller.serialization.FlowSerializationException: Could not parse flow as a VersionedDataflow
at org.apache.nifi.cluster.protocol.StandardDataFlow.parseVersionedDataflow(StandardDataFlow.java:167)
at org.apache.nifi.cluster.protocol.StandardDataFlow.getVersionedDataflow(StandardDataFlow.java:108)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.isFlowEmpty(VersionedFlowSynchronizer.java:1290)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:159)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1817)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:778)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:499)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at org.eclipse.jetty.ee10.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:1587)
at org.eclipse.jetty.ee10.servlet.ServletContextHandler.contextInitialized(ServletContextHandler.java:498)
at org.eclipse.jetty.ee10.servlet.ServletHandler.initialize(ServletHandler.java:675)
at org.eclipse.jetty.ee10.servlet.ServletContextHandler.startContext(ServletContextHandler.java:1321)
at org.eclipse.jetty.ee10.webapp.WebAppContext.startWebapp(WebAppContext.java:1346)
at org.eclipse.jetty.ee10.webapp.WebAppContext.startContext(WebAppContext.java:1304)
at org.eclipse.jetty.ee10.servlet.ServletContextHandler.lambda$doStart$0(ServletContextHandler.java:1047)
at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.call(ContextHandler.java:1452)
at org.eclipse.jetty.ee10.servlet.ServletContextHandler.doStart(ServletContextHandler.java:1044)
at org.eclipse.jetty.ee10.webapp.WebAppContext.doStart(WebAppContext.java:503)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:120)
at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:491)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:120)
at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:491)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:641)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:120)
at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:491)
at org.eclipse.jetty.server.Server.doStart(Server.java:582)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:874)
at org.apache.nifi.runtime.Application.startServer(Application.java:131)
at org.apache.nifi.runtime.Application.run(Application.java:78)
at org.apache.nifi.runtime.Application.run(Application.java:60)
at org.apache.nifi.NiFi.main(NiFi.java:42)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 1, column: 1]
at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2660)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:741)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2752)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:867)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:753)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:5004)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4910)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3921)
at org.apache.nifi.cluster.protocol.StandardDataFlow.parseVersionedDataflow(StandardDataFlow.java:165)
... 38 common frames omitted

[#7]

@Shampy 

You are saying that with no flow.json.gz file present, NIFi throws above exception on startup.
 With No flow.json.gz file, NiFi will create one during startup.

I also suggest using the official Apache NiFi 2.x release and stop using the pre-release Maintenance releases that were put out during development of NiFi 2.x.  There were 4 maintenance (M1 - M4) releases put out before the first official release of Apache NiFi 2.0.0.

The latest official release ofd Apache NiFi 2.x is NiFi 2.2.0 which can be downloaded from here:
https://nifi.apache.org/download/

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#8]

@Trilok Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.