Nifi fails to start on HDF 3.3.0 upgrade

[#1]

While running express upgrade from HDF 3.1.0 to 3.3.0 in Ambari 2.7.3, Nifi fails to restart with the below log. Even after downgrading the upgrade to 3.1.0 Nifi still fails to restart with the same error.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 1006, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 143, in start
    self.configure(env, is_starting = True)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 111, in configure
    self.write_configurations(params, is_starting)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 231, in write_configurations
    self.write_files(params)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 295, in write_files
    boostrap_notification_content=config_utils.append_xml_content(params.nifi_boostrap_notification_content, params.nifi_boostrap_notification_dict)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/config_utils.py", line 27, in append_xml_content
    original_config_root = ElementTree.fromstring(original_evaluated_config)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1301, in XML
    parser.feed(text)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1643, in feed
    self._raiseerror(v)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1507, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 27, column 29

[#2]

You have to edit Advanced nifi-bootstrap-notification-services-env in Ambari and remove the following part from the services section:

<!--
<service>
<id>http-notification</id>
<class>org.apache.nifi.bootstrap.notification.http.HttpNotificationService</class>
<property name="URL"></property>
</service>
-->

[#3]

@pauljoshiva 

The error which you mposted seems to be occurring because of the incorrect content of "nifi-bootstrap-notification-services-env" teamplate. So can you please check the template in ambari UI and please check and verify the contents present inside this tem[late.]

Ambari UI --> NiFi --> Configs (Tab) --> "Advanced nifi-bootstrap-notification-services-env" --> "Template for bootstrap-notification-services.xml"

Also can you please share the output of the following commad from the Ambari Server host:

# /var/lib/ambari-server/resources/scripts/configs.py --protocol=http --unsafe --user=admin --password=admin --port=8080 --action=get --host=localhost --cluster=KerLatest --config-type=nifi-bootstrap-notification-services-env --file=/tmp/nifi-bootstrap-notification-services-env.json

# cat /tmp/nifi-bootstrap-notification-services-env.json

NOTE: Please replace the username & password for ambari admin / Hostname & Port and also the cluster name "KerLatest" according to your environment.