HDP Local Repository update error - API Error 500

[#1]

After an Ambari server upgrade from 2.5.0.3 to 2.6.2.2, I noticed that operations started from Ambari Web UI were hanging without any progress in the status bar. Looking through the logs, showed an error regarding operating system matching. 

28 Nov 2020 16:48:52,372  WARN [ambari-action-scheduler] ActionScheduler:316 - Exception received
java.lang.RuntimeException: org.apache.ambari.server.controller.spi.SystemException: Operating System matching redhat7 could not be found
        at org.apache.ambari.server.actionmanager.ExecutionCommandWrapper.getExecutionCommand(ExecutionCommandWrapper.java:253)
        at org.apache.ambari.server.actionmanager.ActionScheduler.processInProgressStage(ActionScheduler.java:704)
        at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:417)
        at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:310)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ambari.server.controller.spi.SystemException: Operating System matching redhat7 could not be found
        at org.apache.ambari.server.state.stack.upgrade.RepositoryVersionHelper.getOSEntityForHost(RepositoryVersionHelper.java:422)

A colleague's searching revealed that this problem might be related to HDP local repository setup. So I checked the configuration of HDP local repository and realized an issue with its configuration. We inherited this cluster and have no information on how the HDP 2.6 repository was setup before we began the Ambari Server upgrade. 

As can be seen in the picture, the repo versions page was empty and unresponsive. Entries could also not be added.
Using the information given here - https://docs.cloudera.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-installation/content/using_a_local_r... we have tried to use the Ambari rest API to create operating systems entries for HDP but keep getting the 500 server error on each try.

#curl to get the resources
curl -ik -u admin -H "X-Requested-By:ambari" -X GET https://localhost:8080/api/v1/stacks/HDP/versions/2.6/repository_versions?fields=operating_systems/repositories/Repositories/base_url

#response
{
  "items" : [
    {
      "href" : "http://localhost:8080/api/v1/stacks/HDP/versions/2.6/repository_versions/1",
      "RepositoryVersions" : {
        "id" : 1,
        "stack_name" : "HDP",
        "stack_version" : "2.6"
      },
      "operating_systems" : [ ]
    }
  ]
}

#repo.json
{
  "operating_systems" : [
      {
        "OperatingSystems" : {
          "ambari_managed_repositories": false,
          "os_type" : "redhat7",
          "repository_version_id" : 1,
          "stack_name" : "HDP",
          "stack_version" : "2.6"
        },
        "repositories" : [
          {
            "Repositories" : {
              "base_url" : "ftp://localhost/HDP/centos6/2.x/updates",
              "os_type" : "redhat7",
              "repo_id" : "HDP-2.6",
              "repository_version_id" : 1,
              "stack_name" : "HDP",
              "stack_version" : "2.6",
              "unique" : false
            }
          },
          {
            "Repositories" : {
              "base_url" : "ftp://localhost/HDP-UTILS-1.1.0.21/repos/centos6",
              "os_type" : "redhat7",
              "repo_id" : "HDP-UTILS-1.1.0.21",
              "repository_version_id" : 1,
              "stack_name" : "HDP",
              "stack_version" : "2.6",
              "unique" : false
            }
          }
        ]
      }
    ]
}

#curl to make updates
curl -ik -u admin -H "X-Requested-By:ambari" -H "Content-Type: application/json" -d @repo.json -X PUT 'https://localhost:8080/api/v1/stacks/HDP/versions/2.6/repository_versions/1'

#response
Enter host password for user 'admin':
HTTP/1.1 500 Server Error
Strict-Transport-Security: max-age=3156000
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=1w53kyo4ucxx21erviezjlc1z8, Path=/;Secure:HttpOnly
User: admin
Content-Type: test/plain;charset=ISO-8859-1
Content-Length: 48

{
   "status": 500,
   "message": "Server Error"
}

Please help to resolve the issue

Thank you

[#2]

Hi @CaptainJa ,

Thank you for reaching out to community!

It looks like you are hitting Ambari UI Bug (https://issues.apache.org/jira/browse/AMBARI-21854). 

Also, can you please confirm if the operating system you are using is RHEL7? because i see the base url used for repositories is for Centos6. Ensure to update the repository to use RedHat Satellite if the OS is Redhat7. 

[#3]

Hello @Madhur Thanks a lot for the reply.

I can confirm that the operating system is rhel7. The base url used was a configuration setting passed down but we have used it for other clusters without issues. I will nonetheless check with the client to make sure it is correct.

Concerning the link to the bug report, the upgrade was done for Ambari 2.6.2.2 while the mentioned bug was fixed in version 2.6.0.0. Also, the scenarios presented in the bug are a bit different in our case.

Thanks a lot for the help