Is it possible to tell Cloudbreak to create Azure Storage Accounts with "Secure transfer required" enabled?

[#1]

Our digital security team has policy in place that requires storage accounts be created with secure transfer enabled. This is causing cluster deployment to fail almost immediately with the following error:

cloudbreak_1   | 2018-10-15 21:14:05,064 [RxIoScheduler-5] log:55 INFO  c.m.a.m.s.StorageAccounts create - [owner:spring] [type:springLog] [id:] [name:] [flow:] [tracking:] --> PUT https://management.azure.com/subscriptions/<subscription>/resourceGroups/cloudbreak-images/providers...
cloudbreak_1   | 2018-10-15 21:14:05,244 [RxIoScheduler-5] log:55 INFO  c.m.a.m.s.StorageAccounts create - [owner:spring] [type:springLog] [id:] [name:] [flow:] [tracking:] <-- 403 Forbidden https://management.azure.com/subscriptions/<subscription>/resourceGroups/cloudbreak-images/providers... (179 ms, 1581-byte body)
cloudbreak_1   | 2018-10-15 21:14:05,258 [reactorDispatcher-15] prepareImage:77 ERROR c.s.c.c.a.AzureSetup - [owner:11e84560-4e65-4883-ac27-d1229f87a9d0] [type:STACK] [id:6] [name:stefan-test-2] [flow:1549ed56-7093-4a88-8668-0a3583828def] [tracking:] Could not create image with the specified parameters
cloudbreak_1   | com.microsoft.azure.CloudException: Resource 'cbimgne0af2cfef5d804171a' was disallowed by policy. Policy identifiers: '[{"policyAssignment":{"name":"[Custom] Ensure https traffic only for storage account","id":"/providers/Microsoft.Management/managementgroups/Root/providers/Microsoft.Authorization/policyAssignments/ab6676de3e234be08ae27478"},"policyDefinition":{"name":"[Custom] Ensure https traffic only for storage account","id":"/providers/Microsoft.Management/managementgroups/Root/providers/Microsoft.Authorization/policyDefinitions/[Custom] Ensure https traffic only for storage account"}}]'.

Is it possible to change this somewhere such that the storage account is created with secure transfer enabled? We really don't want to go after a policy "exception" if at all possible.

[#2]

Cloudbreak does not support this azure feature yet, but can you please create manually the "cbimgne0af2cfef5d804171a" storage account under "cloudbreak-images" resource group before cluster install? Cloudbreak will recognize the resource group and storage account exists, so cloudbreak will not create this storage account.

[#3]

Hi Peter,

Thanks for the suggestion! That got me a little further, but it still fails. It appears that it is trying to connect with http and not https.

Stefan

cloudbreak_1   | Caused by: com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: can't create container in storage, storage service error occurred
cloudbreak_1   |        at com.sequenceiq.cloudbreak.cloud.azure.client.AzureClient.createContainerInStorage(AzureClient.java:295)
cloudbreak_1   |        at com.sequenceiq.cloudbreak.cloud.azure.AzureSetup.copyVhdImageIfNecessary(AzureSetup.java:96)
cloudbreak_1   |        at com.sequenceiq.cloudbreak.cloud.azure.AzureSetup.prepareImage(AzureSetup.java:75)
cloudbreak_1   |        ... 23 common frames omitted
cloudbreak_1   | Caused by: com.microsoft.azure.storage.StorageException: The account being accessed does not support http.
cloudbreak_1   |        at com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:175)
cloudbreak_1   |        at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:94)
cloudbreak_1   |        at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:315)
cloudbreak_1   |        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
cloudbreak_1   |        at com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:769)
cloudbreak_1   |        at com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:379)
cloudbreak_1   |        at com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:326)
cloudbreak_1   |        at com.sequenceiq.cloudbreak.cloud.azure.client.AzureClient.createContainerInStorage(AzureClient.java:292)
cloudbreak_1   |        ... 25 common frames omitted

<br>

[#4]

@Stefan Garrard

Your issue is valid, I've opened a PR with the fix:

https://github.com/hortonworks/cloudbreak/pull/4086

May I ask which version of Cloudbreak are you using?

[#5]

Hi @pdarvasi, Thanks for submitting the PR for us! We are running Cloudbreak 2.7.1.
Stefan

[#6]

@Stefan Garrard

You can upgrade to the newest version containing the fix following these steps:

1.Navigate to your deployment directory, typically /var/lib/cloudbreak-deployment

2.Edit and then run the following curl command:

export CBD_VERSION=2.7.3-rc.6
curl -Ls public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_${CBD_VERSION}_$(uname)_x86_64.tgz | tar -xz -C /bin cbd

3.Verify the version:

cbd version 

4.Next, restart Cloudbreak by using:

cbd restart  

Hope this helps resolving your issue!

[#7]

@pdarvasi I tried doing this a couple of days and also again today and ran into this error after downloading the new cbd version:

[root@hostname cloudbreak-deployment]# curl -Ls public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_${CBD_VERSION}_$(uname)_x86_64.tgz | tar -xz -C /bin cbd
[root@hostname cloudbreak-deployment]#
[root@hostname cloudbreak-deployment]# cbd version
local version:2.7.3-rc.6-ce93c9d
latest release:2.7.2
docker images:
 hortonworks/haveged:1.1.0
 hortonworks/socat:1.0.0
 hortonworks/logspout:v3.2.2
 hortonworks/logrotate:1.0.1
 hortonworks/cbd-smartsense:0.13.2
 hortonworks/cloudbreak-uaa:3.6.5-pgupdate
 hortonworks/cloudbreak:2.7.3-rc.6
 hortonworks/hdc-auth:2.7.3-rc.6
 hortonworks/hdc-web:2.7.3-rc.6
 hortonworks/cloudbreak-autoscale:2.7.3-rc.6
[root@hostname cloudbreak-deployment]# cbd start
generating docker-compose.yml
generating uaa.yml
Initialize and migrate databases
Starting cbreak_commondb_1 ... done
Pulling uluwatu (hortonworks/hdc-web:2.7.3-rc.6)...
Trying to pull repository registry.access.redhat.com/hortonworks/hdc-web ...
Trying to pull repository docker.io/hortonworks/hdc-web ...
manifest for docker.io/hortonworks/hdc-web:2.7.3-rc.6 not found

Is there something that I might be doing wrong?

Thanks again for your help so far!

Stefan

[#8]

@Stefan Garrard

You are right, the docker image build was not completed for that RC build.

Could you please try it out with 2.7.3-rc.16?

Sorry for the inconvenience!