Error "Could not verify credential" while trying to add credentials in cloudbreak in GCP

[#1]

When I try to add a GCP credential in the UI I get this error:

Failed to verify the credential: Could not verify credential! 403 Forbidden

On console I get this:

# cb credential create from-file --cli-input-json credential.json --name cloudbreakkeyEnter Password: ERROR: status code: 400, message: Failed to verify the credential: Could not verify credential! 403 Forbidden

cb doctor:

# cbd doctor
===> Deployer doctor: Checks your environment, and reports a diagnose.
uname: Linux cbd-deployment-vm 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
local version:2.9.0
latest release:2.9.0
docker command exists: OK
docker client version: 1.13.1
docker client version: 1.13.1
ping 8.8.8.8 on host: OK
ping github.com on host: OK
ping 8.8.8.8 in container: [ERROR]
ping github.com in container: [ERROR] 

I assume there must be some networking issue but don't know what to look for

EDIT: In the logs I get this stacktrace:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1067)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUsingHead(AbstractGoogleClientRequest.java:406)
        at com.google.api.services.compute.Compute$Regions$List.executeUsingHead(Compute.java)
        at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialVerifier.preCheckOfGooglePermission(GcpCredentialVerifier.java:57)
        at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialVerifier$$FastClassBySpringCGLIB$$a658f442.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
        at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
        at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163)
        at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118)
        at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:152)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialVerifier$$EnhancerBySpringCGLIB$$676bf766.preCheckOfGooglePermission(<generated>)
        at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialConnector.verify(GcpCredentialConnector.java:43)
        at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler.accept(CredentialVerificationHandler.java:43)
        at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler.accept(CredentialVerificationHandler.java:20)
        at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler$$FastClassBySpringCGLIB$$f08a8fcf.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler$$EnhancerBySpringCGLIB$$d500084f.accept(<generated>)
        at reactor.bus.EventBus$3.accept(EventBus.java:317)
        at reactor.bus.EventBus$3.accept(EventBus.java:310)
        at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
        at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
        at reactor.bus.EventBus.accept(EventBus.java:591)
        at reactor.bus.EventBus.accept(EventBus.java:63)
        at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
        at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:844)

[#2]

In the logs I get this stacktrace:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden

at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)

at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)

at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)

at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1067)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUsingHead(AbstractGoogleClientRequest.java:406)

at com.google.api.services.compute.Compute$Regions$List.executeUsingHead(Compute.java)

at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialVerifier.preCheckOfGooglePermission(GcpCredentialVerifier.java:57)

at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialVerifier$$FastClassBySpringCGLIB$$a658f442.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)

at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)

at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163)

at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118)

at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:152)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialVerifier$$EnhancerBySpringCGLIB$$676bf766.preCheckOfGooglePermission(<generated>)

at com.sequenceiq.cloudbreak.cloud.gcp.GcpCredentialConnector.verify(GcpCredentialConnector.java:43)

at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler.accept(CredentialVerificationHandler.java:43)

at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler.accept(CredentialVerificationHandler.java:20)

at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler$$FastClassBySpringCGLIB$$f08a8fcf.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

at com.sequenceiq.cloudbreak.cloud.handler.CredentialVerificationHandler$$EnhancerBySpringCGLIB$$d500084f.accept(<generated>)

at reactor.bus.EventBus$3.accept(EventBus.java:317)

at reactor.bus.EventBus$3.accept(EventBus.java:310)

at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)

at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)

at reactor.bus.EventBus.accept(EventBus.java:591)

at reactor.bus.EventBus.accept(EventBus.java:63)

at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)

at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

at java.base/java.lang.Thread.run(Thread.java:844)

[#3 · Accepted solution]

In the end I solved this by creating a new service account and editing the instance to be on the new one.