Member since
02-01-2018
52
Posts
2
Kudos Received
0
Solutions
08-18-2018
08:33 AM
I'm using CloidBreak for deploying hdp clusters. However the requirement is to do it based on hardened image that follows CIS guidelines, so I wanted to use their CentOS 7 image from Azure Marketplace. I did all prerequisites (packer built me an image, I've added new image catalog etc). However, as you can already suspect, the deployment failed because of that: {
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "VMMarketplaceInvalidInput",
"message": "Creating a virtual machine from Marketplace image requires Plan information in the request. VM: '/subscriptions/xxxx/resourceGroups/xxxx/providers/Microsoft.Compute/virtualMachines/xxxxxxxm1'."
}
]
} This is because ARM template does not have plan information, that is required when deploying VMs from marketplace. Unfortunately and super sadly I can't see a workaround for that unless I modify cloudbreak itself...
... View more
Labels:
- Labels:
-
Hortonworks Cloudbreak
08-14-2018
05:57 AM
Hi, By open open ports I mean communication between nodes for hadoop operations.
... View more
08-13-2018
03:41 PM
Thans @pdarvasi. I will let you know how it did go. Also do you mean, that cb scripts will open required ports etc? Cheers!
... View more
08-13-2018
10:22 AM
Hi, I want to deploy HDP 2.6 (using cloudbreak 2.7.1) using prehardened images by CIS in Azure. What should I take into consideration? By default only port 22 is opened by iptables. (I also struggle to deploy cbd 2.7 on that image as docker containers seem to not talk to each other, but will raise another issue if I won't have further luck with it).
... View more
Labels:
07-27-2018
11:05 AM
I have configured HDP with kerberos for the realm DATA.EXAMPLE.COM This generated this config at /etc/krb5.conf [domain_realm]
.data.example.com = DATA.EXAMPLE.COM
[realms]
DATA.EXAMPLE.COM = {
admin_server = data.example.com kdc = data.example.com
}
However I need to allow corp users as well to use kinit, so I've modified this to: [domain_realm]
.data.example.com = DATA.EXAMPLE.COM .corp.example.com = CORP.EXAMPLE.COM
[realms]
DATA.EXAMPLE.COM = {
admin_server = data.example.com kdc = data.example.com
}
CORP.EXAMPLE.COM = {
admin_server = corp.example.com
kdc = corp.example.com
}
So this works in terms of authenticating using kinit. But... It does not, when I try to interact with the cluster. Whenever I type hdfs dfs -ls / I get this message: 18/07/27 11:01:24 INFO util.KerberosName: No auth_to_local rules applied to user@CORP.EXAMPLE.COM
18/07/27 11:01:29 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1532689285894
18/07/27 11:01:33 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1532689285894
18/07/27 11:01:34 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1532689285894
18/07/27 11:01:39 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1532689285894
18/07/27 11:01:43 WARN ipc.Client: Couldn't setup connection for user@CORP.EXAMPLE.COM to ds-beta-prod-02-m3.data.example.com/10.251.2.76:8020
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1620)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:823)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:290)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:202)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:184)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2177)
at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1442)
at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1454)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:265)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1697)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:356)
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 41 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
... 44 more
ls: Failed on local exception: java.io.IOException: Couldn't setup connection for user@CORP.EXAMPLE.COM to ds-beta-prod-02-m3.data.example.com/10.251.2.76:8020; Host Details : local host is: "ds-beta-prod-02-m2.data.exmple.com/10.251.2.74"; destination host is: "ds-beta-prod-02-m3.data.example.com":8020;
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
06-28-2018
07:55 AM
Hi @mmolnar Yea, won't forget. I've automated this process, so in my case it's just changing a value of my Ansible variable. Unfortunately I observed another issue In my kerberized cluster, when trying to start Hive Interactive I get this error: Requested user hive is not whitelisted and has id 982,which is below the minimum allowed 1000 When I checked /etc/passwd I saw that half of HDP services are below 1000 and some are above. This is again after an upgrade from 2.6. Should I create a new thread for that issue? Thanks
... View more
06-27-2018
10:27 PM
Hi @mmolnar I can confirm that the fix in that release candidate works for me. Thanks a lot.
... View more
06-27-2018
10:05 PM
Hi @mmolnar I can confirm that after adding this env variable I don't have this issue anymore and I'm on cbd 2.7.1-rc.13 Thank you!
... View more
06-27-2018
03:26 PM
Hi @mmolnar, Thanks for getting back to me as I'm under huge pressure due to deadlines. This time I've named my test cluster "asdtesttest" (I know not descriptive name) hostname -d Returns nothing hostname -f Returns: asdtesttest-m0 Files you've asked are attached. logs.zip
... View more
06-27-2018
03:05 PM
I confirm, that this is only an issue, when my VNET is using custom DNS (like those provided from AADDS). CloudBreak 2.6 was using unbound service, and hosts could communicate with each other using "example.com". Seems like it's not the case anymore, or there's a missing configuration. This is a massive blocker for us.
... View more
- « Previous
-
- 1
- 2
- Next »