About Indra

Indra · ‎01-13-2018

@pdarvasi Managed disks have Storage Service Encryption enabled by default. Have not had a chance to look at Azure Disk Encryption in this context but need to for a customer as well for HDP. Would help to get some additional details - perhaps via Paige, so we can get product/engineering team involved if needed.

Indra · ‎01-13-2018

@pdarvasi Sorry for the late response. Had a chance to try this out. Version: local version:2.2.0-96149cf latest release:1.16.5 (1) UI looks great (thumbs up!) (2) Ran into a bug - the "create cluster" button would not get enabled whatsoever :)) - with and without "enable Kerberos" checked - template feature is missing - and the SSH key was randomly showing in red or green if I tabbed through it randomly - simple, tried and tested ambari blueprint - existing vnet, subnet, nsg I duodecuple-checked my entries to ensure accuracy and completeness but to no avail. 🙂 Appreciate the effort that has gone into this version. Look forward to fixes and to trying out test KDC and existing KDC.

Indra · ‎01-05-2018

@fschneider @pdarvasi In the latest Azure release of cloudbreak - Jan 2018 (sorry don't have version handy), I noticed that my custom DNS server was not being used. Ran into this when I was attempting to enable dynamic DNS updates, domain join and AD authentication via sssd. I had to do the following to get it to work... sed -i s/127.0.0.1/<my DNS server IP>/g /etc/dhcp/dhclient-enter-hooks sed -i s/"search ${search}"/"search ${search} <my domain> "/g /etc/dhcp/dhclient-enter-hook

Indra · ‎12-14-2017

@pdarvasi Attempted: HDP 2.6, existing Vnet/subnet (with custom DNS server), availability sets, Kerberos with new MIT Kerberos KDC Simple Ambari blueprint- tried and tested Cloudbreak 1.16.5 Issue: After 12+ hours - provisioning did not complete. Some errors: Ambari component version: The cluster's current version could not be determined Services say installation pending Zookeeper shows restart required but when restart is attempted - service invalidate state error with message HOST_SVCCOMP_OP_IN_PROGRESS at INIT Note: I was able to kerberize post cluster provisioning without issues in a separate instance. Any insight is appreciated. Thanks.

Indra · ‎12-14-2017

@pdarvasi: (1) Availability set (1 for multiple host groups - masters): Works fine (2) Linux kernel version: is the one in your screenshot - version with patch (3) Kerberos enabled post-provisioning against MIT Kerberos KDC: No issues (4) Kerberos enabled at provision time with new MIT Kerberos KDC: Did not complete in 12 hours..lots of alerts, install pending, and services in invalid state and unable to start, Ambari version could not be determined...will open a separate post

Indra · ‎12-12-2017

@pdarvasi - testing it today.

Indra · ‎12-07-2017

@pdarvasi Let me take a look and get back to you on this (shocked!)

Indra · ‎12-06-2017

A great feature would be to be able to specify whether to encrypt data disks at provision time with Azure Disk Encryption. Please share if this is in the roadmap. Thanks.

Indra · ‎12-06-2017

@pdarvasi Thanks!

Indra · ‎12-06-2017

@pdarvasi Will try out and get back to you. Thanks so much!

Indra · ‎12-05-2017

@pdarvasi We appreciate the speed at which the Cloudbreak team is rolling out changes for Azure. Will report back any issues I run into. Can you please confirm if it includes a fix for this? - Linux kernel version... Thanks

Indra · ‎12-03-2017

Solution: VM FQDN needs to be shorter than what you get with Azure defaults. This is not a Cloudbreak issue.

Indra · ‎12-03-2017

Hello, Issue: Around mid-November 2017, ran into an issue with datanodes not coming up, and namenode therefore not coming out of safe node when we kerberized (MIT kerberos KDC) a cluster after we provisioned via Cloudbreak. Reference: https://issues.apache.org/jira/browse/HDFS-12029 https://access.redhat.com/errata/RHBA-2017:1674 https://community.hortonworks.com/articles/109940/after-os-patching-all-the-datanodes-nodes-are-up-b.html Temporary Fix: Hortonworks support was consulted, and they applied a temporary fix: Updated hadoop-env.sh configs in HDFS configs in ambari as follows: Original entry: export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true ${HADOOP_OPTS}" With fix: export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Xss1280k ${HADOOP_OPTS}” Restarted services and we were good to go. Strategic solution: Upgrade the underlying OS image to a version that includes a patch. 3.10.0-514.26.2.el7.x86_64 version, per the support engineer Reporting this since an upgrade is being worked, should have done it earlier 😞 @pdarvasi

Indra · ‎12-03-2017

@pdarvasi & @jeff Would be great if we had at least the ARM template out in the marketplace - we currently dont have any offering from Hortonworks to spin up a cluster. Cloudera has Director and an ARM template. Would be great if Hortonworks had a similar model...especially for situations like this.

Indra · ‎12-03-2017

Just found Cloudbreak missing in the marketplace. Please let me know if this is a Microsoft issue so I can escalate internally. I am in the middle of a PoC with strict timelines and am pretty badly impacted. Your support is much appreciated.

Indra · ‎12-02-2017

Attempting to create a HDP cluster with Kerberos at provision time against AD failed. Issue is tied to the same as one reported - very long VM FQDN - exceeding upper limits defined in AD, AAD DS

Indra · ‎11-29-2017

Provisioned a cluster on Azure using Cloudbreak and then... Attempted: Kerberize the cluster using Ambari Kerberos automatic wizard, against an existing Active Directory prepped ahead of time Issue: The kerberos set up fails when it tries to create a SPN for zookeeper. The error seems to point to length of CN exceeding max length limit. STDERR from Ambari Kerberos wizard UI: 2017-11-28 16:41:58,340 - Failed to create principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM - Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM STDOUT from Ambari Kerberos wizard UI: 2017-11-28 16:41:57,944 - Processing identities... 2017-11-28 16:41:58,019 - Processing principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,021 - Principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,048 - Processing principal, ambari-qa-denali@DENALI.COM 2017-11-28 16:41:58,049 - Principal, ambari-qa-denali@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,076 - Processing principal, hdfs-denali@DENALI.COM 2017-11-28 16:41:58,077 - Principal, hdfs-denali@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,104 - Processing principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,106 - Principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,133 - Processing principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,134 - Principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,163 - Processing principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,165 - Principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,193 - Processing principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,195 - Principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,221 - Processing principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,222 - Principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,248 - Processing principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,249 - Principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,276 - Processing principal, jn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,278 - Principal, jn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,306 - Processing principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 2017-11-28 16:41:58,307 - Principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password 2017-11-28 16:41:58,334 - Processing principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM Just to show that several SPs got created, it consistently fails at zookeeper. Troubleshooting attempted: Reduced zookeeper to zk, got past the error, only to fail for amshbase, reduced this to amshb, got past the setup. Failed during smoke testing; We cannot be changing service principal names, this was merely to test the hypothesis that it was length related. Ambari log: 29 Nov 2017 00:47:08,143 INFO [Server Action Executor Worker 464] StackAdvisorRunner:71 - advisor script stderr: 29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SQOOP=[SQOOP] to auth to local mapping 29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SQOOP to auth to local mapping 29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service HDFS=[HDFS_CLIENT, ZKFC, DATANODE, JOURNALNODE, NAMENODE] to auth to local mapping 29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HDFS_CLIENT to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component DATANODE to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component JOURNALNODE to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component NAMENODE to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service TEZ=[TEZ_CLIENT] to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component TEZ_CLIENT to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service MAPREDUCE2=[MAPREDUCE2_CLIENT, HISTORYSERVER] to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HISTORYSERVER to auth to local mapping 29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service ZOOKEEPER=[ZOOKEEPER_SERVER, ZOOKEEPER_CLIENT] to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component ZOOKEEPER_SERVER to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service YARN=[NODEMANAGER, YARN_CLIENT, APP_TIMELINE_SERVER, RESOURCEMANAGER] to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component NODEMANAGER to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component APP_TIMELINE_SERVER to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component RESOURCEMANAGER to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service KERBEROS=[KERBEROS_CLIENT] to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component KERBEROS_CLIENT to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service PIG=[PIG] to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component PIG to auth to local mapping 29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service HIVE=[HIVE_SERVER, MYSQL_SERVER, HIVE_METASTORE, HIVE_CLIENT, WEBHCAT_SERVER] to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HIVE_SERVER to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HIVE_METASTORE to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component WEBHCAT_SERVER to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SLIDER=[SLIDER] to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SLIDER to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service AMBARI_METRICS=[METRICS_MONITOR, METRICS_COLLECTOR] to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component METRICS_COLLECTOR to auth to local mapping 29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SMARTSENSE=[HST_AGENT, HST_SERVER] to auth to local mapping 29 Nov 2017 00:47:08,156 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SPARK2=[SPARK2_CLIENT, SPARK2_JOBHISTORYSERVER] to auth to local mapping 29 Nov 2017 00:47:08,156 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SPARK2_CLIENT to auth to local mapping 29 Nov 2017 00:47:08,156 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SPARK2_JOBHISTORYSERVER to auth to local mapping 29 Nov 2017 00:47:08,557 INFO [Server Action Executor Worker 465] KerberosServerAction:353 - Processing identities... 29 Nov 2017 00:47:08,629 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,657 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hdfs-denali@DENALI.COM 29 Nov 2017 00:47:08,684 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,713 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,740 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,768 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,796 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,824 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,852 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,879 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM 29 Nov 2017 00:47:08,885 ERROR [Server Action Executor Worker 465] CreatePrincipalsServerAction:297 - Failed to create principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM - Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createPrincipal(ADKerberosOperationHandler.java:331) at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.createPrincipal(CreatePrincipalsServerAction.java:256) at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.processIdentity(CreatePrincipalsServerAction.java:159) at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:532) at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:414) at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.execute(CreatePrincipalsServerAction.java:91) at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:555) at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:492) at java.lang.Thread.run(Thread.java:748) Caused by: javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 19 - 00002082: AtrErr: DSID-031519A3, #1: 0: 00002082: DSID-031519A3, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 3 (cn):len 138 ]; remaining name '"cn=zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net,OU=hdpou,DC=denali,DC=com"' at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3149) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3082) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2888) at com.sun.jndi.ldap.LdapCtx.c_createSubcontext(LdapCtx.java:812) at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_createSubcontext(ComponentDirContext.java:341) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.createSubcontext(PartialCompositeDirContext.java:268) at javax.naming.directory.InitialDirContext.createSubcontext(InitialDirContext.java:202) at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createPrincipal(ADKerberosOperationHandler.java:329) ... 8 more 29 Nov 2017 00:47:08,886 INFO [Server Action Executor Worker 465] KerberosServerAction:457 - Processing identities completed. 29 Nov 2017 00:47:09,559 ERROR [ambari-action-scheduler] ActionScheduler:440 - Operation completely failed, aborting request id: 39 29 Nov 2017 00:47:09,560 INFO [ambari-action-scheduler] ActionScheduler:952 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION 29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 466 stageId 2 29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 467 stageId 3 29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-e0.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 468 stageId 4 29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m1.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 469 stageId 4 29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m12.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 470 stageId 4 29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 471 stageId 4 29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m34.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 472 stageId 4 29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s15.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 473 stageId 4 29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 474 stageId 4 29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s17.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 475 stageId 4 29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 476 stageId 5 29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 477 stageId 6 29 Nov 2017 00:48:41,263 INFO [pool-18-thread-1] MetricsServiceImpl:64 - Checking for metrics sink initialization Deduction: The length is beyond the limit acceptable by Active Directory OK: yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM FAILS: zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM amshbase/den-m1.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM Question: (1) Anyone run into this issue that has a solution to share? I know I can pop a MIT Kerberos KDC in front of AD...looking for options. (2) Does the Cloudbreak team have any guidance? Thanks in advance. I am now attempting to provision via Cloudbreak - kerberize at provision-time against existing Active Directory. Fingers crossed.

Indra · ‎11-09-2017

@pdarvasi Availability set support for masters, premium managed disks support, and GA of Kerberos will drive increased adoption of CloudBreak on Azure. We look forward to support for the same.

Indra · ‎11-09-2017

@pdarvasi - thanks so much!

Indra · ‎10-20-2017

@jeff: The context here is Cloudbreak - provisioning HDP using Cloudbreak. Some of our enterprise customers have a requirement to use RHEL. Can Cloudbreak be configured to use a RHEL image available in Azure marketplace for the cluster nodes instead of the default?

Indra · ‎10-18-2017

Community, Looking for an Ambari blueprint that provisions a new mysql database and leverages the same for both hive and oozie. Where can I find a sample or something close - there is some dependency here - for hive, create new, for oozie, use existing... Is this even possible, or is the only way I can do this in one shot is to use an external database? Please advise.

Indra · ‎10-17-2017

Thanks. Tried it, and it did not work, @Geoffrey Shelton Okot

Indra · ‎10-17-2017

Hello, Can someone please share a blueprint for Hive HA? The blueprint I am trying, pasted below, gives me the error - Failed to create cluster: Incorrect number of 'HIVE_SERVER' components are in '[master_2, master_3]' hostgroups: count: 2, min: 1 max: 1 I did see this, and this, new to Ambari blueprints and would like to start with min. conf. Any help is much appreciated. { "Blueprints": { "blueprint_name": "ha-trials", "stack_name": "HDP", "stack_version": "2.6" }, "host_groups": [ { "name": "edge", "cardinality": "1", "components": [ { "name": "HDFS_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }, { "name": "METRICS_MONITOR" }, { "name": "TEZ_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "PIG" }, { "name": "SQOOP" }, { "name": "SLIDER" }, { "name": "HIVE_CLIENT" } ] }, { "name": "master_1", "cardinality": "1", "components": [ { "name": "HISTORYSERVER" }, { "name": "JOURNALNODE" }, { "name": "METRICS_MONITOR" }, { "name": "NAMENODE" }, { "name": "ZKFC" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "SLIDER" } ] }, { "name": "master_2", "cardinality": "1", "components": [ { "name": "APP_TIMELINE_SERVER" }, { "name": "JOURNALNODE" }, { "name": "METRICS_MONITOR" }, { "name": "RESOURCEMANAGER" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "MYSQL_SERVER" }, { "name": "HIVE_SERVER" }, { "name": "HIVE_METASTORE" }, { "name": "WEBHCAT_SERVER" }, { "name": "TEZ_CLIENT" }, { "name": "HIVE_CLIENT" }, { "name": "ZOOKEEPER_CLIENT" } ] }, { "name": "master_3", "cardinality": "1", "components": [ { "name": "JOURNALNODE" }, { "name": "METRICS_MONITOR" }, { "name": "NAMENODE" }, { "name": "ZKFC" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "RESOURCEMANAGER" }, { "name": "HIVE_SERVER" }, { "name": "HIVE_METASTORE" }, { "name": "WEBHCAT_SERVER" }, { "name": "HCAT" }, { "name": "HIVE_CLIENT" } ] }, { "name": "slave_1", "components": [ { "name": "DATANODE" }, { "name": "METRICS_MONITOR" }, { "name": "NODEMANAGER" }, { "name": "TEZ_CLIENT" }, { "name": "HIVE_CLIENT" } ], "cardinality": "3+" }, { "name": "management", "configurations": [], "cardinality": "3+", "components": [ { "name": "METRICS_MONITOR" }, { "name": "METRICS_COLLECTOR" } ] } ], "configurations": [ { "core-site": { "properties": { "fs.defaultFS": "hdfs://mycluster", "ha.zookeeper.quorum": "%HOSTGROUP::master_1%:2181,%HOSTGROUPHOSTGROUP::master_2%:2181,%HOSTGROUP::master_3%:2181", "fs.trash.interval": "4320" } } }, { "hdfs-site": { "properties": { "dfs.client.failover.proxy.provider.mycluster": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "dfs.ha.automatic-failover.enabled": "true", "dfs.ha.fencing.methods": "shell(/bin/true)", "dfs.ha.namenodes.mycluster": "nn1,nn2", "dfs.namenode.http-address": "%HOSTGROUP::master_1%:50070", "dfs.namenode.http-address.mycluster.nn1": "%HOSTGROUP::master_1%:50070", "dfs.namenode.http-address.mycluster.nn2": "%HOSTGROUP::master_3%:50070", "dfs.namenode.https-address": "%HOSTGROUP::master_1%:50470", "dfs.namenode.https-address.mycluster.nn1": "%HOSTGROUP::master_1%:50470", "dfs.namenode.https-address.mycluster.nn2": "%HOSTGROUP::master_3%:50470", "dfs.namenode.rpc-address.mycluster.nn1": "%HOSTGROUP::master_1%:8020", "dfs.namenode.rpc-address.mycluster.nn2": "%HOSTGROUP::master_3%:8020", "dfs.namenode.shared.edits.dir": "qjournal://%HOSTGROUP::master_1%:8485;%HOSTGROUP::master_2%:8485;%HOSTGROUP::master_3%:8485/mycluster", "dfs.nameservices": "mycluster", "dfs.namenode.safemode.threshold-pct": "0.99" } } }, { "yarn-site": { "properties": { "hadoop.registry.rm.enabled": "false", "hadoop.registry.zk.quorum": "%HOSTGROUP::master_3%:2181,%HOSTGROUP::master_2%:2181,%HOSTGROUP::master_1%:2181", "yarn.log.server.url": "http://%HOSTGROUP::master_2%:19888/jobhistory/logs", "yarn.resourcemanager.address": "%HOSTGROUP::master_2%:8050", "yarn.resourcemanager.admin.address": "%HOSTGROUP::master_2%:8141", "yarn.resourcemanager.cluster-id": "yarn-cluster", "yarn.resourcemanager.ha.automatic-failover.zk-base-path": "/yarn-leader-election", "yarn.resourcemanager.ha.enabled": "true", "yarn.resourcemanager.ha.rm-ids": "rm1,rm2", "yarn.resourcemanager.hostname": "%HOSTGROUP::master_2%", "yarn.resourcemanager.recovery.enabled": "true", "yarn.resourcemanager.resource-tracker.address": "%HOSTGROUP::master_2%:8025", "yarn.resourcemanager.scheduler.address": "%HOSTGROUP::master_2%:8030", "yarn.resourcemanager.store.class": "org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore", "yarn.resourcemanager.webapp.address": "%HOSTGROUP::master_2%:8088", "yarn.resourcemanager.webapp.https.address": "%HOSTGROUP::master_2%:8090", "yarn.timeline-service.address": "%HOSTGROUP::master_2%:10200", "yarn.timeline-service.webapp.address": "%HOSTGROUP::master_2%:8188", "yarn.timeline-service.webapp.https.address": "%HOSTGROUP::master_2%:8190", "yarn.resourcemanager.zk-address": "%HOSTGROUP::master_2%:2181,%HOSTGROUP::master_1%:2181,%HOSTGROUP::master_3%:2181", "yarn.resourcemanager.hostname.rm1": "%HOSTGROUP::master_2%", "yarn.resourcemanager.hostname.rm2": "%HOSTGROUP::master_3%", "yarn.acl.enable": "true" } } }, { "hive-env": { "properties": { "cost_based_optimizer": "On", "hcat_log_dir": "/var/log/webhcat", "hcat_pid_dir": "/var/run/webhcat", "hcat_user": "hcat", "hive_ambari_database": "MySQL", "hive_database": "New MySQL Database", "hive_database_name": "hive", "hive_database_type": "mysql", "hive_exec_orc_storage_strategy": "SPEED", "hive_log_dir": "/var/log/hive", "hive_metastore_port": "9083", "hive_pid_dir": "/var/run/hive", "hive_security_authorization": "None", "hive_timeline_logging_enabled": "true", "hive_txn_acid": "Off", "hive_user": "hive", "webhcat_user": "hcat" } } }, { "hive-site": { "hive.exec.compress.output": "true", "hive.merge.mapfiles": "true", "hive.server2.tez.initialize.default.sessions": "true", "hive.server2.transport.mode": "http", "ambari.hive.db.schema.name": "hive", "hive.zookeeper.client.port": "2181", "hive.zookeeper.namespace": "hive_zookeeper_namespace", "hive.zookeeper.quorum": "%HOSTGROUP::master_2%:2181,%HOSTGROUP::master_1%:2181,%HOSTGROUP::master_3%:2181", "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver", "javax.jdo.option.ConnectionURL": "jdbc:mysql://%HOSTGROUP::master_2%/hive?createDatabaseIfNotExist=true", "javax.jdo.option.ConnectionUserName": "hive" } } ] }

Indra · ‎10-15-2017

Hello- Attempted: Create a multi-master cluster, on Azure with masters and workers placed in master node availability set and worker node availability set respectively, Blueprint: Each master is assigned to a host group similar to this. Availability sets: In the "Configure cluster" tab of provisioning steps, created an availability set for masters called "as-masternodes". Issue: The "as-masternodes" availability set, is available for selection for only one master via the GUI after which it becomes unavailable for selection for other master nodes. Only one master ends up in the availability set specified for master nodes. Question: Please let me know how I can achieve adding all master nodes to the same availability set. Thanks.

Indra · ‎10-13-2017

Thanks so much, @fschneider

Indra · ‎10-12-2017

A customer of mine would like to use their own DNS server with Cloudbreak on Azure. Can you please share how this can be configured if supported? If not supported, please share if it is a roadmap item. Thanks.

Indra · ‎10-12-2017

A customer of mine will most likely bring their own RHEL 7.x custom image and put it in the marketplace, or use existing marketplace image with "Bring your own license" (BYOL) or "Pay as you go" (PAYG). Please share how any of these can be supported. Any best practices/tuning/conf that needs to be applied with the images? From a roadmap perspective, would be great if there are options we can choose from and if HWX can have images in the marketplace that are pre-tuned/configured readily available for use. 🙂

Indra · ‎10-12-2017

@pdarvasi: Thanks for the quick response. Yes- would like to use premium managed disks for master nodes. Also, in lower environments for workers, use standard managed disks, and in production, premium. This is a need for an customer. Can this be rolled out quickly? If yes, do share timelines. Thanks.

Indra · ‎10-06-2017

Hello, I am trying to create a custom template and pick premium managed disks for masters and standard managed disks for workers. The documentation details the option of selecting "Volume Type" for making such a distinction. https://hortonworks.github.io/cloudbreak-azure-docs/azure-config/index.html under "Custom template". However, in my Cloudbreak GUI, I dont see the "Volume type" dropdown. What am I missing here? Thanks in advance. Anagha

Indra · ‎09-25-2017

Looking for documentation on installing HDF on Azure. I see that there is no marketplace template and it will be a pure IaaS setup. This is for a PoC. Plan is to set up a 3 node NiFi-only cluster (no Kafka/Storm etc), with one management node for security/operations, leveraging Ambari to install NiFi. Looking for guidance specifically on these areas- OS image to use on Azure Any OS level tuning/configuration that needs to be done Anything networking related besides Azure vnet Recommended foundational software with version – e.g. Java version and anything else Minimum config - VM SKU, disk SKU, for operations and security node, and disk partitioning Minimum config - VM SKU, disk SKU, disk partitioning for NiFi nodes Any best practices Detailed documentation Thanks in advance.

Online	Offline
Last Visited	‎08-25-2017 12:28 AM

Date Registered	‎12-12-2013 10:34 AM
Last Visited	‎08-25-2017 12:28 AM
Total Messages Posted	33
Total Kudos Received	2

Re: Closed - Cloudbreak on Azure: Kerberize a cl...

Re: Cloudbreak on Azure: Ability to encrypt data d...

Re: Cloudbreak on Azure: Provisioning cluster with...

Re: Closed: Cloudbreak on Azure: Support for "br...

Cloudbreak on Azure: Provisioning cluster with new...

Re: Closed: Cloudbreak on Azure: Linux kernel ve...

Re: Closed: Cloudbreak on Azure: Linux kernel ve...

Re: Cloudbreak on Azure: Ability to encrypt data d...

Cloudbreak on Azure: Ability to encrypt data disks

Re: Closed: Cloudbreak on Azure: Not available i...

Re: Closed: Cloudbreak on Azure: Linux kernel ve...

Re: Closed: Cloudbreak on Azure: Not available i...

Re: Closed - Cloudbreak on Azure: Kerberize a cl...

Closed: Cloudbreak on Azure: Linux kernel versio...

Re: Closed: Cloudbreak on Azure: Not available i...

Closed: Cloudbreak on Azure: Not available in Az...

Re: Closed - Cloudbreak on Azure: Kerberize a cl...

Closed - Cloudbreak on Azure: Kerberize a cluste...

Re: Closed: Cloudbreak on Azure - Support for as...

Re: Closed - Cloudbreak on Azure: Support for "b...

Re: Closed - Cloudbreak on Azure: Support for "b...

Ambari blueprint for Oozie with mysql | Cloudbreak

Re: Ambari blueprint for Hive HA | Cloudbreak

Ambari blueprint for Hive HA | Cloudbreak

Closed: Cloudbreak on Azure - Support for assign...

Re: Closed: Cloudbreak on Azure: Support for "br...

Closed: Cloudbreak on Azure: Support for "bring ...

Closed - Cloudbreak on Azure: Support for "bring...

Re: Support for premium managed disks with Cloudbr...

Support for premium managed disks with Cloudbreak ...

HDF on Azure - guidance