About matthew_cable

matthew_cable · ‎08-20-2018

Right, that makes sense. What I don't understand is why a checkpoint wouldn't immediately be taken on startup, since it is well past the HDFS Maximum Checkpoint Delay.

matthew_cable · ‎08-17-2018

Hello all, I'm running CloudBreak through some testing, and came across a situation I'm not quite sure how to solve. We're looking to use Cloudbreak to enable us to stop clusters when no activity is expected (i.e. evenings, weekends, etc). I've noticed that after stopping a cluster for an extended period of time, that the "NameNode Last Checkpoint" alert is being thrown. I'm not sure of the expected behavior (Checkpoint on cluster startup?). Any one else is a similar situation? Thanks!

matthew_cable · ‎08-15-2018

Great, thanks!

matthew_cable · ‎08-13-2018

Is it possible to add multiple security groups to the EC2 instances created via Cloudbreak in AWS? I know I can add them through the AWS console after creation, but trying to do it at cluster creation time.

matthew_cable · ‎08-13-2018

Is it possible to add multiple security groups to the EC2 instances created via Cloudbreak in AWS? I know I can add them through the AWS console after creation, but trying to do it at cluster creation time.

matthew_cable · ‎08-10-2018

Nice! Is there a ballpark estimate on when 2.8 will be available? Not looking for an exact date or anything, just a sense for when it might be out. Thanks again

matthew_cable · ‎08-10-2018

Is it possible to change the AMI that will be used when scaling a cluster through Cloudbreak? Here's the longer scenario: - Using a custom AMI that has various hardening changes applied, let's call it AMI-11111 - I build a cluster using this AMI, everything is good. - Over time, we apply various OS patches, primarily to address security vulnerabilities. - If we need to scale the cluster in the future, it will use the original AMI, which will have the vulnerabilities. Instead, would prefer to use an updated AMI (i.e AMI-22222) that already has the security patches applied. Is this possible? Or is there a better way to handle this scenario? Thanks!

matthew_cable · ‎07-21-2017

Are there plans to support HDP 2.6+ on CentOS/RHEL 7.3? I saw in the support matrix that 7.2 is the latest version currently supported. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_support-matrices/content/ch01.html

matthew_cable · ‎04-10-2017

Thanks Kshitij - will go down this route, as it seems to be a good fit.

matthew_cable · ‎04-06-2017

Currently on HDP 2.5, and looking at options for creating groups of users, based on experience, and limiting access to resources based these groups. For example beginner group users may get 10gb/4vcpu, intermediate gets 50gb/8vcpu, expert users get 200gb/16vcpu. The stack we are interested in is: Zeppelin -> Livy -> Spark -> YARN cluster Initially looking at doing this via YARN queues (i.e. user A can only submit jobs to the beginner queue), but from what I can tell, the Zeppelin Spark interpreter config only allows configuration of one queue, so regardless of user, all jobs would be sent to the same queue. Another option would be to have multiple Zeppelin's, and configure different queues, but from what I can tell, if we have multiple Zeppelins all managed via Ambari, they will all get the same config. Am I off base? Is there a better way to limit resources (primarily memory) based on the user? Thanks!

Online	Offline
Last Visited	‎08-20-2018 05:12 PM

Date Registered	‎04-04-2017 06:38 PM
Last Visited	‎08-20-2018 05:12 PM
Total Messages Posted	10
Total Kudos Received	1

Re: CloudBreak - stopped cluster NameNode Last Che...

CloudBreak - stopped cluster NameNode Last Checkpo...

Re: Cloudbreak - AWS multiple security groups

Cloudbreak - AWS multiple security groups

Cloudbreak - AWS multiple security groups

Re: Change AMI used for autoscaling

Change AMI used for autoscaling

HDP on RHEL 7.3

Re: Limiting user resources for Zep->Livy->Spark->...

Limiting user resources for Zep->Livy->Spark->Yarn