About learninghuman

learninghuman · ‎10-27-2017

I have come across DataPlane service which was announced recently. Few questions in that regard, 1. As far as i understand, it contains Atlas, Ranger and Knox. Is this correct? 2. What is the motivation behind this new product? If i have HDP, i get Ranger, Knox, Atlas anyways. 3. What is the applicability of this product? Can i use this with other Hadoop distributions? 4. Has the sandbox for Dataplane been released? Where can i find the download?

learninghuman · ‎03-14-2017

My question on HDFS using SAN as the backend storage has 3 main parts 1. Is it feasible to use SAN as the back end storage for HDFS? 2. What are the pros and cons of using SAN or NAS for HDFS? 3. Has it been tested for performance and may be other aspects?

learninghuman · ‎01-23-2017

@Devin Pinkston For some reason the link does not work. But essentially, i would to know HWs recommendation of using BlueData for HDP and on that platform i also get the other services running (multi-tenancy).

learninghuman · ‎01-23-2017

I came across this and found the BlueData's BDAAS interesting. Whats Hortownworks recommendation for the same in using HDP on BlueData?

learninghuman · ‎01-06-2017

@Tom McCuch Thanks for the clarification. One another related question is that in general what are the advantages that Mesos would bring over Yarn? Especially given the fact that Hortonworks is making efforts to support HDP on Mesos. I mean why care. If HDP on the cloud, its still YARN thats going to be the cluster manager.

learninghuman · ‎01-05-2017

Is it possible to deploy HDP docker container in Mesos using Marathon? If so, where can i get the docker images from and the Marathon recipes? If its not possible with the combination above, what are the options to deploy HDP on Mesos? How is it going to be better than running on Yarn?

learninghuman · ‎01-02-2017

@Dominika Bialek @rdoktorics The two answers are contradictory "It is not possible to add additional services." and "You can add new service..."

learninghuman · ‎12-30-2016

My understanding along with questions as below, AWS-HDCloud Manual scaling using Ambari or AWS UI possible. Auto Scaling 1. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 1.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. -------------------------------------------------------------------------------------------------------------------------------------------------------------- AWS-HDP on IaaS Manual scaling using Ambari is possible. Auto Scaling-Without CloudBreak 2. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 2.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. Auto Scaling-WithCloudBreak Auto-scaling may be possible, but question 2.1 applies here as well. -------------------------------------------------------------------------------------------------------------------------------------------------------------- Azure-HdInsights Manual scaling using Ambari or Azure UI possible. Auto Scaling 3. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 3.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. -------------------------------------------------------------------------------------------------------------------------------------------------------------- Azure-HDP in MarketPlace Manual scaling using Ambari or Azure UI possible. Auto Scaling 4. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 4.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. -------------------------------------------------------------------------------------------------------------------------------------------------------------- Azure-HDP on IaaS Same questions as AWS-HDP on IaaS

learninghuman · ‎12-30-2016

@Tom McCuch One last question which i got after reading your answer again. WASB in Azure is supported on both HDP on Azure IaaS and HDP in Azure MarketPlace. Does this mean that WASB is natively optimized in Hadoop 2.x? If so, this would also mean that any distribution with Hadoop 2.x deployed on Azure can use WASB for storage?

learninghuman · ‎12-30-2016

I understand that the HDCloud on AWS comes with 2/3 cluster types. What are the difference services available in each of the cluster types? In the documentation, its stated that for example, "Data Science" cluster type comes with Spark and Zeppelin. Obviously HDFS is part of it; Another example is the EDW-ETL cluster type where it comes with Hive. This means that Tez would also be part of it. Likewise, where do i get the complete list of services along with the version on each of the cluster types? My next question is: Is it possible to add more services using Ambari or other means on top of the Services that already comes as part of the cluster type.

learninghuman · ‎12-30-2016

@milind pandit so to summarize (with a question) AWS HDCloud can be set only on the private cloud(not in the public cloud) HDP cluster on IaaS can be created on both the public and the private cloud Azure HDInsights - Its a manage service. Can it be created on both the private and the public cloud? HDP in the marketplace - can be created on both the public and the private cloud HDP cluster on IaaS - can be created on both the public and the private cloud

learninghuman · ‎12-29-2016

@mqureshi I thought Periscope helps to set threshold metrics for auto-scaling. Its now integrated with CloudBreak so that HDP deployed using CloudBreak on AWS, Azure, OpenStack etc can auto-scale. Thats my understanding atleast.

learninghuman · ‎12-29-2016

@mqureshi Thanks. As per link1 and link2 , option 1 i.e. if HDP is deployed using Cloudbreak on-premise with OpenStack, then it can scale to cloud (same cluster) may be possible. Please clarify One more question, Can i create a one single HDP cluster with few nodes on-premise and few nodes from VPC in AWS or VN in Azure? I am not talking about elasticity or auto scaling here but but just want to if i can create a VPC or VN and then create a cluster with 10 nodes on-premise and 20 nodes from VPC or VN.

learninghuman · ‎12-28-2016

The table attached from the link mentions about 4 factors - Scalability, Elasticity, protection of sensitive data and unlimited storage. Few questions on how this is achived First and foremost, my understanding of Hybrid options are the following. Correct me as appropriate 1. With HDP deployed on OpenStack using CloudBreak OnPremise. The cluster can autoscale to cloud based on set metrics. 2. With HDP on-premise (no OpenStack...so no CloudBreak) plus nodes from VPC in AWS or VN in Azure added to the same cluster. Now in option 1, How are the 4 factors mentioned above achieved? With data on-premise, if the cluster auto scales, then the newly added compute nodes may not achieve data locality (performance is impacted). If the HDFS rebalance is done, then the sensitive data is compromised. How do you ensure that the sensitive data is kept on-premise and never moved to cloud? This means that the processing cannot happen in cloud on the data on-premise. How is this enforced? Option 2, How are the 4 factors mentioned above achieved? Is auto-scaling possible in this option? If not manual intervention is required to scale the cluster? How do you ensure that the sensitive data is kept on-premise and never moved to cloud? This means that the processing cannot happen in cloud on the data on-premise. How is this enforced?

learninghuman · ‎12-28-2016

@Tom McCuch So to summarize, please correct as appropriate 1. HDI 3.5 - WASB and ADLS 2. Pre HDI 3.5 - Only WASB 3. HDP on Asure IaaS - Only WASB and HDFS on VHD 4. HDP from Azure Marketplace - Only WASB and HDFS on VHD 5. HDCloud 2.5 - S3 Only 6. HDP on AWS IaaS - HDFS on Ephemeral or EBS

learninghuman · ‎12-28-2016

@rdoktorics 1. yes thats what i meant. One cannot. 2. i thought from your initial answer that if on-premise is managed by OpenStack then you can scale to Cloud based on some metrics (periscope helps). So can i take it as - even with cloudbreak + openstack + on-premise, it cannot scale to cloud but it can scale only within the on-prem openstack cloud. same is the case if you deploy HDP to cloud. So across on-prem and cloud not possbile but within on-prem and cloud possible.

learninghuman · ‎12-28-2016

@Greg Keys My understanding was that CloudBreak can be used to deploy HDP only on the cloud. Can it be used for deploying HDP on-premise, HDC and HDP on Azure market place?

learninghuman · ‎12-28-2016

@rdoktorics No OpenStack yet. I will take look. So you are saying that 1) Its not possible to deploy HDP on-premise unless its done via OpenStack 2) With OpenStack on-premise it can auto scale to cloud (add nodes from cloud to the existing on-prem cluster) 3) same as 2

learninghuman · ‎12-27-2016

My question is very similar to this but may not be the same, 1. Is it possible to deploy HDP on premise using CloudBreak? 2. With CloudBreak and periscope integrated, is it possible to auto scale from on-premise to public cloud (AWS, Azure)? i.e. Can more nodes from cloud be added to the on-premise cluster based on some metrics? 3. Is the above possible on VPC (AWS) and Virtual Network (Azure)? 4. What are the hybrid options available for HDP?

learninghuman · ‎12-27-2016

@Tom McCuch Thanks. Can you also please talk a little bit about ADLS? Do you still recommend WASB over ADLS? And i am not clear on the parallelism factor on s3 and WASB. Are you saying that S3 does not offer parallelism and suitable for larger number of smaller files? whats you take on parallelism when it comes to WASB? And can i use WASB, ADLS and S3 when i install HDP on Azure's IaaS (using CloudBreak) as the HDFS layer?

learninghuman · ‎12-23-2016

My question is around the scalability factor in the HDP flavors on the cloud (specifically Azure and AWS) I understand that HDInsight cluster can scale on demand with few clicks. What about the following? Can i scale these similar to HDInsight or i need to manually spin a new node and attach to the cluster? 1. HDCloud on AWS 2. HDP on Azure market place

learninghuman · ‎12-23-2016

@pdarvasi, Thanks. Few more questions 1. Is is possible to have private cloud hosted on-premise using Azure (not the virtual network) and use the services of HDInsights, HDP from market place? 2. From the link you gave, it talks about blob storage. Is that same as WASB? 3. What about HDInsight and HDP from MarketPlace on the virtual network? I hope i can still do that. 4. How do i get the HDP running in the hybrid architecture in both Azure and AWS? By hybrid architecture, i mean running jobs on 2 situations i. on-premise, cloud (public as well private (VPC, VN) ) combination and ii. public, private (VPC, VN) combination

learninghuman · ‎12-23-2016

@milind pandit When you say private cloud, i assume that its VN in the Azure world and VPC in the AWS world. Is that right?

learninghuman · ‎12-22-2016

What are my options to deploy HDP on the private cloud of both AWS and Azure? As per my understanding, AWS Private Cloud in the AWS world mean VPC. For IaaS on VPC, Can i deploy using CloudBreak? Do i have the option of HDCloud in AWS VPC? Azure Private cloud in the Azure mean hosted private cloud on-premise? Or there is also a flavor similar to VPC? Do i get HDInsight, HDP in Azure marketplace in private cloud? Can i deploy using CloudBreak? In both the AWS and Azure cases, what are my storage options? How do i move data from on-premise to private cloud?

learninghuman · ‎12-22-2016

What are the storage options possible when deploying HDP on Cloud? My understanding as follows, 1. Azure (HDInsight, HDP via CloudBreak, HDP in the MarketPlace) WASB - What about parallelism here? i.e. if i store a file here and run a map reduce job processing this file. Would i achieve the same effect as i achieve in HDFS storage? ADLS - Although not co-located, performance can be improved by means of parallelism. HDFS itself - I can move the data to the edge node then copy into HDFS What are my options to move my data into WASB, ADLS? This thread suggests NI-FI but my requirement is ephemeral and NIFI investment may not sell. 2. AWS (Below questions apply to HDCloud, HDP via CloudBreak to AWS) S3 - What about parallelism here? i.e. if i store a file here and run a map reduce job processing this file. Would i achieve the same effect as i achieve in HDFS storage? HDFS itself - I can move the data to the edge node then copy into HDFS And out of these storage options, which one is better over the other and for what reason?

learninghuman · ‎12-21-2016

@Greg Keys Thanks again. Hopefully last set of questions 1. With HDP in Azure marketplace, we cannot use the OS of our choice. With CloudBreak, can we specify the OS? 2. Storage in Azure - HDFS, WSAB, ADLS are options for all deployment options of HDP IaaS (CloudBreak, Marketplace), HDInsights? 3. With HDC can i choose the OS? 4. What are the storage options for HDCloud? Is it HDFS and S3 (same as that for HDP on AWS IaaS through CloudBreak)? 5. Can i deploy HDP via CloudBreak in AWS VPC similar to the way that i can deploy in the AWS public cloud? 6. Can i deploy HDC on AWS VPC? 7. What are my options to move data from on-premise to AWS public cloud (S3, HDFS) and AWS VPC (S3, HDFS)? (This may not be strictly HDP question!) 8. What are my options to move data from on-premise to Azure public cloud (WASB, ADLS, HDFS) ? 9. Can i spin HDInsights or HDP (Cloudbreak or marketplace) in Azure private cloud? (I assume that Azure offers two flavors of private cloud - on-premise hosted and the other one similar to VPC)

learninghuman · ‎12-21-2016

@Greg Keys Thanks a lot. Few follow up questions 1. Option 2 that i was talking about is what i see in the Azure portal. Please see the attachments. hdponazure.png and hdponazure-clustercreation.png 2. What about the "Data Lake store" as an option for storage on all options? 3. With respect to performance, my question was more around the issues due to compute and storage not colocated. 4. And what is the purpose of HDCoud? Is it similar to CloudBreak for AWS? Is it for HDP on AWS IaaS? 5. And HDC that you mentioned above - is that a HDP as a service Offering from AWS?

learninghuman · ‎12-21-2016

I see 3 different options to deploy HDP on Hadoop HDInsights (Built on top of HDP) HDP as a Service Deploying HDP on Azure's bare metal In my understanding 1 and 2 are managed services where the control is limited when it comes to the choice of OS etc. HDInsight has multiple cluster types (not sure whats the rationale behind this though) Questions: Whats the rationale behind having multiple cluster types for HDInsight? Why are two services (1 and 2 above) offered? When to use what? (apart from this) Are there any performance benchmarks done on HDInsight or HDP on Azure in a production situation? What are the different storage types possible on the above services? Atleast on HDInsight i see that Blob storage and Data Lake Store are options but both are external to the compute nodes. May hit performance, hence curious about question 3 apart from the fact that the cluster run on the virtual machines. What are the option to provision HDP on Azure bare metal nodes (Option 3)? Does CloudBreak help there?

learninghuman · ‎11-03-2016

I have a need to migrate the data from the traditional EDW to Hive. Post migration of the data, i have to validate if the data is migrated successfully or not i.e. running count queries, min, max etc on the tables that are migrated. Ideally, this is required to be run on the entire data set on both the EDW and Hive for comparison. However, there are few factors that may limit this and the data validation queries can be run only on the subset of the data. This means that sampling is required to choose the representative subset. What tables have to be part of the representative subset is one and the other factor is the sample size itself. Now on determining the sample size, the key aspect is the SLA of the queries. So given the SLA, the number of queries, is it possible to arrive at the sample size? I referred the excel sheet at link and arrived at the following formula; This is definitely an approximation only but does this make sense at all? S = ( ( X * R ) / Q ) * F S - Sample Size in GB X - SLA in minutes * 60 R - Disk Scan Rate of the cluster in GB/sec Q - Number of queries F - Adjusting Factor that account for the intermediate data reads (MR), computation within the hive query etc. Example: 10 minute SLA, 28 GB/Sec scan rate, 4 queries, F = 0.65 S = ((600 * 28) /4 ) * 0.65 = 2730 GB Assumption is that the queries are not CPU intensive.

learninghuman · ‎10-24-2016

Not very clear on the key matching technique that you described. Can you please elaborate on what you generate and compare between TD and Hadoop with examples? And also the matched subset. What if the random sampling from TD and Hadoop are far apart?

Online	Offline
Last Visited	‎07-05-2016 08:52 AM

Date Registered	‎06-29-2016 09:30 AM
Last Visited	‎07-05-2016 08:52 AM
Total Messages Posted	81
Total Kudos Received	43

Re: Mapreduce and Hcatalog Integration fails to us...

Dataplane service

Feasibility and recommendation for running HDFS on...

Re: Hortonworks recommendation for Blue Data

Hortonworks recommendation for Blue Data

Re: HDP on Mesos using Marathon, Docker

HDP on Mesos using Marathon, Docker

Re: HDCloud services and options to add more servi...

Scaling and Auto-scaling of HDP on AWS and Azure C...

Re: HDP on Cloud (Azure, AWS) - Storage options an...

HDCloud services and options to add more services

Re: HDP on private cloud (AWS, Azure)

Re: HDP advantages on Hybrid Cloud

Re: HDP advantages on Hybrid Cloud

HDP advantages on Hybrid Cloud

Re: HDP on Cloud (Azure, AWS) - Storage options an...

Re: HDP on Hybrid Cloud

Re: Scalability on HDP on Cloud (AWS, Azure)

Re: HDP on Hybrid Cloud

HDP on Hybrid Cloud

Re: HDP on Cloud (Azure, AWS) - Storage options an...

Scalability on HDP on Cloud (AWS, Azure)

Re: HDP on private cloud (AWS, Azure)

Re: HDP on private cloud (AWS, Azure)

HDP on private cloud (AWS, Azure)

HDP on Cloud (Azure, AWS) - Storage options and da...

Re: HDInsight Vs HDP Service on Azure Vs HDP on Az...

Re: HDInsight Vs HDP Service on Azure Vs HDP on Az...

HDInsight Vs HDP Service on Azure Vs HDP on Azure ...

Data Migration Validation - Sample Size Calculatio...

Re: Data Validation post migration