Sandbox downloads have failed for two days on. 10....

coatespt · ‎02-12-2019

Something wrong with sandbox downloads- They start, load a few meg and then stall. This is Mac 10.134.3 and occurs on multiple machines from multiple locations yesterday and today.

coatespt · ‎02-10-2017

Tom--yes, that all sounds right to me. Great answers. It's quite remarkable that they can do this when you consider the implications. It's easy to see how it's done on the network, as they can allocated bandwidth with a fixed usage guarantee and a known capacity on the link. What is mystifying is how it is possible for the physical infrastructure of the volumes to support this, as the the amount of work they do is highly variable depending on specific workload. Whatever is behind the volumes is clearly not your grandpa's NAS, as it can flatten out huge random demand peaks from many users.

coatespt · ‎02-09-2017

Tom--thanks for the reply. With regard to the EBS, yes-all the nodes in question are EBS optimized. The critical question is, do ALL those optimized instances share a single 10Gb network to the EBS hosts? As near as I can tell empirically, as the number of writers or readers increases, the total throughput approaches a limit of about 1/3 of a 10Gb network asymptotically. No matter how many readers or writers, that's the limit. The docs are clear that "optimized" puts a node's EBS on a different LAN but do not seem to indicate whether that alternate network is also 10Gb/sec. Or possibly there is some fancier topology with multiple alternate LANs, etc. The same general behavior happens with S3, but with the limit about 105MB/sec. instead of 350MB/sec. It takes about 8 or 10 threads to hit the limit. Individual readers never go over about 12MB/sec. My numbers are consistent with benchmarks that I've found online, including the asymptotic thing. But I don't think that can really be the limit--Netflix claims they query a petabyte of S3 daily. It would have to be a very long day!

coatespt · ‎02-05-2017

I cannot get any more than about 105 MB/sec into or out of S3 on an AWS cluster with 8 big nodes. (S3 is in the same region.) It takes many parallel requests to get even this much--each individual mapper seems to cap out at barely more than single digit MBPS. It seems to be a limit on the VPC, but there must be a way to increase this, as I read about people processing petabytes of s3 data. Can anyone offer any suggestions on how to configure for reasonable S3 performance? Also, can anyone shed light on how the networking works underneath? Is the S3 traffic coming over the same 10Gb LAN that the instances are using? Does EBS traffice go over the LAN too? Total EBS traffic seems to be in practice limited to about 350MB/sec across the cluster. Does EBS use the same LAN as inter-node traffic? If so, it would seem to be impossible that you could ever exceed a total of 1.25 MB/sec of disk I/O for everything. That can't be right, given the size of clusters I hear about. What's going on?

coatespt · ‎11-03-2016

To clarify, it makes no sense for HiveServer2 to do a MapReduce for a select(*) because even if it did, it would still have to fetch the results into one process to merge them. So to save time in this one case, HS2 does the API calls itself instead of handing the work off to one or more worker processes. There seems to be a bug in Hive, however, in that it does not check for the HDFS fs.s3a.proxy.host and fs.s3a.proxy.port settings, but instead deals directly with S3. I am told that the hive setting hive.fetch.task.conversion=none should force a map reduce even if HS2 would have otherwise shortcut the process and read directly.

coatespt · ‎11-02-2016

I'm using insert-into to write data up to S3, but it's writing very large files--0.8GB to 1.8 GB plus one of just a few K. I've tried tez.grouping.max-size and min-size, but neither seems to limit either the min or the max size of the files that are generated. I've also tried controlling the number of mappers and reducers, but to no avail.

coatespt · ‎11-01-2016

Chris--thanks. I didn't see your reply. I had only been trying simple selects, for which it reliably fails, but upon complicating the query so that it results in a Tez job, it behaves correctly every time. Of course, there is no reason to do a distributed operation when all of the results go to one client receiving the data, so it seems that Hive disregards the HDFS proxy settings when you do this. I think this would be the only time that the proxy is any of Hive's concern, because normally, all those interactions are delegated to the mappers and reducers, are they not? BTW, it seems that anything that will force a Tez job causes the proxy to be used, including an insert into HDFS, which is what is important in this case. But there should be a Hive setting for the proxies too, shouldn't there?

coatespt · ‎10-24-2016

@Chris Nauroth Can anyone shed any light on the logic surrounding Hive's decision about where to send a data operation? Our proxy apparently never sees S3 read requests while S3 clearly does, and responds. As this all takes place within AWS, is it possible that some hidden network chicanery somehow redirects the requests?

coatespt · ‎10-20-2016

I'm using hive's fs.s3a.proxy.host andfs.s3a.proxy.port to send data through http proxy. Outbound data goes through proxy but reads do not--i.e. they are read directly from S3. Is this a known bug? Is there some other setting I'm missing? Using Hue as the client. Is it possible there is some kind of cache in the path?

coatespt · ‎10-14-2016

Installing a Vormetric gateway server to encrypt S3 traffic. Connections fail because we lack and SSL certificate. I obtained a self-signed certificate from Vormetric but we can't figure out how to install it. This is a small Ambari cluster running 2.3.6 Hortonworks. Please assume a low level of understanding of the intricacies of Java security, etc.

coatespt · ‎10-05-2016

Say you have a two tables connected with a meaningless key, i.e., every row from one of the tables corresponds 1:1 to a row in the other table. Maybe the pairs of rows started life in a single table, but it was convenient to store them in two for some reason. To use the data in its complete form you create a view, V, that unites the data, i.e., the join of T1 and T2 where T1.key=T2.key. When you use this view in a query, the result is as if you actually did this join to create a temp table called V, then used V in the query. What you see seems to be different. Hive is free to rearrange the operations more efficiently so long as the result is the same. What does it actually do and how is the strategy determined? This has some implications for us because we'd like to split our data up along lines of requirements for security. What are the implications of this strategy--is the data in it's joined form ever left laying around somewhere exposed?

coatespt · ‎10-05-2016

The root cause we discovered is interesting (would still like to know if hypothesis above is correct). It turns out not to be the metastore. Creating tables on large S3 data sets takes so long that the Knox idle timeout is exceeded. Increasing the timeout sufficiently eliminates the failure in this case but the underlying cause is that it takes unreasonably long to create a table in this circumstance--a few hundred seconds. (small thousands of files) Overriding the timeouts by such a large margin has its own side effects. Someone should file this as a Jira ticket for this---If you're on S3 you are very likely to have huge numbers of files. Verifying that the party running the create-table has rights on each one makes less sense in S3 than in HDFS. Some folks will have hundreds of thousands or even millions of files.

coatespt · ‎10-04-2016

A create-table statement with and S3 bucket as the backing store works fine with no data in the bucket, or with one year's data, but fails reliably after 59.9 seconds if there are several years data (thousands of files). It leaves the metastore in a bad state as well, with hive unable to complete even simple operations for some time after. What's happening? Does Hive need to verify that the the user has rights to every file in the bucket? Is there some easy cure for this?

coatespt · ‎09-28-2016

I've set up a volume tagged as ARCHIVAL on each of my 4 EC2 nodes. When I make an HDFS directory and give it a COLD policy, running "hdfs storagepolicies -getStoragePolicy -path /zorch" says that the policy is set, but when I "hdfs dfs -put" data in the directory I get the error below. If I remove the policy, I can write data to that directory. The underlying directories at the linux level look fine. I can see that data gets written in the underlying drive on the machine where I'm running "hdfs -put". Any ideas? "COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 4 datanode(s) running and no node(s) are excluded in this operation. [pete.coates@master1 ~]$ hdfs storagepolicies Usage: bin/hdfs storagepolicies [COMMAND] [-listPolicies] [-setStoragePolicy -path <path> -policy <policy>] [-getStoragePolicy -path <path>] [-unsetStoragePolicy -path <path>] [-help <command-name>]"

coatespt · ‎09-27-2016

This brings up an issue. When the S3->S3 moves occur, does the data move across the local LAN link or does this occur entirely within the S3 infrastructure. I.e., if you copy, a NAS-backed file on a server it is read in across the LAN and then written out again. S3 isn't a NAS in that sense--but is this what it does, or does S3 move the data around on its own networks when the move is S3-S3? This matters because network is probably our limiting resource with our query types. @Rajesh Balamohan

coatespt · ‎09-27-2016

Thanks. That's what I thought---it's negligible in HDFS but not always trivial in S3 because it's a copy+delete. Interesting idea about using distcp to transfer the data. Not sure if that would actually help with EBS backing HDFS but it's worth a try.

coatespt · ‎09-26-2016

Can anyone explain exactly what's going on here? When running "set hive.tez.exec.print.summary=true;" with large hive queries over S3, the job is only about half over when Hive/Tez prints all the job stats as if the job is complete. But the following is the final line (slightly obfuscated) and the copy takes as long as the query itself. INFO : Moving data to: s3a://xxxxxxxxxxx/incoming/mha/poc/.hive-staging_hive_2016-09-26_17-49-00_060_4187715327928xxxxxx-3/-ext-10000 from s3a://xxxxxxxxxxxx/incoming/mha/poc/.hive-staging_hive_2016-09-26_17-49-00_060_4187715327928xxxxxx-3/-ext-10002 What is the reason for the data being moved? If the same thing happens with HDFS it's not noticeably, probably because it's just moving pointers around, but on S3 it seems to be actually moving the data. (a) is this true and (b) why the movement?

coatespt · ‎09-25-2016

I feared as much. Thank you for your suggestion--I think it work for us, as this is a cloud cluster, and we can archive to S3, obviating the need to use heterogeneous storage for its intended purpose. However, I would like to suggest a Jira ticket to add a storage class for this purpose. There are significant use-cases where it would be useful to know that a subset of your data is confined to specific drives (a) without the restrictions of the existing policies (b) without abusing a storage class for this purpose.

coatespt · ‎09-23-2016

My existing EBS volumes are transparently encrypted. I added an extra volume that is not encrypted. Now I want to be able to control where HDFS writes a file. I think it must be possible because heterogeneous storage policies tell HDFS where to write. How can I do this?

coatespt · ‎09-20-2016

Do you mean 50Mbps per mapper or for the cluster as a whole? (I assume you mean the former, as the latter would imply almost two days to read a TB of S3 data.) Assuming you do mean 50Mbps per mapper, what is the limit on S3 throughput to the whole cluster—that’s the key information. Do you have a ballpark number for this?

coatespt · ‎08-31-2016

Some relevant details: This is an analytics application with all the data uploaded into S3. Hive runs directly against the S3 data, i.e, we do not move the data into HDFS for processing, If we worked on HDFS we could just use Ranger, but we need both the capacity and the durability provided by S3. Amazon's KMS is logically adequate to our needs, but does not satisfy security requirements because we are not allowed to have any third party with access to encryption keys. What I want to know is whether Vormetric's transparent data encryption works for S3, and if so, is it truly transparent, i.e., everything is the same from the user's point of view with it or without it.

coatespt · ‎08-30-2016

We need SSE for Hive running over S3, but cannot use SSE-S3 because we cannot have our encryption keys accessible to a third party even if it's Amazon. How can we achieve this? Ideas: (1) Ranger using SSE-C would be ideal but Ranger does not support this, and AFAIK there are no immediate plans to correct this shortcoming. Any chance I'm wrong about this? (2) SSE-KMS has an "envelope key" (CMC) that encrypts the master keys. If we could somehow control just the CMC from within Horton, that would probably be sufficient, as the keys managed by Amazon would be encrypted. (3) Third party product that would handle key-management and/or encryption. Does anything fill this niche?

coatespt · ‎08-29-2016

Yes, thanks--aware of Ranger, which would be ideal for our purposes, but it's not easy to tell from the literature whether Ranger can manage encryption at rest and key management on S3. The literature talks about HDFS, but I don't see any reference to S3 and have been told that it does not. Do all the Ranger features work for S3?

coatespt · ‎08-29-2016

Does Vormetric work with Hortonworks over S3? We cannot use Amazon's KMS. Is there another solution that maintains keys entirely under the user's control?

Online	Offline
Last Visited	‎02-12-2019 02:08 PM

Date Registered	‎08-29-2016 06:20 PM
Last Visited	‎02-12-2019 02:08 PM
Total Messages Posted	24
Total Kudos Received	8

Sandbox downloads have failed for two days on. 10....

Re: What are the cluster-wide bandwidth limitation...

Re: What are the cluster-wide bandwidth limitation...

What are the cluster-wide bandwidth limitations on...

Re: Hive using HTTP proxy in only one direction fr...

How can I control the size of the blocks that hive...

Re: Hive using HTTP proxy in only one direction fr...

Re: Hive using HTTP proxy in only one direction fr...

Hive using HTTP proxy in only one direction from S...

Adding an S3 proxy and need to know how to include...

What's really happening in a Hive view that joins ...

Re: Metastore times out on create table if large n...

Metastore times out on create table if large numbe...

Heterogeneous ARCHIVAL storage over EBS looks ok b...

Re: set hive.tez.exec.print.summary=true causes od...

Re: set hive.tez.exec.print.summary=true causes od...

set hive.tez.exec.print.summary=true causes odd be...

Re: How to write HDFS data to a specific device

How to write HDFS data to a specific device

Re: Are there any special considerations or optimi...

Re: Does Vormetric work with S3?

How to achieve SSE for Hive over S3 without SSE-S3

Re: Does Vormetric work with S3?

Does Vormetric work with S3?