Member since
05-30-2018
1322
Posts
715
Kudos Received
148
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4041 | 08-20-2018 08:26 PM | |
| 1943 | 08-15-2018 01:59 PM | |
| 2370 | 08-13-2018 02:20 PM | |
| 4102 | 07-23-2018 04:37 PM | |
| 5010 | 07-19-2018 12:52 PM |
07-09-2016
01:30 AM
6 Kudos
Short Description: Teragen and Terasort Performance testing on AWS Article This article should be used with extreme care. Do not use as benchmark. I performed this test to simply run a quick 1 Terabype teragen test on AWS to determine what type of performance I can get from mapreduce on AWS with VERY LITTLE configuration tweaking/tuning On my github page here you will find the following:
teragen script hadoop,yarn,mapred,capacity scheduler configurations used during testing Hardware: (Master & Datanode) 1 Master, 3 Data nodes d2.4xlarge, 16vCPU, 122GB ram, (max) 12x2000 Storage TeraGen Results: 1hrs, 6mins, 38sec Job Counters: Terasort Results: 1hrs, 34mins, 20sec Teravalidate Results: 25mins, 27sec
... View more
Labels:
07-08-2016
05:24 PM
I have launched cluster on cloudbreak and want to start using hdp services. the documentation on http://sequenceiq.com/cloudbreak-docs/latest/operations/#ssh-to-the-hosts does't provide details on how to do so. Any help?
... View more
Labels:
- Labels:
-
Hortonworks Cloudbreak
07-08-2016
05:11 PM
1 Kudo
@Faisal Hussain please take a look at this post. @Bryan Bende mentioned this: incoming CSV like: h1,h2,h3,h4 v1,v2,v3,v4 You could capture that in ExtractText with a pattern of: (.+),(.+),(.+),(.+)\n(.+),(.+),(.+),(.+) Then in ReplaceText: { "${csv.1}" : "${csv.5}", "${csv.2}" : "${csv.6}", "${csv.3}" : "${csv.7}", "${csv.4}" : "${csv.8}" } Would produce: { "h1" : "v1", "h2" : "v2", "h3" : "v3", "h4" : "v4" }
... View more
07-08-2016
05:04 PM
1 Kudo
@Sri Bandaru you can add the jars in hive.aux.jars.path=/path/to/jar to add them to global location. then you don't need admin access.
... View more
07-08-2016
01:45 PM
I am running HDP cluster on AWS. EBS is getting very expensive. Reading about heterogeneous storage, is possible to use AWS-s3 as lets say cold storage and AWS-EBS as warm/hot storage. If so how would I do that. I don't see any documentation. s3 is much cheaper and hence want to use EBS as hot and s3 as cold storage.
... View more
Labels:
- Labels:
-
Apache Hadoop
07-08-2016
01:32 AM
1 Kudo
@Jay Johnson use Username: root
Password: hawq2016
... View more
07-08-2016
01:28 AM
@slachterman Good catch. fixed.
... View more
07-08-2016
01:00 AM
@rkovacs so does that mean to use cloudbreak.sequenceiq.com with azure i must launch azure vm?
... View more
07-07-2016
08:36 PM
Cloudbreak with cloudbreak.sequenceiq.com seems very easy on AWS. I simply create a role and use that role in cloudbreak to deploy instances. For azure I am not sure where to get App Id, Password, App Owner Tenant Id. Where do a find this info to launch clusters from cloubreak.sequenceiq.com? do I have to launch cloudbreak deployer on azure do use cloudbreak.sequenceiq.com? I don't have to do this with AWS.
... View more
Labels:
- Labels:
-
Hortonworks Cloudbreak