About bleonhardi

bleonhardi · ‎02-17-2016

I only understand half of the s3 problems but it might be that you need to upgrade if a custom url is what you want. https://issues.apache.org/jira/browse/HADOOP-11261 "It also enables using a custom url pointing to an S3-compatible object store." Fix Version/s:2.7.0

bleonhardi · ‎02-17-2016

So you want to read/write files in HDFS? This is what webhdfs is for: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

bleonhardi · ‎02-17-2016

Multicluster mode in Ambari is perhaps one of the most requested features. However its a BIG implementation effort.

bleonhardi · ‎02-17-2016

Good question the yarn documentation says: "Also, there are safe-guards to ensure that users cannot view and/or modify applications from other users." However we have setup the administer settings in our cluster and I can still see all applications. So there must be some other setting to enable these "safeguards".

bleonhardi · ‎02-17-2016

So you definitely have the possibility to restrict control over a subset of queues. ( We had problems getting it to run in a non-kerberized cluster but in a kerberized cluster they work fine. ) Let me see if I find a way to restrict seeing applications as well. yarn.scheduler.capacity.root.<queue-path>.acl_submit_applications The ACL which controls who can submit applications to the given queue. If the given user/group has necessary ACLs on the given queue or one of the parent queues in the hierarchy they can submit applications. ACLs for this property are inherited from the parent queue if not specified. yarn.scheduler.capacity.root.<queue-path>.acl_administer_queue The ACL which controls who can administer applications on the given queue. If the given user/group has necessary ACLs on the given queue or one of the parent queues in the hierarchy they can administer applications. ACLs for this property are inherited from the parent queue if not specified.

bleonhardi · ‎02-17-2016

https://issues.apache.org/jira/browse/HADOOP-11261 Which version of Hadoop are you using? "It also enables using a custom url pointing to an S3-compatible object store."

bleonhardi · ‎02-17-2016

Still works on yarn, the official new one is mapreduce.job.reduces but I always used the one above and he still takes it.

bleonhardi · ‎02-17-2016

OK just to repeat. You can access s3 through hive with simple queries? So it cannot be a connection problem right? Perhaps too many parallel connections timing out when all the mappers spin up? Do you see some tasks successfully completing and then some tasks failing after 3 retries? In this case it sounds like a timeout issue. I have seen some issues in google like this that tried to fix it by increasing connection timeouts and retries. However mostly in presto forums. However there are s3 parameters available in the hdfs-site configuration https://hadoop.apache.org/docs/r2.6.3/hadoop-project-dist/hadoop-common/core-default.xml fs.s3a.connection.timeout

bleonhardi · ‎02-17-2016

Its mapred.reduce.tasks, if you run a mapreduce program from the hadoop client you would set it like this: -Dmapred.reduce.tasks=x Pig and Hive have different ways to predict reducer numbers.

bleonhardi · ‎02-17-2016

https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html The Edit log is the "transaction log" in HDFS. This means a transaction ( create a file, delete it ... ) is committed once it has been persisted to the edit log. In the good old times the edit log was local to the Namenode and merged into the FSImage by the secondary namenode. In HDFS HA the Edit log has been distributed to three Journalnodes. They still write an edit log but now on in a Quorum. ( I.e. the change needs to be persisted by a majority of the journalnodes ). But really the link explains it all very nicely.

Online	Offline
Last Visited	‎08-27-2016 12:14 PM

Member Since	‎09-23-2015 08:23 PM
Last Visited	‎08-27-2016 12:14 PM
Posts	800
Kudos received	888

Cloudera Community

Re: where an when does the fileinputformat() runs...

Re: We perform frequently Cartesian products invo...

Re: Kafka for queue to spark

Re: How new DAGs are submitted to existing Tez App...

Re: What is it meant by "HiveServer cannot handle ...

Re: Hive aggregate query failing for External tabl...

Re: how to call the web-service to read/write the ...

Re: YARN queues need a recipe to have two separate...

Re: YARN queues need a recipe to have two separate...

Re: YARN queues need a recipe to have two separate...

Re: Hive aggregate query failing for External tabl...

Re: reducer tasks long time

Re: Hive aggregate query failing for External tabl...

Re: reducer tasks long time

Re: Difference between journal and edits?