About vincentD

vincentD · ‎07-19-2019

The issue is resolved with these steps: From the CM web page, click YARN (MR2 Included) -> Actions -> Install YARN MapReduce Framework JARs.

vincentD · ‎07-19-2019

Hi experts, I am setting up a small test cluster, using CDH 6.2.0. So far it seems that HDFS, YARN start up fine. And spark-submit runs, and the applications generate results as expected. But teragen does not run. It looks like a MapReduce setup issue. Could you please give some hints? Thank you! $ hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-examples-3.0.0-cdh6.2.0.jar teragen 10000000 testgen_output WARNING: Use "yarn jar" to launch YARN applications. 19/07/19 08:57:19 INFO client.RMProxy: Connecting to ResourceManager at c41-12/172.16.2.121:8032 java.io.FileNotFoundException: File does not exist: hdfs://c41-12:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.2.0-mr-framework.tar.gz at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:145) at org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:488) at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2225) at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2221) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2227) at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:607) at org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:460) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:146) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588) at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:304) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:308) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:313) at org.apache.hadoop.util.RunJar.main(RunJar.java:227)

vincentD · ‎07-01-2019

Hello Experts, Is there a way to read compressed (.gzip, .zip or other formats) CSV files from a directory with pyspark, to create a dataframe? Thank you!

vincentD · ‎06-21-2019

Hi Experts, Is it possible to configure YARN on a data node to use half of cpu cores and half of total memory, and when necessary e.g. we are very busy, modify YARN configuration to expand to all available resources (cpu cores, memory)? And when not that busy, shrink back to half of the resouces? Can YARN be that elastic? Thank you!

vincentD · ‎05-23-2019

Thanks a lot for your reply Harsh. These sound great. Can you give some pointers to some learning materials on both methods, i.e. examples, blogs, URLs or books etc?

vincentD · ‎05-18-2019

We have ten millions image and video files, are looking for efficient ways to store them in Hadoop (HDFS ...), and analyze them with tools available in the Hadoop ecosystem. I understand HDFS prefer big files. These image files are small, they are under ten megabytes. Please advise. Thanks very much!

vincentD · ‎04-29-2019

Hi Wilfred, Without node labels, is it possible to have a few nodes reserved for some users' exclusive usage? Thanks Vincent

vincentD · ‎04-29-2019

It's very nice to know the upcoming updates. So with the feature in place, we would be able to assign some nodes exclusively to certain users (e.g. user userA, application appX) for their processing requirements. The question following naturally is about data storage locations: can other users' applications still read and write to the assigned nodes? will appX output files be written to the assigned nodes only, will appX be allowed to read input file blocks from all nodes in the cluster? This is too much I know. Thanks a lot!

vincentD · ‎04-25-2019

I double that, but I check here anyway. Our system is on CDH-5.15.2. The resource manager and job scheduler is YARN: $ yarn version Hadoop 2.6.0-cdh5.15.2 Subversion http://github.com/cloudera/hadoop -r c97bcbf0cba923467d45f5519b1953f436c64f12 Compiled by jenkins on 2018-11-13T13:53Z Compiled with protoc 2.5.0 From source with checksum 9d2d5b887383c7d4b811372f867c6440 This command was run using /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/jars/hadoop-common-2.6.0-cdh5.15.2.jar There are two users, they want isolated environment to run some experiments. I wonder if we can reserve two nodes for their use. Any suggestions and discussion are very welcome. Best Regards, Vincent

Online	Offline
Last Visited	‎09-20-2019 12:03 PM

Date Registered	‎02-17-2019 02:32 PM
Last Visited	‎09-20-2019 12:03 PM
Total Messages Posted	9

Re: MapReduce setup?

MapReduce setup?

How to read all compressed CSV files from a direct...

Flexibility with YARN

Re: Efficient ways to store many images files

Efficient ways to store many images files

Re: Node Labels

Re: Is it possible to reserve whole nodes for excl...

Is it possible to reserve whole nodes for exclusiv...