About emilysharpe

DianaTorres · ‎10-11-2023

@Srinivascnu As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.

emilysharpe · ‎04-10-2017

Thanks for clearing it up @Bryan Bende, much appreciated.

pminovic · ‎10-10-2016

It refers to active Name Node. For distcp you are providing 2 paths, source and target files/directories, both on HDFS of respective clusters. With some more settings you can refer to NN name service, meaning that you don't need to care which NN is active.

sunile_manjee · ‎07-14-2016

@Emily Sharpe thanks for the insights.

rushikeshdeshmu · ‎03-29-2016

@Emily Sharpe, If the original question is answered then please accept the best answer.

gmarya · ‎05-17-2016

Hi @Neeraj Sabharwal, I am trying to save my output results in Spark using saveAsTextFile(""). The result of which is multiple parts (part-0000, part-00001 ...so on) along with .crc files in the output directory. Do you have any idea how can I avoid forming the .crc files?

amcbarnett · ‎12-10-2015

I believe generally hard coding parallel is a bad idea in your pig script. With Parallel 1, you are effectively having 1 reducer perform the job. This can affect scale and performance. I would allow default parallelism and use the hdfs dfs -getmerge option. For an input point of view, Here is a tip to Combine Small files.

emilysharpe · ‎12-11-2015

Thanks to @Deepesh for the workaround. Also wanted to add (for info) that these steps will not be required after HDP upgrade. We will use ALTER TABLE activeTable CONCATENATE; to combine the many smaller ORC files into fewer larger ones (possible from Hive 0.14+). https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionConcatenate

Online	Offline
Last Visited	‎11-16-2018 12:44 AM

Member Since	‎12-09-2015 11:26 PM
Last Visited	‎11-16-2018 12:44 AM
Posts	35
Kudos received	13

Cloudera Community

Re: Merge and Rename files in HDFS - Pig?

Re: SQOOP EXPORT OF BLANKS VALUES AS BLANK IN TERA...

Re: NiFi templates not persisting unless exported ...

Re: Is the DistCp NameNode path the active HDFS Na...

Re: Best way to ensure null values are not stored ...

Re: How to manually manage number of HBase regions...

Re: Is there an -ignoreCrc equivalent when using g...

Re: Best practice for extract/output data (generat...

Re: Bug in Hive (HDP 2.1), Timestamp data overwrit...