Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.
Hi , we are using cdh5.4 cluser and web logs data volume in HDFS are huge with snappy compression and want to apply LZO compression techniques ,but cdh5.4 installed through packages.I have installed hadoop lzo rpm packages and through maven compile the jar files.In order to apply LZO compression on HDFS data,pulling into local system and then apply lzop compression(lzop file=>file.lzo) and put it back to HDFS path and running the below map reduce job to create lzo.idex to allow split the lzo files .My question is how to apply lzo compression alogeritham in HDFS(instead of copying to local and apply the copmression and put it back into HDFS) and when every run any mapreduce job,the data in HDFS should be lzo format.Any suggestions or idea would be great hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.20-SNAPSHOT.jar com.hadoop.compression.lzo.DistributedLzoIndexer /home/hadoop/test Thanks, Mohan
... View more
I am mohan ,base location is Singapore and working with Singtel pvt limitted(Telecom company),recently joined this company as Big data architecture .Our company using cloudera 5.4 entrerprise edition . Having 10+years experience and cloud era distribution working more than 2 years in sizing ,storage format,data analytics .
... View more