Let's say I have one input file named input.txt and the content of the file is given below Hadoop is good Hortonworks makes the life easy Hadoop is a framework This input.txt contains only 3 lines . Now I need the count of the word "Hortonworks" and line number of occurrence in the input file . For this input file the count of "Hortonworks" is 1 and it is in the line number 2. I can find it running separate MapReduce job for each query. Can we find both queries output in one MapReduce Job ? I do not want to run two separate job for this purpose. It will be IO heat for billions of data. ... View more

I am trying to move data from oracle to HDFS. File format is parquet and compression is lzo .Below sqoop command not working as expected sqoop import --connect jdbc:oracle:thin:@**.***.***.***:1521:**** --username **** --password **** --table MyTable -m 1 --target-dir /user/aps/test105 --fields-terminated-by '|' --as-parquetfile --compress --compression-codec lzop Below is the log 16/09/27 19:42:56 INFO mapreduce.Job: Running job: job_1474971595874_0032 16/09/27 19:43:04 INFO mapreduce.Job: Job job_1474971595874_0032 running in uber mode : false 16/09/27 19:43:04 INFO mapreduce.Job: map 0% reduce 0% 16/09/27 19:43:11 INFO mapreduce.Job: map 100% reduce 0% 16/09/27 19:43:11 INFO mapreduce.Job: Job job_1474971595874_0032 completed successfully 16/09/27 19:43:11 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=163431 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=6342 HDFS: Number of bytes written=2716 HDFS: Number of read operations=50 HDFS: Number of large read operations=0 HDFS: Number of write operations=9 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=4340 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=4340 Total vcore-seconds taken by all map tasks=4340 Total megabyte-seconds taken by all map tasks=8888320 Map-Reduce Framework Map input records=25 Map output records=25 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=101 CPU time spent (ms)=3600 Physical memory (bytes) snapshot=360005632 Virtual memory (bytes) snapshot=3532488704 Total committed heap usage (bytes)=299892736 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0 16/09/27 19:43:11 INFO mapreduce.ImportJobBase: Transferred 2.6523 KB in 18.3874 seconds (147.7096 bytes/sec) 16/09/27 19:43:11 INFO mapreduce.ImportJobBase: Retrieved 25 records. edwweb@ctsc00691239901:~/aps$ hadoop fs -ls /user/aps/test105 Found 2 items drwxr-xr-x - ****** hdfs 0 2016-09-27 19:42 /user/aps/test105/.metadata -rw-r--r-- 3 ****** hdfs 1279 2016-09-27 19:43 /user/aps/test105/53e90168-e46a-4404-a726-063c533e3db2.parquet Output is parquet file. It should be lzo file. Could you please help. ... View more

I have installed Hortonworks Sandbox 2.4 in my system using virtual box . But Spark is not up and running in this sandbox. All other services are working .I have 8 GB RAM and Ubuntu 14.04 LS . Please help. ... View more

I am trying to import data from Teradata to HDFS using lzo compression. Below is the sqoop command. sqoop import --connection-manager org.apache.sqoop.teradata.TeradataConnManager --connect jdbc:teradata://**.***.***.***/DATABASE=**** --username **** --password **** --table employee --target-dir /user/aps/test3 --compress --compression-codec lzop -m 1 This command is executing perfectly but I am getting as text file format as below $ hadoop fs -ls /user/aps/test3 Found 2 items -rw-r--r-- 3 edwweb hdfs 0 2016-09-14 16:24 /user/aps/test3/_SUCCESS -rw-r--r-- 3 edwweb hdfs 18 2016-09-14 16:24 /user/aps/test3/part-m-00000 $ hadoop fs -cat /user/aps/test3/part-m-00000 1,Arumoy 2,Manish It should be part-m-00000.lzo instead of part-m-00000. Am I doing anything wrong? ... View more