I am getting data using gethdfs and did some processing and writing back to hdfs. untill puthdfs data is processing fast, but puthdfs is writing data slow to hdfs. could you please let me know how to improve the speed? ... View more

Hi, I want to understand the below. 1. I have placed a log file in to hdfs 2. Then processed the log file in hdfs to csv format, say location of the csv is "/my/loc/" (both steps (1&2) using nifi). 3. created a table in hive "create external table load_table (col1, col2, ...) fields terminated by ',' lines terminated by '\n' stored as textfile location '/my/loc' -----I want to do as below----- 4. copy the above data from load_table to orig_table(internal table) 5. remove the csv file (as the data is stream data in to csv, it may grow endless). Things need clarification: I have used puthiveql processor to do step 4, as insert overwrite table orig_table select col1, col2, ... from load_table; but I was getting error that "extraneous input ";" expecting EOF near <EOF>. I thought because of ";" in the insert statement, removed it but got "org.apache.hadoop.hive.ql.exec.movetask" error. Also, for step 5, how can I remove csv, as streaming data is coming and storing in csv, this may grow huge. Thank you ... View more

Hi, I am getting error when inserting into hive tables using puthiveql processor. Scenario: I have a file, which I managed to convert in to INSERT INTO statement ( I copied it to a location to see if any syntax mistakes are there, found no errors). Assume: INSERT INTO mytable values ('a', '', 'c', '', 'd'); when I send the above to puthive ql, I am getting the below error. I thought it is because of the ";" in the end of insert into statement, so I removed it, then I am getting the below error Please help me. Thank you. ... View more

I have below data in hdfs a="alphabet_123_a" b="alphabetb" c="alphabet"c" is third one" b="newb" d="alphabet@/d" a="new a" a="changed a", b="changed b" c="changed c" e="alphabet e" My idea is: 1. Make a table in hive as orc, with columns a, b, c,d,e. 2. extract the attributes from the above data. 3. Mapping attributes according to column names in hive and storing them in hive. 4. in first line a,b,c; second line b,d,a; third line a,b,c,e 5. now after extracting all the lines and storing in hive, the values which are not present in lines (e.g. first line dont have "d" and "e"; second line dont have "c" and "e"; third line dont have "d") should be NULL, by the time they store in hive. Approach 1. Table "details" is created with columns a,b,c,d,e 2. Extract text processor is configured with custom properties as (a=)(.*?(?=\s\w+=|$)) --- [This will extract "alphabet_123a" in line 1 along with quotes(") at begening and ending of the values (b=)(.*?(?=\s\w+=|$)) --- [This will extract "aphabetb" in line 1 along with quotes...) 3. I am confused in the replace text processor, as 1. how to remove double quotes? 2. insert NULL values if the corresponding column name is missing in the line? 3. how to generalize the replace text for search value? Also let me know, how can I change the regex in extract text processor(if necessary)? Please help me Thanks ... View more