Member since
06-20-2017
9
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2509 | 06-27-2017 06:48 AM |
11-09-2018
07:09 PM
I'm still a newbie to NIFI, and currently in progress of exploring the uses of some of the processors. Is there any way to push data from s3 to redshift using NIFI? What have I done up to now is, I have simply merged two csvs from s3 and stored it in another bucket as one file. I was able to merge two csvs using the MergeRecord processor. But what I don't know is, how to push data from S3 to Redshift tables?
... View more
Labels:
- Labels:
-
Apache NiFi
07-21-2017
09:21 AM
Thank you @rbiswas
... View more
07-17-2017
11:58 AM
I'm already having a MySQL table in my local machine (Linux) it self, and I have a Hive external table with the same schema as the MySQL table. I'm trying to import data from MySQL table to my Hive external table and I'm using sqoop for this. But then the problem is, whenever a new record is being added to the MySQL table, it doesn't update the Hive external table automatically? This is the sqoop import command I'm using: sqoop import --connect jdbc:mysql://localhost:3306/sqoop --username root -P --split-by id --columns id,name,age,salary --table customer --target-dir /user/chamith/mysqlhivetest/ --fields-terminated-by "," --hive-import --hive-table test.customers Am I missing something over here? Or how can this be done? Any help could be appreciated.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Sqoop
06-28-2017
09:39 AM
@knarendran Thank you so much, this works!
... View more
06-27-2017
06:48 AM
Changing the regex to below worked for me: ^([^0-9]+).* Hope it helps!
... View more
06-27-2017
04:32 AM
Thank you @Andres Koitmäe for your quick reply. I updated my query with the regex you mentioned, but still, I'm getting it as null. Is that the normal behavior when I search in with a SELECT query after executing the hive query? To make sure that I've made the correct changes I'm pasting the hive query here again: CREATE EXTERNAL TABLE LogParserSample(
logtype STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'input.regex' = '^[^0-9]+'
)
STORED AS TEXTFILE
LOCATION '/mypath/'; Thanks again.
... View more
06-26-2017
01:37 PM
I'm still a noob in this world, so bare with me if there are any misconceptions. What have I done so far is, I've setup Hadoop and Hive in my linux enviroment and I'm trying to extract data from an HDFS file into my Hive table. This is how the log line looks like: apache2013-10-09T14:04:32Zphp129.124.201.110/EKEE.php20019705910Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24 This my Hive query: CREATE EXTERNAL TABLE LogParserSample(
logtype STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'input.regex' = '^([\w\-]+)'
)
STORED AS TEXTFILE
LOCATION '/mypath/'; When I did execute the above the table got created successfully. But then when I tried to query the data from that table using a SELECT statement, I'm getting null instead of the word apache. For this instance, I'm trying to extract the word apache from the above log line. But then I tried executing the same regex in rubular, the outcome was correct as I expected. I'm unable to figure out why! Have I missed anything trivial above? Any help could be appreciated. Thanks!
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive