Member since
08-07-2018
23
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
284 | 08-09-2018 11:20 AM |
10-01-2018
01:20 PM
My requirement is to find the time elapsed between two jobs and display it. Log: 2018-09-27 09:27:18,615 71e02f2f-32d5-9509-870a-f80e54dc8775 George Started 2018-09-27 09:29:14,615 71e02f2f-32d5-9509-870a-f80e54dc8775 George Complete filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{UUID:messageId} %{USERNAME:usr} %{WORD:event}"]
add_tag => [ "%{event}" ]
}
date {
match => [ "timestamp", "ISO8601"]
}
# Measures the execution time of system1
elapsed {
unique_id_field => "messageId"
start_tag => "Started"
end_tag => "Complete"
new_event_on_match => true
add_tag => ["%{messageId}"]
add_field => { "User" => "%{usr}"}
}
# Records the execution time of system1
if "%{messageId}" in [tags] {
aggregate {
task_id => "%{messageId}"
code => "map['report'] = [(event['elapsed_time']*1000).to_i]"
map_action => "create"
add_field => { "User" => "%{usr}"}
}
}
}
... View more
09-19-2018
09:26 AM
sqoop job --create incj -- import --connect "jdbc:sqlserver://10.10.5.26:1433;database=***" --username **** --password **** --table dpt --incremental lastmodified -check-column dateimp --target-dir /tmp/user1/bdt/inc -m 1 --merge-key deptid
... View more
Labels:
09-17-2018
06:49 AM
create table department_part (Department_ID int, Department_Name String, Manager_ID int) Partitioned by (Location_ID int)
Row Format delimited
Fields terminated by ','
stored as textfile;
... View more
Labels:
09-17-2018
06:36 AM
@Sindhu If I increase the mappers, will the time consumed be less or How many mappers can we keep in general ??? Thanks!
... View more
09-17-2018
06:29 AM
$ sqoop import \
--connect jdbc:mysql://localhost/userdb \
--username root \
--table emp --m 1
... View more
Labels:
08-31-2018
05:44 AM
@gulshad.ansari Where can we find the snapshot, exacty which location.
... View more
08-30-2018
02:18 PM
@Felix Albani If i delete my files in trash, is there any possibility to have reference anywhere ?
... View more
08-30-2018
07:33 AM
08-27-2018
10:01 AM
@Jay Kumar SenSharma Hi, When u import everything from sql package from 1st statement, will it not import SQLContext class ?
... View more
08-27-2018
09:51 AM
import org.apache.spark.sql._
import org.apache.spark.sql.SQLContext._
... View more
Labels:
08-17-2018
08:35 AM
rddstudent.write.format("csv").saveAsTable("Student_Spark2")
"WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider csv. Persisting data source table `default`.`student_spark2` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive." Please let me know why I am not able to create a table using CSV as format
... View more
Labels:
08-16-2018
05:44 AM
Hi @Sindhu, I tried but i m not getting it.
... View more
08-14-2018
09:41 AM
@Sandeep Nemuri Ya, i m able to use repartition for a different RDD. Is it possible to repartition the same RDD ?
... View more
08-09-2018
11:20 AM
Thanks @Felix Albani. I found it. For Scala code pls check this git hub link https://github.com/phatak-dev/spark2.0-examples/blob/master/src/main/scala/com/madhukaraphatak/examples/sparktwo/streaming/FileStreamExample.scala. This is exactly what i needed.
... View more