Thanks @Richard Dobson I see Socket connection for steaming. Is it possible to do without that? I need to move it to productions. Also, current design:
I have a central repo for rules to be applied and evaluating which ones to apply and send this info in JSON format to Validation Engine. I have defined it in 2 ways : 1) SQL rules - using SparkSQL
{
"file" : "path to file",
"delimiter" : ",", "rules" : [
{
"query" : "select count(*) from file", "threshold" : 20,
},
{
.....
} ]
} Results : Usually count comarision and its results 2) Something like spot check where validate each row on some rules: {
"file" : "path to file", "delimiter" : ",",
"rules" : [ {
"name" : "row_check",
"compare_value" :20
}, {
"name" : "min_value_check",
"column_name" : "abc,
"compare_value" :20
},
{
.....
} ]
} Results : Validation results for each row
eg
row_no/row_key | pass/failure | actual value | threshold | rule_name
Can you please shed some light how can I turn to stream and process these kind of rules?
Also please provide a sample snippet or any link.. [I am entirely new to python and spark].
... View more
