@Benjamin Newell So, the idea of ListSFTP is to provide list of files from SFTP based on filters, etc. This processor is stateful, meaning it will give you list of files that have been modified since last run. And it will maintain a state. That's a reason for not allowing incoming connections. Option 1. FetchSFTP has connection "not.found". You can use it to postpone processing of this file (add processor to penalize it and loop it back to FetchSFTP). And configure your log monitor to ignore all the messages matching "Failed to fetch content for {} from filename {} on remote host {} because the file could not be found on the remote system; routing to not.found" Option 2. First Flow: Use ListSFTP to pull files and put them on landing zone (local or NAS). Second Flow: In your flow - replace FetchSFTP with FetchFile. FetchFile has property "Log level when file not found". Use "DEBUG", so it won't be printing errors into log file. Option 3. Create custom processor extending FetchSFTP and change onTrigger to NOT print error message in case file not found exception. Let me know if that helps. ... View more

@Hanu V To work around your issue, you can explicitly cast to string or int (or whatever your data type should be). I could reproduce your case: select greatest(month, billed) from sales limit 5; Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating greatest(month,billed) (state=,code=0) with workaround: select greatest(int(month), int(billed)) from sales limit 5; +------+--+ | _c0 | +------+--+ | 8 | | 8 | | 8 | | 8 | | 2 | +------+--+ or select greatest(string(month), string(billed)) from sales limit 5; +------+--+ | _c0 | +------+--+ | 8 | | 8 | | 8 | | 8 | | 2 | +------+--+ ... View more

@Bilel Boubakri, I've tested your use case with getSFTP processor, it worked as expected with given regex (skipped MATFIC dir with all the files in it). I cannot test specifically GetFTP, as we don't have FTP servers running, but I'm sure it should give the same result. Tested with NiFi 1.1.0.2.1.1.0-2. Some suggestions: 1. if you copy-pasted your regex - make sure you don't have new line character 2. make sure your regex is in Path Filter Regex and not in File Filter Regex property 3. If 1 and 2 don't help, try to dance with tambourine and call Cthulhu Hope, that will help! ... View more

@Andres Urrego, What you are looking for (UPSERTS) aren't available in SQOOP-import. There are several approaches on how to actually update data in Hive. One of them is described here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-access/content/incrementally-updating-hive-table-with-sqoop-and-ext-table.html Other approaches are also using side load and merge as post-sqoop or scheduled jobs/processes. You can also check Hive ACID transactions, or using Hive-Hbase integration package. Choosing right approach is not trivial and depends on: initial volume, incremental volumes, frequency or incremental jobs, probability of updates, ability to identify uniqueness of records, acceptable latency, etc... ... View more

Alright, so I ended up with simple script and one processor in NiFi. Modules for reload should be provided in "modules_list" property of the processor (comma delimited). Script body: import sys, json def class_reloader(modules_to_reload): reload_msg = "" all_module_names = sys.modules.keys() all_module_names.sort() for mn in all_module_names: m = sys.modules[mn] # -- find full match of names with given modules if mn in modules_to_reload: try: reload(m) reload_msg = reload_msg + mn + "|" except: return 1, reload_msg continue # -- find if mn is submodule of any given one for mtr in modules_to_reload: if mn.startswith(mtr+'.'): try: reload(m) reload_msg = reload_msg + mn + "|" break except: return 1, reload_msg return 0, reload_msg #-------------------------------# flowFile = session.create() if(flowFile != None): modules_prop = modules_list.getValue() ml = [] if modules_prop: ml = modules_prop.split(',') cr = class_reloader(ml) flowFile = session.putAttribute(flowFile, 'class_reload_result', str(cr[0])) flowFile = session.putAttribute(flowFile, 'class_reload_found', str(cr[1])) session.transfer(flowFile, REL_SUCCESS) # implicit return at the end The code can be improved to navigate to FAILURE relationship in case of non-zero response code from the method. It's not perfect solution, but will work in most cases. If you have better one - please share! 🙂 ... View more

@Dhanya Kumar Heballi Shivamurthy, assuming you have both arrays with the same size (per record): select c1, c21, c31, c4 from ( select 100 c1, split('Delta|Alpha|Beta','\\|') c2, split('Source|Varied|Volume','\\|') c3, 'AppData' c4 ) foo LATERAL VIEW posexplode(c2) n1 as c22, c21 LATERAL VIEW posexplode(c3) n2 as c32, c31 where c22=c32; If array lengths can be different, then you need to add more conditions: select c1, c222 c2, c333 c3, c4 from ( select c1, c22, c32, -- keep indices case when c32 < size(c2) then c21 else null end c222, case when c22 < size(c3) then c31 else null end c333, c4 from ( select 100 c1, split('Delta|Alpha|Beta','\\|') c2, split('Source|Varied|Volume|Owner','\\|') c3, 'AppData' c4 ) foo LATERAL VIEW posexplode(c2) n1 as c22, c21 LATERAL VIEW posexplode(c3) n2 as c32, c31 ) bar where c22=c32 or (c222 is null and c22=0) or (c333 is null and c32=0); Result: +------+--------+---------+----------+--+ | c1 | c2 | c3 | c4 | +------+--------+---------+----------+--+ | 100 | Delta | Source | AppData | | 100 | Alpha | Varied | AppData | | 100 | Beta | Volume | AppData | | 100 | Owner | NULL | AppData | +------+--------+---------+----------+--+ ... View more