I understand that it'd be pointless to send a FlowFile to the ListS3 processor, but it still seems like we should be able to attach it to a Wait processor and have it trigger that way. ... View more

This is not working for spark on hive.    no applicaiton id found when grep query id. ... View more

You can use EvaluateJsonPath. You only have to add an extra attribute and name it as you want (e.g. count) and set as value "$.my.array.size.length()". Then select Destination and change it to flowfile-attribute. This processor will produce a flowfile which has your extra attribute ... View more

Hi. From an outlook.live.com account, I want to read another folder other than INBOX, specifically one called VEGA and located inside INBOX. I have tried INBOX/VEGA INBOX\VEGA and others in different ways, but it returns an error. 2022-08-12 18:03:18,309 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.email.ConsumePOP3 ConsumePOP3[id=92a5a620-0182-1000-f4ca-a827c5fd2f4f] Processing halted: yielding [1 sec] java.lang.IllegalStateException: no such folder [INBOX\VEGA] I need help. Thanks.     ... View more

if I want to replace more then one How?    for example I tried this  ${'$1':replace("ClientName","Client Name"),'$2':replace("Ageinyears ","Age in years")} but not work  ... View more

Followed your steps but getting error on ConvertRecord processor /partition_dt is invalid because 'partition_dt' is not an associated property or has no validator associated with it. How can I resolve? ... View more

  @Shu_ashu Which processor do you use to get this attributes? Is it UpdateAttribute? As you can see in above output screenshot both regex matches and gives same results once we extract all the contents as attributes then you can use expression language like ${Attribute 1} to get 1096 value ..etc. ... View more

Hello @priyanshu_soni    You can skip the "Timestamp" part as the same is inserted by HBase implicitly. I tried the same Query as you, excluding the Timestamp & the same was Successful:   hbase(main):018:0> put 'Table_X1','125','Cf1:CheckItem','{"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"}' Took 0.0077 seconds hbase(main):019:0> scan 'Table_X1' ROW COLUMN+CELL 125 column=Cf1:CheckItem, timestamp=1621593487680, value={"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"} 1 row(s) Took 0.0057 seconds As you may see above, the "timestamp" field corresponds to the Epoch Timestamp of the Inserted Row Time of Operation. If you wish to explicitly specify the Timestamp, You can include a EpochTime as shared below: hbase(main):020:0> put 'Table_X1','126','Cf1:CheckItem','{"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"}',1621593487680 Took 0.0202 seconds hbase(main):021:0> scan 'Table_X1' ROW COLUMN+CELL 125 column=Cf1:CheckItem, timestamp=1621593487680, value={"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"} 126 column=Cf1:CheckItem, timestamp=1621593487680, value={"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"} 2 row(s) Took 0.0071 seconds   Let us know if you have any issues with the Put Operation.   - Smarak ... View more

@Shu_ashu I tried this below expression for two job names in my flow and it is throwing error. Could you help? Options I tried: ```${N_JOB:equalsIgnoreCase('DWDMBL0D','DWDMBL0F'):and(${LAST_RUN:equals('${custom_date}')}):and(${C_CURR_STA:equalsIgnoreCase('C')})}```   ```${N_JOB:equalsIgnoreCase("DWDMBL0D","DWDMBL0F"):and(${LAST_RUN:equals('${custom_date}')}):and(${C_CURR_STA:equalsIgnoreCase('C')})}``` ... View more

Hi @Shu_ashu  I have the neir input and i want to extract all attributes but i can't got the attributes in the arrays. i have try the jolttransformation processor but no ok Please can you help me [{ "Num_client__c": "3456", "Code_etablissement__c": "4562", "Mode_de_retrait__c": "xx", "OrderNumber": "000001", "Commande_en_stock__c": "y", "Commande_payee__c": "y", "Montant_Commande_TTC__c":"3000", "Status":"En cours", "OrderedDate": "2021-02-20", "Date__c": "2021-02-21", "Heure_de_debut__c":"10:24", "Heure_de_fin__c": "11:15", "shipping_address": [ { "ShippingStreet": "4 Allé du bois", "ShippingPostalCode": "75008", "ShippingCity": "Paris" } ], "orderItems_request": [ { "Code_Base_Produit__c": "AG45", "Quantity": "75", "UnitPrice": "30" }, { "Code_Base_Produit__c": "FF34", "Quantity": "75", "UnitPrice": "20" } ] }, { "Num_client__c": "3456", "Code_etablissement__c": "4562", "Mode_de_retrait__c": "xx", "OrderNumber": "000002", "Commande_en_stock__c": "y", "Commande_payee__c": "y", "Montant_Commande_TTC__c":"3000", "Status":"En cours", "OrderedDate": "2021-02-20", "Date__c": "2021-02-21", "Heure_de_debut__c":"10:24", "Heure_de_fin__c": "11:15", "shipping_address": [ { "ShippingStreet": "4 Allé du bois", "ShippingPostalCode": "75008", "ShippingCity": "Paris" } ], "orderItems_request": [ { "Code_Base_Produit__c": "AG45", "Quantity": "75", "UnitPrice": "30" } ] } ] ... View more

Iam also having the same issue.I have multiple fields and multiple files coming with this format. Any generic way to handle this?. Are there any sample for the validate processor and conversion of these fields?. Thanks in advance. ... View more

This solution will not work. The failure flow does not go to next putDatabaseRecord processor because the error is exception. Failure flow files will go nowhere ... View more

@Shu_ashu  How to configure the flow if  EvaluateJsonPath Processor flow have recipient mail id and FetchFile has the file path. I have configured as below but two mails are getting triggered. Could you help to modify the flow so that only one mail with attachment will trigger ?      ... View more

great job dude, thanks! 🙂 ... View more

Why doesn't spark work like hive? Just create a temporary directory to store the final files, and finally rename it. ... View more

Do you have this flow available for download? Would be an excellent template ... View more

Hi @calonsca! Please have a look at this spec as well! [ { "operation": "shift", "spec": { "@": "data", "ID": "&", "#${date}": "date", "#${dataset:toLower()}": "dataset" } } ] ... View more

spark-xml package is a good option too. With all options you are limited to only process simple XMLs which can be interpreted as dataset with rows and columns. However, if we make it a little complex, those options won’t be useful.  ... View more

I believe this will fail if you stop your job today and run it tomorrow.. now will change to other day and you will miss the data... ... View more

is there any resolution to this. I am seeing this issue with one of the acid tables which has around 25 M records.  Other tables have 700 M records and are working fine.  Facing this issue only for few tables. ... View more

@SouperDude  Can you tell me the detailed steps? ... View more

Hello @DataGeek_Anish ,   Did you find the solution of this? I also stuck into the same problem and could not able to solve it. I have JSON array data in HIVE tables. The data type of the column is ARRAY. Could anyone please help me to resolve this issue or let me know the other solution to flatten the JSON array data would be really appreciate.   Data: [{ "ts" : 1403781896 , "id" : 14 , "log" : "show" },{ "ts" : 1403781896 , "id" : 14 , "log" : "start" }] [{ "ts" : 1403781911 , "id" : 14 , "log" : "press" },{ "ts" : 1403781911 , "id" : 14 , "log" : "press" }] Logic Used: SELECT get_json_object ( single_json_table . single_json , '$.ts' ) AS ts , get_json_object ( single_json_table . single_json , '$.id' ) AS id , get_json_object ( single_json_table . single_json , '$.log' ) AS log FROM ( SELECT explode ( json_array_col ) as single_json FROM jt ) single_json_table ; Error: Error while compiling statement: FAILED: SemanticException [Error 10014]: line 2:0 Wrong arguments ''$.td'': No matching method for class org.apache.hadoop.hive.ql.udf.UDFJson with (map<string,string>, string). Possible choices:   FUNC(string, string)   PS: I can unfold the array with the explode function but can't able to flatten the JSON array data into columns.   Thank You.     ... View more

Hi Friends    I tried running the curl from my Linux session on which Nifi installed. This is a secured nifi instance . I got the forbidden error. Please see the error below    -bash-4.2$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"id":"e3d11d1d-327b-1b5b-96a7-3cb59c48df17","state":"RUNNING"}' https://XXXXX:8443/nifi//nifi-api/flow/process-groups/e3d11d1d-327b-1b5b-96a7-3cb59c48df17; HTTP/1.1 403 Forbidden Server: squid/3.5.20 Mime-Version: 1.0 Date: Tue, 07 Jul 2020 09:34:57 GMT Content-Type: text/html;charset=utf-8 Content-Length: 3347 X-Squid-Error: ERR_ACCESS_DENIED 0 Vary: Accept-Language Content-Language: en X-Cache: MISS from XXXXX X-Cache-Lookup: NONE from XXXX Via: 1.1 XXXXXsquid/3.5.20) Connection: keep-alive curl: (56) Received HTTP code 403 from proxy after CONNECT   @Shu_ashu - can you please help me here sir. Is it that I need to specify the certs somewhere ? ... View more

How to do this for putHDFS processor? I don't see any Last Modified Time property in that. ... View more

Hi, Would you please elaborate on why Hive configuration is needed?   Thanks ... View more

@Shu_ashu  this approach works has a problem that clear-state is working only on stopped processor.  I am using  ScrollElasticsearch processor and it needs to be cleared before it can be executed again.  I tried    curl -i -X POST http://localhost:8080/nifi-api/processors/0172101b-be82-11aa-1249-d1383cb1ceba/state/clear-requests    but it end-up with conflict status  I must stop processor in order to clear-state Do I really have to stop processor? manually or via API - it doesn't seems to me as a good design. Could you Help or give any advice please? Thank u.  Petr ... View more

@Shu_ashu I don't understand how ScanHBase would work as an alternative when it has an input requirement i.e. it can't be used as a root node in a graph for gathering records from HBase indiscriminately.  Would you agree?   It seems like the only viable solution then is use the RestApi as you've suggested. ... View more

I am also facing the same issue of not able to update mongo records using exiting _id . I just need to update 2-3 fields in existing mongo document ... View more

I have written a blog on this,  Kindly refer to blog to setup dbcp connection pool Lookup controller service and execute same query in multiple databases.   Please follow this link, it is with an example with step by step instructions to setup the same: https://bigdata-galaxy.blogspot.com/2020/04/nifi-querying-multiple-databases-using.html      ... View more

Hi @DarkStar  As this thread was marked 'Solved' in March of 2018 you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your XML source that could aid others in providing a more targeted answer to your question.      ... View more