Member since
06-08-2017
1049
Posts
510
Kudos Received
312
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5675 | 04-15-2020 05:01 PM | |
2969 | 10-15-2019 08:12 PM | |
1171 | 10-12-2019 08:29 PM | |
6099 | 09-21-2019 10:04 AM | |
1998 | 09-19-2019 07:11 AM |
01-30-2023
07:18 AM
I understand that it'd be pointless to send a FlowFile to the ListS3 processor, but it still seems like we should be able to attach it to a Wait processor and have it trigger that way.
... View more
11-10-2022
02:45 AM
This is not working for spark on hive. no applicaiton id found when grep query id.
... View more
10-13-2022
12:28 PM
You can use EvaluateJsonPath. You only have to add an extra attribute and name it as you want (e.g. count) and set as value "$.my.array.size.length()". Then select Destination and change it to flowfile-attribute. This processor will produce a flowfile which has your extra attribute
... View more
08-12-2022
09:46 AM
Hi. From an outlook.live.com account, I want to read another folder other than INBOX, specifically one called VEGA and located inside INBOX. I have tried INBOX/VEGA INBOX\VEGA and others in different ways, but it returns an error. 2022-08-12 18:03:18,309 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.email.ConsumePOP3 ConsumePOP3[id=92a5a620-0182-1000-f4ca-a827c5fd2f4f] Processing halted: yielding [1 sec] java.lang.IllegalStateException: no such folder [INBOX\VEGA] I need help. Thanks.
... View more
10-30-2021
05:15 AM
if I want to replace more then one How? for example I tried this ${'$1':replace("ClientName","Client Name"),'$2':replace("Ageinyears ","Age in years")} but not work
... View more
10-15-2021
06:07 AM
Followed your steps but getting error on ConvertRecord processor /partition_dt is invalid because 'partition_dt' is not an associated property or has no validator associated with it. How can I resolve?
... View more
08-02-2021
04:53 PM
@Shu_ashu Which processor do you use to get this attributes? Is it UpdateAttribute? As you can see in above output screenshot both regex matches and gives same results once we extract all the contents as attributes then you can use expression language like ${Attribute 1} to get 1096 value ..etc.
... View more
05-21-2021
03:43 AM
Hello @priyanshu_soni You can skip the "Timestamp" part as the same is inserted by HBase implicitly. I tried the same Query as you, excluding the Timestamp & the same was Successful: hbase(main):018:0> put 'Table_X1','125','Cf1:CheckItem','{"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"}'
Took 0.0077 seconds
hbase(main):019:0> scan 'Table_X1'
ROW COLUMN+CELL
125 column=Cf1:CheckItem, timestamp=1621593487680, value={"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"}
1 row(s)
Took 0.0057 seconds As you may see above, the "timestamp" field corresponds to the Epoch Timestamp of the Inserted Row Time of Operation. If you wish to explicitly specify the Timestamp, You can include a EpochTime as shared below: hbase(main):020:0> put 'Table_X1','126','Cf1:CheckItem','{"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"}',1621593487680
Took 0.0202 seconds
hbase(main):021:0> scan 'Table_X1'
ROW COLUMN+CELL
125 column=Cf1:CheckItem, timestamp=1621593487680, value={"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"}
126 column=Cf1:CheckItem, timestamp=1621593487680, value={"ID" : "1334134","Name" : "Apparel Fabric","Path" : "Arts, Crafts & Sewing/Fabric/Apparel Fabric"}
2 row(s)
Took 0.0071 seconds Let us know if you have any issues with the Put Operation. - Smarak
... View more
04-20-2021
02:25 PM
@Shu_ashu I tried this below expression for two job names in my flow and it is throwing error. Could you help? Options I tried: ```${N_JOB:equalsIgnoreCase('DWDMBL0D','DWDMBL0F'):and(${LAST_RUN:equals('${custom_date}')}):and(${C_CURR_STA:equalsIgnoreCase('C')})}``` ```${N_JOB:equalsIgnoreCase("DWDMBL0D","DWDMBL0F"):and(${LAST_RUN:equals('${custom_date}')}):and(${C_CURR_STA:equalsIgnoreCase('C')})}```
... View more
02-26-2021
06:38 AM
Hi @Shu_ashu I have the neir input and i want to extract all attributes but i can't got the attributes in the arrays. i have try the jolttransformation processor but no ok Please can you help me [{ "Num_client__c": "3456", "Code_etablissement__c": "4562", "Mode_de_retrait__c": "xx", "OrderNumber": "000001", "Commande_en_stock__c": "y", "Commande_payee__c": "y", "Montant_Commande_TTC__c":"3000", "Status":"En cours", "OrderedDate": "2021-02-20", "Date__c": "2021-02-21", "Heure_de_debut__c":"10:24", "Heure_de_fin__c": "11:15", "shipping_address": [ { "ShippingStreet": "4 Allé du bois", "ShippingPostalCode": "75008", "ShippingCity": "Paris" } ], "orderItems_request": [ { "Code_Base_Produit__c": "AG45", "Quantity": "75", "UnitPrice": "30" }, { "Code_Base_Produit__c": "FF34", "Quantity": "75", "UnitPrice": "20" } ] }, { "Num_client__c": "3456", "Code_etablissement__c": "4562", "Mode_de_retrait__c": "xx", "OrderNumber": "000002", "Commande_en_stock__c": "y", "Commande_payee__c": "y", "Montant_Commande_TTC__c":"3000", "Status":"En cours", "OrderedDate": "2021-02-20", "Date__c": "2021-02-21", "Heure_de_debut__c":"10:24", "Heure_de_fin__c": "11:15", "shipping_address": [ { "ShippingStreet": "4 Allé du bois", "ShippingPostalCode": "75008", "ShippingCity": "Paris" } ], "orderItems_request": [ { "Code_Base_Produit__c": "AG45", "Quantity": "75", "UnitPrice": "30" } ] } ]
... View more
02-24-2021
08:06 AM
Iam also having the same issue.I have multiple fields and multiple files coming with this format. Any generic way to handle this?. Are there any sample for the validate processor and conversion of these fields?. Thanks in advance.
... View more
- Tags:
- The
02-12-2021
12:10 AM
This solution will not work. The failure flow does not go to next putDatabaseRecord processor because the error is exception. Failure flow files will go nowhere
... View more
02-11-2021
10:00 PM
@Shu_ashu How to configure the flow if EvaluateJsonPath Processor flow have recipient mail id and FetchFile has the file path. I have configured as below but two mails are getting triggered. Could you help to modify the flow so that only one mail with attachment will trigger ?
... View more
11-08-2020
01:37 PM
great job dude, thanks! 🙂
... View more
10-12-2020
12:22 AM
Why doesn't spark work like hive? Just create a temporary directory to store the final files, and finally rename it.
... View more
10-06-2020
08:47 AM
Do you have this flow available for download? Would be an excellent template
... View more
09-30-2020
12:47 PM
Hi @calonsca! Please have a look at this spec as well! [ { "operation": "shift", "spec": { "@": "data", "ID": "&", "#${date}": "date", "#${dataset:toLower()}": "dataset" } } ]
... View more
09-26-2020
02:37 AM
spark-xml package is a good option too. With all options you are limited to only process simple XMLs which can be interpreted as dataset with rows and columns. However, if we make it a little complex, those options won’t be useful.
... View more
09-16-2020
03:27 PM
I believe this will fail if you stop your job today and run it tomorrow.. now will change to other day and you will miss the data...
... View more
09-08-2020
07:38 PM
is there any resolution to this. I am seeing this issue with one of the acid tables which has around 25 M records. Other tables have 700 M records and are working fine. Facing this issue only for few tables.
... View more
08-04-2020
01:21 AM
@SouperDude Can you tell me the detailed steps?
... View more
07-30-2020
11:51 PM
Hello @DataGeek_Anish ,
Did you find the solution of this? I also stuck into the same problem and could not able to solve it.
I have JSON array data in HIVE tables. The data type of the column is ARRAY.
Could anyone please help me to resolve this issue or let me know the other solution to flatten the JSON array data would be really appreciate.
Data:
[{ "ts" : 1403781896 , "id" : 14 , "log" : "show" },{ "ts" : 1403781896 , "id" : 14 , "log" : "start" }]
[{ "ts" : 1403781911 , "id" : 14 , "log" : "press" },{ "ts" : 1403781911 , "id" : 14 , "log" : "press" }]
Logic Used:
SELECT
get_json_object ( single_json_table . single_json , '$.ts' ) AS ts , get_json_object ( single_json_table . single_json , '$.id' ) AS id , get_json_object ( single_json_table . single_json , '$.log' ) AS log
FROM ( SELECT explode ( json_array_col ) as single_json FROM jt ) single_json_table ;
Error:
Error while compiling statement: FAILED: SemanticException [Error 10014]: line 2:0 Wrong arguments ''$.td'': No matching method for class org.apache.hadoop.hive.ql.udf.UDFJson with (map<string,string>, string). Possible choices: FUNC(string, string)
PS: I can unfold the array with the explode function but can't able to flatten the JSON array data into columns.
Thank You.
... View more
07-07-2020
02:42 AM
Hi Friends I tried running the curl from my Linux session on which Nifi installed. This is a secured nifi instance . I got the forbidden error. Please see the error below -bash-4.2$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"id":"e3d11d1d-327b-1b5b-96a7-3cb59c48df17","state":"RUNNING"}' https://XXXXX:8443/nifi//nifi-api/flow/process-groups/e3d11d1d-327b-1b5b-96a7-3cb59c48df17; HTTP/1.1 403 Forbidden Server: squid/3.5.20 Mime-Version: 1.0 Date: Tue, 07 Jul 2020 09:34:57 GMT Content-Type: text/html;charset=utf-8 Content-Length: 3347 X-Squid-Error: ERR_ACCESS_DENIED 0 Vary: Accept-Language Content-Language: en X-Cache: MISS from XXXXX X-Cache-Lookup: NONE from XXXX Via: 1.1 XXXXXsquid/3.5.20) Connection: keep-alive curl: (56) Received HTTP code 403 from proxy after CONNECT @Shu_ashu - can you please help me here sir. Is it that I need to specify the certs somewhere ?
... View more
06-17-2020
04:59 AM
How to do this for putHDFS processor? I don't see any Last Modified Time property in that.
... View more
06-02-2020
05:14 AM
Hi, Would you please elaborate on why Hive configuration is needed? Thanks
... View more
05-29-2020
08:11 AM
@Shu_ashu this approach works has a problem that clear-state is working only on stopped processor. I am using ScrollElasticsearch processor and it needs to be cleared before it can be executed again. I tried curl -i -X POST http://localhost:8080/nifi-api/processors/0172101b-be82-11aa-1249-d1383cb1ceba/state/clear-requests but it end-up with conflict status I must stop processor in order to clear-state Do I really have to stop processor? manually or via API - it doesn't seems to me as a good design. Could you Help or give any advice please? Thank u. Petr
... View more
05-05-2020
11:45 AM
@Shu_ashu I don't understand how ScanHBase would work as an alternative when it has an input requirement i.e. it can't be used as a root node in a graph for gathering records from HBase indiscriminately. Would you agree? It seems like the only viable solution then is use the RestApi as you've suggested.
... View more
04-22-2020
08:39 AM
I am also facing the same issue of not able to update mongo records using exiting _id . I just need to update 2-3 fields in existing mongo document
... View more
04-19-2020
10:41 PM
1 Kudo
I have written a blog on this, Kindly refer to blog to setup dbcp connection pool Lookup controller service and execute same query in multiple databases. Please follow this link, it is with an example with step by step instructions to setup the same: https://bigdata-galaxy.blogspot.com/2020/04/nifi-querying-multiple-databases-using.html
... View more
04-19-2020
05:13 PM
Hi @DarkStar
As this thread was marked 'Solved' in March of 2018 you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your XML source that could aid others in providing a more targeted answer to your question.
... View more