Member since
04-19-2018
32
Posts
4
Kudos Received
0
Solutions
04-20-2021
02:25 PM
@Shu_ashu I tried this below expression for two job names in my flow and it is throwing error. Could you help? Options I tried: ```${N_JOB:equalsIgnoreCase('DWDMBL0D','DWDMBL0F'):and(${LAST_RUN:equals('${custom_date}')}):and(${C_CURR_STA:equalsIgnoreCase('C')})}``` ```${N_JOB:equalsIgnoreCase("DWDMBL0D","DWDMBL0F"):and(${LAST_RUN:equals('${custom_date}')}):and(${C_CURR_STA:equalsIgnoreCase('C')})}```
... View more
11-27-2019
12:51 AM
did you solve the issue?, because I am facing a similar one with a different automatically generated pom
... View more
06-07-2018
11:25 AM
1 Kudo
@aman
mittal
Great, Good to know that..!! Other way of doing by checking the length of fragment.index attribute then using nested ifelse statements to determine the prepend by 00,0. but the expression will become complex using Advanced property will be good approach. If the Answer addressed your question, Take a moment to Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.
... View more
05-26-2018
07:14 PM
@aman
mittal
Yes, it's possible. Take a look into the below sample flow Flow overview: 1.SelectHiveQL //to list tables from specific database in avro format HiveQL Select Query
show tables from default //to list all tables from default database 2.ConvertAvroToJson //to convert the list of tables from avro format to json format 3.SplitJson //split each table into individual flowfiles 4.EvaluateJsonPath //extract tab_name value and keep as attribute to the flowfile 5.RemoteProcessorGroup //as you are going to do for 3k tables it's better to use RPG for distributing the work. if you don't want to use RPG then skip both 5,6 processors feed success relationship from 4 to 7. 6.InputPort //get the RPG flowfiles 7.SelectHiveQL //to pull data from the hive tables 8.EncryptContent 9.RouteOnAttribute //as selecthiveql processor writes query.input.tables attribute, so based on this attribute and NiFi expression language add two properties in the processor. Example: azure
${query.input.tables:startsWith("a")} //only tablenames starts with a
gcloud
${query.input.tables:startsWith("e"):or(${query.input.tables:startsWith("a")})} //we are going to route table names starts with e(or)a to gcloud Feed the gcloud relationship to PutGCSobject processor and azure relationship to PutAzureBlobStorage processor. Refer to this link for NiFi expression language and make your expression that can route only the required tables to azure,gcs. In addition i have used only single database to list all the tables but if your 3k tables are coming from different databases then use GenerateFlowfile processor and add all the list of databases.Extract each database name as attribute --> feed the success relationship to SelectHiveQL processor. Refer to this link dynamically pass database attribute to first select hiveql processor. Reference flow.xml load-hivetables-to-azure-gcs195751.xml - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.
... View more
05-15-2018
04:22 AM
@Matt Burgess Thank you for clarification ! Cheers
... View more
04-27-2018
05:43 AM
Thanks . I was not accessing the api with correct syntax .
... View more
10-23-2018
01:34 PM
@aman
mittal : Did you find a solution for this ? Thanks !
... View more
09-06-2018
01:58 PM
There is a known issue in the PutGCSObject if running behind a secured network and accessing GCS through a proxy. The Proxy Host and Proxy Port added to the PutGCSObject will proxy the object itself, but DOES NOT proxy the GCPCredentialsControllerService, the service that authenticates your connection using yous Service AccountJSON file. At this time, this is a Community Supported Processor, and this issue exists in HFD 3.2 (Nifi v.1.7).
... View more