Member since
02-01-2022
269
Posts
95
Kudos Received
59
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1909 | 06-12-2024 06:43 AM | |
2666 | 04-12-2024 06:05 AM | |
1977 | 12-07-2023 04:50 AM | |
1177 | 12-05-2023 06:22 AM | |
2077 | 11-28-2023 10:54 AM |
12-15-2022
06:12 AM
@Jaimin7 assuming "gt" is your parameter, that would be referenced as #{gt} and i think would be: {"updatedAt" : {#{gt} : 2022-12-10}} If "gt" is your variable, that would be represented as ${gt} and that would then be: {"updatedAt" : {${gt} : 2022-12-10}} Preference going forward is to use Paramaters as Variables will be going away soon.
... View more
12-12-2022
05:42 AM
1 Kudo
@hegdemahendra Awesome to see you making custom processors. A few things that might help: Make sure this controller service exists in your nifi, some nifi builds do not include all nars Make sure this controller service exists in your project dependencies That said, you may find more advanced nifi help engaging with the nifi developer community on slack or the mailing list. You can find the slack invite link at the bottom of that page.
... View more
12-09-2022
06:03 AM
@Althotta I do not think this is possible for the create to reference schema registry, only insert/update. To do this table creation in NiFi you will need to do some flowfile content manipulation work to fabricate the CREATE TABLE statement. For example if the flowfile content is the avro schema (you could get this form the schema registry API) then you can parse out the columns and data types to get the middle of the create statement: (col_name data_type [COMMENT 'col_comment'],, ...) then a ReplaceText processor to add the top lines: CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name and bottom lines: [COMMENT 'table_comment'] [ROW FORMAT row_format] [FIELDS TERMINATED BY char] [STORED AS file_format]; This would give you a full create statement which you can then execute against Hive.
... View more
12-09-2022
05:54 AM
@quangbilly79 The top screen shot is for CDP Public Cloud and the bottom screen shot is for CDP Private Cloud Base. As such, the CDE (Cloudera Data Engineering) "Schedule Job" is not available in the Cloudera Manager UI. You could use something like Oozie: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/configuring-oozie/topics/oozie-introduction.html
... View more
12-09-2022
05:39 AM
@abdebja Have you completed HUE configuration for SSL of each services host(s)? The docs are as follows: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/securing-hue/topics/hue-configuring-tls-ssl.html
... View more
12-09-2022
05:33 AM
@KPG1 Here is a good post i found that describes the required steps to move postgres to mysql including a php to do the postgres to mysql conversion. https://cloudera.ericlin.me/2016/10/hive-metastore-migration-from-embedded-postgres-to-mysql/ You may need to find a more modern tool for that conversion, but the process to dump, convert, and move metastores should be very similar.
... View more
11-18-2022
09:02 AM
@Mosunmola In order to start Hive, you will need yarn and hdfs started first. Do not restart all. The sandbox environment is going to require a very robust computer to run the entire stack, like 32gb+ memory. That said, a 16gbs or less memory may not be enough to run multiple services depending on what else may be running on your machine. So start and stop only what you need to test, not the entire stack.
... View more
10-28-2022
06:15 AM
@Vickey You will need to investigate the differences in execution from nifi vs hue. You should be able to see some differences in tez between both queries. The return code 1 is often due to tez resource availability. Also, i would suggest that you NOT do anything with retry logic in nifi until you are sure the processor works as expected the first time. Bringing in the retry logic only hides the issue if it is intermittent. Sounds like it is consistently not working, so retry is not helpful. Another suggestion is to turn the process logging level to DEBUG. Last but not least, in future, try to screen shot your flow, and processor configs so all of us can see more details
... View more
10-28-2022
06:07 AM
@D5ha To specifically answer. your question:
is there any way to identify the specific content_repository location for each processor?
No, there is no such manner to do this.
... View more
10-28-2022
05:44 AM
1 Kudo
@sathish3389 Define a parameter context and parameter ("parameter_password") for your flow with your password string, define that as sensitive value, then use the parameter in the processor property value : ${http.headers.Authorization:equals(#{parameter_password}) This will hide the password and make it easy to update the password by just updating the parameter.
... View more