Member since
02-01-2022
126
Posts
30
Kudos Received
24
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
220 | 03-16-2023 05:29 AM | |
93 | 03-02-2023 06:44 AM | |
219 | 03-01-2023 04:13 AM | |
100 | 02-24-2023 05:55 AM | |
81 | 02-22-2023 08:28 AM |
01-23-2023
06:31 AM
@BRinxen First, I feel your pain, as this sandbox was always an issue. Some advice below. Second, i would highly recommend you find a way to do something with hive,spark in another more modern form factor (not old hortonworks sandbox). That said, you are going to need like 32 gb of ram on a very beefy machine to role the whole sandbox even then it will struggle. If you have less resources, you willy only be able to run a few services, not the whole stack. Turn everything else off/maintenance mode. Start yarn, mapreduce, hdfs first. Then begin to start hive. Expect things to take a long time so be patient. Make sure nothing else is running on the main machine.
... View more
01-23-2023
06:22 AM
Very good article, this will definitely help me in the future!
... View more
12-21-2022
05:55 AM
@zIfo based on what i see, you need to completely fill out the keystore and truststore using cacerts. Make sure nifi user has permissions to read file. If your https end point is not a public cert it will not work with cacerts, you will need to make your own keystore and truststore which contains your specific certs. Here is a post with more details: https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/td-p/302719 And another one that goes into much greater detail: https://community.cloudera.com/t5/Support-Questions/RESOLVED-NIFI-LISTENHTTP-SSL/td-p/146985
... View more
12-16-2022
05:19 AM
The hive libraries are quite large and are not included in the base binary distribution. You can add them manually, or use github to clone and build the full project. Reference: https://github.com/apache/nifi/blob/rel/nifi-1.19.0/nifi-assembly/pom.xml#L1054
... View more
12-16-2022
04:43 AM
Ahh my confusion, it is hard to assume without a lot more detail. Reference: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#gt ${fileSize:gt( 1024 )} You need to use the expression "gt( some value )" against a parameter or variable. In the case above, "file size greater than 1024".
... View more
12-15-2022
06:31 AM
@wallacei There are roles attached to the main control plane and roles attached to specific environment. Work with you environmentAdmin to make sure you have all the correct roles at the control plane level and then deeper at the environment level if necessary. Below is the EnvironmentAdmin [
{
"crn": "crn:altus:iam:us-west-1:altus:policy:EnvironmentAdminPolicy",
"policyStatements": [
{
"rights": [
"environments/getFreeipaOperationStatus",
"environments/repairFreeIPA",
"environments/upgradeFreeIPA",
"environments/createDatahub",
"datahub/read",
"datahub/write",
"datalake/read",
"datalake/write",
"environments/read",
"environments/write"
],
"resources": [
"*"
]
}
]
}
] You may need more specific roles to access additional services around the environment.
... View more
12-15-2022
06:16 AM
@saicharan You can find the hive jdbc driver for mac here which allows the operating system drop down: https://www.cloudera.com/downloads/connectors/hive/jdbc/2-5-4.html Also be sure to share your connection string and specific error message for better responses.
... View more
12-15-2022
06:12 AM
@Jaimin7 assuming "gt" is your parameter, that would be referenced as #{gt} and i think would be: {"updatedAt" : {#{gt} : 2022-12-10}} If "gt" is your variable, that would be represented as ${gt} and that would then be: {"updatedAt" : {${gt} : 2022-12-10}} Preference going forward is to use Paramaters as Variables will be going away soon.
... View more
12-12-2022
05:42 AM
1 Kudo
@hegdemahendra Awesome to see you making custom processors. A few things that might help: Make sure this controller service exists in your nifi, some nifi builds do not include all nars Make sure this controller service exists in your project dependencies That said, you may find more advanced nifi help engaging with the nifi developer community on slack or the mailing list. You can find the slack invite link at the bottom of that page.
... View more
12-09-2022
06:03 AM
@Althotta I do not think this is possible for the create to reference schema registry, only insert/update. To do this table creation in NiFi you will need to do some flowfile content manipulation work to fabricate the CREATE TABLE statement. For example if the flowfile content is the avro schema (you could get this form the schema registry API) then you can parse out the columns and data types to get the middle of the create statement: (col_name data_type [COMMENT 'col_comment'],, ...) then a ReplaceText processor to add the top lines: CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name and bottom lines: [COMMENT 'table_comment'] [ROW FORMAT row_format] [FIELDS TERMINATED BY char] [STORED AS file_format]; This would give you a full create statement which you can then execute against Hive.
... View more
12-09-2022
05:54 AM
@quangbilly79 The top screen shot is for CDP Public Cloud and the bottom screen shot is for CDP Private Cloud Base. As such, the CDE (Cloudera Data Engineering) "Schedule Job" is not available in the Cloudera Manager UI. You could use something like Oozie: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/configuring-oozie/topics/oozie-introduction.html
... View more
12-09-2022
05:39 AM
@abdebja Have you completed HUE configuration for SSL of each services host(s)? The docs are as follows: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/securing-hue/topics/hue-configuring-tls-ssl.html
... View more
12-09-2022
05:33 AM
@KPG1 Here is a good post i found that describes the required steps to move postgres to mysql including a php to do the postgres to mysql conversion. https://cloudera.ericlin.me/2016/10/hive-metastore-migration-from-embedded-postgres-to-mysql/ You may need to find a more modern tool for that conversion, but the process to dump, convert, and move metastores should be very similar.
... View more
11-18-2022
09:02 AM
@Mosunmola In order to start Hive, you will need yarn and hdfs started first. Do not restart all. The sandbox environment is going to require a very robust computer to run the entire stack, like 32gb+ memory. That said, a 16gbs or less memory may not be enough to run multiple services depending on what else may be running on your machine. So start and stop only what you need to test, not the entire stack.
... View more
10-28-2022
06:15 AM
@Vickey You will need to investigate the differences in execution from nifi vs hue. You should be able to see some differences in tez between both queries. The return code 1 is often due to tez resource availability. Also, i would suggest that you NOT do anything with retry logic in nifi until you are sure the processor works as expected the first time. Bringing in the retry logic only hides the issue if it is intermittent. Sounds like it is consistently not working, so retry is not helpful. Another suggestion is to turn the process logging level to DEBUG. Last but not least, in future, try to screen shot your flow, and processor configs so all of us can see more details
... View more
10-28-2022
06:07 AM
@D5ha To specifically answer. your question:
is there any way to identify the specific content_repository location for each processor?
No, there is no such manner to do this.
... View more
10-28-2022
05:44 AM
1 Kudo
@sathish3389 Define a parameter context and parameter ("parameter_password") for your flow with your password string, define that as sensitive value, then use the parameter in the processor property value : ${http.headers.Authorization:equals(#{parameter_password}) This will hide the password and make it easy to update the password by just updating the parameter.
... View more
10-28-2022
05:31 AM
@Griggsy I would have to have working sample to test and provide better solution, that said i think you need to use function inside of the value, not the property. There you could also use ifElse making the NiFi Expression language operate for both scenarios. Chaining expressions together can be a challenge but should get you where you need to go.
... View more
10-28-2022
05:24 AM
@Ekodar You will need to use a driver to connect php to impala. Quick search and this looks promising: https://docs.cloudera.com/documentation/other/connectors/impala-jdbc/latest/Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf Here is another example with more details showing actual php code: https://www.cdata.com/kb/tech/impala-odbc-php.rst
... View more
10-28-2022
05:20 AM
If you are able to get the token and communicate with the api from postman, thats a great start. The PUT request must have some other issue. Are you passing the token correctly? Have you made sure that you are able to get the token, and make the put request from nifi node with command line? You always want to make sure you have working samples for any api system before you try to convert them to postman.
... View more
10-28-2022
05:16 AM
Are you sure the code is the same? Sounds like the clustered version is kicking off job and its never finishing (endless loop?) or the task is creating some performance or connectivity issue.. Depending on your processor, it may need to be flagged as Primary Only.
... View more
10-26-2022
05:41 AM
I believe so. Were you able to get a token using that user?
... View more
10-25-2022
01:42 PM
share the relationship error? I believe you need to complete the flowFIle transfer and committ first, then do the other stateFlowFile transfer and commit
... View more
10-25-2022
01:28 PM
You will need to complete the session.commit() call with right details to fit your scenario.
... View more
10-25-2022
12:53 PM
@ryu CDP Public Cloud Azure or CDP Private Cloud on Azure VMs? To link a NiFi outside of the cluster, you will need to provide that nifi with the files from the CDP Cluster. For example core-site.xml, hdfs-site.xml. Outside of that configuration, you will need to do some networking to allow access between systems, and then last but not least deal with access/auth and kerberos. If you are already working on some of these areas, be sure to include screen shots of processors, controller services, configs, etc.
... View more
10-25-2022
09:30 AM
@D5ha I have had a recent similar need and I learned that you use session.commit() after a session.transfer to send a flowfile in an inner loop. In a custom script, without the commit specifically, nifi will assume and do the commit sending all the data in a single end execution flowfile. session.transfer(flowFile, REL_SUCCESS) session.commit()
... View more
10-24-2022
08:50 AM
@MaarufB Using nifi-cli outside of curl on a nifi node is definitely a challenge. There are a lot of things you need to check. Most of them are addressed in a similar post here: https://community.cloudera.com/t5/Support-Questions/Calling-nifi-Api-using-Postman/td-p/343993 Hopefully you can find the solution within that post.
... View more
10-24-2022
06:27 AM
@yoiun Going to go out on a ledge here: It seems like the the sqoop command and hue/sqoop command are executed on different hosts. Does the new host have permissions to mysql? This error here: Access denied for user 'demo'@'152.30.119.754' leads me to believe it does not.
... View more
10-24-2022
06:18 AM
@i_am_dba This is a very difficult one to explain. I think the issue is the string schema, or removing the avro schema you mentioned. My first suggestion would be to try to specific the schema which should help getting the data into the right formats. An alternate solution is to try and do that manually by replacetext/regex,etc but that is not the ideal solution. That said, another higher level suggesting is to update the upstream datasource to permanently solve the instability from '' (blank string), 'null' (string), or actual NULL (not a blank, '', or string at all).
... View more
10-24-2022
06:09 AM
@MaarufB Please make a new post with as much detail as you can around your question and use case. This is an old topic and will not get a good response in the comments. Feel free to @ tag me in the new post.
... View more