About steven-matison

steven-matison · ‎01-23-2023

@BRinxen First, I feel your pain, as this sandbox was always an issue. Some advice below. Second, i would highly recommend you find a way to do something with hive,spark in another more modern form factor (not old hortonworks sandbox). That said, you are going to need like 32 gb of ram on a very beefy machine to role the whole sandbox even then it will struggle. If you have less resources, you willy only be able to run a few services, not the whole stack. Turn everything else off/maintenance mode. Start yarn, mapreduce, hdfs first. Then begin to start hive. Expect things to take a long time so be patient. Make sure nothing else is running on the main machine.

steven-matison · ‎01-23-2023

Very good article, this will definitely help me in the future!

steven-matison · ‎12-21-2022

@zIfo based on what i see, you need to completely fill out the keystore and truststore using cacerts. Make sure nifi user has permissions to read file. If your https end point is not a public cert it will not work with cacerts, you will need to make your own keystore and truststore which contains your specific certs. Here is a post with more details: https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/td-p/302719 And another one that goes into much greater detail: https://community.cloudera.com/t5/Support-Questions/RESOLVED-NIFI-LISTENHTTP-SSL/td-p/146985

steven-matison · ‎12-16-2022

The hive libraries are quite large and are not included in the base binary distribution. You can add them manually, or use github to clone and build the full project. Reference: https://github.com/apache/nifi/blob/rel/nifi-1.19.0/nifi-assembly/pom.xml#L1054

steven-matison · ‎12-16-2022

Ahh my confusion, it is hard to assume without a lot more detail. Reference: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#gt ${fileSize:gt( 1024 )} You need to use the expression "gt( some value )" against a parameter or variable. In the case above, "file size greater than 1024".

steven-matison · ‎12-15-2022

@wallacei There are roles attached to the main control plane and roles attached to specific environment. Work with you environmentAdmin to make sure you have all the correct roles at the control plane level and then deeper at the environment level if necessary. Below is the EnvironmentAdmin [ { "crn": "crn:altus:iam:us-west-1:altus:policy:EnvironmentAdminPolicy", "policyStatements": [ { "rights": [ "environments/getFreeipaOperationStatus", "environments/repairFreeIPA", "environments/upgradeFreeIPA", "environments/createDatahub", "datahub/read", "datahub/write", "datalake/read", "datalake/write", "environments/read", "environments/write" ], "resources": [ "*" ] } ] } ] You may need more specific roles to access additional services around the environment.

steven-matison · ‎12-15-2022

@saicharan You can find the hive jdbc driver for mac here which allows the operating system drop down: https://www.cloudera.com/downloads/connectors/hive/jdbc/2-5-4.html Also be sure to share your connection string and specific error message for better responses.

steven-matison · ‎12-15-2022

@Jaimin7 assuming "gt" is your parameter, that would be referenced as #{gt} and i think would be: {"updatedAt" : {#{gt} : 2022-12-10}} If "gt" is your variable, that would be represented as ${gt} and that would then be: {"updatedAt" : {${gt} : 2022-12-10}} Preference going forward is to use Paramaters as Variables will be going away soon.

steven-matison · ‎12-12-2022

@hegdemahendra Awesome to see you making custom processors. A few things that might help: Make sure this controller service exists in your nifi, some nifi builds do not include all nars Make sure this controller service exists in your project dependencies That said, you may find more advanced nifi help engaging with the nifi developer community on slack or the mailing list. You can find the slack invite link at the bottom of that page.

steven-matison · ‎12-09-2022

@Althotta I do not think this is possible for the create to reference schema registry, only insert/update. To do this table creation in NiFi you will need to do some flowfile content manipulation work to fabricate the CREATE TABLE statement. For example if the flowfile content is the avro schema (you could get this form the schema registry API) then you can parse out the columns and data types to get the middle of the create statement: (col_name data_type [COMMENT 'col_comment'],, ...) then a ReplaceText processor to add the top lines: CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name and bottom lines: [COMMENT 'table_comment'] [ROW FORMAT row_format] [FIELDS TERMINATED BY char] [STORED AS file_format]; This would give you a full create statement which you can then execute against Hive.

steven-matison · ‎12-09-2022

@quangbilly79 The top screen shot is for CDP Public Cloud and the bottom screen shot is for CDP Private Cloud Base. As such, the CDE (Cloudera Data Engineering) "Schedule Job" is not available in the Cloudera Manager UI. You could use something like Oozie: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/configuring-oozie/topics/oozie-introduction.html

steven-matison · ‎12-09-2022

@abdebja Have you completed HUE configuration for SSL of each services host(s)? The docs are as follows: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/securing-hue/topics/hue-configuring-tls-ssl.html

steven-matison · ‎12-09-2022

@KPG1 Here is a good post i found that describes the required steps to move postgres to mysql including a php to do the postgres to mysql conversion. https://cloudera.ericlin.me/2016/10/hive-metastore-migration-from-embedded-postgres-to-mysql/ You may need to find a more modern tool for that conversion, but the process to dump, convert, and move metastores should be very similar.

steven-matison · ‎11-18-2022

@Mosunmola In order to start Hive, you will need yarn and hdfs started first. Do not restart all. The sandbox environment is going to require a very robust computer to run the entire stack, like 32gb+ memory. That said, a 16gbs or less memory may not be enough to run multiple services depending on what else may be running on your machine. So start and stop only what you need to test, not the entire stack.

steven-matison · ‎10-28-2022

@Vickey You will need to investigate the differences in execution from nifi vs hue. You should be able to see some differences in tez between both queries. The return code 1 is often due to tez resource availability. Also, i would suggest that you NOT do anything with retry logic in nifi until you are sure the processor works as expected the first time. Bringing in the retry logic only hides the issue if it is intermittent. Sounds like it is consistently not working, so retry is not helpful. Another suggestion is to turn the process logging level to DEBUG. Last but not least, in future, try to screen shot your flow, and processor configs so all of us can see more details

steven-matison · ‎10-28-2022

@D5ha To specifically answer. your question: is there any way to identify the specific content_repository location for each processor? No, there is no such manner to do this.

steven-matison · ‎10-28-2022

@sathish3389 Define a parameter context and parameter ("parameter_password") for your flow with your password string, define that as sensitive value, then use the parameter in the processor property value : ${http.headers.Authorization:equals(#{parameter_password}) This will hide the password and make it easy to update the password by just updating the parameter.

steven-matison · ‎10-28-2022

@Griggsy I would have to have working sample to test and provide better solution, that said i think you need to use function inside of the value, not the property. There you could also use ifElse making the NiFi Expression language operate for both scenarios. Chaining expressions together can be a challenge but should get you where you need to go.

steven-matison · ‎10-28-2022

@Ekodar You will need to use a driver to connect php to impala. Quick search and this looks promising: https://docs.cloudera.com/documentation/other/connectors/impala-jdbc/latest/Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf Here is another example with more details showing actual php code: https://www.cdata.com/kb/tech/impala-odbc-php.rst

steven-matison · ‎10-28-2022

If you are able to get the token and communicate with the api from postman, thats a great start. The PUT request must have some other issue. Are you passing the token correctly? Have you made sure that you are able to get the token, and make the put request from nifi node with command line? You always want to make sure you have working samples for any api system before you try to convert them to postman.

steven-matison · ‎10-28-2022

Are you sure the code is the same? Sounds like the clustered version is kicking off job and its never finishing (endless loop?) or the task is creating some performance or connectivity issue.. Depending on your processor, it may need to be flagged as Primary Only.

steven-matison · ‎10-26-2022

I believe so. Were you able to get a token using that user?

steven-matison · ‎10-25-2022

share the relationship error? I believe you need to complete the flowFIle transfer and committ first, then do the other stateFlowFile transfer and commit

steven-matison · ‎10-25-2022

You will need to complete the session.commit() call with right details to fit your scenario.

steven-matison · ‎10-25-2022

@ryu CDP Public Cloud Azure or CDP Private Cloud on Azure VMs? To link a NiFi outside of the cluster, you will need to provide that nifi with the files from the CDP Cluster. For example core-site.xml, hdfs-site.xml. Outside of that configuration, you will need to do some networking to allow access between systems, and then last but not least deal with access/auth and kerberos. If you are already working on some of these areas, be sure to include screen shots of processors, controller services, configs, etc.

steven-matison · ‎10-25-2022

@D5ha I have had a recent similar need and I learned that you use session.commit() after a session.transfer to send a flowfile in an inner loop. In a custom script, without the commit specifically, nifi will assume and do the commit sending all the data in a single end execution flowfile. session.transfer(flowFile, REL_SUCCESS) session.commit()

steven-matison · ‎10-24-2022

@MaarufB Using nifi-cli outside of curl on a nifi node is definitely a challenge. There are a lot of things you need to check. Most of them are addressed in a similar post here: https://community.cloudera.com/t5/Support-Questions/Calling-nifi-Api-using-Postman/td-p/343993 Hopefully you can find the solution within that post.

steven-matison · ‎10-24-2022

@yoiun Going to go out on a ledge here: It seems like the the sqoop command and hue/sqoop command are executed on different hosts. Does the new host have permissions to mysql? This error here: Access denied for user 'demo'@'152.30.119.754' leads me to believe it does not.

steven-matison · ‎10-24-2022

@i_am_dba This is a very difficult one to explain. I think the issue is the string schema, or removing the avro schema you mentioned. My first suggestion would be to try to specific the schema which should help getting the data into the right formats. An alternate solution is to try and do that manually by replacetext/regex,etc but that is not the ideal solution. That said, another higher level suggesting is to update the upstream datasource to permanently solve the instability from '' (blank string), 'null' (string), or actual NULL (not a blank, '', or string at all).

steven-matison · ‎10-24-2022

@MaarufB Please make a new post with as much detail as you can around your question and use case. This is an old topic and will not get a good response in the comments. Feel free to @ tag me in the new post.

Online	Offline
Last Visited	‎03-29-2023 08:51 AM

Date Registered	‎02-01-2022 01:27 PM
Last Visited	‎03-29-2023 08:51 AM
Total Messages Posted	126
Total Kudos Received	29

Re: Decryption Failed with Algorithm [PBEWITHMD5AN...

Re: Changing data location from GCS to local disks...

Re: LookupRecord queue processor suddenly turns fu...

Re: Automatically running selected processor group...

Re: Can't locate files uploaded through ambari

Re: Need help running HortonWorks Sandbox on PC.

Re: Importing additional python modules while maki...

Re: InvokeHTTP - HTTPS API calls

Re: Missing Hive Processors In Nifi 1.19.1

Re: Issues with query parameter in getmongo proces...

Re: What Roles to view both Environment and Data L...

Re: Hive JDBC Connection on Mac OS

Re: Issues with query parameter in getmongo proces...

Re: java.lang.NoClassDefFoundError: org/apache/nif...

Re: Create Hive Tables in NIFI using Schema from s...

Re: How to schedule a job inside Cloudera WebUI (n...

Re: hue can't connect to hive over TLS

Re: Migrating Hive database from Postgress to Mys...

Re: Difficulty in starting program from my browser

Re: Nifi PutHiveQL fails with Return Code 1

Re: Is there any way to identify content storage l...

Re: How to make authenticate credentials sensitive...

Re: NiFi updateRecord - dealing with Blank fields

Re: How to use PHP to connect to coudera Impala?

Re: Controlling Apache NiFi components(ex. process...

Re: NIFI custom processor return multipleflow file...

Re: Controlling Apache NiFi components(ex. process...

Re: NIFI custom processor return multipleflow file...

Re: NIFI custom processor return multipleflow file...

Re: How to use NIFI to connect to the CDP cluster ...

Re: NIFI custom processor return multipleflow file...

Re: Controlling Apache NiFi components(ex. process...

Re: Oozie + sqoop action fails when import records...

Re: Nifi load data from CSV to mySQL

Re: Creating NiFi Template via Rest API