Member since
02-01-2022
223
Posts
71
Kudos Received
48
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
141 | 08-17-2023 06:07 AM | |
177 | 08-14-2023 05:54 AM | |
236 | 08-07-2023 06:51 AM | |
194 | 07-17-2023 08:43 AM | |
311 | 07-17-2023 08:38 AM |
02-09-2023
05:44 AM
@Iwantkakao There are 2 things i see right off bat: Tue Jan 31 19:47:13 UTC 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. ^^ consider the recommendation: useSSL=false in your sqoop command. 23/01/31 19:47:14 ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Access denied for user ''@'localhost' (using password: NO) java.sql.SQLException: Access denied for user ''@'localhost' (using password: NO) ^^ this error is saying that your user does not have access to mysql. You are going to need to provide a specific user, password, and host, with permissions and grants accordingly. If your user is root, add the username and password to the command. Last but not least, sqoop project is now "in the attic" which means the project is no longer actively getting support and developement from the open source community. I recommend that you learn other techniques to complete the same outcome.
... View more
02-09-2023
05:34 AM
@Techie123 You are going to need to provide credentials for the nifi calls against any s3 bucket with access controls. I would recommend working in lower nifi dev environments and use a public S3 buckets to get comfortable. This will test basics of your flow without access issues. This will also remove confusion around nifi flow functionality vs AWS access issues and help you learn when/where to use the different ways (key, or a controler service w/ credentials) to provide access from nifi to s3.
... View more
02-07-2023
05:02 AM
@Abdulrahmants if you need to talk to someone about getting those added, please reach out in direct message. Another approach could be to create an API input endpoint on nifi (handleHttpRequest/handleHttpResponse), and make a scripted (python,java,etc) process to send the file to the nifi endpoint.
... View more
02-06-2023
07:25 AM
1 Kudo
I believe this is a job for MiNiFi https://nifi.apache.org/minifi/index.html Basically, you create a small minifi flow to run on the server/network with privelaged access, and this flow will send its results to NiFi.
... View more
02-02-2023
06:40 AM
@samrathal Per the docs you need to provide the ID: Request consumes: */* Name Location Type Description id path string The connection id. flowfile-uuid path string The flowfile uuid. clusterNodeId query string The id of the node where the content exists if clustered. You should be getting that ID from the previous get call ( Gets a FlowFile from a Connection ) within the response is an object: FlowFileEntity Inside is the "clusterNodeId" corresponding to which node that flowfile exists on. { "flowFile": { "uri": "value" , "uuid": "value" , "filename": "value" , "position": 0 , "size": 0 , "queuedDuration": 0 , "lineageDuration": 0 , "penaltyExpiresIn": 0 , "clusterNodeId": "value" , "clusterNodeAddress": "value" , "attributes": { "name": "value" } , "contentClaimSection": "value" , "contentClaimContainer": "value" , "contentClaimIdentifier": "value" , "contentClaimOffset": 0 , "contentClaimFileSize": "value" , "contentClaimFileSizeBytes": 0 , "penalized": true } } Those details are exposed in the nifi api doc, just be sure to click into the entity.
... View more
01-23-2023
06:52 AM
@phaelax ExecuteScript ExecuteProcess ExecuteStreamCommand etc are some of the hardest configs in nifi. It is very hard to give guidance without exact templates, configs, scripts, etc. That said, i would recommend ExecuteScript and python over bash. If that is interesting to. you, you should spend some time consuming the 3 part series on ExecuteScript by @MattWho . I believe the first part explains how to get flowfile attributes (filename) or flowfile content from previous processor flowfile (getFile) into the script. https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html https://community.hortonworks.com/articles/75545/executescript-cookbook-part-2.html https://community.hortonworks.com/articles/77739/executescript-cookbook-part-3.html
... View more
01-23-2023
06:43 AM
@prakashrulez This sounds like a job for new nifi relationship feature. This feature allows you to indicate the number of retries before a failure.
... View more
01-23-2023
06:31 AM
@BRinxen First, I feel your pain, as this sandbox was always an issue. Some advice below. Second, i would highly recommend you find a way to do something with hive,spark in another more modern form factor (not old hortonworks sandbox). That said, you are going to need like 32 gb of ram on a very beefy machine to role the whole sandbox even then it will struggle. If you have less resources, you willy only be able to run a few services, not the whole stack. Turn everything else off/maintenance mode. Start yarn, mapreduce, hdfs first. Then begin to start hive. Expect things to take a long time so be patient. Make sure nothing else is running on the main machine.
... View more
01-23-2023
06:22 AM
Very good article, this will definitely help me in the future!
... View more
12-21-2022
05:55 AM
@zIfo based on what i see, you need to completely fill out the keystore and truststore using cacerts. Make sure nifi user has permissions to read file. If your https end point is not a public cert it will not work with cacerts, you will need to make your own keystore and truststore which contains your specific certs. Here is a post with more details: https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/td-p/302719 And another one that goes into much greater detail: https://community.cloudera.com/t5/Support-Questions/RESOLVED-NIFI-LISTENHTTP-SSL/td-p/146985
... View more