Member since
02-07-2019
2690
Posts
235
Kudos Received
30
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1088 | 04-15-2025 10:34 PM | |
3277 | 10-28-2024 12:37 AM | |
1417 | 09-04-2024 07:38 AM | |
3241 | 06-10-2024 10:24 PM | |
1382 | 02-01-2024 10:51 PM |
02-05-2025
06:58 AM
@odst, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @MattWho @Shelton who may be able to assist you further. Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
... View more
01-27-2025
11:06 AM
@vystar Considering the breaking changes that are part of Apache NiFi 2.0/1, there is considerably more work in preparing for an upgrade to the that new major release. So I would recommend upgrading to the latest offering in the Apache NiFi 1.x branch. You'll want to review all the release notes from 1.13 to the latest release: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes/#ReleaseNotes-Version1.12.0 You'll want to pay close attention to any mentions of components being moved to optional build profiles. This means that these nars and the components they contain are no longer include with the Apache NiFi download and if needed must be downloaded form Maven Central and manually added to NiFi. Deprecated components still exist in the Download, but will not exist in NiFi 2.x releases. Make sure to maintain a copy of your flow.xml.gz/flow.json.gz (newer releases). The newer Apache NiFi 1.x load a flow.json.gz instead of the older flow.xml.gz on startup. However, in the absence of a flow.json.gz and the presence of flow.xml.gz, NiFi 1.x will load from the flow.xml.gz and produce the new flow.json.gz. After upgrade, you'll still need to review your dataflows. There are some bad practices that are now blocked by Apache NiFi that may leave some components invalid until manual action is taken to resolve the bad configuration (such as using "primary node" execution on any processor that has an inbound connection). As far as FlowFile distribution, do it early in your dataflows as possible. Utilize list/fetch still processors instead of get style. (Example: use ListSFTP and FetchSFTP in place of GetSFTP. This allows you to load-balance the 0 byte listed files before the content is fetched for the files). Other options like Remote Process Groups can be used (they come with some overhead, but do some target NiFi Cluster load based distribution when dealing with large volumes of FlowFiles. Not so great for low volumes.). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-27-2025
01:18 AM
@ose_gold The SFTP issues appear to stem from incorrect permissions and ownership in your Docker setup. Here's the analysis and solution: Key Issues: Root ownership instead of user 'foo' for /home/foo Incorrect chroot directory permissions Docker volume mount permissions You have a volume nifi-conf:/opt/nifi/nifi-current/conf for the NiFi container, but it's not declared in the volumes section at the bottom check the addition below Docker Compose has newer versions 3.8 It might be a good idea to update to a more recent version depending on the features you need. version: '3' #Docker Compose has newer versions 3.8
services:
nifi:
image: apache/nifi:latest # Consider specifying a version
container_name: nifi
ports:
- "8089:8443"
- "5656:5656"
volumes:
- nifi-conf:/opt/nifi/nifi-current/conf
environment:
NIFI_WEB_PROXY_HOST: localhost:8089
SINGLE_USER_CREDENTIALS_USERNAME: admin
SINGLE_USER_CREDENTIALS_PASSWORD: {xxyourpasswdxx}
sftp:
image: atmoz/sftp
volumes:
- ./sftp/upload:/home/foo/upload
ports:
- "2222:22"
command: foo:pass:1001
# Add these permissions
user: "1001:1001"
environment:
- CHOWN_USERS=foo
- CHOWN_DIRS=/home/foo
volumes:
nifi-conf: Before starting containers # Set correct permissions on host
mkdir -p ./sftp/upload
chown -R 1001:1001 ./sftp/upload
chmod 755 ./sftp/upload This configuration: Sets proper user/group ownership Maintains correct chroot permissions Ensures volume mount permissions are preserved Prevents permission conflicts between host and container Happy Hadooping
... View more
01-24-2025
11:53 AM
Hi @CloudSeeker7 , If you can ask the question with example, it would be helpful to check the issue you faced. Then we can find the possible root causes. Please provide the bad record and good record example you are facing.
... View more
01-23-2025
02:07 AM
@itninja, Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. However, if you still have concerns, could you please provide the information that @MattWho has requested?
... View more
01-23-2025
02:05 AM
@Shw123, Welcome to our community! To help you get the best possible answer, I have tagged our NiFi experts @MattWho @SAMSAL who may be able to assist you further. Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
... View more
01-16-2025
10:21 PM
Hi All, It was issue related to config setting. After setting the environment properly issue got resolved. It seems due to some misconfiguration of JAAS and krb5.conf. Following below blog has resolved the issue http://shantanu01.blogspot.com/2020/06/how-to-setup-beeline-on-linux-for.html
... View more
01-12-2025
04:31 PM
@azrulafiq901 Thank you for bringing this to our community. Did you get help on this? If not, You may also follow: [0a] Managing Licenses: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-managing-licenses.html I highly recommend you discussing potential partnership with our Sales Executives. https://www.cloudera.com/contact-sales.html If you still need help, Please feel free to update this thread, so I can have our team approach you on your behalf. I hope this helps. If this answers your question, You may mark this answer as Accept as Solution. You can also say thanks by clicking on the thumbs up button. V
... View more
01-12-2025
05:32 AM
@VidyaSargur Any update here?
... View more
01-06-2025
04:27 PM
Hi, Did you find any solution?
... View more