Member since
02-04-2025
14
Posts
0
Kudos Received
0
Solutions
03-07-2025
06:58 AM
Thank you for the very detailed reply! Just some quick notes. When i bump up the batch size to 50, it causes an error with the PutUDP. "Caused by: java.io.IOException: A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram into was smaller than the datagram itself" When i tried bumping up the concurrent threads from 1 to 2, it caused the video to be extremely blurry. And this is what i am using to test. ffmpeg.exe -stream_loop -10000 -re -i %FILE% -map 0:0 -map 0:1 -c copy -f mpegts udp://224.1.1.2:10000 And then PutUDP ouputs to this: ffplay -i udp://localhost:8090 You can test most of this by testing: ListenUDP -> PutUDP and then blurriness and packets drop occur when doing just 2: ListenUDP -> PutUDP ListenUDP -> PutUDP
... View more
03-06-2025
02:26 PM
I am now running into another problem. Say i want to have 5 different sources getting fed into 5 different ListenUDP and going to PutUDP. 1 works fine. As soon as i add 2 or more, the source starts to lose packets. E.g ListenUDP -> PutUDP (works fine) _________________ ListenUDP -> PutUDP ListenUDP -> PutUDP ListenUDP -> PutUDP ListenUDP -> PutUDP This scenario does not work. Say for a video feed, the video feed becomes very blurry. If i increase the size of the batch size from 1 to 30, this problem is mostly fixed, but i still see some packets being dropped. Anyway of resolving this? @MattWho
... View more
03-06-2025
02:22 PM
I wanted to give an update on my findings, finally able to get multicast to work. I also had to comment out this line: datagramChannel.connect(new InetSocketAddress(sendingHost, sendingPort)); @Override public void open(final InetAddress nicAddress, final int port, final int maxBufferSize) throws IOException { stopped = false; // Use INET protocol family for IPv4 multicast if (isMulticast) { datagramChannel = DatagramChannel.open(StandardProtocolFamily.INET); } else { datagramChannel = DatagramChannel.open(); } datagramChannel.configureBlocking(false); if (maxBufferSize > 0) { datagramChannel.setOption(StandardSocketOptions.SO_RCVBUF, maxBufferSize); final int actualReceiveBufSize = datagramChannel.getOption(StandardSocketOptions.SO_RCVBUF); if (actualReceiveBufSize < maxBufferSize) { logger.warn("Attempted to set Socket Buffer Size to {} bytes but could only set to {} bytes. You may want to consider changing the Operating System's maximum receive buffer", maxBufferSize, actualReceiveBufSize); } } // we don't have to worry about nicAddress being null here because InetSocketAddress already handles it datagramChannel.setOption(StandardSocketOptions.SO_REUSEADDR, true); datagramChannel.socket().bind(new InetSocketAddress(port)); // if a sending host and port were provided then connect to that specific address to only receive // datagrams from that host/port, otherwise we can receive datagrams from any host/port if (sendingHost != null && sendingPort != null) { // This seems to not allow the multicast to work. Therefore, I commented it out. // datagramChannel.connect(new InetSocketAddress(sendingHost, sendingPort)); } selector = Selector.open(); datagramChannel.register(selector, SelectionKey.OP_READ); // Join multicast group if specified if (isMulticast) { InetAddress group = InetAddress.getByName(multicastGroup); // Determine which network interface to use for multicast NetworkInterface networkInterface; if (multicastInterface != null && !multicastInterface.isEmpty()) { // Use specified interface networkInterface = NetworkInterface.getByName(multicastInterface); if (networkInterface == null) { // Try as an IP address networkInterface = NetworkInterface.getByInetAddress(InetAddress.getByName(multicastInterface)); } } else { // Use the NIC address interface if none specified networkInterface = NetworkInterface.getByInetAddress(nicAddress); } if (networkInterface == null) { throw new IOException("Could not find network interface for multicast"); } // Join the multicast group on the selected interface membershipKey = datagramChannel.join(group, networkInterface); logger.info("Joined multicast group {} on interface {}", multicastGroup, networkInterface.getDisplayName()); } }
... View more
02-12-2025
11:51 AM
hey @MattWho My nifi instance and little python script run on local host. My python script sends a packet as ip address 224.1.1.1 on port 10000. When i configure my processor, omitting the sending host and sending port, it still doesn't receive the packet from 224.1.1.1. If i omit the send host and port, and just have 10000 for port. It also doesn't work, since that assumes the packet is being sent from 127.0.0.1 (localhost). I removed the site to site properties like you recommended. Essentially i am just trying to send a multicast message to nifi, and trying to figure out how to configure the current processor to do that. Or I will just try and create a custom udp listener to handle multicast messages. Thanks!
... View more
02-11-2025
12:40 PM
Trying to demonstrate that nifi can listen to multicast traffic. I have this python script: import socket import struct import sys # Multicast group details multicast_group = '224.1.1.1' server_address = ('', 10000) # Create a UDP socket sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # Set the time-to-live for multicast packets ttl = struct.pack('b', 1) sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, ttl) try: # Send data to the multicast group message = b'Hello, multicast!' print(f"sending {message} to {multicast_group}:{server_address[1]}") sent = sock.sendto(message, (multicast_group, server_address[1])) except Exception as e: print(f"Error occurred: {e}") finally: sock.close() And then my listenudp processor is: I can receive the message if I change the multicast_group to 127.0.0.1. How do I get it to receive it with 224.1.1.1. I also added this to my nifi properties # Site to Site properties nifi.remote.input.host=224.1.1.1 nifi.remote.input.secure=true nifi.remote.input.socket.port=10000 @MattWho
... View more
Labels:
- Labels:
-
Apache NiFi
02-07-2025
10:44 AM
@MattWho I have amazing news to report. I decided to try to do everything outside of docker containers, and was successfully able to get it to work. Using the exact configs as i pasted above. Still would like to figure out why it wasn't able to work with docker. nifi.security.user.saml.idp.metadata.url=http://localhost:8080/realms/master/protocol/saml/descriptor Maybe localhost has to be replaced with some other hostname
... View more
02-07-2025
06:42 AM
@MattWho Caused by: org.apache.nifi.web.client.api.WebClientServiceException: Request execution failed HTTP Method [GET] URI [http://localhost:8080/realms/master/protocol/saml/descriptor] using these settings: nifi.security.user.saml.idp.metadata.url=http://localhost:8080/realms/master/protocol/saml/descriptor nifi.security.user.saml.sp.entity.id=org:apache:nifi:saml:sp nifi.security.user.saml.identity.attribute.name= nifi.security.user.saml.group.attribute.name= nifi.security.user.saml.request.signing.enabled=false nifi.security.user.saml.want.assertions.signed=true nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256 nifi.security.user.saml.authentication.expiration=12 hours nifi.security.user.saml.single.logout.enabled=false nifi.security.user.saml.http.client.truststore.strategy=JDK nifi.security.user.saml.http.client.connect.timeout=30 secs nifi.security.user.saml.http.client.read.timeout=30 secs Following instructions from here, except i am using the latest version of keycloak Apache NiFi SAML Authentication with Keycloak Here is what the url looks like in my browser:
... View more
Labels:
- Labels:
-
Apache NiFi
02-06-2025
02:21 PM
Thank you for all of your feedback so far. I have put LDAP on the back burner for now and will try that again soon. Im trying to see if i can get it to work with Keycloak, but i am getting this error: Caused by: org.apache.nifi.web.client.api.WebClientServiceException: Request execution failed HTTP Method [GET] URI [http://localhost:8080/realms/master/protocol/saml/descriptor] using these settings: nifi.security.user.saml.idp.metadata.url=http://localhost:8080/realms/master/protocol/saml/descriptor nifi.security.user.saml.sp.entity.id=org:apache:nifi:saml:sp nifi.security.user.saml.identity.attribute.name= nifi.security.user.saml.group.attribute.name= nifi.security.user.saml.request.signing.enabled=false nifi.security.user.saml.want.assertions.signed=true nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256 nifi.security.user.saml.authentication.expiration=12 hours nifi.security.user.saml.single.logout.enabled=false nifi.security.user.saml.http.client.truststore.strategy=JDK nifi.security.user.saml.http.client.connect.timeout=30 secs nifi.security.user.saml.http.client.read.timeout=30 secs Following instructions from here, except i am using the latest version of keycloak Apache NiFi SAML Authentication with Keycloak
... View more
02-05-2025
09:24 AM
I changed my setting to use LDAPS. I have now changed it to ANONYMOUS. I get this error code: Caused by: javax.naming.NoPermissionException: [LDAP: error code 50 - No user currently bound] 2025-02-05 09:20:33 at java.naming/com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3268) login-identity-providers.xml <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">ANONYMOUS</property> <property name="Manager DN">cn=localhost</property> <property name="Manager Password">password</property> <property name="TLS - Keystore">./conf/keystore.p12</property> <property name="TLS - Keystore Password">password</property> <property name="TLS - Keystore Type">PKCS12</property> <property name="TLS - Truststore">./conf/truststore.p12</property> <property name="TLS - Truststore Password">password</property> <property name="TLS - Truststore Type">PKCS12</property> <property name="TLS - Client Auth">NONE</property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ldap:3890</property> <property name="User Search Base">cn=users</property> <property name="User Search Filter">sAMAccountName={0}</property> <!--<property name="User Search Filter">cn={0}</property>--> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> </provider> authorizers.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">cn=localhost,ou=ngc</property> <property name="Initial User Identity 2">CN=localhost,O=ngc,L=San Diego,ST=California,C=US</property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">ANONYMOUS</property> <property name="Manager DN">cn=localhost</property> <property name="Manager Password">password</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ldap:3890</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">CN=users</property> <property name="User Object Class">person</property> <property name="User Search Scope">ONE_LEVEL</property> <!--<property name="User Search Filter">(uid=*)</property> <property name="User Identity Attribute">sAMAccountName</property>--> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base"></property> <property name="Group Object Class">group</property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute"></property> <property name="Group Member Attribute"></property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <userGroupProvider> <identifier>composite-configurable-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">composite-configurable-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">cn=localhost,ou=ngc</property> <property name="Legacy Authorized Users File"></property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers>
... View more
02-04-2025
01:21 PM
is there a tls toolkit command in version 2.1? ./bin/tls-toolkit.sh standalone -h
... View more