Dear All, I am encountering issue with LDAP integration. I have completed LDAP (ldap-provider) and Certificate configurations according to documentation. I added IU certificate in NiFi (keystore, truststore etc.) and configured other pieces of the MS AD LDAP integration (authorizers.xml, login-identity-providers.xml and nifi.properties). I logged on NiFi from HTTPS UI with initial admin (admin1) and assigned the policies one of the LDAP users (nifiadmin) which is located on MS AD LDAP. I checked LDAP user (nifiadmin) from NiFi UI it is exist in the NiFi. It seems Ok. I added all screenshots (nifi_policies.jpg) about that. When I try to login initial admin (admin1) there is no error: nifi-user.log: 2020-08-13 10:46:43,544 INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Thu Aug 13 10:46:43 MSK 2020 2020-08-13 10:46:43,684 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Thu Aug 13 10:46:43 MSK 2020 2020-08-13 11:21:28,051 INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifiportal.abc.example.com/nifi-api/flow/current-user (source ip: 10.0.2.15) 2020-08-13 11:21:28,062 INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for admin1 2020-08-13 11:21:28,167 INFO [NiFi Web Server-118] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifiportal.abc.example.com/nifi-api/flow/client-id (source ip: 10.0.2.15) 2020-08-13 11:21:28,170 INFO [NiFi Web Server-118] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for admin1 2020-08-13 11:21:28,170 INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifiportal.abc.example.com/nifi-api/flow/config (source ip: 10.0.2.15) 2020-08-13 11:21:28,179 INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for admin1 2020-08-13 11:21:28,206 INFO [NiFi Web Server-118] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifiportal.abc.example.com/nifi-api/flow/banners (source ip: 10.0.2.15)       6. But, when I try to login with LDAP User (nifiadmin) who was already assigned NiFi UI access by me I am getting permission error. I added all screenshots (nifi_policies.jpg) about that: nifi-user.log: 2020-08-13 11:51:52,255 INFO [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifiportal.abc.example.com/nifi-api/flow/current-user (source ip: 10.0.2.15) 2020-08-13 11:51:52,258 INFO [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for nifiadmin 2020-08-13 11:51:52,260 INFO [NiFi Web Server-16] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[nifiadmin], groups[] does not have permission to access the requested resource. Unknown user with identity 'nifiadmin'. Returning Forbidden response.       7. When I check the nifi-app.log there is no error: nifi-app.log: 2020-08-13 10:46:52,310 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@1b8354aa(fa3f2599-3d3b-43c9-9e7a-ea26375d4470,h=[nifiportal.abc.example.com],w=[]) for SslContextFactory@378a5302[provider=null,keyStore=file:///C:/nifi/certificates/private-keystore1,trus tStore=file:///C:/nifi/certificates/public-keystore1] 2020-08-13 10:46:52,325 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@2794eab6{SSL,[ssl, http/1.1]}{nifiportal.abc.example.com:443} 2020-08-13 10:46:52,325 INFO [main] org.eclipse.jetty.server.Server Started @31030ms 2020-08-13 10:46:52,419 INFO [main] org.apache.nifi.nar.NarAutoLoader Starting NAR Auto-Loader for directory .\extensions ... 2020-08-13 10:46:52,419 INFO [main] org.apache.nifi.nar.NarAutoLoader NAR Auto-Loader started 2020-08-13 10:46:52,419 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs: 2020-08-13 10:46:52,419 INFO [main] org.apache.nifi.web.server.JettyServer https://nifiportal.abc.example.com:443/nifi       8. What I did for solving the problem I deleted user.xml and authorizations.xml several times. Nifi creates automatically them but problem is still continue. I tried different kind of the configurations in the related files (authorizers.xml, login-identity-providers.xml and nifi.properties). But no change I also tried another LDAP user than nifiadmin (admin2) but there is no any solution for ldap user login issue I added all configuration files (authorizations, authorizers, login-identity-providers, nifi.properties and users) with jpeg format. I also added screenshots (nifi_policies.jpg) about access and user policies. My environment details are below: Apache NiFi 1.11.3 (single, not cluster) Windows Server 2016 Java JRE 1.8.0_251 (64 Bit) MS Active Directory 2016 for LDAP   Do you have any comment or idea?   nifi_policies authorizations.xml authorizers-1.xml authorizers-2.xml authorizers-3.xml login-identity-providers.xml nifi.properties users.xml ... View more

Dear All, I need your suggestion huge number data transfer to Syslog through Apache NiFi My aim is transferring all IIS logs (located on Azure Blob Storage) to syslog line by line. Therefore I am using split text for parsing lines. Extract Text is transferring split line on the attribute, in this way I can say to syslog processor "Message Body: IISHttp${msg}". I will explain details at the below Actually my problem is bottleneck on the Extract Text. I have to transfer IIS Logs near-real time (less than 1 hour delay). But Extract Text isn't draining the messages (flow files) from the queue in proper time. I tried increasing Thread Number, Run Duration, increasing/reducing Queue size but I couldn't achieve my target. The queue between split text and extract text are always full. I have log gap about 13 hours. I am trying find a way for that My flow is that; 1. Getting IIS Log files from Azure Blob Storage 1.1. List Azure Blob Storage Processor 1.2. Route on Attribute Processor (I have date filter RegEx on it) 1.3. Fetch Azure Blob Storage Processor 2. Split Text Processor: Splitting each IIS Log File to line by line with Split Text Processor. 2.1. Line Split Count: 1 2.2. Maximum Fragment Size: No value set 2.3. Header Line Count: 0 2.4. Header Line Marker Characters: No value set 2.5. Remove Trailing Newlines: True 3. Extract Text Processor: Transferring new flow files which is produced by Split Text Processor to Extract Text Processor. Extract Text Processor is the problematic point 3.1. All Properties are Default 3.2. I added one RegEx in the Properties. I would like to carry on Flow Files attributes to Syslog 3.2.1. Property Name: msg 3.2.2. Value: (.*) 4. Put Syslog Processor: Transferring all flow files where is coming from Extract Text to Put Syslog Processor. 4.1. All Properties are Default or configured properly for requirements (such as IP address of the Syslog, port etc.) 4.2. Message Body: IISHttp${msg} 4.3. There is no flow file waiting on the Put Syslog Processor queue (between extract text and put syslog).   I tried those options also; 1. Route Text Processor usage instead of Extract Text. But I failed 1.1. Removed Extract Text Processor 1.2. Added Route Text but I didn't accomplish to transfer line by line to syslog.   2. Put TCP Processor usage instead of Put Syslog. But I couldn't complete the configuration my knowledge is not enough for that. 2.1. Removed Split Text Processor 2.2. Removed Extract Text Processor 2.3. Removed Syslog Processor  2.4. Added Put TCP Processor 2.4.1. Hostname: Syslog Server 2.4.2. Port: Syslog Server Port (TCP) 2.4.3. Outgoing Message Delimiter: \n (for splitting each line from entire IIS Log file. I want to have just 1 line to syslog transfer for each time) 2.4.4. SSL Context Service --> StandardRestrictedSSLContextService (configuring for mutual authentication) 2.4.5. Rest of the Properties are default 2.5. I need to add some prefixes to each line which is produced by \n delimiter for Syslog Server. How will I do these? 2.5.1. Each Line should be begin these prefixes: 2.5.1.1. Message Timestamp: ${now():format('MMM d HH:mm:ss')} --- Default Property of Put Syslog 2.5.1.2. Message Hostname: ${hostname(true)} --- Default Property of Put Syslog 2.5.2. After these two prefix I need to append IISHttp (Message Body: IISHttp ${msg}) wording. 2.5.3. I don't want to rewrite or completely replace each line. I just want to append some things 2.5.3.1. For Example: My IIS Log File line like this: 2020-03-13 13:59:19 XXX-YYY GET /Maintenance/Status.svc X-ARR-LOG-ID=267ed22c-f1b 200 0 0 1005 1086 46 My line will be like this: Jun 26 23:29:09 SERVER1 IISHttp 2020-03-13 13:59:19 XXX-YYY GET /Maintenance/Status.svc X-ARR-LOG-ID=267ed22c-f1b 200 0 0 1005 1086 46   I guess, Put TCP Processor is better way but I can't continue to configuration due to lack of my knowledge Do you have any comment or suggestion?   My environment details are below: Apache NiFi 1.11.3 (Single Node, not cluster) Windows Server 2016 Java JRE 1.8.0_241 (64 Bit) ... View more