Hello community, I'm trying to setup a Nifi cluster with external certifcates (used tinycerts.org) and after setup SSL and LDAP authentication and add my nodes SSL CNs to authorizations.xml via ambari, I have the following message when trying to access to nifi console: Insufficient Permissions log outhome Untrusted proxy CN=node04.nifi.int, OU=Laboratorio, O=Arq de Sistemas, L=Tres Cantos, ST=Madrid, C=ES I have tried what is told in this link https://community.hortonworks.com/questions/80246/nifi-untrusted-proxy.html reading the pkcs12 certificate with keytool and getting the CN of the owner part of the certificate: Alias name: 1 Creation date: Sep 13, 2017 Entry type: PrivateKeyEntry Certificate chain length: 1 Certificate[1]: Owner: CN=node01.nifi.int, OU=Laboratorio, O=Arq de Sistemas, L=Tres Cantos, ST=Madrid, C=ES Issuer: CN=Arq de Sistemas CA, OU=Secure Digital Certificate Signing, O=Arq de Sistemas, L=Tres Cantos, ST=Madrid, C=ES Serial number: 2cbd Valid from: Tue Sep 12 11:14:33 CEST 2017 until: Wed Sep 12 11:14:33 CEST 2018 Even with that I still having the same issue so after a bit of research I found this post https://community.hortonworks.com/questions/110527/nifi-hdf30-untrusted-proxy.html When I remove users.xml and authorizations.xml nifi is not able to create from authorizers.xml, and create an empty ones after that the nifi instances are unable to start and shows this error: 2017-09-13 17:26:47,480 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Error creating bean with name 'niFiWebApiSecurityConfiguration': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void org.apache.nifi.web.NiFiWebApiSecurityConfiguration.setX509AuthenticationProvider(org.apache.nifi.web.security.x509.X509AuthenticationProvider); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'x509AuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Cannot provide an Initial Admin Identity and a Legacy Authorized Users File 2017-09-13 17:26:47,491 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down... SSL works fine with the certificates.. my authorizers.xml is the following: <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <!-- This file lists the authority providers to use when running securely. In order to use a specific provider it must be configured here and it's identifier must be specified in the nifi.properties file. --> <authorizers> <!-- The FileAuthorizer is NiFi"s provided authorizer and has the following properties: - Authorizations File - The file where the FileAuthorizer will store policies. - Users File - The file where the FileAuthorizer will store users and groups. - Initial Admin Identity - The identity of an initial admin user that will be granted access to the UI and given the ability to create additional users, groups, and policies. The value of this property could be a DN when using certificates or LDAP, or a Kerberos principal. This property will only be used when there are no other users, groups, and policies defined. If this property is specified then a Legacy Authorized Users File can not be specified. NOTE: Any identity mapping rules specified in nifi.properties will also be applied to the initial admin identity, so the value should be the unmapped identity. - Legacy Authorized Users File - The full path to an existing authorized-users.xml that will be automatically converted to the new authorizations model. If this property is specified then an Initial Admin Identity can not be specified, and this property will only be used when there are no other users, groups, and policies defined. - Node Identity [unique key] - The identity of a NiFi cluster node. When clustered, a property for each node should be defined, so that every node knows about every other node. If not clustered these properties can be ignored. The name of each property must be unique, for example for a three node cluster: "Node Identity A", "Node Identity B", "Node Identity C" or "Node Identity 1", "Node Identity 2", "Node Identity 3" NOTE: Any identity mapping rules specified in nifi.properties will also be applied to the node identities, so the values should be the unmapped identities (i.e. full DN from a certificate). --> <authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">/var/lib/nifi/conf/authorizations.xml</property> <property name="Users File">/var/lib/nifi/conf/users.xml</property> <property name="Initial Admin Identity">cn=testuser,ou=Users,dc=nifi,dc=int</property> <property name="Legacy Authorized Users File">/root/authorized-users.xml</property> <!-- Provide the identity (typically a DN) of each node when clustered (see tool tip for detailed description of Node Identity). Must be specified when Ranger Nifi plugin will not be used for authorization. --> <property name="Node Identity 1">CN=node01.nifi.int, OU=Laboratorio, O=Arq de Sistemas, L=Tres Cantos, ST=Madrid, C=ES</property> <property name="Node Identity 2">CN=node03.nifi.int, OU=Laboratorio, O=Arq de Sistemas, L=Tres Cantos, ST=Madrid, C=ES</property> <property name="Node Identity 3">CN=node04.nifi.int, OU=Laboratorio, O=Arq de Sistemas, L=Tres Cantos, ST=Madrid, C=ES</property> </authorizer> </authorizers> Do you know what maybe happening? Thank you in advance. Best regards. ... View more

Hello, I'm doing some test with a nifi cluster (HDF 3), and I wanted to configure ldap as authentication service, I've not configured SSL yet but I would like to test the ldap authentication. But when I try to access the cluster it directly logs me as anonymous and I can see the flows without any login screen. My configuration is the following login-identity-providers.xml <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Manager,dc=nifi,dc=int</property> <property encryption="aes/gcm/256" name="Manager Password">mIV4TPuSpfOGzd3E||FZnVyewmvoWGEmf1sF5cCTCy4tztrwo</property> <property name="TLS - Keystore"/> <property name="TLS - Keystore Password"/> <property name="TLS - Keystore Type"/> <property name="TLS - Truststore"/> <property name="TLS - Truststore Password"/> <property name="TLS - Truststore Type"/> <property name="TLS - Client Auth"/> <property name="TLS - Protocol"/> <property name="TLS - Shutdown Gracefully"/> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://node03.nifi.int:389</property> <property name="User Search Base">ou=Users,dc=nifi,dc=int</property> <property name="User Search Filter">uid={0}</property> <property name="Authentication Expiration">12 hours</property> </provider> nifi.properties: # Generated by Apache Ambari. Tue Sep 12 12:27:33 2017 nifi.administrative.yield.duration=30 sec nifi.authorizer.configuration.file=/usr/hdf/current/nifi/conf/authorizers.xml nifi.bored.yield.duration=10 millis nifi.cluster.flow.election.max.candidates=3 nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.is.node=true nifi.cluster.node.address=node01.nifi.int nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.event.history.size=25 nifi.cluster.node.protocol.max.threads= nifi.cluster.node.protocol.port=9088 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.read.timeout=5 sec nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.is.secure=False nifi.components.status.repository.buffer.size=1440 nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository nifi.components.status.snapshot.frequency=1 min nifi.content.claim.max.appendable.size=10 MB nifi.content.claim.max.flow.files=100 nifi.content.repository.always.sync=false nifi.content.repository.archive.enabled=true nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.directory.default=/var/lib/nifi/content_repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.viewer.url=/nifi-content-viewer/ nifi.database.directory=/var/lib/nifi/database_repository nifi.documentation.working.directory=/var/lib/nifi/work/docs/components nifi.flow.configuration.archive.dir=/var/lib/nifi/archive/ nifi.flow.configuration.archive.enabled=true nifi.flow.configuration.archive.max.count= nifi.flow.configuration.archive.max.storage=500 MB nifi.flow.configuration.archive.max.time=30 days nifi.flow.configuration.file=/var/lib/nifi/conf/flow.xml.gz nifi.flowcontroller.autoResumeState=true nifi.flowcontroller.graceful.shutdown.period=10 sec nifi.flowfile.repository.always.sync=false nifi.flowfile.repository.checkpoint.interval=2 mins nifi.flowfile.repository.directory=/var/lib/nifi/flowfile_repository nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository nifi.flowfile.repository.partitions=256 nifi.flowservice.writedelay.interval=500 ms nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE nifi.kerberos.krb5.file= nifi.kerberos.service.keytab.location= nifi.kerberos.service.principal= nifi.kerberos.spnego.authentication.expiration=12 hours nifi.kerberos.spnego.keytab.location= nifi.kerberos.spnego.principal= nifi.login.identity.provider.configuration.file=/usr/hdf/current/nifi/conf/login-identity-providers.xml nifi.nar.library.directory=/usr/hdf/current/nifi/lib nifi.nar.working.directory=/var/lib/nifi/work/nar nifi.provenance.repository.always.sync=false nifi.provenance.repository.buffer.size=100000 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.debug.frequency=1_000_000 nifi.provenance.repository.directory.default=/var/lib/nifi/provenance_repository nifi.provenance.repository.encryption.key= nifi.provenance.repository.encryption.key.id= nifi.provenance.repository.encryption.key.provider.implementation= nifi.provenance.repository.encryption.key.provider.location= nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository nifi.provenance.repository.index.shard.size=500 MB nifi.provenance.repository.index.threads=1 nifi.provenance.repository.indexed.attributes= nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship nifi.provenance.repository.journal.count=16 nifi.provenance.repository.max.attribute.length=65536 nifi.provenance.repository.max.storage.size=1 GB nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.query.threads=2 nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.rollover.time=30 secs nifi.queue.swap.threshold=20000 nifi.remote.input.host= nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.input.secure=False nifi.remote.input.socket.port= nifi.security.identity.mapping.pattern.dn= nifi.security.identity.mapping.pattern.kerb= nifi.security.identity.mapping.value.dn= nifi.security.identity.mapping.value.kerb= nifi.security.keyPasswd= nifi.security.keystore=/usr/hdf/current/nifi/conf/keystore.jks nifi.security.keystorePasswd= nifi.security.keystoreType=jks nifi.security.needClientAuth=False nifi.security.ocsp.responder.certificate= nifi.security.ocsp.responder.url= nifi.security.truststore=/usr/hdf/current/nifi/conf/truststore.jks nifi.security.truststorePasswd= nifi.security.truststoreType=jks nifi.security.user.authorizer=file-provider nifi.security.user.login.identity.provider=ldap-provider nifi.sensitive.props.additional.keys= nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL nifi.sensitive.props.key=wSdxEcJ0QRZGwFfr||CVtSGQsYIUSOXzAQEQBvu+IQFiwFpM/ZldwZgA nifi.sensitive.props.key.protected=aes/gcm/256 nifi.sensitive.props.provider=BC nifi.state.management.configuration.file=/usr/hdf/current/nifi/conf/state-management.xml nifi.state.management.embedded.zookeeper.properties=/usr/hdf/current/nifi/conf/zookeeper.properties nifi.state.management.embedded.zookeeper.start=false nifi.state.management.provider.cluster=zk-provider nifi.state.management.provider.local=local-provider nifi.swap.in.period=5 sec nifi.swap.in.threads=1 nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager nifi.swap.out.period=5 sec nifi.swap.out.threads=4 nifi.templates.directory=/var/lib/nifi/templates nifi.ui.autorefresh.interval=30 sec nifi.ui.banner.text= nifi.variable.registry.properties= nifi.version=1.2.0.3.0.1.0-43 nifi.web.http.host=node01.nifi.int nifi.web.http.network.interface.default= nifi.web.http.port=9090 nifi.web.https.host= nifi.web.https.network.interface.default= nifi.web.https.port= nifi.web.jetty.threads=200 nifi.web.jetty.working.directory=/var/lib/nifi/work/jetty nifi.web.war.directory=/usr/hdf/current/nifi/lib nifi.zookeeper.connect.string=node02.nifi.int:2181,node01.nifi.int:2181,node03.nifi.int:2181 nifi.zookeeper.connect.timeout=3 secs nifi.zookeeper.root.node=/nifi nifi.zookeeper.session.timeout=3 secs Do you have any idea about what is happening? Thank you in advance. ... View more

Hello, For some reason after enable kerberos security, ambari have changed my datanode ports to 1019, anyone know why is this happening? Because of this for some reason now my namenode is detecting the blocks of a full datanode as underreplicated and is replicated them. ... View more

Hello community, I've a cluster secured with one way trust relationship with an AD, before enable security in the cluster I was able to excute spark via oozie using a shell-action. Is there a way to keep doing it without have to propagate my keytab in every nodemamanager? I've seen that for hive from shell-action you can pass the HADOOP_TOKEN_FILE_LOCATION variable to use it, can I do something similar with spark? if not what alternatives do I have? The problem with the keytab is that I've to change the password every moth so I would have to copy the keytab everytime it changes... Thank you in advance. ... View more

Hello community, I'have a cluster with kerberos and after a restart I'm having the following error when trying to reach the ATS <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title> </head> <body><h2>HTTP ERROR 403</h2> <p>Problem accessing /applicationhistory. Reason: <pre> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr /><i><small>Powered by Jetty://ll></i><br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> </body> </html> I've tried with diferent principals but the issue persist. The resource manager and the rest of the spnego authenticated web consoles still working properly. Any idea about what is going on? Thank you in advance ... View more