Hello, I have a kerberos-enabled cluster and trying to enable SASL/PLAIN as well on the same broker. SASL (GSSAPI) works fine. These are the steps i took: 1) Added PlainLoginModule to kafka_jaas.conf (all other sections already there due to kerberos) KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="{{kafka_keytab_path}}" storeKey=true useTicketCache=false serviceName="{{kafka_bare_jaas_principal}}" principal="{{kafka_jaas_principal}}"; org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret" user_alice="alice-secret"; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="{{kafka_bare_jaas_principal}}"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="{{kafka_keytab_path}}" storeKey=true useTicketCache=false serviceName="zookeeper" principal="{{kafka_jaas_principal}}"; }; I've also validated, -Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf is being loaded (ps -ef | grep kafka_jaas.conf) 2) Created a kafka_plain_jaas_client.conf KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required username="alice" password="alice-secret"; }; 3) Update to server.properties sasl.enabled.mechanisms=GSSAPI,PLAIN advertised.listeners=PLAINTEXTSASL://ip-123-0-0-12.ec2.internal:6667 4) Producer.proerties security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN 5) Restarted Kafka When I use the old kafka_client_jaas that references com.sun.security.auth.module.Krb5LoginModule, everything still works but using the new client_jaas with plainLoginModule I get: kafka@ip-170-0-0-12:/usr/hdp/current/kafka-broker/bin$ /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list ip-170-0-0-12.ec2.internal:6667 --topic ssl_plain_test -producer.config /usr/hdp/current/kafka-broker/conf/producer.properties --security-protocol PLAINTEXTSASL d [2017-09-06 18:13:56,982] WARN Error while fetching metadata with correlation id 0 : {ssl_plain_test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-09-06 18:13:57,183] WARN Error while fetching metadata with correlation id 1 : {ssl_plain_test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-09-06 18:13:57,284] WARN Error while fetching metadata with correlation id 2 : {ssl_plain_test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-09-06 18:13:57,385] WARN Error while fetching metadata with correlation id 3 : {ssl_plain_test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-09-06 18:13:57,485] WARN Error while fetching metadata with correlation id 4 : {ssl_plain_test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) I edited: /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh to point to my client_jaas: export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_plain_jaas_client.conf" Any ideas? Thanks! ... View more

Hi - Is it possible to setup a Hierarchy of tags that can be searchable (either API or UI) through the parent tag? If for example I have 4 tags, company1,company2,company3,company4 and a parent tag of vendors. Is there a way for me to search vendors and it to come up with the tag list of company1-4? I have created tags that inheirt from others but cant see this relationship in the UI ... View more

Hi - upgraded from HDF 2.x -> 3.0 (ran into issue with upgrade so wiped everything 2.x related and installed 3). Getting the below error in app log during startup. Everything done through Ambari 2017-06-29 20:11:31,391 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Error creating bean with name 'niFiWebApiSecurityConfiguration': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void org.apache.nifi.web.NiFiWebApiSecurityConfiguration.setJwtAuthenticationProvider(org.apache.nifi.web.security.jwt.JwtAuthenticationProvider); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'jwtService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtService' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'keyService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyService' defined in class path resource [nifi-administration-context.xml]: Cannot resolve reference to bean 'keyTransactionBuilder' while setting bean property 'transactionBuilder'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyTransactionBuilder' defined in class path resource [nifi-administration-context.xml]: Cannot resolve reference to bean 'keyDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyDataSource': FactoryBean threw exception on object creation; nested exception is org.h2.jdbc.JdbcSQLException: Error while creating file "/data/1/nifi/database_repository_rock" [90062-176] I do see: 2017-06-29 20:11:10,586 - Creating directory Directory['/data/1/nifi/database_repository'] since it doesn't exist. 2017-06-29 20:11:10,586 - Changing owner for /data/1/nifi/database_repository_rock from 0 to nifi 2017-06-29 20:11:10,586 - Changing group for /data/1/nifi/database_repository_rock from 0 to nifi Permissions and everything looks to be set correctly. Have tried completely removing /data/1/* and clearing other remnants of previous install. any ideas or places I should look? Only happens ... View more

Hello, When i run hive commands, ranger audit is picking up my user name with Capitals e.g "John.Doe". When I do HDFS Commands, its lower case "john.doe" My Principal is: John.Doe@CORP.AD and we have auth-to-local rules to convert this to all lower case. (john.doe) In ranger we are also doing ranger.user.sync case conversion to lower so if we use user policies, only hdfs will work (e.g. i appear as john.doe in users and since Hive comes in as "John.Doe" user policies dont get applied).  Example: CREATE TABLE test.permtest (field1 int); - the location of this folder is /data/2017 [john.doe@edge1 ~]$ hdfs dfs -ls /data/2017/ drwxr-xr-x - John.Doe hdfs 0 2017-05-02 20:43 /data/2017/permtest As you can see from the above, the table gets created with the ACL permissions as John.Doe. ------- Now when I do HDFS commands, e.g. it comes up as expected (john.doe - lower case) [john.doe@edge1 ~]$ hdfs dfs -mkdir /data/2017/permtest1 drwxr-xr-x - John.Doe hdfs 0 2017-05-02 20:43 /data/2017/permtest drwxr-xr-x - john.doe hdfs 0 2017-05-02 20:44 /data/2017/permtest1 The John.Doe and john.doe is what gets passed to ranger for authorization and this is a problem since user ranger sync brings over "john.doe" and so any Hive policies wont work. Any ideas? ... View more

Hi I have the processors unpackContent -> MergeContent. I use this to untar a file and then zip the files. I am using the defragment merge strategy and have been noticing that when MergeContent has to handle many flowfiles at once (flowfile queue builds up before MergeContent) from many different fragments I get "Expected number of fragments is X but only getting Y". Simply routing failures back to merge content or creating a run schedule delay helped solve this but wondering why this would be happening. Thanks, ... View more

Hello, I have secured HDP cluster and a nifi flow that ingests JSON and inserts it into a Phoenix Table Getfile->ConvertJsonToSQL->ReplaceText(insert to update)->PutSQL. Prior to enabling kerberos the flow was working fine. After enabling kerberos, I changed the connection pool to: jdbc:phoenix:localhost:2181:/hbase-secure:hbase-dev@DEV.COM:/etc/security/keytabs/hbase.headless.keytab This connection URL works fine with sqlline/phoenix client Now when I start the flow, it initially hangs for a while and I end up with the below logs Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: Thu Jan 26 06:36:52 UTC 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68021: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=ip-123-431-1-123.ec2.internal,16020,1485391803237, seqNum=0 at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) ~[na:na] at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) ~[na:na] at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[na:na] at org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:231) ~[na:na] ... 18 common frames omitted Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68130: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=ip-172-40-1-51.ec2.internal,16020,1485391803237, seqNum=0 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159) ~[na:na] at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) ~[na:na] ... 3 common frames omitted Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to ip-172-40-1-51.ec2.internal/172.40.1.51:16020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to ip-172-40-1-51.ec2.internal/172.40.1.51:16020 is closing. Call id=9, waitTime=3 at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1258) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229) ~[na:na] at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ~[na:na] at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) ~[na:na] at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32741) ~[na:na] at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:373) ~[na:na] at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:200) ~[na:na] at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) ~[na:na] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[na:na] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364) ~[na:na] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338) ~[na:na] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ~[na:na] ... 4 common frames omitted Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to ip-172-40-1-51.ec2.internal/172.40.1.51:16020 is closing. Call id=9, waitTime=3 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1047) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:846) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:574) ~[na:na] Any ideas on what I could be missing? Similar behaviour with executeSQL. Thanks! ... View more

Hello, After enabling HA on the name node, hive is unable to access the hive databases whose metastore name got updated to the new FS path: Error: Error while compiling statement: FAILED: SemanticException java.lang.IllegalArgumentException: java.net.UnknownHostException: cluster1 (state=42000,code=40000) hive metastore tool shows: hive --service metatool -listFSRootListing FS Roots.. hdfs://cluster1/apps/hive/warehouse/test2.db hdfs://cluster1/apps/hive/warehouse/raw.db hdfs://cluster1/apps/hive/warehouse/test.db hdfs://cluster1/apps/hive/warehouse hdfs://cluster1/apps/hive/warehouse/lookup.db cluster1 is the correct fs.defaultFS that was setup during HA. <property> <name>fs.defaultFS</name> <value>hdfs://cluster1</value> <final>true</final> </property> If I create a new database in hive, it gets created using the actual name node host name: 0: jdbc:hive2://ip-10-555-2-555.ec2.internal:> create database test3; No rows affected (0.148 seconds) Listing FS Roots.. hdfs://ip-10-123-5-42.ec2.internal:8020/apps/hive/warehouse/test3.db any ideas on what could be missing? hdfs client works fine with the HA name. ... View more

Hello, I am trying to access hive JDBC through Knox in a secured cluster (kerberos). When accessing them directly, it works fine. I am able to connect with hiveserver directly in HTTP mode passing in kerbeos principal and creating kerberos ticket beeline -u 'jdbc:hive2:/<hive_server>:10001/;transportMode=http;httpPath=cliservice;principal=hive/_HOST@DEV.COM' and access WEBHDFS fine directly connecting to namenode: curl -i --negotiate -u : 'http://<namenode>:50070/webhdfs/v1/?op=LISTSTATUS' Going through Knox gateway (using sample LDAP for simplicity), I get: curl -iku guest:guest-password -X GET 'https://<knox_gateway>:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS' <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 401 Authentication required</title> </head><body><h2>HTTP ERROR 401</h2> <p>Problem accessing /webhdfs/v1/. Reason:<pre> Authentication required</pre></p><hr/><i><small>Powered by Jetty://</small></i><br/> In gateway-audit i do see the request getting translated to the actual internal request but its returning 401. audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable|Request method: GET audit|WEBHDFS|guest|||authentication|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success| audit|WEBHDFS|guest|||authentication|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Groups: [] audit|WEBHDFS|guest|||authorization|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success| audit|WEBHDFS|guest|||dispatch|uri|http://<name_node>:50070/webhdfs/v1/?op=LISTSTATUS&doAs=guest|unavailable|Request method: GET audit|WEBHDFS|guest|||dispatch|uri|http://<name_node>:50070/webhdfs/v1/?op=LISTSTATUS&doAs=guest|success|Response status: 401 audit|WEBHDFS|guest|||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response status: 401 Similarly in Hive, I can connect to hiveserver directly but when I attempt through knox I get: 16/10/04 22:31:34 [main]: ERROR jdbc.HiveConnection: Error opening sessionorg.apache.thrift.transport.TTransportException: HTTP Response code: 401 In Hive server logs: 2016-10-04 22:31:34,063 INFO [HiveServer2-HttpHandler-Pool: Thread-299]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(398)) - Failed to authenticate with http/_HOST kerberos principal, trying with hive/_HOST kerberos principal 2016-10-04 22:31:34,063 ERROR [HiveServer2-HttpHandler-Pool: Thread-299]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(406)) - Failed to authenticate with hive/_HOST kerberos principal 2016-10-04 22:31:34,064 ERROR [HiveServer2-HttpHandler-Pool: Thread-299]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(209)) - Error: org.apache.hive.service.auth.HttpAuthenticationException: java.lang.reflect.UndeclaredThrowableException at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:407) at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:159) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:349) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:404) ... 23 more Caused by: org.apache.hive.service.auth.HttpAuthenticationException: Authorization header received from the client is empty. at org.apache.hive.service.cli.thrift.ThriftHttpServlet.getAuthHeader(ThriftHttpServlet.java:548) at org.apache.hive.service.cli.thrift.ThriftHttpServlet.access$100(ThriftHttpServlet.java:74) at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:449) at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:412) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) ... 24 more FYI I have: hadoop.proxyuser.knox.hosts=<knox_gateway>hadoop.proxyuser.knox.groups=* Thanks for any help! ... View more

Hello, I've created an HDF instance that authenticates with LDAP, the initial admin was setup using a SSL certificate so I can get into the NiFi console as the admin user. I am trying to grant access to another non-admin user and getting the below error when trying to login from another host that does not have certificate: 2016-09-27 17:56:55,897 INFO [NiFi Web Server-28] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for cn=test user,ou=users,dc=hadoop,dc=com 2016-09-27 17:56:55,898 INFO [NiFi Web Server-28] o.a.n.w.a.c.AccessDeniedExceptionMapper cn=test user,ou=users,dc=hadoop,dc=com does not have permission to access the requested resource. Returning Forbidden response. It looks like its authenticating fine with my LDAP server but running to issues with authorization. In the NiFi console i've created that user, "cn=test user,ou=users,dc=hadoop,dc=com" and granted access policy to "view the component. Here is login-identity provider: <property name="User Search Base">OU=users,DC=hadoop,DC=com</property> <property name="User Search Filter">uid={0}</property> and the results of ldapsearch: # test user, users, instream.com dn: cn=test user,ou=users,dc=hadoop,dc=com uid: tuser Am I creating the user incorrectly in NiFi or require any additional settings in nifi.properties? Thanks ... View more