Member since
11-13-2017
31
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1751 | 10-30-2018 11:08 AM |
09-23-2019
02:44 PM
Hello Haris, we are in the same situation, in my opinion the number of processors is a hard limit in a NiFi cluster. Given the timer-driven architecture, the number of processors running at the same time is heavily influenced by the amount of threads available. In this regard, any suggestion from NiFi developers is welcome. For the other questions: the type of processors used and the overall flow design will also have an impact on performance, I would not consider the input volume as a problem since you can scale horizontally. R.
... View more
- Tags:
- limitation
- NiFi
08-08-2019
08:14 AM
Hello Satish, this error is not related to the procedure you followed, you need to kinit as kafka's user.
... View more
11-06-2018
08:25 AM
Which version of Kafka are you using? (not ConsumeKafka) Can you also post the configuration?
... View more
11-02-2018
09:12 AM
@siddharth pande The default behavior is the one you described at the beginning, with each node consuming from a different partition. You should share the Processor's configuration and a describe of the topic. Check also that the ConsumeKafka processor is compatible with the version of Kafka you are using.
... View more
10-30-2018
11:08 AM
Properly setting up the nifi.security.identity.mapping.pattern.kerb and nifi.security.identity.mapping.pattern.dn fixed the problem. Also, while debugging these kind of problems, it's best to delete ranger plugin cache (under /etc/ranger/SERVICE_NAME/policycache/) to ensure that there are no communication problem between NiFi and Ranger.
... View more
10-30-2018
11:01 AM
For anybody looking at the same problem, everything works with the above mentioned configuration.
... View more
10-30-2018
10:55 AM
@Soumitra Sulav This is already similar to my configuration The problem is that this configuration is not enforced by Ranger for everything related to creating/deleting topics (i.e. Zookeeper). Ranger is only enforcing Publishing and Consuming from Kafka.
... View more
10-30-2018
08:19 AM
Hello @Soumitra Sulav, the Ranger plugin for Kafka is enabled and working, I am able to control who can Produce/Consume on a Kafka topic but I am not able to control who can create/list/remove them.
... View more
10-29-2018
10:37 AM
This seems to have worked for me. 1) kinit 2) source the env 3) execute zkCli.sh. Thanks
... View more
10-29-2018
09:55 AM
Hello Geoffrey, first of all thanks, this looks promising. This seems to be completely outside Ambari control. Does this mean that I need to manually create a zookeeper\{HOST}@{DOMAIN} keytab for each Kafka Node? Is there some way to automate this?
... View more
10-26-2018
03:26 PM
Hello, I am doing some tests on a Kerberized HDF cluster with Ranger enabled. Using Kafka, I noticed that everybody can create/describe/delete topics from zookeeper without being authenticated. This is an example, I used a server that is not part of the HDF cluster, and doesn't have Kerberos installed: [root@test_node ~/kafka_2.11-1.1.1/bin]# ./kafka-topics.sh --zookeeper zk_node:2181 --create -topic test_topic --partitions 1 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "test_topic".
[root@test_node ~/kafka_2.11-1.1.1/bin]# ./kafka-topics.sh --zookeeper zk_node:2181 --topic test_topic --describe
Topic:test_topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test_topic Partition: 0 Leader: 1004 Replicas: 1004 Isr: 1004
[root@test_node ~/kafka_2.11-1.1.1/bin]# ./kafka-topics.sh --zookeeper zk_node:2181 --delete --topic test_topic
Topic test_topic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[root@test_node ~/kafka_2.11-1.1.1/bin]# ./kafka-topics.sh --zookeeper zk_node:2181 --topic test_topic --describe
[root@test_node ~/kafka_2.11-1.1.1/bin]# klist
-bash: klist: command not found
I have also been able to delete all the topics created/autocreated by authenticated users. As you can see Kerberos is enabled when I try to consume/produce data on a topic: [root@test_node2 /usr/hdf/current/kafka-broker/bin]# ./kafka-console-producer.sh --broker-list kafka_node:6668 --topic test_topic2 --security-protocol SASL_SSL --producer.config /root/client-ssl.properties
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:456)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:303)
at kafka.producer.NewShinyProducer.<init>(BaseProducer.scala:40)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:50)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a pa
ssword from the user. not available to garner authentication information from the user
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:125)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:141)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:65)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:88)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:413)
... 4 more
Is there a way to prevent this from happening?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Ranger
08-30-2018
03:35 PM
There aren't many information in the manual is it possible I have to manually configure all the options under "Advanced ranger-nifi-plugin-properties" (in the ambari console)? Currently only a few of those properties are configured.
... View more
08-30-2018
02:44 PM
Thanks, I will use this configuration while testing in the future.
... View more
08-30-2018
02:43 PM
Hello @Steven Matison, thanks for replying. I believe that everything is setup as you proposed, I also added the NiFi proxy users to their own policy but nothing changed. Tailing the usersync.log doesn't provide any additional evidence.
... View more
08-30-2018
02:32 PM
Hello, I am setting up an HDF 3.2.0 cluster and I noticed that NiFi CA doesn't start up. Trying to manually start NiFi CA leads to the following errors: 2018/08/30 16:20:52 INFO [main] org.eclipse.jetty.util.log: Logging initialized @789ms to org.eclipse.jetty.util.log.Slf4jLog
2018/08/30 16:20:52 INFO [main] org.eclipse.jetty.server.Server: jetty-9.4.3.v20170317
2018/08/30 16:20:52 INFO [main] org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@1ce24091{SSL,[ssl, http/1.1]}{0.0.0.0:10443}
2018/08/30 16:20:52 INFO [main] org.eclipse.jetty.server.Server: Started @921ms
Server Started
2018/08/30 16:20:52 WARN [qtp1904253191-9] org.eclipse.jetty.server.HttpChannel: /v1/api
javax.servlet.ServletException: Server error
at org.apache.nifi.toolkit.tls.service.server.TlsCertificateAuthorityServiceHandler.handle(TlsCertificateAuthorityServiceHandler.java:99)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
2018/08/30 16:20:52 WARN [qtp1904253191-9] org.eclipse.jetty.server.HttpChannel: /v1/api
javax.servlet.ServletException: Server error
at org.apache.nifi.toolkit.tls.service.server.TlsCertificateAuthorityServiceHandler.handle(TlsCertificateAuthorityServiceHandler.java:99)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:369)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
2018/08/30 16:20:52 WARN [qtp1904253191-9] org.eclipse.jetty.server.HttpChannel:
java.lang.IllegalStateException: Error already set
at org.eclipse.jetty.server.HttpChannelState.onError(HttpChannelState.java:743)
at org.eclipse.jetty.server.HttpChannel.handleException(HttpChannel.java:514)
at org.eclipse.jetty.server.HttpChannelOverHttp.handleException(HttpChannelOverHttp.java:463)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:448)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: Server error
at org.apache.nifi.toolkit.tls.service.server.TlsCertificateAuthorityServiceHandler.handle(TlsCertificateAuthorityServiceHandler.java:99)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
... 10 more
Is there a known solution for this? Thanks, Raffaele
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
08-29-2018
04:32 PM
Hello, I am setting up an HDF 3.2 cluster that's fully Kerberized and I am trying to handle authorization through Ranger. We have a single Active Directory which also acts as KDC. The technical users (for example the service principals automatically created by HDF) are mapped in the following organization unit: OU=HDF,DC=example,DC=com while "normal" users (devs/admins) are mapped in OU=USERS,DC=example,DC=com The problem is that after enabling NiFi plugin and Kafka plugin I have not been able to use any of the two (I added my username to the "admin" policies of both services in ranger). Since the two problems are probably linked, I will start from NiFi and if necessary expand on Kafka in another post. NiFi authentication works (the user is recognized) but I receive the following error: "Unable to view the user interface. Contact the system administrator." When I check the audit log I notice that the User is indicated with the full qualified domain name USER@EXAMPLE.COM (instead of just the username) and the access is denied. nifi.security.user.login.identity.provider=kerberos-provider
I tried the following NiFi properties without success: nifi.security.identity.mapping.pattern.kerb=^(.?)@(.?)$
nifi.security.identity.mapping.value.kerb=$1
Could you help me solve this problem? Thanks
... View more
Labels:
08-29-2018
08:24 AM
Hello @Sandeep Nemuri This is the error that I received: ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) java.lang.IllegalArgumentException: requirement failed: inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are SASL_PLAINTEXT,SASL_SSL at scala.Predef$.require(Predef.scala:224) at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1374) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1350) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1010) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:990) at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28) at kafka.Kafka$.main(Kafka.scala:59) at kafka.Kafka.main(Kafka.scala) Seems that the two values are not completely interchangeable. When Ambari automatically changes the configuration, it changes only "security.inter.broker.protocol" and not "listeners". If I manually change "listeners" to the same value included in "security.inter.broker.protocol" everything works. Let me be more clear: I started with a Kerberized cluster where Kafka configuration is the following:
"listeners" contains "SASL_PLAINTEXT" and "SASL_SSL" "security.inter.broker.protocol" contains "SASL_PLAINTEXT" I installed Knox with the default configuration Ambari believes "SASL_PLAINTEXT" is not a correct value for "security.inter.broker.protocol" and forcefully changes it to "PLAINTEXTSASL". Restarting Kafka will lead to the above mentioned error. Manually modifying "security.inter.broker.protocol" to "SASL_PLAINTEXT" solves the problem (in alternative modifying "listeners" also works) Thanks
... View more
08-28-2018
12:15 PM
Hello, it seems like on HDF 3.2.0 installing kerberos and other services afterwards automatically updates security.inter.broker.protocol to the value "PLAINTEXTSASL" which leads to an error while starting Kafka. The correct value would be SASL_PLAINTEXT. Probable steps to reproduce, Install HDF, Enable Kerberos, Install Knox, try to start Kafka. Update: While changing any configuration this message appears in Ambari (and it's clearly an error): If kerberos is enabled listeners need to contain PLAINTEXTSASL as one of the protocol
host and port where kafka broker will be accepting connections. localhost will be substituted with hostname.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Cloudera DataFlow (CDF)
08-28-2018
10:10 AM
Great, this was actually my case. Thanks
... View more
08-27-2018
02:23 PM
Hello, I have been able to connect properly Ranger to AD with LDAPS. Currently syncing all the users requires a huge amount of time, currently 4+ hours for the initial sync. What are the available options to reduce this time? Regarding the access to Ranger UI, is it possible to assign roles to groups instead of users? Thanks
... View more
Labels:
- Labels:
-
Apache Ranger
03-07-2018
09:51 AM
I am using the reposync -r command without the ppc64le architecture, is it safe to use on x86-64?
... View more
03-07-2018
08:40 AM
@Vipin Rathor Still having problems here unfortunately, for example: ranger_3_1_1_0_35-hbase-plugin FAILED ] 5.0 MB/s | 866 MB 00:06:10 ETA ranger_3_1_1_0_35-usersync-0.7 FAILED ============= ] 1.0 kB/s | 2.0 GB 194:45:05 ETA ranger_3_1_1_0_35-yarn-plugin- FAILED ============= ] 8.3 kB/s | 2.0 GB 22:58:31 ETA zookeeper_3_1_1_0_35-3.4.6.3.1 FAILED This happens only with those specific packages, all the other HDF packages have been downloaded correctly. Trying to download one of the rpms with the browser usually works: http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.1.0/ranger/ranger_3_1_1_0_35-usersync-0.7.0.3.1.1.0-35.x86_64.rpm
... View more
02-14-2018
03:58 PM
Hello,
in the various official guides and articles around the community, there are different (conflicting) suggestions regarding NiFi sizing. HDF documentations suggests that to achieve a sustained throughput of 200MB/s, 4GB of RAM per node are required. Should I consider this throughput to be per node or per cluster? This article here, seems to suggests 16GB of RAM, for the same use case. For a similar one, 64GB of RAM are suggested here. Let's consider a medium-difficulty topology. Which one is the correct sizing for the RAM? I am also wondering about the number of cores, I noticed from the official documentation that scaling from "3-4 nodes" (8 cores, 100MB/s) to "5-7 nodes" (24 cores, 200MB/s), increases the minimum number of cores required 3 times. Starting from 3 nodes and scaling up to 6 would mean an increase in costs of 6 times, why should I prefer this configuration to just 2 smaller clusters (aside from the obvious management issues)? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
02-05-2018
04:01 PM
Hello, we are not allowed to create "local users" on our *nix servers, all our users are defined on active directory and automatically synced with the server (using something similar to SSSD), even the ones needed by 3rd party products. We are now planning to use Hortonworks Data Flow, is there any problem if we define the HDF users (kafka, nifi, ambari-qa, ams, infra-solr, storm.. etc.) together with the proper primary group (hadoop) on Active Directory instead of /etc/passwd? Will this configuration be compatible with Kerberos*? Is there any other limitation I should keep in account? * (Kerberos and Unix Users would be created in two different Organization Units)
... View more
Labels:
02-02-2018
09:48 AM
Did you backup the database, resource folders etc. as described in the upgrade guide? If you did, restore all the backups, downgrade ambari to the previous version and follow the proper order (as I described above).
... View more
01-16-2018
01:35 PM
Hello @Kevin Risden, I have not been able to solve it with configs.py, (probably) because I already tried without success other solutions (that may have interfered). Anyway your hint about inverting the order of the upgrades has been helpful, the correct order is: first upgrade the mpack then upgrade Ambari. Thanks. On this matter the Hortonworks HDF Upgrade documentation is confusing, it seems to suggest the wrong order (and leaves the idea that you can follow both orders). Raffaele
... View more
01-08-2018
02:45 PM
1 Kudo
Hello, just happened again on our development cluster (RHEL 7.3). In this case reinstalling the cluster is not feasible. All my agents are version: ambari-agent-2.6.0.0-267.x86_64. My ambari-server: ambari-server-2.6.0.0-267.x86_64 A bit of context, I reached this point of the ambari upgrade procedure: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_ambari-upgrade/content/upgrade_ambari.html After accessing the console I noticed Zookeeper, Storm and metrics were down (this is not correct, the services are running). Restarting zookeeper led to this error: stderr: /var/lib/ambari-agent/data/errors-1202.txt Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py", line 40, in <module>
BeforeStartHook().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py", line 28, in hook
import params
File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/params.py", line 76, in <module>
hadoop_libexec_dir = stack_select.get_hadoop_dir("libexec")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/stack_select.py", line 365, in get_hadoop_dir
stack_root = Script.get_stack_root()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 659, in get_stack_root
stack_root = json.loads(stack_root_json)
File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-1202.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-1202.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] stdout: /var/lib/ambari-agent/data/output-1202.txt 2018-01-08 15:40:58,562 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=3.0.1.1-5 -> 3.0.1.1-5
2018-01-08 15:40:58,564 - Cannot find stack features for the stack named HDF
2018-01-08 15:40:58,566 - Cannot find stack features for the stack named HDF
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-1202.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-1202.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
2018-01-08 15:40:58,792 - Cannot find stack features for the stack named HDF
Command failed after 1 tries
I tried upgrading the management pack after, but didn't help. In this case, should I open a JIRA?
... View more
01-03-2018
07:57 AM
Thanks @Geouffrey Erasmus. Luckily this happened on a test cluster, we ended up reinstalling from scratch. I will write here if the same happens on our development cluster. Raffaele
... View more
12-19-2017
05:33 PM
Hello @Geouffrey Erasmus, I have exactly the same problem, any update? Thanks.
... View more