Member since
09-28-2015
73
Posts
26
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7264 | 01-20-2017 01:27 PM | |
2650 | 06-01-2016 08:24 AM | |
2813 | 05-28-2016 01:33 AM | |
1925 | 05-17-2016 03:44 PM | |
1075 | 12-22-2015 01:50 AM |
06-19-2017
05:16 AM
After manually executed /usr/hdf/current/streamline/bootstrap/sql/postgresql/create_tables.sql, I was able to start SAM successfully.
... View more
06-19-2017
04:20 AM
After re-configured SAM against Postgresql 9.6, this error is gone. However, I got another error while starting SAM. Exception in thread "main" com.hortonworks.streamline.storage.exception.StorageException: com.google.common.util.concurrent.UncheckedExecutionException: com.hortonworks.streamline.storage.exception.StorageException: org.postgresql.util.PSQLException: ERROR: relation "topology_version" does not exist
Position: 15
at com.hortonworks.streamline.storage.cache.impl.GuavaCache.get(GuavaCache.java:72)
at com.hortonworks.streamline.storage.cache.impl.GuavaCache.get(GuavaCache.java:41)
at com.hortonworks.streamline.storage.CacheBackedStorageManager.get(CacheBackedStorageManager.java:74)
at com.hortonworks.streamline.streams.catalog.service.StreamCatalogService.getTopologyVersionInfo(StreamCatalogService.java:245)
at com.hortonworks.streamline.streams.service.StreamsModule.setupPlaceholderTopologyVersionInfo(StreamsModule.java:202)
at com.hortonworks.streamline.streams.service.StreamsModule.setupPlaceholderEntities(StreamsModule.java:198)
at com.hortonworks.streamline.streams.service.StreamsModule.getResources(StreamsModule.java:104)
at com.hortonworks.streamline.webservice.StreamlineApplication.registerResources(StreamlineApplication.java:293)
at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:100)
at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:74)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
at io.dropwizard.cli.Cli.run(Cli.java:75)
at io.dropwizard.Application.run(Application.java:79)
at com.hortonworks.streamline.webservice.StreamlineApplication.main(StreamlineApplication.java:78)
... View more
06-16-2017
08:54 AM
1 Kudo
Hi, I installed HDF3.0 on an existing HDP2.6.0 cluster using Ambari 2.5.1. Installation was succeed, but failed to start SAM. My configuration uses Postgresql for SAM. I got the below error. Does anyone happen to see this before? Exception in thread "main" com.hortonworks.streamline.storage.exception.StorageException: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ON"
Position: 112
at com.hortonworks.streamline.storage.impl.jdbc.provider.sql.factory.AbstractQueryExecutor$QueryExecution.executeUpdate(AbstractQueryExecutor.java:223)
at com.hortonworks.streamline.storage.impl.jdbc.provider.sql.factory.AbstractQueryExecutor.executeUpdate(AbstractQueryExecutor.java:180)
at com.hortonworks.streamline.storage.impl.jdbc.provider.postgresql.factory.PostgresqlExecutor.insertOrUpdateWithUniqueId(PostgresqlExecutor.java:150)
at com.hortonworks.streamline.storage.impl.jdbc.provider.postgresql.factory.PostgresqlExecutor.insertOrUpdate(PostgresqlExecutor.java:69)
at com.hortonworks.streamline.storage.impl.jdbc.JdbcStorageManager.addOrUpdate(JdbcStorageManager.java:81)
at com.hortonworks.streamline.storage.cache.writer.StorageWriteThrough.addOrUpdate(StorageWriteThrough.java:37)
at com.hortonworks.streamline.storage.CacheBackedStorageManager.addOrUpdate(CacheBackedStorageManager.java:68)
at com.hortonworks.streamline.streams.catalog.service.StreamCatalogService.addOrUpdateTopologyVersionInfo(StreamCatalogService.java:262)
at com.hortonworks.streamline.streams.service.StreamsModule.setupPlaceholderTopologyVersionInfo(StreamsModule.java:210)
at com.hortonworks.streamline.streams.service.StreamsModule.setupPlaceholderEntities(StreamsModule.java:198)
at com.hortonworks.streamline.streams.service.StreamsModule.getResources(StreamsModule.java:104)
at com.hortonworks.streamline.webservice.StreamlineApplication.registerResources(StreamlineApplication.java:293)
at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:100)
at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:74)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
at io.dropwizard.cli.Cli.run(Cli.java:75)
at io.dropwizard.Application.run(Application.java:79)
at com.hortonworks.streamline.webservice.StreamlineApplication.main(StreamlineApplication.java:78)
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ON"
Position: 112
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168)
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:135)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at com.hortonworks.streamline.storage.impl.jdbc.provider.sql.factory.AbstractQueryExecutor$QueryExecution.executeUpdate(AbstractQueryExecutor.java:221)
... 18 more
... View more
Labels:
- Labels:
-
Cloudera DataFlow (CDF)
01-26-2017
10:23 AM
1 Kudo
@dbains, @mthiele, @Daniel Kozlowski, Thank you. It works when created as kafka user. To summarize all the steps -- from Ambari configs, to creating topic, granting permission and testing on kafka console producer/consumer scripts, I created this article Step by Step Recipe for Securing Kafka with Kerberos. Hope it saves others' time 🙂
... View more
01-26-2017
10:14 AM
7 Kudos
I found it is a little tricky to get started with a Kerberos enabled Kafka cluster. I created this step by step recipe for securing Kafka with Kerberos, sending and receiving data on console. This is tested on HDP2.5.0 and Ambari 2.4.1. Enabled Kerberos using the Ambari Kerberos setup wizard under Admin -- Kerberos menu. On Ambari Kafka Config UI, change "listeners" property to "PLAINTEXTSASL://localhost:6667". Restart Kafka as requested by Ambari. Create a test topic in Kafka. Must use the kafka service user to do this. $ cd /usr/hdp/current/kafka-broker/bin
$ sudo su kafka
$ kinit -k -t /etc/security/keytabs/kafka.service.keytab kafka/ip-10-0-1-130.ap-northeast-1.compute.internal
$ ./kafka-topics.sh --zookeeper ip-10-0-1-130.ap-northeast-1.compute.internal:2181 --create --topic foo --partitions 1 --replication-factor 1
Created topic "bar".
Grant permission to user. This can be done using Kafka native ACL mechanism or Apache Ranger. In the example, we use Kafka ACL. User bob needs to be existing in KDC. # Grant user bob as producer on topic foo
./kafka-acls.sh --authorizer-properties zookeeper.connect=ip-10-0-1-130.ap-northeast-1.compute.internal:2181 \
--add --allow-principal User:bob \
--producer --topic foo
Adding ACLs for resource `Topic:foo`:
User:bob has Allow permission for operations: Describe from hosts: *
User:bob has Allow permission for operations: Write from hosts: *
Adding ACLs for resource `Cluster:kafka-cluster`:
User:bob has Allow permission for operations: Create from hosts: *
Current ACLs for resource `Topic:foo`:
User:bob has Allow permission for operations: Describe from hosts: *
User:bob has Allow permission for operations: Write from hosts: *
# Grant user bob as consumer
./kafka-acls.sh --authorizer-properties zookeeper.connect=ip-10-0-1-130.ap-northeast-1.compute.internal:2181 \
--add --allow-principal User:bob \
--consumer --topic foo --group *
Adding ACLs for resource `Topic:foo`:
User:bob has Allow permission for operations: Read from hosts: *
User:bob has Allow permission for operations: Describe from hosts: *
Adding ACLs for resource `Group:connect-distributed.sh`:
User:bob has Allow permission for operations: Read from hosts: *
Current ACLs for resource `Topic:foo`:
User:bob has Allow permission for operations: Read from hosts: *
User:bob has Allow permission for operations: Describe from hosts: *
User:bob has Allow permission for operations: Write from hosts: *
Current ACLs for resource `Group:connect-distributed.sh`:
User:bob has Allow permission for operations: Read from hosts: *
Confirm the above works using the kafka console producer and consumer scripts. # Switch to bob user and log in to KDC.
$ kinit bob
# Start console producer
$ ./kafka-console-producer.sh --broker-list ip-10-0-1-130.ap-northeast-1.compute.internal:6667 --topic foo --security-protocol PLAINTEXTSASL
# On another terminal, start console consumer
./kafka-console-consumer.sh --zookeeper ip-10-0-1-130.ap-northeast-1.compute.internal:2181 --topic foo --security-protocol PLAINTEXTSASL
{metadata.broker.list=ip-10-0-1-130.ap-northeast-1.compute.internal:6667, request.timeout.ms=30000, client.id=console-consumer-57797, security.protocol=PLAINTEXTSASL}
# Type something on the producer terminal, it should appears on the console terminal immediately.
... View more
Labels:
01-23-2017
02:44 AM
I created the topic using a normal user (not the 'kafka' service user). Do I need to use 'kafka' user to create the topic?
... View more
01-23-2017
02:43 AM
I created the topic using a normal user (not the 'kafka' service user). Do I need to use 'kafka' user to create the topic? Below is my server.properties. advertised.listeners=PLAINTEXTSASL://ip-10-0-0-149.ap-northeast-1.compute.internal:6667
authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer
auto.create.topics.enable=true
auto.leader.rebalance.enable=true
compression.type=producer
controlled.shutdown.enable=true
controlled.shutdown.max.retries=3
controlled.shutdown.retry.backoff.ms=5000
controller.message.queue.size=10
controller.socket.timeout.ms=30000
default.replication.factor=1
delete.topic.enable=false
external.kafka.metrics.exclude.prefix=kafka.network.RequestMetrics,kafka.server.DelayedOperationPurgatory,kafka.server.BrokerTopicMetrics.BytesRejectedPerSec
external.kafka.metrics.include.prefix=kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetCommit.98percentile,kafka.network.RequestMetrics.ResponseQueueTimeMs.request.Offsets.95percentile,kafka.network.RequestMetrics.ResponseSendTimeMs.request.Fetch.95percentile,kafka.network.RequestMetrics.RequestsPerSec.request
fetch.purgatory.purge.interval.requests=10000
kafka.ganglia.metrics.group=kafka
kafka.ganglia.metrics.host=localhost
kafka.ganglia.metrics.port=8671
kafka.ganglia.metrics.reporter.enabled=true
kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter
kafka.timeline.metrics.host=ip-10-0-0-229.ap-northeast-1.compute.internal
kafka.timeline.metrics.maxRowCacheSize=10000
kafka.timeline.metrics.port=6188
kafka.timeline.metrics.protocol=http
kafka.timeline.metrics.reporter.enabled=true
kafka.timeline.metrics.reporter.sendInterval=5900
kafka.timeline.metrics.truststore.password=bigdata
kafka.timeline.metrics.truststore.path=/etc/security/clientKeys/all.jks
kafka.timeline.metrics.truststore.type=jks
leader.imbalance.check.interval.seconds=300
leader.imbalance.per.broker.percentage=10
listeners=PLAINTEXTSASL://ip-10-0-0-149.ap-northeast-1.compute.internal:6667
log.cleanup.interval.mins=10
log.dirs=/kafka-logs
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.bytes=-1
log.retention.hours=168
log.roll.hours=168
log.segment.bytes=1073741824
message.max.bytes=1000000
min.insync.replicas=1
num.io.threads=8
num.network.threads=3
num.partitions=1
num.recovery.threads.per.data.dir=1
num.replica.fetchers=1
offset.metadata.max.bytes=4096
offsets.commit.required.acks=-1
offsets.commit.timeout.ms=5000
offsets.load.buffer.size=5242880
offsets.retention.check.interval.ms=600000
offsets.retention.minutes=86400000
offsets.topic.compression.codec=0
offsets.topic.num.partitions=50
offsets.topic.replication.factor=3
offsets.topic.segment.bytes=104857600
port=6667
principal.to.local.class=kafka.security.auth.KerberosPrincipalToLocal
producer.purgatory.purge.interval.requests=10000
queued.max.requests=500
replica.fetch.max.bytes=1048576
replica.fetch.min.bytes=1
replica.fetch.wait.max.ms=500
replica.high.watermark.checkpoint.interval.ms=5000
replica.lag.max.messages=4000
replica.lag.time.max.ms=10000
replica.socket.receive.buffer.bytes=65536
replica.socket.timeout.ms=30000
security.inter.broker.protocol=PLAINTEXTSASL
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
super.users=User:kafka
zookeeper.connect=ip-10-0-0-149.ap-northeast-1.compute.internal:2181
zookeeper.connection.timeout.ms=25000
zookeeper.session.timeout.ms=30000
zookeeper.set.acl=true
zookeeper.sync.time.ms=2000
... View more
01-22-2017
11:49 AM
Hi, After enabled Kerberos using Ambari, I got problem creating topics in Kafka using the kafka-topics.sh script. The topic was created, but its status is wrong without leader. It seems the topic is created with PLAINTEXT, while there is only PLAINTEXTSASL broker in the cluster after enabled Kerberos. The only configuration change I made is to chagne broker listener from 'PLAINTEXT://localhost:6667' to 'PLAINTEXTSASL://localhost:6667'. As posted in this question, I also changed the kafka-topics.sh to make it work with Kerberos. I am using HDP2.5.3. $ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal --create --partitions 1 --replication-factor 1 --topic mytopic
Created topic "mytopic".
$ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal --describe --topic mytopic
Topic:mytopic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: mytopic Partition: 0 Leader: none Replicas: 1001 Isr:
... View more
Labels:
- Labels:
-
Apache Kafka
01-20-2017
01:27 PM
4 Kudos
Solved by the below workaround. This looks like a bug in kafka-topics.sh. 1. Add KAFKA_CLIENT_KERBEROS_PARAMS before executing actual TopicCommand if running in a Kerberos enabled cluster. $ cat kafka-topics.sh
# check if kafka_jaas.conf in config , only enable client_kerberos_params in secure mode.
KAFKA_HOME="$(dirname $(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ))"
KAFKA_JAAS_CONF=$KAFKA_HOME/config/kafka_jaas.conf
if [ -f $KAFKA_JAAS_CONF ]; then
export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_client_jaas.conf"
fi
exec $(dirname $0)/kafka-run-class.sh kafka.admin.TopicCommand "$@"
2. Use Zookeeper server FQDN instead of localhost in command line. $ kinit
$ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal:2181 --create --topic foo --partitions 1 --replication-factor 1
Created topic "foo".
... View more
01-20-2017
12:32 PM
Also tried the below, didn't work... $ export JVMFLAGS="-Djava.security.auth.login.config=/etc/kafka/conf/kafka_client_jaas.conf"
$ ./kafka-topics.sh --zookeeper localhost:2181 --create --topic foo --partitions 1 --replication-factor 1
... View more