@Funamizu Koshi import re from pyspark.sql.functions import UserDefinedFunction from pyspark.sql.types import * udf = UserDefinedFunction(lambda x: re.sub(',','',x), StringType()) new_df = df.select(*[udf(column).alias(column) for column in df.columns]) new_df.collect() ... View more

Hi Jasper, I don't think you have a problem while logging in. https://github.com/streamsets/datacollector/blob/master/apache-kafka_0_9-lib/src/main/java/org/apache/kafka/common/security/kerberos/Login.java After skimming through the above link, I guess if there was any error w.r.t. having a valid ticket, you would've got a log in KerberosLogin itself. What principal are you using? Can you check contents of kafka_client_jaas.conf. Is it of the form below: KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true; }; Or you have keytab configuration? If former, please see kafka_jaas.conf for Client section and kinit with user/keytab mentioned there. Try running the command again as: /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer -Djava.security.auth.login.config= /etc/kafka/kafka_jaas.conf If above command does not work, try exporting that variable. ... View more

Hi @pp z Check whether console consumer works with --bootstrap-server = broker instead of --zookeeper. Check advertised listener in server.properties. Is it of the form PLAINTEXTSASL://host:port? Check whether security_protocol = PLAINTEXTSASL exists in server.properties. Are you getting the exception: WARN SASL configuration failed:javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file. If yes, then modify kafka_client_jaas.conf to include a Client section, you'll most likely get that in kafka_jaas.conf. If not, then ignore this. Use --security-protocol PLAINTEXTSASL instead of SASL_PLAINTEXT. Check whether you're able to start a authenticated connection to zookeeper-client. If not, give the path of conf file that has client section as a JVM param. For example: export JVMFLAGS="-Djava.security.auth.login.config= /usr/hdp/2.6.1.0-129/kafka/conf/kafka_client_jaas.conf" Try these and let me know. Thanks ... View more

@Sofian Benabdelhak Please check the answer here, It may help. https://community.hortonworks.com/questions/114024/invalid-kdc-administrator-credentials.html?childToView=117774#answer-117774 Also use that API and set credentials to persisted. ... View more

@Mugdha One more thing, I was seeing the config versions for YARN, and in v1 (just after installation), yarn.acl.enable is set to false. Is this property set to true while Kerberizing? ... View more

@Ann A You can use the concept of time window. These two links may help you: http://blog.madhukaraphatak.com/introduction-to-spark-two-part-5/ https://stackoverflow.com/questions/37632238/how-to-group-by-time-interval-in-spark-sql ... View more

Naveen, Can you check Kerberos ACL? RHEL/CentOS/Oracle Linux vi /var/kerberos/krb5kdc/kadm5.acl SLES vi /var/lib/kerberos/krb5kdc/kadm5.acl Ubuntu/Debian vi /etc/krb5kdc/kadm5.acl Default settings would be similar to: */admin@EXAMPLE.COM* or in your case */admin@DEV.DATAQUEST.COM* This means that only principals matching the above regex would be considered as admins. So try changing your principal to kadmin/admin@DEV.DATAQUEST.COM instead. Or add a line in the acl giving permission to kadmin. Let me know if this works. ... View more

Hi, I'm not very sure but you could use flume to get data into HDFS by using an hdfs sink. https://flume.apache.org/FlumeUserGuide.html The location in hdfs is mentioned in flume-agent.conf file, for example: agent_foo.sinks.hdfs-Cluster1-sink.hdfs.path = hdfs://namenode/flume/webdata You could write a script to modify this directory with a timestamp and restart the flume agent. And then run that every week through cron. ... View more