About hadooptom

hadooptom · ‎05-26-2016

Thanks @hubbarja. Spent the afternoon trying this out on the CDH 5.7.0 QuickStart VM, with a kerberos enabled cluster and Cloudera Kafka 2.0.0. I think perhaps I didn't quite phrase my question clearly, but what I was trying to ask was whether the spark-streaming-kafka client would support consuming from a Kafka cluster that has client SSL authentication required enabled. For anyone else who tries this, the summary is it won't work due to upstream Spark issue [SPARK-12177], which deals with support for the new Kafka 0.9 consumer / producer API. SSL, SASL_PLAINTEXT or SASL_SSL connections to Kafka all require use of the new API. In fact, this issue is referenced in the known issues released with CDH 5.7.0, I just didn't spot it in time. There's a pull request which appears to support SSL (but no form of Kerberos client authentication) in Github here, if anyone feels brave. Looking at the comments on the Spark ticket, it's going to be at least post Spark 2.0.0 release that this feature gets merged in, and probably not until 2.1.0. Back to the drawing board for me!

hadooptom · ‎05-25-2016

Thanks @srowen. Is the source-code to Cloudera's Spark distribution publically available somewhere, so I can take a look at how to configure it?

hadooptom · ‎05-24-2016

Hi, Before I go to the effort of setting up a Cloudera 5.7 cluster with Kafka, Spark and Kerberos enabled to test it out, can anyone give me the answer to the following: - Does Cloudera's distribution of Spark 1.6.0 support SSL or Kerberos on a Kafka broker? It looks like vanilla Spark 1.6.0 (and spark-streaming-kafka jar) builds against Kafka 0.8, while I assume CDH's Spark is built against 0.9 (as this is the version that ships with CDH Kafka 2.0.1). It looks like vanilla Spark doesn't support SSL or Kerberos authentication to Kafka topics. Many thanks

hadooptom · ‎02-26-2015

Try adding source /etc/hadoop/conf/hadoop-env.sh source /etc/spark/conf/spark-env.sh to the top of a shell-script that submits your Spark job. Don't have a VM with Spark / Hadoop handy right now, but IIRC that's what I've needed to do in the past.

hadooptom · ‎01-07-2015

Ah, this got me on the right track. I've switched the 'hostname' command for 'hostname -f', redeployed the CSD jar and it works now. I think I'm a bit confused about how this works then. CM deploys a fresh config, it supplies the principal name to this initialisation script (presumably it's got the right one, as CM is aware of the keytabs it's distributing). That principal is regexed to remove the hostname and replace it with the template value of _HOST. Accumulo starts and replaces _HOST with the principal again. I guess the problem here is that 'hostname' returns an unqualified hostname whereas CM knows the server running the service by is FQDN. Maybe I'm missing something, but it seems like all the switching of hostnames an template values is what's causing the problem here. Thanks for pointing me in the right direction.

hadooptom · ‎01-07-2015

That didn't work so well... so much for posting from an iPad. 😞 Will try again, otherwise I'll post from my laptop later. <pre> Hi, Having a few problems with Kerberos and the Accumulo 1.6 package installation in CDH5.2. Wondering if anyone can advise. When I try and start any of the services, I get a Kerberos error - the one that usually indicates that you're using the wrong principal name for the keytab. To try and diagnose this I've tracked down the configuration files in /var/run. The keytab that's been distributed correctly initialises with kinit. I've noticed that the accumulo-site.xml has the principal name set as: accumulo/_HOST.domain.local@DOMAIN.LOCAL whereas all other *-site.xml files have the principal set as: service/_HOST@DOMAIN.LOCAL i.e. without the .doman.local following the _HOST placeholder. Is this likely the source of my error or do I need to look elsewhere? I'm not entirely clear how the placeholder values get replaced - it looks like the file is regenerated everytime the service starts up. It looks like it might be controlled by the script in the ACCUMULO16 CM extension at: https://github.com/cloudera/cm_csds/blob/master/ACCUMULO16/src/scripts/accumulo.sh Thanks in advance. </pre>

hadooptom · ‎01-07-2015

Hi, Having a few problems with Kerberos and the Accumulo 1.6 package installation in CDH5.2. Wondering if anyone can advise. When I try and start any of the services, I get a Kerberos error - the one that usually indicates that you're using the wrong principal name for the keytab. To try and diagnose this I've tracked down the configuration files in /var/run. The keytab that's been distributed correctly initialises with kinit. I've noticed that the accumulo-site.xml has the principal name set as: accumulo/_HOST.domain.local@DOMAIN.LOCAL whereas all other *-site.xml files have the principal set as: service/_HOST@DOMAIN.LOCAL i.e. without the .doman.local following the _HOST placeholder. Is this likely the source of my error or do I need to look elsewhere? I'm not entirely clear how the placeholder values get replaced - it looks like the file is regenerated everytime the service starts up. It looks like it might be controlled by the script in the ACCUMULO16 CM extension at: <a target="_blank" href="https://github.com/cloudera/cm_csds/blob/master/ACCUMULO16/src/scripts/accumulo.sh">https://github.com/cloudera/cm_csds/blob/master/ACCUMULO16/src/scripts/accumulo.sh</a> Thanks in advance.

hadooptom · ‎09-30-2014

Perfect, thanks for this. I was also having a look at something like: { setValues { _attachment_body : "@{myInputField}" } } { readClob { charSet : "UTF-8" } } Out of interest, are there any disadvantages with that approach? Thanks.

hadooptom · ‎09-30-2014

I've got a record in Morphlines that includes a byte array field. I want to convert that to a UTF-8 string, i.e. the equivalent of String(field, "UTF-8) in Java. I can see the readClob command exists, but that works on a whole record rather than a single field. Is there an alternative? Appreciate that I should have stored my data differently, but it's not my format and that's what ETL tools (morphlines) are for! 😉 Thanks in advance.

Online	Offline
Last Visited	‎10-21-2023 06:38 AM

Member Since	‎03-04-2014 11:57 PM
Last Visited	‎10-21-2023 06:38 AM
Posts	22
Kudos received	4

Cloudera Community

Re: Spark and Kafka broker with SSL (or Kerberos) ...

Re: Scheduling Spark with Crontab

Re: Accumulo 1.6 package - kerberos error

Re: Spark and Kafka broker with SSL (or Kerberos) ...

Re: Spark and Kafka broker with SSL (or Kerberos) ...

Spark and Kafka broker with SSL (or Kerberos) auth...

Re: Scheduling Spark with Crontab

Re: Accumulo 1.6 package - kerberos error

Re: Accumulo 1.6 package - kerberos error

Accumulo 1.6 package - kerberos error

Re: Morphlines - converting a byte array field to ...

Morphlines - converting a byte array field to a UT...