I unfortunately don't know too many of the details of LDAP. Impala doesn't do anything sophisticated to create the directories - it just mkdir() with S_IRWXU|S_IRWXG|S_IRWXO to create the impala-scratch subdirectory and any missing parent directories. ... View more

The same can be peformed in hive using concat_ws('.',from_unixtime(cast(epochmillis/1000 as BIGINT),'yyyy-MM-dd HH:mm:ss'),cast(floor(epochmillis % 1000) as STRING)) to get the timestamp with milliseconds.   Is this efficient way of doing it ? ... View more

This is something I found on Stackoverflow..   https://stackoverflow.com/questions/36508553/how-to-specify-consumer-group-in-kafka-spark-streaming-using-direct-stream.   Is there any other way of monitoring the backlog in this approach. I tried exploring Spark Metrics API. Even it is of no big use in this case. ... View more

God bless you,  munna143 ... View more

@manuspark3 @SupriyaPS    If scale is not specified, it defaults to 0 . If no precision is supplied , it defaults to 10. Yes decimal works without precison.  ... View more

We're also facing same issue ... and any pointers will be useful.   The issue is by default dataframe assigns null values to non existing fields. The problem is there could be a valid use case where and upsert statement wants to actually update the value of a column to null i.e. delete the value. So I think the issue is not with KuduContext but with DataFrame.   I'm a Spark newbie; is there a way to control how DataFrame is created ? ... View more

Hi,   Can you post the code on how you are trying to write to the output?    Thanks, Ravi ... View more

Hi Tim,   Can we have the flexibility of declaring the variable globally in UDF? Globally, I mean outside the function? And, the reason I am declaring a static variable is to restore the value of timestamp for every record so that I can perform a comparison of the timestamps. Is there an alternative approach for this?   Thanks ... View more

Hi munna143,  You may have to recreate the hash-file for the parcel by completing the steps below:     Download the manually deleted parcel and add it to parcel repo location /opt/cloudera/parcel-repo . Create the hash-file based on parcel: $sha1sum /opt/cloudera/parcel-repo/CDH-parcel-file.parcel | cut -d ' ' -f 1 > /opt/cloudera/parce-repo/CDH-parcel-file.parcel.sha Set the ownership of the files to cloudera-scm $chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/CDH-parcel-file.parcel /opt/cloudera/parcel-repo/CDH-parcel-file.parcel.sha Delete the parcel through Cloudera Manager from the Parcels page.   Thanks,  Jordan  ... View more