@Marcel-Jan Krijgsman have you configured hive.server2.enable.doAs=false ? If you have it set to true then all access to hdfs is going to be made as caller user. However if set to false then access to hdfs is made as hive. I believe the recommended approach is to set this value to false for ranger hive sql authorization. HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Carlton Patterson You can use this to write whole dataframe to single file: myresults.coalesce(1).write.csv("/tmp/myresults.csv") HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Bhushan Kandalkar To recover and and change notebook permissions manually you should do the following: 1. Login to zeppelin server host and switch to zeppelin user. If kerberized, kinit as zeppelin principal using zeppelin keytab 2. Make a backup of the following hdfs file hdfs dfs -cp /user/zeppelin/conf/notebook-authorization.json /user/zeppelin/conf/notebook-authorization.json.orig 3. Get the file from hdfs to local file system hdfs dfs -get /user/zeppelin/conf/notebook-authorization.json /tmp/notebook-authorization.json 4. Edit the file and replate all occurences of the username sed -i -e 's/bhushan-kandalkar@test.com/bhushan-kandalkar/g' /tmp/notebook-authorization.json 5. Upload the file to hdfs hdfs dfs -put -f /tmp/notebook-authorization.json /user/zeppelin/conf/ 6. Restart zeppelin server using ambari Let me know if that works for you. HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Mark I suggest you take the NetworkWordCount example as starting point. Then to transform the stream rdd into dataframe I recommend you look into flatMap, as you can map single column RDD into multiple columns after parsing the json content of each object. Finally when saving to hdfs you should consider a good batch size/repartition to avoid having small files in hdfs. 1. The NetworkWordCount code in github is located here: https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala 2. Here is an example of how to parse JSON using map and flatmap https://github.com/holdenk/learning-spark-examples/blob/master/src/main/scala/com/oreilly/learningsparkexamples/scala/BasicParseJson.scala 3. Saving Dataframe as ORC is very well documented. Just avoid writing small files as this will hurt namenode and your hdfs overall. HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

Hi @Bin Ye Does this snapshot continuously track the changes of this table? No, snapshot is like a photo of the table data at certain point of time. Snapshot will also mark data to prevent it from being deleted. For more information please review this HCC link posts https://es.hortonworks.com/blog/coming-hdp-2-5-incremental-backup-restore-apache-hbase-apache-phoenix/ https://community.hortonworks.com/questions/102843/hbase-snapshot-or-backup.html There are differences between snapshot and backup and incremental back up which you can review in the above link. Hopefully this will help you decide if snapshot is the correct solution for you or if you should go for backup. HTH ... View more