Hue Directory Monitoring

[#1]

Hello,

is it possible to monitor a Hue directory using, say, a User Defined Function?
If so, can you provide an example?

Many thanks.

[#2]

@phir1 , 

Can you give us more context on the use-case, please? What type of monitoring you're planning to implement? And, by Hue directory, which path are you referring to? 

[#3]

I'd like to write Scala/Java or Python code which would monitor Home Directories (e.g., /Users/phil) for:

- files larger than, say, 10GB

- files containing private data, such as bank accounts, email addresses, etc.

[#4]

I'm not sure if we have example scripts or custom functions, but the general idea to achieve what you're trying to do would be - 

1. Referring the HDFS reporting in CM or parse the fsimage (for large files).
2. Write a MapReduce/Spark job to scan files, or run a query in Hive/Impala/SparkSQL to see if data files are mapped to Hive tables.

You can also implement access controls through Ranger, enable transparent disk encryption (TDE) using Ranger KMS, tag sensitive hive columns (containing PII, PCI, PHI) using Atlas classification and assign tag-based masking policies from Ranger, implement navencrypt to encrypt the spill files, et. - if you're planning to protect the sensitive data from unauthorised access. 

[#5]

@phir1, Did @Sean464's responses assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.