@Johnny Fugers From what I understand is that you have set of files inside a hdfs directory which is split by dates, would the directory structure look like this: /user/test/data/2016-03-12/<FILES> /user/test/data/2016-03-13/<FILES> /user/test/data/2016-03-14/<FILES> Like these, if yes then you will not be able to create a partition based table, what you can do is to create a regular table pointing the location to /user/test/data and set the below properties, this will look at all the files inside the subdirectory and get the result set. set hive.input.dir.recursive=true; set hive.mapred.supports.subdirectories=true; set hive.supports.subdirectories=true; set mapreduce.input.fileinputformat.input.dir.recursive=true; ... View more

@Sagar Shimpi By default the shell actions are not allowed to run as another user as sudo is blocked. If you want a yarn application to run as someone other than yarn (i.e. the submitter), then you need to enable the linux container executor so that the containers are started up by the submitting user. Also note the below setting information which also needs to be changed as well to achieve this. With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false (default), it runs as yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (default is 'nobody') With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=true, it runs as the user submitting the workflow. Stating that there are issues around this also where it does not work as expected because of the issues https://issues.apache.org/jira/browse/YARN-2424 https://issues.apache.org/jira/browse/YARN-3462 The current suggestion that I can make is to add line to change the ownership of the file which was created using shell. ... View more