@martinbo   Regarding multiple user creation on multiple nodes, you have to use configuration tools like puppet, chef, ansible, etc   You were asking only about creating a new user in each node, but in real time, your requirement will be extended as follows: 1. Create/modify user at each node 2. Setup temporary password if you don't have sso 3. Create/modify multiple user-groups at each node (admin group, developer group, tester group, analyst, etc) 4. Assign each user to the corresponding user-groups 5. Create a home directory to the each user, setup quota if needed 6. Setup permission & owner to each home directory (as other user should not access) 7. etc   There are so many other activities we can do with this tool, but i've listed few based on your requirement... hope it will help     ... View more

  @Rod   CDH 6.0 will support Spark 2.2.0 and Spark 2.2.0 will support Spark SQL as per below link   https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#sql   Not sure this is the answer for your question, if not, please give some more details ... View more

@KeepCalmNCode   You have mentioned there are lot of small files. And you set the block.size as 445644800 (which is 445 MB approx)   If your block.size > small file  then  you will not find any difference   Ex: All the below will give the same result 445 MB > 1 MB  400 MB > 1 MB 300 MB > 1 MB 200 MB > 1 MB 100 MB > 1 MB 10 MB > 1 MB 2 MB > 1 MB   may be you will find difference in file size when you set the block.size < small file       ... View more

@Sumit   The link that you have provided says "If you have selected IAM authentication, no additional steps are needed" - which includes editing the core-site.xml   You have confirmed that added "AWS Credentials" in "External Accounts" from "Administration" menu - which is IAM role based authentication.    so you don't need to do both ... View more

@Sumit   1. What is your Cloudera Version? If you are using Cloudera 5.10 or above then you can follow the instruction from the link that i've given above espectially 'Adding AWS Credentials' . You didn't mentioned whether you have already tried or not 2. Not sure you have restarted your cluster after the configuration change 3. If you follow this option, it will be applicable for all the users 4. I don't know which blog that you are following and how old it is... if you are using cloudera then use cloudera document       ... View more

@Sumit   Pls refer the below link, it will explain  S3 as storage for Impala tables S3 as a source or destination for HDFS and Hive/Impala replication and for cluster storage Where to update the credentials, etc https://www.cloudera.com/documentation/enterprise/5-12-x/topics/cm_auth_aws.html#concept_tmd_nsh_2y     ... View more

@AcharkiMed   Basically this is a recommendation, still it will be considered as Mandatory or Optional depends upon the environment that you are using Ex: *) For Prod Env - It is mandatory. Otherwise it will create performance difference if there is any switch between active and standby NN *) For test/POC Env - It is optional if you don't have right choice ... View more

  @hendry   There could be two possibilities for this scenario   1. May be the hive and impala tables are referring to the two different files. But chances are less for this scenario unless any minor mistakes in the tables (or) some other internal error   You can confirm this by  > describe formatted db.tablename  Run this command from both hive and impala and get the location and compare   2. Your file has duplicate records. I mean some key values are same but other columns may have different value. So it may return different value when you filter. So check your data in detail       ... View more