Member since
04-03-2019
100
Posts
8
Kudos Received
7
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 224 | 06-02-2026 10:40 AM | |
| 2019 | 01-13-2025 11:17 AM | |
| 8751 | 01-21-2022 04:31 PM | |
| 7879 | 02-25-2020 10:02 AM | |
| 5889 | 02-19-2020 01:29 PM |
02-25-2020
10:02 AM
I got the following responses from Cloudera Certification. Regarding Question #1, the FAQ page has the most the up-to-date information. So right now I'd better hold off purchasing the exam until the DE575 is relaunched. Regarding Question #2, the course is the "Spark and Hadoop Developer" training course is the one I should take for preparing DE575. Regarding Question #3, the environment for the exam is fixed and only available on CDH. Candidates do not have the option to take the exam in an HDP environment. The skills tested are applicable to HDP development as well, it is in the developer track, so it should have nothing to do with the environment that it is running in. It is primarily interested in transforming data that sits on the cluster.
... View more
02-19-2020
01:29 PM
1 Kudo
Finally, I figured out what is going on. The root cause is that, I only set up testuser on edge nodes, not the name node. I looked into this page, https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/GroupsMapping.html, which shows that "For HDFS, the mapping of users to groups is performed on the NameNode. Thus, the host system configuration of the NameNode determines the group mappings for the users." After I created the user on the NameNode and ran the command hdfs dfsadmin -refreshUserToGroupsMappings the copy is successful and there is no permission-denied error.
... View more
02-10-2020
11:51 AM
@GangWar Here it is. $ id -Gn testuser hadoop wheel hdfs
... View more
02-10-2020
09:05 AM
I have run the following test case several times and got the same result. Context: 1. My HDP cluster uses the simple mode to determine user identity. Kerberos is not enabled. 2. Below is the permission on hdfs folder /data/test
drwxrwxr-x - hdfs hadoop 0 2020-02-07 13:33 /data/test
So hdfs (the super user) is the owner and hadoop is the owner group. Both the owner user and owner group has write permission on the /data/test folder.
Steps:
On an edge node, I used ID command to confirm that the logged on user "testuser" is in hadoop group.
$ id
uid=1018(testuser) gid=1003(hadoop) groups=1003(hadoop),10(wheel), 1002(hdfs) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
However, testuser still ran into "Permission Denied".
$ hadoop fs -put ./emptyfile1.txt /data/test
put: Permission denied: user=testuser, access=WRITE, inode="/data/test":hdfs:hadoop:drwxrwxr-x
Then I used hdfs account to changed the folder owner to be testuser.
$ hadoop fs -chown testuser /data/test
From the same edge node, now testuser ran put command successfully.
Here is my question: why testuser cannot write to the hdfs folder via the owner group permissions?
... View more
Labels:
- Labels:
-
HDFS
-
Hortonworks Data Platform (HDP)
01-31-2020
09:07 AM
@cjervis Thanks. I reviewed the FAQ page, but it does not answer my questions. I guess I'd better wait until tomorrow, because the page mentioned the date February 1, 2020 several times for new launches or other changes.
... View more
01-31-2020
08:22 AM
I plan to get a Cloudera certification and need help on following questions: Question #1. I reviewed the page https://www.cloudera.com/about/training/certification.html, It looks like that CCP Data Engineer is the only certification that has not been suspended or retired. Am I right on this? Question #2. To prepare DE575, the only recommended Cloudera course is "Spark and Hadoop Developer" training course. according to this page. https://www.cloudera.com/about/training/certification/ccp-data-engineer.html. Should I consider other courses? Questions #3. My workplace uses HDP. Do I need to get familiar with products like CDH before taking the exam?
... View more
Labels:
- Labels:
-
Certification
01-15-2020
10:02 AM
@Shelton @EricL Thank you both. the correct ACL spec is group::r-x Now the following command works. sudo -u zeppelin hadoop fs -ls /warehouse/tablespace/managed/hive/test1 From what I just ran into, I feel that, by design, Hive takes extra effort to prevent users from accessing managed table files directly. I will follow that design and access Hive managed table only through Hive.
... View more
01-14-2020
05:09 PM
I tried the following command # sudo -u hdfs hadoop fs -setfacl -m g::rx /warehouse/tablespace/managed/hive/test1 But I got the error -setfacl: Invalid type of acl in <aclSpec> :g::rx The acl spec is to modify the owning group permission to rx. Any suggestion?
... View more
01-14-2020
09:49 AM
I might have found the reason. I ran the following command as hdfs, which is the superuser of hdfs. $ hadoop fs -getfacl /warehouse/tablespace/managed/hive/test1 # file: /warehouse/tablespace/managed/hive/test1 # owner: hive # group: hadoop user::rwx user:hive:rwx group::--- mask::rwx other::--- default:user::rwx default:user:hive:rwx default:group::--- default:mask::rwx default:other::--- The output, as I understand, shows that the group owner has no permission on the folder. My guess is that, HDP Hive uses ACL to limit direct access to files behind managed tables. HDP Hive tries to force accessing to managed tables only through Hive.
... View more
01-14-2020
08:05 AM
# hdfs groups zeppelin zeppelin : hadoop zeppelin On the name node, # id zeppelin uid=1018(zeppelin) gid=1003(hadoop) groups=1003(hadoop),1005(zeppelin)
... View more