I saved thousands of small json files in SequenceFile format to resolve the "small file issue". I use the following pyspark code to parse the json data from saved sequence files.     reader= sc.sequenceFile("/mysequencefile_dir", "org.apache.hadoop.io.Text", "org.apache.hadoop.io.Text") rdd=reader.map(lambda x: x[1]) mydf=spark.read.schema(myschema).json(rdd) mydf.show(truncate=False)    The code worked. However, I do not know how to put the key value from the sequence file, which is actually the original json file name, into the mydf dataframe. Please advise. Thank you.   Regards, ... View more

Here is the code. create external table testtable1 (code string, codesystem string) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = "(.{27)(.{50})" ) LOCATION '/data/raw/testtable1'; The error message is: ParseException: Syntax error in line 3:undefined: ROW FORMAT SERDE 'org.apache.hadoop.hiv... ^ Encountered: IDENTIFIER Expected: DELIMITED CAUSED BY: Exception: Syntax error It looks like Impala table only accepts "Row Format Delimited". Then how can I create an hive table with fixed width layout? Should I just do it outside Impala, bu through Hive, and then do other data operation on this table via Impala? Thanks. ... View more

I saw this kind of notation/style using double curly braces everywhere in HDP(Ambari) or CDP (CMS) UI. Below is a configuration value under zeppelin.shiro.knox.main.block for Zeppelin configuration. (This is a random sample I picked and this question is not about Zeppelin.) ++ krbRealm.signatureSecretFile={{CONF_DIR}}/http_secret ++ I understand that I can simply overwrite  {{CONF_DIR}} with the actual path. However, I wonder whether  {{CONF_DIR}}  an ansible variable? If yes, how do I define the variable CONF_DIR in CDP Cloudera Manager? https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#defining-simple-variables   Regards, ... View more

I am using CDP 7.1.7 and the cluster has not enabled Kerbores yet. Ranger is not enabled either. I followed the step in this post https://community.cloudera.com/t5/Support-Questions/CDP-7-1-3-Zepplin-not-able-to-login-with-default-username/td-p/303717 to be able to log in as admin. But this "admin" account has no permission to access the configuration or interpreter page.    According to CDP documentation, https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/configuring-zeppelin/topics/enabling_access_control_for_interpreter__configuration__and_credential_settings.html, to configure shiro.ini for Zeppelin security, I have to go through Zeppelin web UI.    What should I do?   Regards,   ... View more

I am trying to parse a nested json document using RDD rather than DataFrame. The reason I cannot use DataFrame (the typical code is like spark.read.json) is that the document structure is very complicated. The schema detected by the reader is useless because child nodes at the same level have different schemas. So I try the script below.   import json s='{"key1":{"myid": "123","myname":"test"}}' rdd=sc.parallelize(s).map(json.loads)   My next step will be using map transformation to parse json string but I do not know where to start. I tried the script below but it failed.   rdd2=rdd.map(lambda j: (j[x]) for x in j)   I would appreciate any resource on using RDD transformation to parse json. ... View more

I will use Spark2 in CDP and need to install Python3. Do I need to installation Python3 on every node in the CDP cluster, just only need to install it on one particular node?   Spark2 job is executed in JVM containers that could be created on any worker node. I wonder whether the container is created upon a template? If yes, then how the template is created and where is it?   Thanks.   ... View more

I have run the following test case several times and got the same result. Context: 1. My HDP cluster uses the simple mode to determine user identity. Kerberos is not enabled. 2. Below is the permission on hdfs folder /data/test   drwxrwxr-x - hdfs hadoop 0 2020-02-07 13:33 /data/test   So hdfs (the super user) is the owner and hadoop is the owner group. Both the owner user and owner group has write permission on the /data/test folder. Steps: On an edge node, I used ID command to confirm that the logged on user "testuser" is in hadoop group.   $ id uid=1018(testuser) gid=1003(hadoop) groups=1003(hadoop),10(wheel), 1002(hdfs) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023   However, testuser still ran into "Permission Denied".   $ hadoop fs -put ./emptyfile1.txt /data/test put: Permission denied: user=testuser, access=WRITE, inode="/data/test":hdfs:hadoop:drwxrwxr-x   Then I used hdfs account to changed the folder owner to be testuser.     $ hadoop fs -chown testuser /data/test     From the same edge node, now testuser ran put command successfully.   Here is my question: why testuser cannot write to the hdfs folder via the owner group permissions?   ... View more

I plan to get a Cloudera certification and need help on following questions: Question #1. I reviewed the page https://www.cloudera.com/about/training/certification.html, It looks like that CCP Data Engineer is the only certification that has not been suspended or retired.  Am I right on this? Question #2. To prepare DE575, the only recommended Cloudera course is "Spark and Hadoop Developer" training course. according to this page. https://www.cloudera.com/about/training/certification/ccp-data-engineer.html. Should I consider other courses? Questions #3. My workplace uses HDP. Do I need to get familiar with products like CDH before taking the exam?  ... View more