Hi All I am trying to get all hive queries for last 30 days ran on my cluster. I selected all the jobs on cloudera YARN application UI and filtered for last 30 days and also selected attribute of hive_query_string, which allows me to see the actual query. The only issue is that cloudera restricts UI to show only last 100 jobs at a time. Because of this i can't get details on all the other queries. I tried to hit this api http://cluster:8088/ws/v1/cluster/apps to get all the details, two issue: there is no filter and other is there is no hive_query_string variable to filter. It just shows me all the job details. Is there an API exposed by cloudera where i can filter and get all the hive_query_string or if i can configure the cloudera UI to show me more than 100 jobs(or just export it).   Let me know.   Thanks AB ... View more

Hi All   I have been trying to submit below spark job in cluster mode through a bash shell. Client mode submit works perfectly fine. But when i switch to cluster mode, this fails with error, no app file present. App file refers to missing application.conf.   spark-submit \ --master yarn \ --deploy-mode cluster \ --class myCLASS \ --properties-file /home/abhig/spark.conf \ --files /home/abhig/application.conf \ --conf "spark.executor.extraJavaOptions=-Dconfig.resource=application.conf -Dlog4j.configuration=/home/abhig/log4.properties" \ --driver-java-options "-Dconfig.file=/home/abhig/application.conf -Dlog4j.configuration=/home/abhig/log4.properties" \ /loca/project/gateway/mypgm.jar   I followed the link below on similar post https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-File-not-found-error-works-fine-in-local-mode-but-failed/m-p/32306#M1149  This solution mentioned is still not clear.   I even tried   --files $CONFIG_FILE#application.conf Still it doesn't work. Any help will be appreciated.   Thanks AB ... View more

Hi All How do i ensure that the child dir and files created by a member of a group having rwx permissions on hdfs have the same rwx permission as parent? I tried chmod and acls both as suggested by apache and cloudera. All the new dirs created by a user in a group having permission to write are still having the r-x permissions instead of rwx which i want. I have also enabled dfs.namenode.posix.acl.inheritance.enabled to true and dfs.permissions also to true as mentioned in https://issues.apache.org/jira/browse/HDFS-6962. fs.permissions.umask-mode=000 dfs.umaskmode, fs.permissions.umask-mode=022 [root@dev ~]# id abhig uid=515(abhig) gid=519(abhig) groups=519(abhig),525(low_priority),528(devgrp) ******************************************** [abhig@dev ~]$ hdfs dfs -setfacl -m default:group:devgrp:rwx /test [abhig@dev ~]$ hdfs dfs -getfacl /test # file: /test # owner: abhig # group: devgrp user::rwx group::r-x other::r-x default:user::rwx default:group::r-x default:group:devgrp:rwx default:mask::rwx default:other::r-x ******************************************** [abhig@dev ~]$ hdfs dfs -mkdir /test/tst1 [abhig@dev ~]$ hdfs dfs -getfacl /test/tst1 # file: /test/tst1 # owner: abhig # group: devgrp user::rwx group::r-x group:devgrp:rwx #effective:r-x mask::r-x other::r-x default:user::rwx default:group::r-x default:group:devgrp:rwx default:mask::rwx default:other::r-x ********************************************* This doesn't help much https://community.cloudera.com/t5/Storage-Random-Access-HDFS/HDFS-ACL-Inheritance/m-p/25494#M1092    Please give a workaround if any. ... View more

Hi Has anyone faced issue with impala catalog server creating jar files and storing it in /tmp dir on the node where impala catalog server is running? Like below. Everytime i run invalidate metadata on any database, it creates 2 jar files like below and doesnt delete those after it finishes. These files keep on increarsing and overtime start causing below issue Query: invalidate metadata ERROR: FSError: java.io.IOException: No space left on device CAUSED BY: IOException: No space left on device   Also resulting in failure of catalog server and impala daemon eventually. This happens with only one of my cluster. All my clusters are running on same version. No change done recently. Using CDH 5.8.3, Impala Shell v2.6.0-cdh5.8.3.   Jar files created on running invalidate metadata command. -rw-r--r-- 1 impala impala 68246195 Apr 4 12:54 0079d271-f044-46be-9580-7d98cd4fced2.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:53 1dd950d5-2db9-4a88-9043-02dff869697c.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:53 2c910b59-83fc-4cde-96d3-3ac086f9fcb2.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 13:02 318ef680-2ea2-47fb-b688-8df439af676c.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:50 3b15f36a-5353-4553-bee6-96c5ac807703.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:53 43e13701-dad8-4892-a19c-43125dbaf1e1.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:50 51a5b806-fb0c-444a-a989-87e832501711.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:55 54ef3cd7-ea8d-4932-a032-9b3f63f5a60b.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:58 81dd054d-d720-4d11-826a-eba2518e4381.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:53 93cc7752-c80c-4f70-a47c-14681fd5c487.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:58 9a25bded-1cd7-4773-8962-0fc41705f728.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:52 a70b6d71-191e-494a-a78e-63a03642e0e9.jar -rw-r--r-- 1 impala impala 68246195 Apr 4 12:54 a7a102f8-9abe-4388-90a4-8abd85bb9c09.jar   ... View more