Q&A

There is a really good article in IBM documentation about managing queue access with ACLs (https://developer.ibm.com/hadoop/2017/06/30/managing-queue-access-acls/)   However when I change the ACL config and add specific group: yarn.scheduler.capacity.root.default.acl_submit_applications=yarn,ambari-qa,clusterusers   And after I refresh queues with: $mapred queue -showacls   I get the following output. Queue acls for user : myuser Queue Operations=====================rootdefault   My temporal solution was to set: yarn.scheduler.capacity.root.default.acl_submit_applications=*   Which works (see the output): Queue acls for user : myuser Queue Operations=====================rootdefault SUBMIT_APPLICATIONS   Note: myuser is a part of a group clusterusers   I would like to know why this config does not work if I set some groups on my own. Any ideas?

 We have nifi flows like belowListSFTP >> ... >> FetchSFTP >> ... >> MergeContent >> ... >> PutFTP- FetchSFTP fetch all files in yyyymmdd folder( ex : ./A1/20191018 ) one by one- MergeContent make all files into one zipfile.It works well, but fetching all files one by one takes long time. also it needs lots of memory. Therefore we want to change above process into belowstep1) Compress all files of one folder( ex : ./A1/20191018 --> 20191018.zip )step2) Fetch this one zipfilestep3) PutFTPIn this process, I dont know about step1How can I Compress all files in folder?Thanks All files are in Windows.Nifi is in Linux. 

We installed new ambari cluster with the following details ( we moved to redhat 7.5 instead 7.2 )   Redhat – 7.5HDP version – 2.6.4Ambari – 2.6.2   After we complete the installation , we notice about very strange behavior ( please note that this is new cluster )   On HDFS status summary, I see the following messages about under-replicated blocks   We see under replicated blocks is 12 ( while its should be 0 on new installation )   Any suggestion – why this ? I just want to say that this behavior not appears on redhat 7.2

stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in AmsCollector().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 50, in start ams_service('collector', action = 'stop') File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 61, in ams_service hbase_service('regionserver', action=action) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/hbase_service.py", line 49, in hbase_service on_timeout = format("{no_op_test} && {sudo} -H -E kill -9 `{sudo} ca

I know we have Hive properties hive.metastore.server.max.threads and hive.metastore.server.min.threads. But how to find the Active threads of HMS. And what (job or user) causing the more active threads? 

I have a requirement to move the previous day processed and merged json files into new hdfs path. The requirement is to recursively search unprocessed files and move the pending unprocessed files. Path 1 → /data/nifi/working/2019/10/source_2019_10_15.json — Daily processed files are merged under this path and gets added on daily basis. Path 2 → /data/nifi/incoming/ — The code should search if folders doesn't exist then create and move the files are just move the files if the folders are already present.   Currently, I am using nifi flow -- ListHDFS→ MoveHDFS but unable to achieve it.   Need help how this can be achieved.   Thank you for the help.    

Hi, I am moving data between hdfs directories to pick the latest updated flow file. The code should verify on the source hdfs directories to pick the latest merged json files time greater than 2 hrs and process to the target along with the sub folders if the target directory doesn't have the directories available. Push the files and append sequence number for every new file received on the same date and delete it from the source directory after processing it. If new files received, then reprocess with new sequence number. Source hdfs path:-/data/json/incoming/year=2019/month=10/day=22/$flow-fileTarget hdfs path:-/data/json/final/$path/$flow-fileFilename(received) :- source_es_2019_10_21.jsonlFilename(required post processing) :-source_es_2019_10_21_1.jsonlsource_es_2019_10_21_2.jsonlsource_es_2019_10_21_3.jsonl I am currently using nififlow listhdfs->updateattribute->puthdfs->deletehdfs I have completed all the activates except generating sequence numb

While starting the spark2history in amzon2 os hdp 3.1 i am getting the below error.    19/10/24 10:44:48 INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions19/10/24 10:44:49 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.Exception in thread "main" java.lang.reflect.InvocationTargetException   at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)Caused by: java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-539041147636939375.8: /tmp/libleveldbjni-64-1-539041147636939375.8: failed to map segment from shared object]at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)at org.fusesource.leveldb

Hello,I need to download HDP 2.6.5.When I try to download it from  https://www.cloudera.com/downloads/hdp.htmlthere is only Older Versions3.0.13.0.0

We are running a non-kerberized cluster with around ten nodes. However, the workbench was displaying Kerberos configuration for Hadoop Authentication, because the host of the workbench had a krb5.conf file. As described in the docs (https://www.cloudera.com/documentation/data-science-workbench/1-5-x/topics/cdsw_kerberos.html) we stoppend the workbench, deleted the file, restarted the service. However, now we're encountering this roblem when starting new sessions on the workbench. Which configuration has to be reset to get rid of this?  

when am connecting to thrift server using beeline it's showing  below error Beeline version 1.2.1.spark2 by Apache Hivebeeline> !connect jdbc:hive2://host:10000Connecting to jdbc:hive2://host:10000Enter username for jdbc:hive2://host:10000: admin-sbarik2Enter password for jdbc:hive2://host:10000: ********org/apache/hive/service/cli/thrift/TCLIService$Iface0: jdbc:hive2://host:10000 (closed)>   it's automatically closed.

Spark2 2.3.0 on cdh.5.13 deployed with CSD and parcels. by default it uses /etc/spark2/conf,but it seems it should use /etc/spark2/conf.cloudera.spark2_on_yarn/ is there a way to set it properly?i could set it in spark2-env.sh but it seems not the appropriate place.

Hi, I'm going to install Cloudera on a customer required Oracle for store CDH Metadata. System DBA asked me if the property ALTER user hue quota 100m on system;is mandatory and why. Someone knows if I can skip safely this command? Thanks in advance.Roberto

Hi Team, I have an impala table column under this format, 2019-Oct-14 20:00:01.027898 as string but I want to insert in to another table as timestamp. I tried so many ways but its giving me null. Could you please guide me the easiest way to store this formatted string as timestamp in impala. Thanks,RNN

I installed CDH 6.3.0 from the repo given in the documentation. Configured a local permanent repository, followed all. Edited /etc/hosts with only one server and localhost "3.211.103.106 ec2-3-211-103-106.compute-1.amazonaws.com foo1 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6". "netstat -nap |grep 7182|grep LISTEN" is also showing up  tcp 0 0 0.0.0.0:7182 0.0.0.0:* LISTEN 10662/java "Main and jps" up too  Curling also shown up  Now the question is "Web UI is not up on 7180 nor 7182". Pasting the log as well 2019-10-22 13:39:04,316 INFO main:com.cloudera.server.cmf.Main: ================================================================================2019-10-22 13:39:04,323 INFO main:com.cloudera.server.cmf.Main: Starting SCM Server. JVM Args: [-Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties, -Dfile.encoding=UTF-8, -Dcmf.root.logger=INFO,LOGFILE, -D

Hello,   My goal is provide documentation on remote access to Hive and we primarily use Mac in our department. I was able to install beeline based on the following blog, link   I know beeline is working correctly as I’m able to connect to small CDH 5.14.2 cluster that does not have Kerberos or TLS enabled.   When connecting to our Kerberos and TLS enable cluster, I get the following error message:     readlink: illegal option -- fusage: readlink [-n] [file ...]scan complete in 2msConnecting to jdbc:hive2://FQDN.HOST:10000/default;principal=hive/FQDN_HOSTNAME@REALMSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/Users/user_name/opt/apache-hive-1.1.0-bin/lib/hive-jdbc-1.1.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/Users/user_name/opt/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See <a href="<a href="http://ww

Hi All, I am moving data between hdfs directories to pick the latest updated flow file. The requirement is to move the merged json files. The code should verify on the source hdfs directories to pick the json files and process to the target along with the sub folders if the target directory doesn't have the directories available.Source hdfs path:-/data/json/incoming/year=1970/month=01/day=18/$flow-fileTarget hdfs path:-/data/json/final/$path/$flow-fileI am currently using listhdfs->updateattribute->movehdfsIn update attribute I have provided property(filename): ${path}.substringAfterLast("/")path: ${filename:substringBeforeLast('/')} Movehdfs output directory has Output Directory - /data/json/final/${path} Could you please verify the flow and help to resolve the issue as I am unable to pull the files into target using movehdfs along with copying the sub-folders if it not already present. 

I am building a 4 node machine and i want to enable hdfs and rm HA. The hdp version is 3.1.0 and has the yarn ats-hbase app always running.  I wanted to know is it mandatory to destroy the app as mentioned https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/data-operating-system/content/remove_ats_hbase_before_switching_between_clusters.html before enabling HA? Thanks a lot for your help in advance 

I can't find where could I disable authentication for a Web UI. I'm using HDP 3.1. Any help appreciated!  

hiI created a CentOS 7.7 Linux machine (but also happens with RHEL 7.4) on Azure.I installed the cloudera-manager RPM packages via yumWhen i try to start the cloudera-scm-server-db service I get "permission denied" on /var/lib/cloudera-scm-server-db/dataI added -x flag to the /usr/share/cmf/bin/initialize_embedded_db.sh, and added the "strace" command before the initdb command.I see that the command is running as cloudera-scm user (in the /var/log/messages file)and Is see in the strace output, that it fails on "chmod("/var/lib/cloudera-scm-server-db/data", 0700) = -1 EACCES (Permission denied)"When I run the command from Linux CLI as user cloudera-scm it runs OK.Does the Linux behave differently on Azure (i.e. more security features)I don't think it relates to firewallI see in the /var/log/messages the following messages:Oct 22 12:53:14 cdh-doxi-1 systemd: Created slice User Slice of cloudera-scm.Oct 22 12:53:14 cdh-doxi-1 systemd: Started Session c1 of user cloudera-scm

Hi  Team,   Does kudu tables show up in Cloudera Navigator search for metadata. These tables are created in Impala as Kudu storage type and I'm unable to see them in Navigator. I did not apply any filters in search.   Any possible reason?   Thanks, Meher  

Hello, Friends.   I have a problem with Ranger, I have a Kerberized Cluster, I try to access Ranger UI with user admin:admin and I get error" The username or password you entered is incorrect". likewise changing.   Ranger Admin username for AmbariRanger Admin user's password   Even changing and restarting ambari I can't get in. Could you help me?   Greetings

I am tyring to get this output. Nevertheless some articles points out this is due to a version matter, which I doubt, since I've seen this behavior on multiple environments with different versions is just that I don't know how to enable it. If someone can point me to the right path I would highly appreciate it. Regards!

I would to write a script which will check the status of Hivserver2 is running or not. I would like to connect beeline and show databases. if i get the output then Hiveserver 2 is running. Can any one help in writing the script for this ?

Hi All, I'm trying to submit following Spark2 job on CDH 5.16 cluster and it's only taking first parameter of --packages option and throwing error for second parameter spark2-submit --packages com.databricks:spark-xml_2.11:0.4.1, com.databricks:spark-csv_2.11:1.5.0 /path/to/python-script spark2-submit --packages com.databricks:spark-xml_2.11:0.4.1, com.databricks:spark-csv_2.11:1.5.0 /path/to/python-script.pyException in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR com.databricks:spark-csv_2.11:1.5.0 with URI com.databricks. Please specify a class through --class. at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657) at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:224) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116) at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(