Solved

Hey, folks, I have a problem, I just Kerberizar my installation of my cluster in horton but I get an error that can not create a PID, try to do it by hand but delete the file and gives error. could give me a help.   Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/ranger_tagsync.py", line 133, in <module> RangerTagsync().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 355, in execute self.execute_prefix_function(self.command_name, 'post', env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 382, in execute_prefix_function method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 424, in post_start raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file)) resource_management.core.exceptions.Fail: Pid file /var/run/ranger/tagsync.pid doesn't exist after s

Hi   I try to install a fresh CFM instalation in a CM&CDH 6.2.0 cluster.   I install NiFi CA and NiFi Registry and NiFi (Parcel distribution with csd) in the cluster.   NiFi CA and NiFi Registry works fine, both services are started and looks healthy but i can't start NiFi, I install in 2 nodes to made a cluster but i can't start, always the same message:     19/10/31 11:58:12 ERROR o.a.n.b.Command: The sensitive.key file /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/sensitive.key already exists. That shouldn't have been. Aborting.Thu Oct 31 11:58:16 CET 2019JAVA_HOME=/usr/java/jdk1.8.0_121Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/nifi_nifi-NIFI_NODE-6ef7e4e859120410b823067891375e2e_pid45532.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTSUsing /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE as conf dirUsing scripts/control.sh as process scriptCONF_DIR=/var/run/cloudera-scm

Hello, I have a websocket on which I receive a text message (listenWebsocket Processor) , I want to put the whole text message as an attribute into the FetchFile processor. I want to call the value of this extracted attribute in the filepath of the processor?  ExtractText works well, the data provenance shows that there's an new attribute after ExtractText has run, but the attribute value is not set into the file path in the FetchFile Processor ?? What's wrong, did I miss some configuration? A custom attribute is called ${attributename} , right? Thanks a lot for your help Henriette  

I'm using the ConvertJSONToSQL processor, and the MySql table has a space in the name.  It appears to be the problem because when I remove the name from the table it works as expected.   I can successfully query the table from various tools and the mysql command line using back ticks "`" around the table name.   The table name:personal information   I've tried the following: `personal information` "personal information" 'personal information' \"personal information\"[personal information]   Any suggestions?

I am trying to understand how to integrate NiFi within an existing SDLC. The following are some challenges that I am trying to resolve:   1) I am aware that NiFi Registry 0.2.0 allows us to add a git flow persistence provider. This means on every commit to the Registry, the flow file will be automatically be saved to the git repo. Is it possible to take the NiFi Registry out of the picture entirely and work solely with git. I have the feeling that there is a good reason to tightly integrate NiFi with NiFi registry, I would be really helpful if some one can explain the rational behind this tight integration.   2) Is it possible to deploy a NiFi flow from an artifactory instead of the NiFi Registry? A reference to an example would be really helpful.   3) Lets say I have a bunch of NiFi custom processors that needs to be hosted on git and there is Jenkins in this SDLC. Jenkins does the auto build of each custom processors and deploys them in the artifactory. Also the flow f

Dear community,we have a problem when connecting to impala with the impala jdbc connector.Our cluster setup is:cdh 6.2/impala 3.2kerberos enabledssl activated. Our cluster 's configuration has been checked thoroughly and the cluster is correctly processing high volumes of data, using kerberos authentication and has security/sentry etc in place. SSL and kerberos processes work fine, in other words this is not a cluster we just started setting up.We have an issue when making a connection from a java program using the impala jdbc connector.I am following the documentation on https://docs.cloudera.com/documentation/other/connectors/impala-jdbc/2-6-11/Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf (-> page 12 and 88 contain detailed explanation about the connection strings and each possible option for Kerberos ).When we use a connection string including ssl and authentication mechanism 3 and provide the ldap user/password this works fine:#WORKS FINEimpalaConnectionURL=jdbc:impala

I like to change file permissions for security hardening These files have 'w' bit in others, is it possible to change to 220 manually? --w--w--w- 1 root root 0 Oct 29 18:41 /var/run/cloudera-scm-agent/cgroups/memory/<service_name>/cgroup.event_control--w--w--w- 1 root root 0 Oct 29 18:43 /var/run/cloudera-scm-agent/cgroups/memory/<service_name>/cgroup.event_control   These files are newly created every time that the service is started, can permissions be set to 220 permanently instead of manually? This change may cause problems with the service?   Thanks

I am getting an error installing ambari-server-2.7.4.0 on Centos8.   It seems as though it can't resolve conflict with python.  Although I have Python36 installed, it is still asking for python2.6 or greater.   I read your many responses but they did not help with my situation.  Any support on this matter will be appreciated.  Thanks.

There is a really good article in IBM documentation about managing queue access with ACLs (https://developer.ibm.com/hadoop/2017/06/30/managing-queue-access-acls/)   However when I change the ACL config and add specific group: yarn.scheduler.capacity.root.default.acl_submit_applications=yarn,ambari-qa,clusterusers   And after I refresh queues with: $mapred queue -showacls   I get the following output. Queue acls for user : myuser Queue Operations=====================rootdefault   My temporal solution was to set: yarn.scheduler.capacity.root.default.acl_submit_applications=*   Which works (see the output): Queue acls for user : myuser Queue Operations=====================rootdefault SUBMIT_APPLICATIONS   Note: myuser is a part of a group clusterusers   I would like to know why this config does not work if I set some groups on my own. Any ideas?

 We have nifi flows like belowListSFTP >> ... >> FetchSFTP >> ... >> MergeContent >> ... >> PutFTP- FetchSFTP fetch all files in yyyymmdd folder( ex : ./A1/20191018 ) one by one- MergeContent make all files into one zipfile.It works well, but fetching all files one by one takes long time. also it needs lots of memory. Therefore we want to change above process into belowstep1) Compress all files of one folder( ex : ./A1/20191018 --> 20191018.zip )step2) Fetch this one zipfilestep3) PutFTPIn this process, I dont know about step1How can I Compress all files in folder?Thanks All files are in Windows.Nifi is in Linux. 

We installed new ambari cluster with the following details ( we moved to redhat 7.5 instead 7.2 )   Redhat – 7.5HDP version – 2.6.4Ambari – 2.6.2   After we complete the installation , we notice about very strange behavior ( please note that this is new cluster )   On HDFS status summary, I see the following messages about under-replicated blocks   We see under replicated blocks is 12 ( while its should be 0 on new installation )   Any suggestion – why this ? I just want to say that this behavior not appears on redhat 7.2

stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in AmsCollector().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 50, in start ams_service('collector', action = 'stop') File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 61, in ams_service hbase_service('regionserver', action=action) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/hbase_service.py", line 49, in hbase_service on_timeout = format("{no_op_test} && {sudo} -H -E kill -9 `{sudo} ca

I know we have Hive properties hive.metastore.server.max.threads and hive.metastore.server.min.threads. But how to find the Active threads of HMS. And what (job or user) causing the more active threads? 

I have a requirement to move the previous day processed and merged json files into new hdfs path. The requirement is to recursively search unprocessed files and move the pending unprocessed files. Path 1 → /data/nifi/working/2019/10/source_2019_10_15.json — Daily processed files are merged under this path and gets added on daily basis. Path 2 → /data/nifi/incoming/ — The code should search if folders doesn't exist then create and move the files are just move the files if the folders are already present.   Currently, I am using nifi flow -- ListHDFS→ MoveHDFS but unable to achieve it.   Need help how this can be achieved.   Thank you for the help.    

Hi, I am moving data between hdfs directories to pick the latest updated flow file. The code should verify on the source hdfs directories to pick the latest merged json files time greater than 2 hrs and process to the target along with the sub folders if the target directory doesn't have the directories available. Push the files and append sequence number for every new file received on the same date and delete it from the source directory after processing it. If new files received, then reprocess with new sequence number. Source hdfs path:-/data/json/incoming/year=2019/month=10/day=22/$flow-fileTarget hdfs path:-/data/json/final/$path/$flow-fileFilename(received) :- source_es_2019_10_21.jsonlFilename(required post processing) :-source_es_2019_10_21_1.jsonlsource_es_2019_10_21_2.jsonlsource_es_2019_10_21_3.jsonl I am currently using nififlow listhdfs->updateattribute->puthdfs->deletehdfs I have completed all the activates except generating sequence numb

While starting the spark2history in amzon2 os hdp 3.1 i am getting the below error.    19/10/24 10:44:48 INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions19/10/24 10:44:49 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.Exception in thread "main" java.lang.reflect.InvocationTargetException   at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)Caused by: java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-539041147636939375.8: /tmp/libleveldbjni-64-1-539041147636939375.8: failed to map segment from shared object]at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)at org.fusesource.leveldb

Hello,I need to download HDP 2.6.5.When I try to download it from  https://www.cloudera.com/downloads/hdp.htmlthere is only Older Versions3.0.13.0.0

We are running a non-kerberized cluster with around ten nodes. However, the workbench was displaying Kerberos configuration for Hadoop Authentication, because the host of the workbench had a krb5.conf file. As described in the docs (https://www.cloudera.com/documentation/data-science-workbench/1-5-x/topics/cdsw_kerberos.html) we stoppend the workbench, deleted the file, restarted the service. However, now we're encountering this roblem when starting new sessions on the workbench. Which configuration has to be reset to get rid of this?  

when am connecting to thrift server using beeline it's showing  below error Beeline version 1.2.1.spark2 by Apache Hivebeeline> !connect jdbc:hive2://host:10000Connecting to jdbc:hive2://host:10000Enter username for jdbc:hive2://host:10000: admin-sbarik2Enter password for jdbc:hive2://host:10000: ********org/apache/hive/service/cli/thrift/TCLIService$Iface0: jdbc:hive2://host:10000 (closed)>   it's automatically closed.

Spark2 2.3.0 on cdh.5.13 deployed with CSD and parcels. by default it uses /etc/spark2/conf,but it seems it should use /etc/spark2/conf.cloudera.spark2_on_yarn/ is there a way to set it properly?i could set it in spark2-env.sh but it seems not the appropriate place.

Hi, I'm going to install Cloudera on a customer required Oracle for store CDH Metadata. System DBA asked me if the property ALTER user hue quota 100m on system;is mandatory and why. Someone knows if I can skip safely this command? Thanks in advance.Roberto

Hi Team, I have an impala table column under this format, 2019-Oct-14 20:00:01.027898 as string but I want to insert in to another table as timestamp. I tried so many ways but its giving me null. Could you please guide me the easiest way to store this formatted string as timestamp in impala. Thanks,RNN

I installed CDH 6.3.0 from the repo given in the documentation. Configured a local permanent repository, followed all. Edited /etc/hosts with only one server and localhost "3.211.103.106 ec2-3-211-103-106.compute-1.amazonaws.com foo1 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6". "netstat -nap |grep 7182|grep LISTEN" is also showing up  tcp 0 0 0.0.0.0:7182 0.0.0.0:* LISTEN 10662/java "Main and jps" up too  Curling also shown up  Now the question is "Web UI is not up on 7180 nor 7182". Pasting the log as well 2019-10-22 13:39:04,316 INFO main:com.cloudera.server.cmf.Main: ================================================================================2019-10-22 13:39:04,323 INFO main:com.cloudera.server.cmf.Main: Starting SCM Server. JVM Args: [-Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties, -Dfile.encoding=UTF-8, -Dcmf.root.logger=INFO,LOGFILE, -D

Hello,   My goal is provide documentation on remote access to Hive and we primarily use Mac in our department. I was able to install beeline based on the following blog, link   I know beeline is working correctly as I’m able to connect to small CDH 5.14.2 cluster that does not have Kerberos or TLS enabled.   When connecting to our Kerberos and TLS enable cluster, I get the following error message:     readlink: illegal option -- fusage: readlink [-n] [file ...]scan complete in 2msConnecting to jdbc:hive2://FQDN.HOST:10000/default;principal=hive/FQDN_HOSTNAME@REALMSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/Users/user_name/opt/apache-hive-1.1.0-bin/lib/hive-jdbc-1.1.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/Users/user_name/opt/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See <a href="<a href="http://ww

Hi All, I am moving data between hdfs directories to pick the latest updated flow file. The requirement is to move the merged json files. The code should verify on the source hdfs directories to pick the json files and process to the target along with the sub folders if the target directory doesn't have the directories available.Source hdfs path:-/data/json/incoming/year=1970/month=01/day=18/$flow-fileTarget hdfs path:-/data/json/final/$path/$flow-fileI am currently using listhdfs->updateattribute->movehdfsIn update attribute I have provided property(filename): ${path}.substringAfterLast("/")path: ${filename:substringBeforeLast('/')} Movehdfs output directory has Output Directory - /data/json/final/${path} Could you please verify the flow and help to resolve the issue as I am unable to pull the files into target using movehdfs along with copying the sub-folders if it not already present.