Q&A

Hi I am running out of space on one of our DN, and found that Yarn container logs are consuming around 50GB, though there are 3 Spark jobs/applications running which can be seen under Yarn > Applications tab, but I see a lot of other application ids also listed, would like to know if these could be deleted to reclaim space (the ones which are not running)Secondly, any precautions to be taken into consideration while deleting these container logs. Any guidance would be of great help. RegardsWert  

Hello,I am working with cloudera flow manager version 1.0.1. I cannot got login page when i enable ssl & ldap in nifi registry instead of this page, and there is an node identity.  I have checked the config that may be correctly. and try more time such as remove /var/lib/nifiregister/* or /var/run/cloudera-scm-agent/process/***-nifiregistry-NIFI_REGISTRY_SERVER/* the blow pic shows the login user is node id. the related ldap info is below, but there is not userDN config option in cloudera manager. there is same setting between nifi and nifi registry.2019-11-02 10:24:59,167 INFO org.springframework.ldap.core.support.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations The behavior is very strange. Who could help me what i missed?Thanks,Paul

Hi, I scheduled exam before 1 month and it got cancelled due to Examination cluster issue.After waiting for certain time i cancelled my registration to find the alternatives.Now i am about to schedule alternat exam but before that i would like to know availability of CCA175.Please responde if any one gave exam in last week. @cjervis : It would be really helpful if you can contact your sources to identify the status of exam cluster issue.

Hi Team, We are facing an issue where mapreduce jobs are failing with below error.Distribution: Cloudera (CDH 5.14.4) ====Exception in thread "main" java.io.FileNotFoundException: /tmp/hadoop-unjar5272208588996002870/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore$AsyncClient$revoke_privileges_call.class (No space left on device)        at java.io.FileOutputStream.open0(Native Method)        at java.io.FileOutputStream.open(FileOutputStream.java:270)        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:110)        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)       

Is there a difference between installing ambari via the apache docs vs Hortonworks docs? I assume that the end result is exactly the same since Hortonworks labels the distribution in the docs as "Apache" and the repo they instruct to add uses apache.org as the package URL:[root@HW001 ~]# yum info ambari-serverInstalled PackagesName        : ambari-serverArch        : x86_64Version     : 2.7.3.0Release     : 139Size        : 418 MRepo        : installedFrom repo   : ambari-2.7.3.0Summary     : Ambari ServerURL         : http://www.apache.orgLicense     : (c) Apache Software FoundationDescription : Maven Recipe: RPM Package. However, the installation instructions that the ambari project site links to are different and only involve building from source via maven (and seem to nowhere mention installation opt

Hi i have below json file and i need to use Evaluate Json Path    {"busId":1,"speed":80,location":{"lat":21.336742,"lon":39.6303}}   is working fine but when i add location Evaluate Json Path return error transferring  failure

I am quite new to nifi. We want to download a csv file from an FtpServer and insert in into HDFS. I have got this working with a small test file by using the getSFTP and putHDFS processors. We now want to work with our first actual data file of around 15Gb, and later date files of possibly in the 200-500 GB range. Are there any restrictions on the size of the files that can be ingested, and if so, how can these be overcome ? Thanks very much in advance

Background: I have installed an HDP (v3.1.0.0-78) cluster with Ambari 2.7.0.0 on Azure via Cloudbreak 2.9.1 using the deployment template available on this Quickstart link - https://docs.cloudera.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.1/quickstart-azure/content/cb_launc.... Question- How do I access zeppelin, Knox, spark history server UI, and other UIs over internet. I can access the Ambari Admin UI. Clicking on other UIs takes to the browser but times out. Please suggest. Thanks,Ani

While doing a sqoop-export to load data from HDFS to Oracle hosted on cloud, I face the below issue.   ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: ORA-01017: invalid username/password; logon denied java.sql.SQLException: ORA-01017: invalid username/password; logon denied   I have done a kinit username@domainname already before initializing the sqoop command and kinit has generated a valid kerberos ticket as well. Even though I do this and run the command from the location where the kerberos user keytab file is located, it still gives me the above issue. As I can see in the sqoop.properties file, parameter org.apache.sqoop.security.authentication.type=KERBEROS is commented out. Is this the reason for the above issue? How can we resolve the above issue where we need sqoop to export data from HDFS to kerberos authenticated DB?

We need to build a Kafka production cluster with 3-5 nodes in cluster , We have the following options: Kafka in Docker containers (Kafka cluster include zookeeper and schema registry on each node) Kafka cluster not using docker (Kafka cluster include zookeeper and schema registry on each node) Since we are talking on production cluster we need good performance as we have high read/write to disks (disk size is 10T), good IO performance, etc. So does Kafka using Docker meet the requirements for productions clusters? more info - https://www.infoq.com/articles/apache-kafka-best-practices-to-optimize-your-deployment

        Hi all   On ambari 2.6.2 version we have the following logsearch files     find / -name *-logsearch-conf.xml   /var/lib/ambari-server/resources/common-services/ACCUMULO/1.6.1.2.2.0/configuration/accumulo-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/FLUME/1.4.0.2.0/configuration/flume-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/HBASE/0.96.0.2.0/configuration/hbase-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/configuration/hdfs-logsearc

Hello,My new CDF cluster include 3 nifi nodes. I enable SSL in nifi . but there is not SSL on LDAP. I don't what is the reason got the error:  2019-11-01 13:42:28,489 WARN org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: Failed to replicate request GET /nifi-api/flow/current-user to xx-cmf02.beta1.nn/10.202.252.92:8080 due to java.net.ConnectException: Failed to connect to xx-cmf02.beta1.nn/10.202.252.92:8080 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:242) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160)......Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method)The behavior is very strange that one node access else node 8080 port but i did not to config it.So, what did i miss it?Thank you for your advice first.Paul

In working with a particular HDP 3.1 cluster, with Spark 2.3 installed, I am finding that the Spark client libraries (ex: spark-cli command, as well as libraries under jars) are not available on every node.  They are only installed on the nodes the customer refers to as "client nodes" (I believe this is analogous to "edge nodes").  They also have data nodes in the cluster, which are able to run Spark executors (and, in fact, YARN does distribute tasks to executors on them), but those nodes do not have Spark client libraries installed.   Is this a normal setup?  Can I not assume that the Spark client is installed on every node, even if it is generally available on the cluster?  Thanks for any insight.

Hello everyone,    I have a question about enabling the TLS communication between the hosts in my cluster. The installation procedure recommends to enable the TLS over the cluster, but when I try to run the following command:    sudo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services   I receive the following errors:   The first warning is:    could not generate CSR     When I check the log I got two errors:   req failed for /var/lib/cloudera-scm-server/certmanager/CMCA/private/ca_key.pem. Exit code: 1 Output:problems making Certificate Request139727014807440:error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long:a_mbstr.c:158:maxsize=64     Does anyone have the same problem? I am struggling to figure out this issue and after having googled it I have not found much...   If I skip this step, unfortunately, I will not be able to pass the&nbsp

Hello! Urgent problemI'm working on Cloudera Flow Management 1.0.1 and to evaluate the feasibility to move HDF to CDF.I'm meeting a problem that is how to config the Multi-Tenant Authorization with kerberos/LDAP in cloudera manager.Could you please help me for the following questions ? For HDF, there is apache ranger  can config and implement the Multi-Tenant Authorization. It can be config in Ranger Admin GUI. How can I do the Multi-Tenant Authorization like HDF with CFM?    Thanks, Paul

Hi, I have a general question. All of our metadata (CM, Hive MS, Oozie, Navigator, etc) is stored in a local MySQL instance on the Cloudera Manager host (see list of DB's below). If this host were to fail, or the MySQL service were to become unavailable for a period of time, would all of the core cluster services (excluding CM services) continue to function?  I am unclear if we need to look at implementing a MySQL cluster to maintain cluster uptime.  Thank you! Edit: I should add that we are currently replicating MySQL to a slave.amonrmanmetastoremetastoresentrynavnavmsooziesqoophue

Hello All,      I have a two nifi nodes and while taking backup moved the flow files to backup folder in both the nodes and restarted NIFI service and my NIFI UI is blank. I restored the flow files back and after restart the NIFI node crashes with the below error. ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.   Please help.

Hey, folks, I have a problem, I just Kerberizar my installation of my cluster in horton but I get an error that can not create a PID, try to do it by hand but delete the file and gives error. could give me a help.   Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/ranger_tagsync.py", line 133, in <module> RangerTagsync().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 355, in execute self.execute_prefix_function(self.command_name, 'post', env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 382, in execute_prefix_function method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 424, in post_start raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file)) resource_management.core.exceptions.Fail: Pid file /var/run/ranger/tagsync.pid doesn't exist after s

Hi   I try to install a fresh CFM instalation in a CM&CDH 6.2.0 cluster.   I install NiFi CA and NiFi Registry and NiFi (Parcel distribution with csd) in the cluster.   NiFi CA and NiFi Registry works fine, both services are started and looks healthy but i can't start NiFi, I install in 2 nodes to made a cluster but i can't start, always the same message:     19/10/31 11:58:12 ERROR o.a.n.b.Command: The sensitive.key file /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/sensitive.key already exists. That shouldn't have been. Aborting.Thu Oct 31 11:58:16 CET 2019JAVA_HOME=/usr/java/jdk1.8.0_121Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/nifi_nifi-NIFI_NODE-6ef7e4e859120410b823067891375e2e_pid45532.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTSUsing /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE as conf dirUsing scripts/control.sh as process scriptCONF_DIR=/var/run/cloudera-scm

Hello, I have a websocket on which I receive a text message (listenWebsocket Processor) , I want to put the whole text message as an attribute into the FetchFile processor. I want to call the value of this extracted attribute in the filepath of the processor?  ExtractText works well, the data provenance shows that there's an new attribute after ExtractText has run, but the attribute value is not set into the file path in the FetchFile Processor ?? What's wrong, did I miss some configuration? A custom attribute is called ${attributename} , right? Thanks a lot for your help Henriette  

I'm using the ConvertJSONToSQL processor, and the MySql table has a space in the name.  It appears to be the problem because when I remove the name from the table it works as expected.   I can successfully query the table from various tools and the mysql command line using back ticks "`" around the table name.   The table name:personal information   I've tried the following: `personal information` "personal information" 'personal information' \"personal information\"[personal information]   Any suggestions?

I am trying to understand how to integrate NiFi within an existing SDLC. The following are some challenges that I am trying to resolve:   1) I am aware that NiFi Registry 0.2.0 allows us to add a git flow persistence provider. This means on every commit to the Registry, the flow file will be automatically be saved to the git repo. Is it possible to take the NiFi Registry out of the picture entirely and work solely with git. I have the feeling that there is a good reason to tightly integrate NiFi with NiFi registry, I would be really helpful if some one can explain the rational behind this tight integration.   2) Is it possible to deploy a NiFi flow from an artifactory instead of the NiFi Registry? A reference to an example would be really helpful.   3) Lets say I have a bunch of NiFi custom processors that needs to be hosted on git and there is Jenkins in this SDLC. Jenkins does the auto build of each custom processors and deploys them in the artifactory. Also the flow f

Dear community,we have a problem when connecting to impala with the impala jdbc connector.Our cluster setup is:cdh 6.2/impala 3.2kerberos enabledssl activated. Our cluster 's configuration has been checked thoroughly and the cluster is correctly processing high volumes of data, using kerberos authentication and has security/sentry etc in place. SSL and kerberos processes work fine, in other words this is not a cluster we just started setting up.We have an issue when making a connection from a java program using the impala jdbc connector.I am following the documentation on https://docs.cloudera.com/documentation/other/connectors/impala-jdbc/2-6-11/Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf (-> page 12 and 88 contain detailed explanation about the connection strings and each possible option for Kerberos ).When we use a connection string including ssl and authentication mechanism 3 and provide the ldap user/password this works fine:#WORKS FINEimpalaConnectionURL=jdbc:impala

I like to change file permissions for security hardening These files have 'w' bit in others, is it possible to change to 220 manually? --w--w--w- 1 root root 0 Oct 29 18:41 /var/run/cloudera-scm-agent/cgroups/memory/<service_name>/cgroup.event_control--w--w--w- 1 root root 0 Oct 29 18:43 /var/run/cloudera-scm-agent/cgroups/memory/<service_name>/cgroup.event_control   These files are newly created every time that the service is started, can permissions be set to 220 permanently instead of manually? This change may cause problems with the service?   Thanks

I am getting an error installing ambari-server-2.7.4.0 on Centos8.   It seems as though it can't resolve conflict with python.  Although I have Python36 installed, it is still asking for python2.6 or greater.   I read your many responses but they did not help with my situation.  Any support on this matter will be appreciated.  Thanks.