Solved

Hello all,   I am having an odd issue with NIFI. The cluster stands, and it works but I am unable to log in. I am getting "Untrusted proxy CN=server.domain.com, OU=NIFI" That is not the OU for the zookeeper node I am using. It should be OU=nifi_test. I discovered this in nifi-user.log:   2019-10-28 14:36:26,958 INFO [NiFi Web Server-132] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=nifi_admin, OU=nifi_test) GET <a href="https://server.domain:port/nifi-api/flow/current-user" target="_blank">https://server.domain:port/nifi-api/flow/current-user</a> (source ip: ipaddress)2019-10-28 14:36:26,962 INFO [NiFi Web Server-132] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=nifi_admin, OU=nifi_test2019-10-28 14:36:27,273 INFO [NiFi Web Server-133] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=nifi_admin, OU=nifi_test><CN=server.domain:port, OU=NIFI>) GET <a href="https://server.domain:port/nifi-api/flow

I just noticed that the SmartSense Activity Explorer Zeppelin Notebooks have been failing to run on a production HDP 2.6.1 cluster. I'm not sure how long the issue has been occurring since the dashboards haven't been used much until now. Whenever we try to run the paragraphs, we immediately get an error about unable to make a connection. No other information is given. We are able to connect to Phoenix through psql.py, so we know Phoenix is working properly, just not the dashboard. We've tried restarting the activity explorer, which hasn't fixed the issue. Has someone seen this issue? Any ideas? I'm including the logs we are seeing below. ==> activity-explorer.log <==2019-11-05 10:34:42,555 INFO [qtp1209702763-1653] NotebookServer:711 - New operation from 10.142.131.4 : 62057 : admin : GET_NOTE : 2BPD7951H2019-11-05 10:34:42,558 WARN [qtp1209702763-1653] VFSNotebookRepo:292 - Get Note revisions feature isn't supported in class org.apache.zeppelin.notebook.repo.VFSNotebookRe

Hello everyone,    I can't complete the first installation of my Cluster on AWS. When I try to install my instances on my nodes, the error given by Cloudera Manager is the following:     HTTP ERROR 502Problem accessing /cmf/process/287/logs. Reason: Connection refused (Connection refused)Could not connect to host.     I know the error is very simple, but I am struggling to find out the solution.   My first thought was about the /etc/hosts file, that it is the following for each host in my cluster:     127.0.0.1 localhost.localdomain localhost13.48.140.49 master.sysdatadigital.it master13.48.181.38 slave1.sysdatadigital.it slave113.48.185.39 slave2.sysdatadigital.it slave213.53.62.160 slave3.sysdatadigital.it slave313.48.18.0 slave4.sysdatadigital.it slave4     Moreover, I checked with the following command the hostname in each node:     python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(sock

Hi all, Is it possible to save the uuid of a processor in an attribute? For example,   i have the processor ExecuteSQL, which has this uuid I wish I could save the uuid like this:id = $ .uuid It's possible? Thanks!

Where can I see the list of questions I posted? I tried the two buttons - My Account and My Profile. They do not have a list of questions. Also, I actually got the same "My Profile" page from the two buttons. Is it supposed to be like that? In addition, I am from HCC and I could not find my list of questions posted in HCC either. Thanks.

Error: Unable to instantiate class [org.apache.zeppelin.server.LdapRealm] for object named 'ldapRealm'.  The same is true for 'ldapGroupRealm'Zeppelin 0.8 on HDP 3.1.0.0 Configuration:# authentication settingsldapRealm = org.apache.zeppelin.server.LdapRealmldapRealm.contextFactory.environment[ldap.searchBase] = DC=tantor,DC=netldapRealm.userDnTemplate = uid={0},OU=users,DC=tantor,DC=netldapRealm.contextFactory.url = ldap://infra01.tantor.net:389ldapRealm.contextFactory.authenticationMechanism = SIMPLE# general settingssessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManagersecurityManager.cacheManager = $cacheManagersecurityManager.sessionManager = $sessionManagersecurityManager.sessionManager.globalSessionTimeout = 86400000shiro.loginUrl = /api/login Exception: ERROR [2019-11-06 12:49:24,565] ({main} EnvironmentLoader.java[initEnvironment]:146) - Shiro environment initialization failedorg.apache.shiro.config.ConfigurationException: Unable to

I'm trying to load druid-avro-extensions by setting it on druid.extensions.loadList config property, but the broker is not starting beause the extension is not found: Exception in thread "main" io.druid.java.util.common.ISE: Extension [/usr/hdp/current/druid-broker/extensions/druid-parquet-extensions] specified in "druid.extensions.loadList" didn't exist!? Edit: the extensions is not located on the spcified directory, but it should be there since it is a core extension. What can I do to fix this issue and why is not the extension there?   What is happening? Thanks.  

I'm trying to create a custom CUDA-capable engine image with the CDSW GPU-Guideon our airgapped CDSW-Cluster. We have a CDSW Cluster with 1 Master-Node and 2 Woker-Nodes (one of the Worker-Nodes is equipped with NVIDIA GPUs)I used the following command on the GPU-Host, which is also given in CSDW GPU-Guide, to build the CUDA docker image:docker build --network host -t <company-registry>/cdsw-cuda:8 . -f cuda.DockerfileBeforehand we configured these firewall rules on our GPU-Host:http://developer.download.nvidia.com/*http://archive.ubuntu.com/ubuntu/*http://security.ubuntu.com/ubuntu/*Unfortunately I get these error messages: (Full output on stdout is attached)Reading package lists...W: The repository 'http://archive.cloudera.com/kudu/ubuntu/xenial/amd64/kudu xenial-kudu5 Release' does not have a Release file.W: The repository 'http://ppa.launchpad.net/deadsnakes/ppa/ubuntu xenial Release' does not have a Release file.E: Failed to fetch http://archive.cloudera.com/kudu/u

Hi I am running out of space on one of our DN, and found that Yarn container logs are consuming around 50GB, though there are 3 Spark jobs/applications running which can be seen under Yarn > Applications tab, but I see a lot of other application ids also listed, would like to know if these could be deleted to reclaim space (the ones which are not running)Secondly, any precautions to be taken into consideration while deleting these container logs. Any guidance would be of great help. RegardsWert  

Hello,I am working with cloudera flow manager version 1.0.1. I cannot got login page when i enable ssl & ldap in nifi registry instead of this page, and there is an node identity.  I have checked the config that may be correctly. and try more time such as remove /var/lib/nifiregister/* or /var/run/cloudera-scm-agent/process/***-nifiregistry-NIFI_REGISTRY_SERVER/* the blow pic shows the login user is node id. the related ldap info is below, but there is not userDN config option in cloudera manager. there is same setting between nifi and nifi registry.2019-11-02 10:24:59,167 INFO org.springframework.ldap.core.support.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations The behavior is very strange. Who could help me what i missed?Thanks,Paul

Hi, I scheduled exam before 1 month and it got cancelled due to Examination cluster issue.After waiting for certain time i cancelled my registration to find the alternatives.Now i am about to schedule alternat exam but before that i would like to know availability of CCA175.Please responde if any one gave exam in last week. @cjervis : It would be really helpful if you can contact your sources to identify the status of exam cluster issue.

Hi Team, We are facing an issue where mapreduce jobs are failing with below error.Distribution: Cloudera (CDH 5.14.4) ====Exception in thread "main" java.io.FileNotFoundException: /tmp/hadoop-unjar5272208588996002870/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore$AsyncClient$revoke_privileges_call.class (No space left on device)        at java.io.FileOutputStream.open0(Native Method)        at java.io.FileOutputStream.open(FileOutputStream.java:270)        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:110)        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)       

Is there a difference between installing ambari via the apache docs vs Hortonworks docs? I assume that the end result is exactly the same since Hortonworks labels the distribution in the docs as "Apache" and the repo they instruct to add uses apache.org as the package URL:[root@HW001 ~]# yum info ambari-serverInstalled PackagesName        : ambari-serverArch        : x86_64Version     : 2.7.3.0Release     : 139Size        : 418 MRepo        : installedFrom repo   : ambari-2.7.3.0Summary     : Ambari ServerURL         : http://www.apache.orgLicense     : (c) Apache Software FoundationDescription : Maven Recipe: RPM Package. However, the installation instructions that the ambari project site links to are different and only involve building from source via maven (and seem to nowhere mention installation opt

Hi i have below json file and i need to use Evaluate Json Path    {"busId":1,"speed":80,location":{"lat":21.336742,"lon":39.6303}}   is working fine but when i add location Evaluate Json Path return error transferring  failure

I am quite new to nifi. We want to download a csv file from an FtpServer and insert in into HDFS. I have got this working with a small test file by using the getSFTP and putHDFS processors. We now want to work with our first actual data file of around 15Gb, and later date files of possibly in the 200-500 GB range. Are there any restrictions on the size of the files that can be ingested, and if so, how can these be overcome ? Thanks very much in advance

Background: I have installed an HDP (v3.1.0.0-78) cluster with Ambari 2.7.0.0 on Azure via Cloudbreak 2.9.1 using the deployment template available on this Quickstart link - https://docs.cloudera.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.1/quickstart-azure/content/cb_launc.... Question- How do I access zeppelin, Knox, spark history server UI, and other UIs over internet. I can access the Ambari Admin UI. Clicking on other UIs takes to the browser but times out. Please suggest. Thanks,Ani

While doing a sqoop-export to load data from HDFS to Oracle hosted on cloud, I face the below issue.   ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: ORA-01017: invalid username/password; logon denied java.sql.SQLException: ORA-01017: invalid username/password; logon denied   I have done a kinit username@domainname already before initializing the sqoop command and kinit has generated a valid kerberos ticket as well. Even though I do this and run the command from the location where the kerberos user keytab file is located, it still gives me the above issue. As I can see in the sqoop.properties file, parameter org.apache.sqoop.security.authentication.type=KERBEROS is commented out. Is this the reason for the above issue? How can we resolve the above issue where we need sqoop to export data from HDFS to kerberos authenticated DB?

We need to build a Kafka production cluster with 3-5 nodes in cluster , We have the following options: Kafka in Docker containers (Kafka cluster include zookeeper and schema registry on each node) Kafka cluster not using docker (Kafka cluster include zookeeper and schema registry on each node) Since we are talking on production cluster we need good performance as we have high read/write to disks (disk size is 10T), good IO performance, etc. So does Kafka using Docker meet the requirements for productions clusters? more info - https://www.infoq.com/articles/apache-kafka-best-practices-to-optimize-your-deployment

        Hi all   On ambari 2.6.2 version we have the following logsearch files     find / -name *-logsearch-conf.xml   /var/lib/ambari-server/resources/common-services/ACCUMULO/1.6.1.2.2.0/configuration/accumulo-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/FLUME/1.4.0.2.0/configuration/flume-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/HBASE/0.96.0.2.0/configuration/hbase-logsearch-conf.xml /var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/configuration/hdfs-logsearc

Hello,My new CDF cluster include 3 nifi nodes. I enable SSL in nifi . but there is not SSL on LDAP. I don't what is the reason got the error:  2019-11-01 13:42:28,489 WARN org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: Failed to replicate request GET /nifi-api/flow/current-user to xx-cmf02.beta1.nn/10.202.252.92:8080 due to java.net.ConnectException: Failed to connect to xx-cmf02.beta1.nn/10.202.252.92:8080 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:242) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160)......Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method)The behavior is very strange that one node access else node 8080 port but i did not to config it.So, what did i miss it?Thank you for your advice first.Paul

In working with a particular HDP 3.1 cluster, with Spark 2.3 installed, I am finding that the Spark client libraries (ex: spark-cli command, as well as libraries under jars) are not available on every node.  They are only installed on the nodes the customer refers to as "client nodes" (I believe this is analogous to "edge nodes").  They also have data nodes in the cluster, which are able to run Spark executors (and, in fact, YARN does distribute tasks to executors on them), but those nodes do not have Spark client libraries installed.   Is this a normal setup?  Can I not assume that the Spark client is installed on every node, even if it is generally available on the cluster?  Thanks for any insight.

Hello everyone,    I have a question about enabling the TLS communication between the hosts in my cluster. The installation procedure recommends to enable the TLS over the cluster, but when I try to run the following command:    sudo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services   I receive the following errors:   The first warning is:    could not generate CSR     When I check the log I got two errors:   req failed for /var/lib/cloudera-scm-server/certmanager/CMCA/private/ca_key.pem. Exit code: 1 Output:problems making Certificate Request139727014807440:error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long:a_mbstr.c:158:maxsize=64     Does anyone have the same problem? I am struggling to figure out this issue and after having googled it I have not found much...   If I skip this step, unfortunately, I will not be able to pass the&nbsp

Hello! Urgent problemI'm working on Cloudera Flow Management 1.0.1 and to evaluate the feasibility to move HDF to CDF.I'm meeting a problem that is how to config the Multi-Tenant Authorization with kerberos/LDAP in cloudera manager.Could you please help me for the following questions ? For HDF, there is apache ranger  can config and implement the Multi-Tenant Authorization. It can be config in Ranger Admin GUI. How can I do the Multi-Tenant Authorization like HDF with CFM?    Thanks, Paul

Hi, I have a general question. All of our metadata (CM, Hive MS, Oozie, Navigator, etc) is stored in a local MySQL instance on the Cloudera Manager host (see list of DB's below). If this host were to fail, or the MySQL service were to become unavailable for a period of time, would all of the core cluster services (excluding CM services) continue to function?  I am unclear if we need to look at implementing a MySQL cluster to maintain cluster uptime.  Thank you! Edit: I should add that we are currently replicating MySQL to a slave.amonrmanmetastoremetastoresentrynavnavmsooziesqoophue

Hello All,      I have a two nifi nodes and while taking backup moved the flow files to backup folder in both the nodes and restarted NIFI service and my NIFI UI is blank. I restored the flow files back and after restart the NIFI node crashes with the below error. ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.   Please help.