Solved

Installed Ambari 2.7 HDP 3.1 version, after added Ranger service I couldn't able to login. From Ranger UI "Unable to connect db" and in ambari UI it shows "User:admin credentials on Ambari UI are not in sync with Ranger" I checked Ranger access logs, Checked DB connection, grant permissions everythings is fine. Also from Ranger Config, passwords for "Ranger Admin username for Ambari" and "Ranger Admin user's password" are same and I thought this was a bug. Can anyone help me on this? Thanks in advance.

@Shelton@deepesh1, I need your help in figuring out why this Hive Metastore and Hive Server 2 are not working. Please find the ambari-error.log below: Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart self.status(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 87, in status check_process_status(status_params.hive_metastore_pid) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning()ComponentIsNotRunningThe above exception was the cause of the following exception:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 201, in <module> HiveMetastore().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/scr

Hi everyone.   Before anything I'm going to describe the environment where I'm trying to install the Cluster on Amabri:   Static hostname: sandbox-hdp.hortonworks.comIcon name: computer-vmChassis: vmMachine ID: d8323e61d06a489f97b6175ebe8ff9c7Boot ID: a0b2c8c30d5345f796a51dc754e807e9Virtualization: vmwareOperating System: CentOS Linux 7 (Core)CPE OS Name: cpe:/o:centos:centos:7Kernel: Linux 3.10.0-1127.13.1.el7.x86_64Architecture: x86-64   Amabri Agent Version: ambari-agent.x86_64 2.7.4.0-118 @Ambari-2.7.4.0   Ambari Server Version: ambari-server.x86_64 2.7.4.0-118 @Ambari-2.7.4.0   Ambari Agent ini file: [server]hostname=sandbox-hdp.hortonworks.comurl_port=8440secured_url_port=8441connect_retry_delay=10max_reconnect_retry_delay=30   [agent] .... public_hostname_script=/var/lib/ambari-agent/hostname.shhostname_script=/var/lib/ambari-agent/hostname.sh   [security] ....force_https_protocol=PROTOCOL_TLSv1_2    /var/lib/ambari-agent/hostname.sh

Hi,Im new to Nifi and Groovy. I have been trying to run a groovy script using the ExecuteScript Processor but not matter what my groovy code is I keep getting the exception "Java.Lang.NoClassDefFoundError: Couldnot Initialize class groovy.lang.groovySystem" not sure why? I have installed the groovy sdk and set the environment variable PATH accordingly. Not sure what else I have to do? Thanks for your help.

Failed to Start Infra Solr Instance in Ambari,  Please find the logs, -------------------------------------- Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/infra_solr.py", line 171, in <module>InfraSolr().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/infra_solr.py", line 58, in startsetup_solr_znode_env()File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/setup_infra_solr.py", line 142, in setup_solr_znode_envjava_opts=java_optsFile "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/solr_cloud_util.py", line 162, in setup_kerberos_pluginExecute(setup_kerberos_plugin_cmd)File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib

Hi All, We're trying to install new cluster with Cloudera Manager 6.2.0 ( CDH 6.2.1 ), while adding Hue service, I am getting an error as " Unexpected error. Unable to verify database connection." in Cloudera Manager. I checked cloudera manager server logs in /var/log/cloudera-scm-server/cloudera-scm-server.log Here is the log I found at the time Hue test connection failure  2020-08-05 11:49:55,499 INFO scm-web-436:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult2020-08-05 11:49:55,506 INFO scm-web-436:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:2002020-08-05 11:49:57,541 INFO scm-web-370:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult2020-08-05 11:49:57,548 INFO scm-web-370:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operatio

Consumer IMAP - Reset State?Can anyone tell me how to control state on Consume IMAP (v1.11.4)I have successfully implemented a flow where I receive emails using the ConsumeIMAP ExtractEmailHeaders and ExtractEmailAttachements, all is working as expected.I have set the properties:Delete Messages = falseMark Message as Read = false so my emails remain in the INBOX.I would like a way to "reset" -  so that the processor would receive all the emails again, however I can't seem to find where state is controlled.Stopping and starting the processor does not reset;Stopping/Starting Nifi does not reset. Does this processor hold some other state or does it query my provenance to workout where it is up to? Would I need to delete the provenance to reset it? I was expecting some kinds of state similar to ListSFTP? Any help appreciated John

Hi, I need to add new hosts to an existing cluster using Ambari but the new hosts have more disks than the old nodes that I want to add to the HDFS (old nodes have /data/disk1& /data/disk2 while new nodes have/data/disk1, /data/disk2, /data/disk3 & /data/disk4). How can I add those disks after adding the nodes? can I just update  dfs.datanode.data.dir?

The NiFi Toolkit CLI has a neat option for setting variables in a deployed NiFi flow. This is very useful for deploying a flow to different environments and then changing a few necessary variables. The commands are nifi pg-get-vars and nifi pg-set-var and would be used as follows; #> nifi pg-get-vars -pgid 34x015f8-0284-1000-5488-a7c6511e5656 -u http://nifi-prod:8080#> nifi pg-set-var -pgid 34x015f8-0284-1000-5488-a7c6511e5656 -var ListFilesInputDir -val /prod/temp -u http://nifi-prod:8080However, using Parameters in  Parameter Contexts seems to be the new and better way of doing things and can be used among different groups in a flow, I have implemented many in my 140 processor project! So my question is this: How do I now set different values for these Parameters for different environments when I deploy my flow. My deployment will be automated using the CLI in the NiFi toolkit, so I'm looking for a solution that is automated/scripted. There appears to be Lis

Hello Experts,I am doing an upgrade from CM 5.16 to CM 7.1.2. There is an instruction to enter userid:password in the url in the .repo file:Using the Cloudera public repositorySubstitute your USERNAME and PASSWORD in the Package Repository URL where indicated in the URL in /etc/yum.repos.d/cloudera-manager.repo baseurl=https://USERID:PASSWORD@archive.cloudera.com/p/cm7/7.1.2/redhat6/yum/After setting above and running the command: yum deplist cloudera-manager-agent it gives an error below:........@archive.cloudera.com/p/cm7/7.1.2/redhat6/yum/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'xyz.com:<password>@archive.cloudera.com'"Anyone seen this error how to resolve. Thanks!

Hi, Im new to Nifi. Im trying to parse csv records with the SplitRecord Processor utilizing the CSVReader controller service, however Im getting IllegalArgumentException an exception on the file despite the CSV seems to be well formatted and Im able to parse\read through other means. Im getting the file  through the GetFile processor and when I view the content from the queue I notice a red dot (see screenshot) in the beginning which seems to indicate invalid character but when I open the file in a text editor I dont see anything abnormal. This file is coming from a third party and it would be hard to change. When I copy the content and place it into other file and save, the new file works! Can someone help please to find what is causing the issue and how to fix? Thanks

I am installing HDP 3.1.0 on RHEL 7....I have setup a local repository and installation of all other services works fine except Ranger.  When i run the wizard to install Ranger, I get an error '0 status code received on POST method for API: /api/v1/stacks/HDP/versions/3.1/recommendations' . Here is the stack trace : ERROR [ambari-client-thread-34] ContainerResponse:423 - The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP containerorg.eclipse.jetty.io.EofException: Early EOFat org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1138)at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1126)at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)at java.io.BufferedInputStream.read(BufferedInputStream.java:345)at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)at sun.nio.cs.StreamDecoder.implRead(S

I have been trying a full installation of CDH6.1 and there are issues when i start the scm server. Installing on Centos 7 (AWS EC2 instance) with no prior CDH installed The logs say something like this:INFO main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: MYSQL database engine and table mapping: {InnoDB=[AUDITS, COMMANDS, CONFIGS, SCHEMA_VERSION]}2019-03-25 16:27:22,021 WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version. This is expected if you are starting Cloudera Manager for the first time. Please also ignore any error messages about missing tables. Moving ahead assuming no upgrade: org.hibernate.exception.SQLGrammarException: could not extract ResultSet2019-03-25 16:27:22,031 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.2019-03-25 16:27:22,032 INFO main:com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 1.2019-03-25 16:27:22,039 WARN main:org.springframework.

Im trying to install the trial version of CDP Data Center on my linux (ubuntu 18.04) virtual machine. So i followed the instructions/commands provided on the Cloudera website. 1- $ wget https://archive.cloudera.com/cm7/7.1.1/cloudera-manager-installer.bin2- $ chmod u+x cloudera-manager-installer.bin3- $ sudo ./cloudera-manager-installer.bin Once i have executed the 3rd instruction, the installation starts then i get the following error:Refreshing repository metadata failed. See /var/log/cloudera-manager-installer/2.refresh-repo.log for details. Click OK to revert this installation.And this is what's on that log file: Atteint :1 http://fr.archive.ubuntu.com/ubuntu bionic InReleaseAtteint :2 http://fr.archive.ubuntu.com/ubuntu bionic-updates InReleaseAtteint :3 http://security.ubuntu.com/ubuntu bionic-security InReleaseAtteint :4 http://fr.archive.ubuntu.com/ubuntu bionic-backports InReleaseIgn :5 https://archive.cloudera.com/cm7/7.1.1/ubuntu1804/apt bionic-cm7.1.1 InRelea

Good day,    I'm trying to upload nifi template and getting this kind of error:   org.apache.nifi.processors.kite.ConvertAvroSchema is not known to this NiFi instance   I suppose that my Nifi Instance misses some component - but where can I get it and where on file system I need to add missing file? I'm little bit confused.   Thanks! 

Hi friends,   I have cloudera trail version 6.2. In the command prompt when i tried to initiate spark shell using   spark-shell, im getting the below error:   [root@cloudera tmp]# spark-shellSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).20/04/09 08:19:33 ERROR spark.SparkContext: Error initializing SparkContext.java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:345)at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179)at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:60)at org.apache.spark.scheduler.TaskSchedulerImpl.sta

Befor I post this issue, we have already readed all the same issue's solutions that we can find.   Our cluster is installed with cdh6.2, after install we use HUE to use the cluster. Job is submited via Hue.   When spark code need to import numpy,  got error below:   Traceback (most recent call last): File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/test.py", line 79, in <module> from pyspark.ml.linalg import Vectors File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/python/lib/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module> File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/python/lib/pyspark.zip/pyspark/ml/base.py", line 24, in <module> File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_0

We updated HDP 2.4 to 2.6. We are using Hive version 1.2 as our default as we are not ready to migrate to 2.1.1 yet. Everything looked fine at first after the update. But later we tried to restart Hive MetaStore and am getting a error. It appears to be trying to update the MetaStore Schema in the DB even though the Upgrade it done   Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 203, in <module> HiveMetastore().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 56, in start create_metastore_schema() File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 417, in create_metastore_schema user = params.hive_user File "/usr/lib/py

Hi Team,We are hitting a bug BUG-77340 where HS2 failover requires knox restart if cookie use is enabled for HS2. We checked https://my.cloudera.com/knowledge/quotError-opening?id=273455 and it is mentioned to set hive.server2.thrift.http.cookie.auth.enabled to false. We would like to know if there is any known performance impact if we set this parameter to false.  Also, we have below parameters set to 600s but still we are seeing server 500 error.gateway.httpclient.connectionTimeout 600sgateway.httpclient.socketTimeout 600s 

We are building a new Cloudera cluster and replicating the HDFS data from an existing cluster. This existing cluster is on two sites and the rack awareness is configured accordingly, with a default replication factor of 3. If we are building this new cluster at one of these two sites, is it possible to ensure that HDFS is replicating from the same physical location and not from the other site? The background: we don't want to cause a big network load between the two sites if all the data is already locally available.

Hello, I am working on a project where I need to run multiple HiveQL statements in a sequence, then store the output for each of those HiveQL statement in a variable and then inserting those output variables in a SQL Database - using NiFi processors.Also Can I use SelectHiveQL processor for running multiple Hive Queries in a sequence?Ideally what should be the NiFi Flow for the above work?Please let me know as early as possible. Thanks!

Hi, i've heard Cloudera Manager is open source.Where can i find its source? Couldn't find it in github.

Hello Team, We just want to know like, From what version of CDH is supporting by cloudera if we go for licenced version of CDH ?Can someone please let me know. Thanks,Vinod

Does anyone know what the error in the subject line of this message means?  I am using SplitRecord processor, checking for duplicate records (removed for testing), then trying to use MergeRecord to put all the splits back into one file, but MergeRecord fails with "Field Eccentricity Cannot Be Null" Error.  I have not found information on the meaning of this error. Thanks in advance if you know. 

we set `retention bytes` value - `104857600` for topic - `topic_test`[root@confluent01 ~]# kafka-topics --zookeeper localhost:2181 --alter --topic topic_test --config retention.bytes=104857600WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.Going forward, please use kafka-configs.sh for this functionalityUpdated config for topic "topic_test".Now we verify the `retention bytes` from the zookeeper:[root@confluent01 ~]# zookeeper-shell confluent01:2181 get /config/topics/topic_testConnecting to confluent1:2181{"version":1,"config":{"retention.bytes":"104857600"}}cZxid = 0xb30a00000038WATCHER::WatchedEvent state:SyncConnected type:None path:nullctime = Mon Jun 29 11:42:30 GMT 2020mZxid = 0xb31100008978mtime = Wed Jul 22 19:22:20 GMT 2020pZxid = 0xb30a00000038cversion = 0dataVersion = 7aclVersion = 0ephemeralOwner = 0x0dataLength = 54numChildren = 0now we performed `reboot` to the kafka confluent01 machineafter machines started