Q&A

The NiFi Toolkit CLI has a neat option for setting variables in a deployed NiFi flow. This is very useful for deploying a flow to different environments and then changing a few necessary variables. The commands are nifi pg-get-vars and nifi pg-set-var and would be used as follows; #> nifi pg-get-vars -pgid 34x015f8-0284-1000-5488-a7c6511e5656 -u http://nifi-prod:8080#> nifi pg-set-var -pgid 34x015f8-0284-1000-5488-a7c6511e5656 -var ListFilesInputDir -val /prod/temp -u http://nifi-prod:8080However, using Parameters in  Parameter Contexts seems to be the new and better way of doing things and can be used among different groups in a flow, I have implemented many in my 140 processor project! So my question is this: How do I now set different values for these Parameters for different environments when I deploy my flow. My deployment will be automated using the CLI in the NiFi toolkit, so I'm looking for a solution that is automated/scripted. There appears to be Lis

Hello Experts,I am doing an upgrade from CM 5.16 to CM 7.1.2. There is an instruction to enter userid:password in the url in the .repo file:Using the Cloudera public repositorySubstitute your USERNAME and PASSWORD in the Package Repository URL where indicated in the URL in /etc/yum.repos.d/cloudera-manager.repo baseurl=https://USERID:PASSWORD@archive.cloudera.com/p/cm7/7.1.2/redhat6/yum/After setting above and running the command: yum deplist cloudera-manager-agent it gives an error below:........@archive.cloudera.com/p/cm7/7.1.2/redhat6/yum/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'xyz.com:<password>@archive.cloudera.com'"Anyone seen this error how to resolve. Thanks!

Hi, Im new to Nifi. Im trying to parse csv records with the SplitRecord Processor utilizing the CSVReader controller service, however Im getting IllegalArgumentException an exception on the file despite the CSV seems to be well formatted and Im able to parse\read through other means. Im getting the file  through the GetFile processor and when I view the content from the queue I notice a red dot (see screenshot) in the beginning which seems to indicate invalid character but when I open the file in a text editor I dont see anything abnormal. This file is coming from a third party and it would be hard to change. When I copy the content and place it into other file and save, the new file works! Can someone help please to find what is causing the issue and how to fix? Thanks

I am installing HDP 3.1.0 on RHEL 7....I have setup a local repository and installation of all other services works fine except Ranger.  When i run the wizard to install Ranger, I get an error '0 status code received on POST method for API: /api/v1/stacks/HDP/versions/3.1/recommendations' . Here is the stack trace : ERROR [ambari-client-thread-34] ContainerResponse:423 - The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP containerorg.eclipse.jetty.io.EofException: Early EOFat org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1138)at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1126)at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)at java.io.BufferedInputStream.read(BufferedInputStream.java:345)at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)at sun.nio.cs.StreamDecoder.implRead(S

I have been trying a full installation of CDH6.1 and there are issues when i start the scm server. Installing on Centos 7 (AWS EC2 instance) with no prior CDH installed The logs say something like this:INFO main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: MYSQL database engine and table mapping: {InnoDB=[AUDITS, COMMANDS, CONFIGS, SCHEMA_VERSION]}2019-03-25 16:27:22,021 WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version. This is expected if you are starting Cloudera Manager for the first time. Please also ignore any error messages about missing tables. Moving ahead assuming no upgrade: org.hibernate.exception.SQLGrammarException: could not extract ResultSet2019-03-25 16:27:22,031 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.2019-03-25 16:27:22,032 INFO main:com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 1.2019-03-25 16:27:22,039 WARN main:org.springframework.

Im trying to install the trial version of CDP Data Center on my linux (ubuntu 18.04) virtual machine. So i followed the instructions/commands provided on the Cloudera website. 1- $ wget https://archive.cloudera.com/cm7/7.1.1/cloudera-manager-installer.bin2- $ chmod u+x cloudera-manager-installer.bin3- $ sudo ./cloudera-manager-installer.bin Once i have executed the 3rd instruction, the installation starts then i get the following error:Refreshing repository metadata failed. See /var/log/cloudera-manager-installer/2.refresh-repo.log for details. Click OK to revert this installation.And this is what's on that log file: Atteint :1 http://fr.archive.ubuntu.com/ubuntu bionic InReleaseAtteint :2 http://fr.archive.ubuntu.com/ubuntu bionic-updates InReleaseAtteint :3 http://security.ubuntu.com/ubuntu bionic-security InReleaseAtteint :4 http://fr.archive.ubuntu.com/ubuntu bionic-backports InReleaseIgn :5 https://archive.cloudera.com/cm7/7.1.1/ubuntu1804/apt bionic-cm7.1.1 InRelea

Good day,    I'm trying to upload nifi template and getting this kind of error:   org.apache.nifi.processors.kite.ConvertAvroSchema is not known to this NiFi instance   I suppose that my Nifi Instance misses some component - but where can I get it and where on file system I need to add missing file? I'm little bit confused.   Thanks! 

Hi friends,   I have cloudera trail version 6.2. In the command prompt when i tried to initiate spark shell using   spark-shell, im getting the below error:   [root@cloudera tmp]# spark-shellSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).20/04/09 08:19:33 ERROR spark.SparkContext: Error initializing SparkContext.java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:345)at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179)at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:60)at org.apache.spark.scheduler.TaskSchedulerImpl.sta

Befor I post this issue, we have already readed all the same issue's solutions that we can find.   Our cluster is installed with cdh6.2, after install we use HUE to use the cluster. Job is submited via Hue.   When spark code need to import numpy,  got error below:   Traceback (most recent call last): File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/test.py", line 79, in <module> from pyspark.ml.linalg import Vectors File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/python/lib/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module> File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/python/lib/pyspark.zip/pyspark/ml/base.py", line 24, in <module> File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_0

We updated HDP 2.4 to 2.6. We are using Hive version 1.2 as our default as we are not ready to migrate to 2.1.1 yet. Everything looked fine at first after the update. But later we tried to restart Hive MetaStore and am getting a error. It appears to be trying to update the MetaStore Schema in the DB even though the Upgrade it done   Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 203, in <module> HiveMetastore().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 56, in start create_metastore_schema() File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 417, in create_metastore_schema user = params.hive_user File "/usr/lib/py

Hi Team,We are hitting a bug BUG-77340 where HS2 failover requires knox restart if cookie use is enabled for HS2. We checked https://my.cloudera.com/knowledge/quotError-opening?id=273455 and it is mentioned to set hive.server2.thrift.http.cookie.auth.enabled to false. We would like to know if there is any known performance impact if we set this parameter to false.  Also, we have below parameters set to 600s but still we are seeing server 500 error.gateway.httpclient.connectionTimeout 600sgateway.httpclient.socketTimeout 600s 

We are building a new Cloudera cluster and replicating the HDFS data from an existing cluster. This existing cluster is on two sites and the rack awareness is configured accordingly, with a default replication factor of 3. If we are building this new cluster at one of these two sites, is it possible to ensure that HDFS is replicating from the same physical location and not from the other site? The background: we don't want to cause a big network load between the two sites if all the data is already locally available.

Hello, I am working on a project where I need to run multiple HiveQL statements in a sequence, then store the output for each of those HiveQL statement in a variable and then inserting those output variables in a SQL Database - using NiFi processors.Also Can I use SelectHiveQL processor for running multiple Hive Queries in a sequence?Ideally what should be the NiFi Flow for the above work?Please let me know as early as possible. Thanks!

Hi, i've heard Cloudera Manager is open source.Where can i find its source? Couldn't find it in github.

Hello Team, We just want to know like, From what version of CDH is supporting by cloudera if we go for licenced version of CDH ?Can someone please let me know. Thanks,Vinod

Does anyone know what the error in the subject line of this message means?  I am using SplitRecord processor, checking for duplicate records (removed for testing), then trying to use MergeRecord to put all the splits back into one file, but MergeRecord fails with "Field Eccentricity Cannot Be Null" Error.  I have not found information on the meaning of this error. Thanks in advance if you know. 

we set `retention bytes` value - `104857600` for topic - `topic_test`[root@confluent01 ~]# kafka-topics --zookeeper localhost:2181 --alter --topic topic_test --config retention.bytes=104857600WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.Going forward, please use kafka-configs.sh for this functionalityUpdated config for topic "topic_test".Now we verify the `retention bytes` from the zookeeper:[root@confluent01 ~]# zookeeper-shell confluent01:2181 get /config/topics/topic_testConnecting to confluent1:2181{"version":1,"config":{"retention.bytes":"104857600"}}cZxid = 0xb30a00000038WATCHER::WatchedEvent state:SyncConnected type:None path:nullctime = Mon Jun 29 11:42:30 GMT 2020mZxid = 0xb31100008978mtime = Wed Jul 22 19:22:20 GMT 2020pZxid = 0xb30a00000038cversion = 0dataVersion = 7aclVersion = 0ephemeralOwner = 0x0dataLength = 54numChildren = 0now we performed `reboot` to the kafka confluent01 machineafter machines started

I tried creating a simple table , i am not sure what the error is.Can some one please help me.CREATE TABLE stocks_eod (> stockticker STRING,> tradedate INT,> openprice FLOAT,> highprice FLOAT,> lowprice FLOAT,> volume BIGINT> ) ROW FORMAT DELIMTED FIELDS TERMINATED BY ','; NoViableAltException(232@[])at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormat(HiveParser.java:36025)at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6865)at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:4122)at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1786)at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1152)at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:211)at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:171)at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:447)at org.apache.hadoop.hive.ql.Driver.compile(Driver.java

I am new to this. i've tried installing ambari and on the first try, i was able to start the ambari-server. but when i tried again, an error occurred.Using python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start.........Unable to determine server PID. Retrying.........Unable to determine server PID. Retrying.........Unable to determine server PID. Retrying...ERROR: Exiting with exit code -1.REASON: Ambari Server java process died with exitcode 255.the /var/log/ambari-server/ambari-server.out gave me:Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0Jul 10, 2019 10:02:23 PM com.google.inject.assistedinj

Hi, I'm using the Impala function "show table stats": show table stats table1;  I get following resultsbusiness_datetec_execution_date#Rows#FilesSizeBytes CachedCache ReplicationFormat13/05/202013/05/2020 20:08000BNOT CACHEDNOT CACHEDPARQUET14/07/202015/07/2020 16:396116.08KBNOT CACHEDNOT CACHEDPARQUET15/07/202016/07/2020 16:077702114.68KBNOT CACHEDNOT CACHEDPARQUET16/07/202020/07/2020 16:50272141.88KBNOT CACHEDNOT CACHEDPARQUET17/07/202021/07/2020 12:38290145.33KBNOT CACHEDNOT CACHEDPARQUET20/07/202020/07/2020 12:16379151.10KBNOT CACHEDNOT CACHEDPARQUET21/07/202022/07/2020 12:45156131.50KBNOT CACHEDNOT CACHEDPARQUET24/07/202024/07/2020 15:15278140.26KBNOT CACHEDNOT CACHEDPARQUETTotal 24318340.84KB0B  However if I sum up the rows in column "#Rows" I get 2151...!Am I'm missing something? How is that possible?ThanksThanks

Can anyone tell me what I'm doing wrong. I'm getting the following error from the table creation below -  FAILED: ParseException line 8:37 mismatched input 'TREX' expecting ) near ''\\', bp.step_parameter_name) \nWHERE prm.data_source = '' in function specification")  CREATE TABLE new_table AS SELECT DISTINCTprm.source AS source,bp.process_stage AS process_stageFROM table1 AS prmLEFT JOIN table2 AS bpON (prm.parameter = CONCAT(bp.step_path, '\\', bp.step_parameter_name))WHERE (prm.data_source = 'TDATA' OR prm.data_source = 'Docs')

Hello we have secured Nifi cluster with 3 nodes. When we click to list or empty queue on connection, there is error message Node is unable to fulfill this request due to: Unable to view the data for Processor with ID 904e5254-8129-32ce-967e-03876fc61926. Contact the system administrator. Returning Forbidden response We grant user policy to view and modify data, but no success. Admin user got the same error.message Regards Kamil 

I'm facing an issue during the upgrade of HDP 3.1.0.0-78 to 3.1.4.0-315 on Ubuntu 18The upgrade process is not able to restart the datanodes.I get the error java.lang.RuntimeException: Cannot start datanode because the configured max locked memory size (dfs.datanode.max.locked.memory) of 2147483648 bytes is more than the datanode's available RLIMIT_MEMLOCK ulimit of 16777216 bytes.I don't understand why this error happens. The datanodes were well started before the starting of the upgrade process and the system setting RLIMIT_MEMLOCK hasn't been changed.Thanks in advance for your help  

HelloI would like to knowHow to reset Cloudera Manager Admin Password from the Backend DatabaseDatabase version: Mysql 5.7.29CM version: 6.3.xAuthentication method  for the "Authentication Backend Order": External then Cloudera Manager Thanks in advanced!

Hi all,can I install an HDFS datanode role in machine with installed HDFS NameNode or into machine with SecondaryNameNode?I use CDH 6.3.2 Thanks,Francesco