Q&A

Hello everyone, I am new to NiFi, so please help me to debug the issue of printing flow file statistics in the log file of the NiFi. So I have an input port that connects to a remote port and gets the data. Currently, the input port is able to receive data from remote ports and forwards it to the next processor for further processing. Initially, I used to get some statistics from the NiFi log file about the number of flow files it received and sent for the next processor at the input port in the below format.  nifi_amount_flowfiles_sent{instance="test-Nifi",component_type="InputPort",component_name="TENANT1",component_id="d931ff3e-0178-1000-0000-2b4d2d1a7acd",parent_id="d5221ecd-0178-6666-7354-4d2d7e6aaddd",} 0.0 Currently, the input port is receiving data and forwarding data to the next processor but it always prints count 0 in a NiFi log file. Not sure where things are going wrong. Any help/reference will be helpful.Thanks. 

Hello, I am trying to find an API command for enabling the Ranger plugin for HDFS (step 1f in https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/installation/topics/cdpdc-additional-steps-ranger.html). I do not see it in https://archive.cloudera.com/cm7/7.1.4/generic/jar/cm_api/apidocs/index.html . Is it possible to use API for this task? Thank you,Pawel

Hello Everyone, Has anyone tested the Distcp editor available in HUE with CDP 7.1.5? Currently I see the below screen:  In this, I'm not able to manually enter the source cluster path. As soon as I click on the source path text box, it opens a dialog box to select the path from the current cluster HDFS. Has anybody tested pulling data from another cluster? What config changes are required for getting this to work? Thanks,Megh

Hello Everyone, I'm facing a rather strange problem with my Ranger policies. I'm not able to save modifications to any policies once they get created. When I create a policy for the first time, it get created successfully. Subsequently, when I try to modify that policy, and click on save, the UI simply doesn't submit. Checked the Ranger Admin Logs as well, no entries. Any suggestions? Thanks,Megh

Hello,I have a question regarding connecting a webportal which will act a data source. I need to know how can i do connect it to my cluster in cloudera.Regards,Mubashar

In the Yarn Queue Manager UI view below, there is 3 levels: root parent and childthe color between root and parent is green but the color between child and parent is orange,What those colors mean? 

Hi All,I have a problem when add service nifi in my cluster, when nifinode starting it failed :Code: 400 Output: {"statusCode":400,"msgDesc":"Operation denied. Group name: nifi specified in policy does not exist in ranger admin."}Failed to create ranger policy...Ensure Ranger has following groups:nifiEnsure Ranger has following users:Code: 400 Output: {"statusCode":400,"msgDesc":"Operation denied. Group name: nifi specified in policy does not exist in ranger admin."}Failed to create ranger policy...Ensure Ranger has following groups:nifiEnsure Ranger has following users:It says nifi group not exist in ranger instead user nifi has exist in ranger, but when i try to create manually nifi group in ranger, it solved.Is it any ways to create automatically nifi group without create manually in ranger? Thanks n Regards Meidi

Hi experts,I am trying to install hive using ambari which i downloaded the bigtop repo. Here is the error:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py", line 79, in <module> HCatClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py", line 35, in install self.install_packages(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 821, in install_packages retry_count=agent_stack_retry_count) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/

Hello,there are problems with Hive Compactor. We can see in hivemetastore.log this message "Max block location exceeded for split" and it´s appearing more and more times.After that, the "compactor.Cleaner" is not launched.We observed that after a Hive Metastore restart, the "compactor.Cleaner" has not been launched nevermore, but logs doesn´t display any message about it.Could be a degradation of the Hive Compactor when delta files are growing in the partitions?Regards. Hive 3.1.0Hadoop 3.1.1    hive.compactor.initiator.on = true hive.compactor.job.queue = compactions hive.compactor.worker.threads = 6 hive.metastore.thrift.compact.protocol.enabled = true hive.compactor.check.interval = 120s hive.compactor.delta.num.threshold = 5 hive.compactor.cleaner.run.interval = 5s hive.compactor.worker.timeout = 300s hive.compactor.max.num.delta = 100 hive.compactor.abortedtxn.threshold = 100 hive.compactor.initiator.failed.compacts.threshold = 10 hive.compactor.history.re

I have successfully installed Oracle VM Virtual Box Manager. I am trying to do very simple tutorials to get familiar with the hadoop framework  https://www.cloudera.com/tutorials.html but i can't seem to find the cloudera machine learning, cloudera data platform and so on where i can i find them please? Also what tutorial can you recommend to get familiar with Hadoop, MapReduce and Yarn? there are so many.. Thank you

Hello,     We're using Cloudera Manager 6.3.1 to deploy CDH6.3.2, and it componet of Zookeeper version is 3.4.5+cdh6.3.2. but we fond a vulnerability CVE-2019-0201 from https://www.cvedetails.com/cve/CVE-2019-0201/?q=CVE-2019-0201  (An issue is persent in Apache Zookeeper 1.0.0 to 3.4.13 and 3.5.0-alpha to 3.5.4-beta. ZooKeeper's getACL() command doesn't check any permission when retrieves the ACLs of the requested node and returns all information contained in the ACL Id field as plaintext string).     Now, i want to upgrade Zookeeper to deal this vulnerability in CDH6.3.2, but i don't kown what should i do. Should i use a independence Zookeeper-3.4.14 replace Zookeeper-3.4.5+cdh6.3.2?    Thank you in advance !     Best Regards !     Yushuo

Hi experts,I am trying to deploy a cluster with the apache bigtops package via ambari.I was able to deploy the basic components such as hdfs, yarn, mapreduce, and zookeeper via ambari.Now I am trying to install hive.I am able to install hive through the command line with "yum install hive*" Ambari is currently failing to install hcatalog.Looking at the ambari logs, I noticed this error:resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install hcatalog' returned 1. Error: Nothing to do However, when installing via command line, I am able to install hcatalog with"yum install hive-hcatalog" Can someone please let me know how I can have ambari run the command /usr/bin/yum -d 0 -e 0 -y install hive-hcataloginstead of /usr/bin/yum -d 0 -e 0 -y install hcatalog Any help is much appreciated.Thanks,

I have a tar file with .csv, .xml, and .txt files that I want to unpack using the UnpackContent processor.   Can I specify all three file types in the file filter?

I have an 8 GB RAM and I understand that Cloudera express won't be working on my computer.Is there any alternative? How can I work with Cloudera express using my computer? Is there any way to use the Cloud to get more RAM power?Thank you for your help and guidance

Hello Everyone, I have an external table (Text with Gzip compression) partitioned on a date column. I have created another external table with the same structure but with Parquet format with Snappy compression. I want to copy all the data from the text table to parquet table, and I'm using the following query for testing it out for one day.insert into parquet_table partition(asdt) select * from text_table where partition_col='somedate'; The partition contains almost 100M rows. This query runs for almost half an hour and then results into this error:ERROR : Vertex failed, vertexName=Reducer 2, vertexId=vertex_1619005705788_0004_2_01, diagnostics=[Task failed, taskId=task_1619005705788_0004_2_01_000240, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #1 at org.apache.tez.runtime.library.common.shuffle.ordered

I am trying to deploy a HDP cluster with ambari. I met an error when deployingIt reported the error: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 37, in <module> BeforeInstallHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 33, in hook install_packages() File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/shared_initialization.py", line 37, in install_packages retry_count=params.agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 125, in __new__ cls(names_list.pop(0), env, provider, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/env

Hello,while attempting to start ListenHTTP (port=10000) on a fresh cluster, I obtain the following error. Do you know what to change to get it work? ListenHTTP[id=6222ab5d-0177-1000-ffff-ffffaeaba266]Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed.Failure is due to java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:10000: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:10000Cloudera Flow Management (CFM) 2.0.4.01.11.4.2.0.4.0-80 built 09/27/2020 09:52:49 CESTTagged nifi-1.11.4-RC1 Best regardsJaro

Hello Everyone,   I'm trying to distcp from an HDP 3.1.5 cluster (non-kerberized) to CDP 7.1.5 cluster (kerberized).   I'm running the distcp command on the secure cluster like follows: hadoop distcp -Ddfs.client.use.datanode.hostname=true -Ddfs.datanode.use.datanode.hostname=true -Dipc.client.fallback-to-simple-auth-allowed=true webhdfs://<source-nameservice>/path/to/source/folder webhdfs://<destination-nameservice>/path/to/destination/folder   It starts fine and then fails after 40 mins (Map progress reaches 97% and then gets stuck).   When I looked into the task attempt logs, I find this error:     2021-04-19 17:00:30,916 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration2021-04-19 17:00:30,985 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.Met

I need to know how to sort my HDFS directories based on their size.

Dear All, I am looking for mini exercises/projects in Hadoop Cloudera to get familiar with the framework. Do you know any website or youtube channel that provides that? I d like to start basic from 'copy data from local to HDFS' to intermediate level projects. Thank you

Hi there.I'm using NiFi to get data from remote windows servers using the GetSmbFile processor.It works but I have a number of servers exposing data on SMB share and I'm trying to understand if I can catalog all my remote servers in a SQL table with needed parameters (eg. hostname, user, pass, share) to be passed to that processor.I see that GetSmbFiles doesn't support EL so I haven't found a working solution nor anything helpful on line.Is this achievable without writing my own processor or running external code?Any help is appreciated

I tried to move my HSI to another node on my production cluster without stopping hiveserver or metastore components, so i disabled it first from ambari UI (no problem at this stage).Now when i try to enable Interactive Query it shows a pop with an error message : You cannot enable Interactive Query now because Interactive Query is in the process of being disabled. This may take a few minutes. Try again later. Nothing in ambari logs neither in hive-interactive logs because it was uninstalled. Any advise please ? 

I'm facing a severe performance issue on a job that suddenly(no code changes) takes 4x time to complete, after debugging and investigating I found most of the data is read to a single executor(13 GB one executor vs 200 MB the rest). Initially i thought it was an classical uneven partitions issue so i started to test different partition numbers (and criteria) but the issue was not fixed, I did a rows per partition analysis and found all partitions have similar number of rows so this is not the problem, it seems the scheduler assigns most partitions to a single executor instead of evenly, question is how spark decides which partitions go to which executor, and how to control that behavior to make distribution even? I asked this on SO too: https://stackoverflow.com/questions/67133177/how-spark-distributes-partitions-to-executors

Hi,I want to  remove some old disk and added new disk with new mount points. Now the new disk are part of  CDH. Now I want to  transfer the blocks from old disk /data01 to new disk /data05 and then remove /data01 after move of all blocks.Is there a way to do this ?can I take datanode services down and then manually copy? will Cloudera able to find the new blocks on /data05 (new disk) ?

I'm currently working on CDH5.13 (Cloudera Distribution Hadoop), and i have a couple of questions: 1- I want to get the latest version of CDH(6.3.3). When i try to download it, i have this message (Your MyCloudera account is not currently connected with a CDH subscription which is required for CDH 6.3.3 and later.). Where do I go to get a subscription? And what is the cost of the subscription?  2- I tried to download older versions of CDH, but it says ( As of February 1, 2021, all downloads of CDH and Cloudera Manager require a username and password and use a modified URL. You must use the modified URL, including the username and password when downloading the repository contents described below.). I'm assuming i have to get a license to have this username and password. Where should I go to buy this? 3- I also tried to download Hortonworks Data Platform, but i have this message (You must be a HDP customer to access these downloads. If you believe