Solved

Hello everebody! Pls i need to your advise. I need to allows a number of flow-files together into a single flow-file with use general key.For example, i have two flow-files with next attributes:manager_name = John Dowcostumer_info = Matt Damoncostumer_location = LA and manager_name = John Dowcostumer_info = Tony Starkcostumer_location = LA Next, I need to recieve single flow-file with count result of the flow-files with general key = John DowFinally single flow-file whith attributes:manager_name = John Dowtotal = 2 How i can do it in nifi? 

How do I upload multiple nifi templates at once?Nifi only allows to upload one single template/xml file at a time. How do I upload all the templates in  bulk?

I am new to nifi and I want nifi logs to roll daily with no retention policyIs this configuration correct?Another question was do we need to restart nifi after updating logback.xml file?   <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"><!---<file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>--><rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"><fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log</fileNamePattern><maxFileSize>1GB</maxFileSize><!-- keep 30 log files worth of history --><maxHistory>1</maxHistory></rollingPolicy><immediateFlush>true</immediateFlush><encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"><pattern>%date %level [%thread] %logger{40} %msg%n</pattern></encoder></appender>      &nbs

Hi Guys, I am trying to migrate an ORC table from one cluster to another. The export and import does not throw any error and I could see folders and files getting created. But the imported table does not contain any data. Table is getting created with all the correct columns. The same works properly for table stored as textfile. Hive version is 3.1. Please provide your inputs.  Regards,Magudeswaran R.

Recently, MapReduce job sometimes failed, the details as below: after check map tasks , the log like below: Log Type: syslogLog Upload Time: Sun Apr 25 13:54:17 +0800 2021Log Length: 55072021-04-25 13:51:01,806 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties2021-04-25 13:51:01,893 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).2021-04-25 13:51:01,893 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started2021-04-25 13:51:01,895 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:2021-04-25 13:51:01,895 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1618548626214_99981, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@732c2a62)2021-04-25 13:51:02,182 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.

I have installed NiFi and MiNiFi on a remote unix server. When I run the command: tail -f nifi-app.logI get all repositories: content_repository, database_repository, flowfile_repository, provenance_repository for NiFi.quamd I run the same command for MiNiFi: tail -f minifi-app.log, get three repositories: content_repository, flowfile_repository, provenance_repository.  the installation was successful. I run the command: http://ip:8080. I can't get the NiFi interface, I'm told that this page is not accessible.I do the same thing for MiNiFi and I don't get  interface.this page is not accessible. i need help    

Hi may I know what version of Python does the new 6.3.3 version of Cloudera supported?

Hi,I have configured Ranger with AD for user and group sync but not able to see the users in the Ranger. Ldapsearch  is working fine and able to retrieve the users from group hdpadmin. Please find below snap for the user configuration.Common ConfigUser ConfigGroup Config  

Hi , I am trying to pull impala running query with status /services/Impala/impalaQueries#'?filter=(query_state=RUNNING) but it also displaying all Finished query detials. I need to pull information of Only RUNNING query which duration exceed with certain time period. Can some one help me. ThanksAt

I have a question about regionservers that go to bad health when I am writing data from Spark.My question is: how do I know (in what logs to look exactly) what the cause is of the bad health?BACKGROUND:My spark job processes a month of some data and writes to hbase. This runs 1/month. For most months there are no problems.For some months (probably with slightly higher traffic), I notice the regionservers go into bad health. The master notices this and when the server goes down, it moves all regions to another regionserver, and then it becomes healthy again. But as my writing is going on, the same happens to other regionservers and eventually my spark job fails.I am confident my error in write is not due to corrupt data (like impossible UTC time or so), since sometimes this happens and then I clearly see in my spark logs "caused by value out of bounds". Now I see21/03/17 15:54:22 INFO client.AsyncRequestFutureImpl: id=5, table=sometable, attempt=10/11, failureCount=2048ops, last excepti

I'm sending data from MiNiFi to the NiFi Cluster. Is there any way to keep MiNiFi backpressure data in the disk rather than the memory? I've also disabled the Swap feature for increasing performance. I'm worried about consuming lots of the memory by MiNiFi. Nifi version: 1.8.0MiNiFi version: 0.5.0 java

Hi,  I've been using cloudera manager for the last one year with running of 20+ nodes. Recently i started to see heap memory size issue in Service monitor roles. I've increased from 3 to 4, then 4 to 5, and then 5 to 6 GB.But still i sometime get the service monitor crashed and restarted. During the time, the entire dashboard seems bad. What i need to do here to fix the issue?.  Logs are2021-04-26 16:10:34,938 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 20583ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=182ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=20877ms2021-04-26 16:11:34,862 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 19870ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=131m

I have 5 datanode where impala-demons was installed and in all it shows me this error:**************************************************************************************log4j:ERROR Could not instantiate appender named "redactorForRootLogger".E0426 13:25:49.818779 12933 RequestPoolService.java:257] Unable to stop AllocationFileLoaderService after failed start.Java exception follows:java.lang.IllegalStateExceptionat com.google.common.base.Preconditions.checkState(Preconditions.java:492)at org.apache.impala.util.FileWatchService.stop(FileWatchService.java:136)at org.apache.impala.util.RequestPoolService.stopInternal(RequestPoolService.java:281)at org.apache.impala.util.RequestPoolService.start(RequestPoolService.java:255)F0426 13:25:49.820109 12933 request-pool-service.cc:148] RuntimeException: org.apache.impala.yarn.server.resourcemanager.schedule$CAUSED BY: AllocationConfigurationException: Could get past last queue placement rule without assigning. Impalad exiting.******************

Hi all,I've run few oozie workflow through Hue. But once the jobs finished, I cannot see the logs anymore. I'm wondering whether this is normal behaviour or I can config something on yarn to make the logs available even after job finishes.  yarn.log-aggregation-enable is set to enable on the cluster Using this command is fine. yarn logs -applicationId <Application ID>. Only cannot see in the hue web ui after the job finished  Thanks

Hello Everyone, Has anyone tested the Distcp editor available in HUE with CDP 7.1.5? Currently I see the below screen:  In this, I'm not able to manually enter the source cluster path. As soon as I click on the source path text box, it opens a dialog box to select the path from the current cluster HDFS. Has anybody tested pulling data from another cluster? What config changes are required for getting this to work? Thanks,Megh

Hello Everyone, I'm facing a rather strange problem with my Ranger policies. I'm not able to save modifications to any policies once they get created. When I create a policy for the first time, it get created successfully. Subsequently, when I try to modify that policy, and click on save, the UI simply doesn't submit. Checked the Ranger Admin Logs as well, no entries. Any suggestions? Thanks,Megh

In the Yarn Queue Manager UI view below, there is 3 levels: root parent and childthe color between root and parent is green but the color between child and parent is orange,What those colors mean? 

Hi All,I have a problem when add service nifi in my cluster, when nifinode starting it failed :Code: 400 Output: {"statusCode":400,"msgDesc":"Operation denied. Group name: nifi specified in policy does not exist in ranger admin."}Failed to create ranger policy...Ensure Ranger has following groups:nifiEnsure Ranger has following users:Code: 400 Output: {"statusCode":400,"msgDesc":"Operation denied. Group name: nifi specified in policy does not exist in ranger admin."}Failed to create ranger policy...Ensure Ranger has following groups:nifiEnsure Ranger has following users:It says nifi group not exist in ranger instead user nifi has exist in ranger, but when i try to create manually nifi group in ranger, it solved.Is it any ways to create automatically nifi group without create manually in ranger? Thanks n Regards Meidi

Hi experts,I am trying to install hive using ambari which i downloaded the bigtop repo. Here is the error:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py", line 79, in <module> HCatClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py", line 35, in install self.install_packages(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 821, in install_packages retry_count=agent_stack_retry_count) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/

I have successfully installed Oracle VM Virtual Box Manager. I am trying to do very simple tutorials to get familiar with the hadoop framework  https://www.cloudera.com/tutorials.html but i can't seem to find the cloudera machine learning, cloudera data platform and so on where i can i find them please? Also what tutorial can you recommend to get familiar with Hadoop, MapReduce and Yarn? there are so many.. Thank you

Hello,     We're using Cloudera Manager 6.3.1 to deploy CDH6.3.2, and it componet of Zookeeper version is 3.4.5+cdh6.3.2. but we fond a vulnerability CVE-2019-0201 from https://www.cvedetails.com/cve/CVE-2019-0201/?q=CVE-2019-0201  (An issue is persent in Apache Zookeeper 1.0.0 to 3.4.13 and 3.5.0-alpha to 3.5.4-beta. ZooKeeper's getACL() command doesn't check any permission when retrieves the ACLs of the requested node and returns all information contained in the ACL Id field as plaintext string).     Now, i want to upgrade Zookeeper to deal this vulnerability in CDH6.3.2, but i don't kown what should i do. Should i use a independence Zookeeper-3.4.14 replace Zookeeper-3.4.5+cdh6.3.2?    Thank you in advance !     Best Regards !     Yushuo

Hi experts,I am trying to deploy a cluster with the apache bigtops package via ambari.I was able to deploy the basic components such as hdfs, yarn, mapreduce, and zookeeper via ambari.Now I am trying to install hive.I am able to install hive through the command line with "yum install hive*" Ambari is currently failing to install hcatalog.Looking at the ambari logs, I noticed this error:resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install hcatalog' returned 1. Error: Nothing to do However, when installing via command line, I am able to install hcatalog with"yum install hive-hcatalog" Can someone please let me know how I can have ambari run the command /usr/bin/yum -d 0 -e 0 -y install hive-hcataloginstead of /usr/bin/yum -d 0 -e 0 -y install hcatalog Any help is much appreciated.Thanks,

I have a tar file with .csv, .xml, and .txt files that I want to unpack using the UnpackContent processor.   Can I specify all three file types in the file filter?

I have an 8 GB RAM and I understand that Cloudera express won't be working on my computer.Is there any alternative? How can I work with Cloudera express using my computer? Is there any way to use the Cloud to get more RAM power?Thank you for your help and guidance

Hello Everyone, I have an external table (Text with Gzip compression) partitioned on a date column. I have created another external table with the same structure but with Parquet format with Snappy compression. I want to copy all the data from the text table to parquet table, and I'm using the following query for testing it out for one day.insert into parquet_table partition(asdt) select * from text_table where partition_col='somedate'; The partition contains almost 100M rows. This query runs for almost half an hour and then results into this error:ERROR : Vertex failed, vertexName=Reducer 2, vertexId=vertex_1619005705788_0004_2_01, diagnostics=[Task failed, taskId=task_1619005705788_0004_2_01_000240, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #1 at org.apache.tez.runtime.library.common.shuffle.ordered