Solved

There have some rows data with header in txt file, like this: test_a|test_b|test_c|test_d|test_ea|b|3.0|4.0|5.0a|b|3.0|4.0|5.0a|b|3.0|4.0|5.0 now , i want  remove the value after test_c and test_d  decimal point,  the result change this: test_a|test_b|test_c|test_d|test_ea|b|3|4|5.0a|b|3|4|5.0a|b|3|4|5.0 how could i do? thanks.

hi there,I have encountered a common error as follows:Memory left in process limit: 180.41 GBMemory left in query limit: -467.80 KBQuery(7e4794a439e5f26d:748d4c7c00000000): memory limit exceeded. Limit=1.00 GB Reservation=635.75 MB ReservationLimit=819.20 MB OtherMemory=388.71 MB Total=1.00 GB Peak=1.00 GBI don't quite understand the difference between the "process limit" and "query limit", and what's the reservation memory for?many thanks!

Our sales team are extracting the months sales opportunities, actuals etc. Within the one CSV file there are potentially 100+ rows. Now, we need to create new flow files (array?) containing individual sales people. We have 20+ sales people each working on many opportunities.salesOpId,salesRepId, salesOpStage, .....142435,135235,inception ....142436,572856,inception ....142437,135235,pipelining ....142435,572856,inception....149468,135235,contract....149464,135653,contract.... As you can seesalesRepId 135235 is working against 3 opportunities, (142436,142437,149468)salesRepId 572856 is working against 2 opportunities (142436, 142435)salesRepId 135653 is working on a single opportunity (149464) For this example I need 3 FlowFiles created (which will be sent to their respective managers); I'm capable of reading the CSV changing it to Json; but now I'm completely stumped on the next step - split the json? Any help would be greatly appreciated. Ron M. &nbs

Hi, We have a CDH cluster with version 5.15 and we have migrated the db used by Cloudera from postgres to Oracle. We have followed the migration steps and took the cluster down. When we tried to start the cloudera server again it's failing with the below error.  022-01-13 09:53:19,028 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.2022-01-13 09:53:19,029 INFO main:com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 51300.2022-01-13 09:53:19,029 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 51300WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version. This is expected if you are starting Cloudera Manager for the first time. Please also ignore any error messages about missing tables. Moving ahead assuming no upgrade: org.hibernate.MappingException: No Dialect mapping for JDBC type: -92022-01-13 09:53:19,043 ERROR main:com.cloudera.server.cmf.bootst

Hive Table:Internal tableROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' STORED AS TEXTFILEData ingestion is successfulFacing memory issues while querying!Input file: JSON fileWhile researching on memory configuration, found some information here to calculate recommended memory configuration against available resources.Our POC cluster size: 4 node cluster - 8 GB, 2 cores and 2 disks on each nodePlease refer below information and let me know if memory needs to be extended or do TEZ memory configuration needs to be changed to overcome "out of memory" issue?Calculated memory configuration uisng yarn-utils.py:$ python yarn-utils.py -c 2 -m 8 -d 2 -k True Using cores=2 memory=8GB disks=2 hbase=True Profile: cores=2 memory=5120MB reserved=3GB usableMem=5GB disks=2 Num Container=4 Container Ram=1024MB Used Ram=4GB Unused Ram=3GB yarn.scheduler.minimum-allocation-mb=1024 yarn.scheduler.maximum-allocation-mb=4096 yarn.nodemanager.resource.memory-mb=4096 mapreduce.map.memory.mb=1

I am using CDP 7.1.7 and the cluster has not enabled Kerbores yet. Ranger is not enabled either. I followed the step in this post https://community.cloudera.com/t5/Support-Questions/CDP-7-1-3-Zepplin-not-able-to-login-with-default-username/td-p/303717 to be able to log in as admin. But this "admin" account has no permission to access the configuration or interpreter page.  According to CDP documentation, https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/configuring-zeppelin/topics/enabling_access_control_for_interpreter__configuration__and_credential_settings.html, to configure shiro.ini for Zeppelin security, I have to go through Zeppelin web UI.  What should I do? Regards, 

how to use ListFTP and FetchFTP processors get all files two hours before the current time in FTP server.There will be more and more files in the FTP server named by dateThese filename in the FTP contains a date as follows: xxxx-xxxx-xxxx-xxxx-20211231143000-xxxx-xxxx.csvxxxx-xxxx-xxxx-xxxx-20211231153000-xxxx-xxxx.csv......    

Hi,   our CDH CM is down (difficult issue, related to certificates renewal), so we would like to access everything related to the workbench (sessions, projects, ...) from terminal(I guess there is a database on the CDSW host related to that...)  thanks for help

While creating table , its failing with below error.Query: CREATE TABLE REPORTKUDUPRIMARY KEY(Marketing_Cloud_Visitor_ID)PARTITION BY HASH(Marketing_Cloud_Visitor_ID) PARTITIONS 10STORED AS KUDU TBLPROPERTIES('kudu.master_addresses'='X.X.X.X','kudu.num_tablet_replicas'='1')AS SELECT * FROM REPORTN  WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation: required 52236317, size limit 10485760Key already present in Kudu table 'impala::db.REPORTKUDU'. (1 of 9006 similar)

Hello  I'm new to the community and to cloudera/big data in general,  I am having issues with hive performance I have for example a table of 600 records and when I use a select * it runs in .05 seconds but if I use for example a count(*) or any function it runs in like 17 seconds, do any have any tip or trick to check performance or what parameter to check/modify in order to improve this execution time? My enviroment are CDH 6.1.0 withHive 2.1.1-cdh6.1.0 Thank you in advance  Ulises Rangel

Hello Guys, We have 3 node NiFi cluster and the NiFi version is 1.12.1nifi-hatest-01nifi-hatest-02nifi-hatest-03 All the 3 nodes are in cluster, but not in UI.In all the 3 NiFi nodes, we see this 2 nodes in cluster mode:Now, as we see nifi-hatest-02 is the primary and cooridantor. And nifi-hatest-03 is not being shown here. In nifi-hatest-01, we see nifi-hatest-01 is connecting to nifi-hatest-022022-01-19 16:02:09,840 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2022-01-19 16:02:09,820 and sent to nifi-hatest-02:9088 at 2022-01-19 16:02:09,840; send took 20 millis In nifi-hatest-03, we see nifi-hatest-03 is also connecting to nifi-hatest-022022-01-19 16:02:09,840 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2022-01-19 16:02:09,820 and sent to nifi-hatest-02:9088 at 2022-01-19 16:02:09,840; send took 20 millis In nifi-hatest-02

Hi Team ,  We have upgraded from CDH  to CDP 7.4.4,Atlas has been installed and Navigator also running parallely .Is it safe to remove and clean up databases of cloudera navigator and from CM ?Please share the steps involved. Thanks,Syed

I have a JSON text coming in a flowfile. { "_id":{ "_data":"8261DB7CD7000000362B022C0100296E5A1004B9491C697A7B45E1BCDF455EB59ABC6F46645F6964006461DB7CD7564AB600124193730004" }, "operationType":"insert", "clusterTime":{ "$timestamp":{ "t":1641774295, "i":54 } }} I would like to be able to replace all JSON attribute names with lowercased names.Nifi is quite new to me. Can anyone please give a sample implementation?Thank you very much.

Please help me, I found log files making full on path after I deleted some logs in /var/log/cloudera-scm-agent and back to CM. I try to restart Cloudera Management Service and then I face Event Server cannot start anymore   My stdout err message show as below :  [14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29239+ date+ dateFri Jan 14 19:41:20 +07 2022+ umask 027+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'+ MGMT_HOME_ARG_NAME=--mgmt-home++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh+ cloudera_config=/opt/

HI,We have a HDFS Capacity Utilization alert on Ambari. The alert says that we have a 81% Disk usage. After check in HDFS we realized that most of the data used comme from the following folders:- /tmp- /user/hive/checkpoints_tmpCould you please give us a clear procedure to clean up those folders without losing any data?Thank you  Environnement infos:HDP-3.0.1.0HDFS 3.1.0YARN 3.1.0MapReduce2 3.0.0.3.0Hive 3.0.0.3.0HBase 2.0.0.3.0ZooKeeper 3.4.9.3.0Ambari Metrics 0.1.0Atlas 0.7.0.3.0Kafka 1.0.0.3.0Knox 0.5.0.3.0Ranger 1.0.0.3.0Kerberos 1.10.3-30 

Hi, I am getting error when I run kudu ksck command Errors:==================Corruption: master consensus error: there are master consensus conflictsFAILEDRuntime error: ksck discovered errors All 3 Masters & Tablet server are reported as healthy, Any assistance / pointer in fixing this error. Thanks  

Hi,We are trying to run some code in parallel using Zeppelin two notebooks in two Zeppelin UI open in two tabs in the browser login with the same user. However, even if we run both notebooks at the same time, one of them gets executed and the other one waits until the first one finishes to start running. Is it normal behavior of Zeppelin? Is there any way it supports parallel execution of notebooks with the same user? Thanks.

We have a new 1.13.2 NIFI Cluster.  The cluster has nodes randomly restarting and we are unable to determine the cause.  We do see 'connection refused' errors intermittantly without explanation in the logs (attached).  We don't know what additional steps are to take at this point.  nifi.properties and latest nifi-app.log. Cluster Details12 x 8 cores - 32gb of Memory These servers leverage a 5 node Zookeeper Quorum, we have not seen latency or file handler issues there.  NIFI Properties File (redacted to protect the innocent)# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/l

Hello All, I am trying to access the NiFi bulletin-board through NiFi API with InvokeHTTP process in order to consume the failure message. Referencing these curl command of this post: Creating NiFi Template via Rest API  I am at the V1.15 with single-user login mode.My current attempt is:1. User GenerateFlowFile to simulate the content for API /access/token.{"Custom Text" : "username=${username}&password=${password}"} 2. Use InvokeHTTP to request the access token.HTTP MethodPOSTRemote URLhttps://localhost:8443/nifi-api/access/tokenContent Typex-www-form-urlencodedThis will response an access token which will be sent within the header.  3. Use InvokeHTTP to request the bulletin-board.HTTP MethodGETRemote URLhttps://localhost:8443/nifi-api/bulletin-boardAuthenticationBearer ${access_token}This will response the bulletin-board I need.HoweverIn my current approach, the password will appear in the flowfile content as plain-text witch doesn't  make it a

Hello,I use yarn (base on HDP3.1.5.0-152),when I view the nodemanager log,I found the log doesn't note the log likes:INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 28532 for container-id container_e51_1637148502450_21618293_01_000024: 4.1 GB of 4.5 GB physical memory used; 6.3 GB of 9.4 GB virtual memory usedI don't remember there are some conf in yarn-site.xml can control this.Any one help?

I have the following CSV: mobileusernamebill_idbill_dateproduct_idproduct_price9876543210JohnAB102020-12-23X345009876543210JohnAB102020-12-23X352309876543210JohnAB222020-11-14X897009999999999AliceAC542019-12-20X10109 And I want to convert them to the following structure: { "users": [ { "mobile": "9876543210", "username": "John", "bills": [ { "bill_id": "AB10", "bill_date": "2020-12-23", "products": [{ "product_id": "X34", "product_price": "500", }, { "product_id": "X35", "product_price": "230", }] }, { "bill_id": "AB22", "bill_date": "2020-11-14", "products": [ { "product_id": "X89", "product_price": "700", }] }] }, { "mobile": "9999999999", "username": "Alice", "bills": [ {

Hello, we have Impala issue last days with regularly Impala Catalog Server restart.Access data via Impala works, but Cloudera Manager sent alerts every 5 minutes with IllegalStateException: Table.toThrift() called without holding the table lock: table_name We tried to invalidate metadata on table and compute stats, but no success.It works normally before and there were no changes on cluster. Have you any idea what is the problem? RegardsKamil

Hi,We are getting the following alert from ambari notifications .NameNode High Availability HealthActive['xx-xxx-x1-xx02.xxxxx.xx:50470'], Standby[], Unknown['xx-xxx-x1-xx01.xxxxx.xx:50470'] Could you please help us solve this issue? Thank you

Hi All, I'm new person is a hadoop world. Can not find answer on this question. How to gracefully stop YARN role on a data node and  save or pause running jobs So the jobs will not exit with killer or crash status. I know that in ClouderaManager you can decommission yarn role when you can stop it. Is this a safe way to keep the jobs running and not failed state? Is this a gracefull yarn role shutdown or where is other way to do this?   Please advice me how to do this. Thank you.  

OS WINDOWS11Java: d:\Java\jdk1.8.0_202hadoop: d:\hadoop-3.1.3datanodes : D:\hadoop-3.1.3\data\datanode & D:\hadoop-3.1.3\data\namenodedone "gsudo chwon 777 -R data" %PATH%I have overwritten bin files from here apache-hadoop-3.1.3-winutils/bin at master · s911415/apache-hadoop-3.1.3-winutils (github.com) hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.disk.balancer.enabled</name> <value>false</value></property> <property> <name>dfs.data.transfer.protection</name> <value>false</value> </property> <property> <name>dfs.encrypt.data.transfer</name> <value>true</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/D:/hadoop-3.1.3/data/namenode</va