Q&A

I have a flow that extracts an ID from a flow file, then does an ExectueSQL to fetch some additional data.   Since this is one at a time, it's very inefficient (~300 records per minute - which doesn't sound bad until it's processing a million records). Is there a way to run flow files through a process and extract the required value in batches of 50 or 100 so that it returns a comma-separated value (like 123, 124, 125, 126, ... etc.) to pass to the ExecuteSQL so it's returning data in larger batches instead of one at a time? Currently, it's doing an EvaluateJsonPath, pulling that one value to a property so the ExecuteSQL processor can construct the SQL like "SELECT * FROM TABLE_NAME WHERE KEY = ${key}".  I'd like to modify that so it's "...WHERE KEY IN (${keyList})"  and keyList is the comma-separated list created above.  Not sure how to approach this, however. 

I have a JSON text coming in a flowfile. { "_id":{ "_data":"8261DB7CD7000000362B022C0100296E5A1004B9491C697A7B45E1BCDF455EB59ABC6F46645F6964006461DB7CD7564AB600124193730004" }, "operationType":"insert", "clusterTime":{ "$timestamp":{ "t":1641774295, "i":54 } }} I would like to be able to replace all JSON attribute names with lowercased names.Nifi is quite new to me. Can anyone please give a sample implementation?Thank you very much.

Please help me, I found log files making full on path after I deleted some logs in /var/log/cloudera-scm-agent and back to CM. I try to restart Cloudera Management Service and then I face Event Server cannot start anymore   My stdout err message show as below :  [14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29239+ date+ dateFri Jan 14 19:41:20 +07 2022+ umask 027+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'+ MGMT_HOME_ARG_NAME=--mgmt-home++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh+ cloudera_config=/opt/

Is it possible to send hive variables to multiple hql files/queries and execute them in a single hive session?when I try something like this, only the first query is picking up the variables and other is returning no result. $beeline -u "{$url}" --hivevar FROM=2020 --hivevar TO=2021 -e "$(cat loadTable1.hql loadTable2.hql)" Where both the hqls take these two variables as parameters.My goal is to run my hqls in parallel in one hive sessions, are there any cons to this approach? -Thanks

HI,We have a HDFS Capacity Utilization alert on Ambari. The alert says that we have a 81% Disk usage. After check in HDFS we realized that most of the data used comme from the following folders:- /tmp- /user/hive/checkpoints_tmpCould you please give us a clear procedure to clean up those folders without losing any data?Thank you  Environnement infos:HDP-3.0.1.0HDFS 3.1.0YARN 3.1.0MapReduce2 3.0.0.3.0Hive 3.0.0.3.0HBase 2.0.0.3.0ZooKeeper 3.4.9.3.0Ambari Metrics 0.1.0Atlas 0.7.0.3.0Kafka 1.0.0.3.0Knox 0.5.0.3.0Ranger 1.0.0.3.0Kerberos 1.10.3-30 

Hi, I am getting error when I run kudu ksck command Errors:==================Corruption: master consensus error: there are master consensus conflictsFAILEDRuntime error: ksck discovered errors All 3 Masters & Tablet server are reported as healthy, Any assistance / pointer in fixing this error. Thanks  

Hi Everyone, I need to delete hbase znode from zokkeeper CLI but getting follwing error when I try that:[zk: localhost:2181(CONNECTED) 1] rmr /hbase-secureAuthentication is not valid : /hbase-secure/replication[zk: localhost:2181(CONNECTED) 2] This is the ACL set:[zk: localhost:2181(CONNECTED) 11] getAcl /hbase-secure'world,'anyone: r'sasl,'hbase: cdrwa I have even tried to set the ACL as follows but it didn't help either:[zk: localhost:2181(CONNECTED) 7] setAcl /hbase-secure world:anyone:cdrwaAuthentication is not valid : /hbase-secure[zk: localhost:2181(CONNECTED) 8] rmr /hbase-secureAuthentication is not valid : /hbase-secure/replication The Cluster is kerberized, what am I missing here? Appreciate all the help! 

Hi guys, good afternoon!I'm using NiFi to load a Hive table.I would like to know what alternatives I have and how I need to proceed so that a FlowFile from the ExecuteSQL Processor is transformed from AVRO to TXT (CSV).I tried to use ConvertRecord Processor but AvroReader, even using Embedded Avro Schema, gives the following error:ConvertRecord[id=be0735bf-babb-134a-acfa-71df3952b30d] Failed to process StandardFlowFileRecord[uuid=610e5f55-7040-45d5-8c79-d734dbd589d2,claim=StandardContentClaim [resourceClaim=179,sectionidresource12,container=12 container=12 =112], offset=53226, length=1281],offset=0,name=46842655031134935,size=1281]; will route to failure: org.apache.nifi.processor.exception.ProcessException: Cannot write Schema Name As Attribute because the Schema Name is not knownI have an HDP version 2.6.5 and the NiFi version is 1.5.0.3.1.2.0-7Thanks in advance for any help.Luis Leite

Hi,We are trying to run some code in parallel using Zeppelin two notebooks in two Zeppelin UI open in two tabs in the browser login with the same user. However, even if we run both notebooks at the same time, one of them gets executed and the other one waits until the first one finishes to start running. Is it normal behavior of Zeppelin? Is there any way it supports parallel execution of notebooks with the same user? Thanks.

We have a new 1.13.2 NIFI Cluster.  The cluster has nodes randomly restarting and we are unable to determine the cause.  We do see 'connection refused' errors intermittantly without explanation in the logs (attached).  We don't know what additional steps are to take at this point.  nifi.properties and latest nifi-app.log. Cluster Details12 x 8 cores - 32gb of Memory These servers leverage a 5 node Zookeeper Quorum, we have not seen latency or file handler issues there.  NIFI Properties File (redacted to protect the innocent)# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/l

Hello All, I am trying to access the NiFi bulletin-board through NiFi API with InvokeHTTP process in order to consume the failure message. Referencing these curl command of this post: Creating NiFi Template via Rest API  I am at the V1.15 with single-user login mode.My current attempt is:1. User GenerateFlowFile to simulate the content for API /access/token.{"Custom Text" : "username=${username}&password=${password}"} 2. Use InvokeHTTP to request the access token.HTTP MethodPOSTRemote URLhttps://localhost:8443/nifi-api/access/tokenContent Typex-www-form-urlencodedThis will response an access token which will be sent within the header.  3. Use InvokeHTTP to request the bulletin-board.HTTP MethodGETRemote URLhttps://localhost:8443/nifi-api/bulletin-boardAuthenticationBearer ${access_token}This will response the bulletin-board I need.HoweverIn my current approach, the password will appear in the flowfile content as plain-text witch doesn't  make it a

Hi,I am facing issues with HDFS in my Cloudera Manager cluster. I have a cluster of 4 virtual machines (1 master and 3 slaves) on cloud and at every shutdown and restart of the cluster/vms HDFS shows some missing blocks.I have found a "workaround" as deleting the missing files and restarting the service would solve the problem but sometimes the missing files contain important data related to other hadoop services that generate issues if they are removed. Is there another solution to avoid deleting data ?Thank you.

Hello,I use yarn (base on HDP3.1.5.0-152),when I view the nodemanager log,I found the log doesn't note the log likes:INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 28532 for container-id container_e51_1637148502450_21618293_01_000024: 4.1 GB of 4.5 GB physical memory used; 6.3 GB of 9.4 GB virtual memory usedI don't remember there are some conf in yarn-site.xml can control this.Any one help?

FileSystem Operation Failed: File /tmp/xxx.txt could only be written to 0 of the 1 minReplication nodes. There are 1 datanode(s) running and 1 node(s) are excluded in this operation. I'm using Cloudera Trial version installed in single node. Does anyone have this problem too?

I'm just starting to learn NiFi. I need to read parquet data from s3 bucket, I don’t understand how to set up lists3 and fetchs3object processors for reading data. full path looks like this: s3://inbox/prod/export/date=2022-01-07/user=100/cro.parquet I'll write data to sql database - I don't have problems with it, but not sure)) I tried to configure the lists3 processor myself and I think is not very good bucket inbox aws_access_key_id aws_secret_access_key region US EAST endpoint override URL http://s3.wi-fi.ru:8080

I have the following CSV: mobileusernamebill_idbill_dateproduct_idproduct_price9876543210JohnAB102020-12-23X345009876543210JohnAB102020-12-23X352309876543210JohnAB222020-11-14X897009999999999AliceAC542019-12-20X10109 And I want to convert them to the following structure: { "users": [ { "mobile": "9876543210", "username": "John", "bills": [ { "bill_id": "AB10", "bill_date": "2020-12-23", "products": [{ "product_id": "X34", "product_price": "500", }, { "product_id": "X35", "product_price": "230", }] }, { "bill_id": "AB22", "bill_date": "2020-11-14", "products": [ { "product_id": "X89", "product_price": "700", }] }] }, { "mobile": "9999999999", "username": "Alice", "bills": [ {

Hi,I've a single Kafka node configured which receives data from telegraf agents and then it is passed to Influx Database.I've a log retention set for 1h in kafka.server.properties file as - log.retention.hours=1But Kafka is having log queues formed and log getting full in 24 hours of time:  kafka/bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server localhost:9101 --describe --group kafka_consumer --command-config admin.props | awk '/<topic>/{print $5}' -20333796490 Is it because of the throughput when logs can grow more than for an hour if not consumed?In another environment where the load is even more, this issue is not observed.Any suggestion?

Hello, we have Impala issue last days with regularly Impala Catalog Server restart.Access data via Impala works, but Cloudera Manager sent alerts every 5 minutes with IllegalStateException: Table.toThrift() called without holding the table lock: table_name We tried to invalidate metadata on table and compute stats, but no success.It works normally before and there were no changes on cluster. Have you any idea what is the problem? RegardsKamil

Hi,We are getting the following alert from ambari notifications .NameNode High Availability HealthActive['xx-xxx-x1-xx02.xxxxx.xx:50470'], Standby[], Unknown['xx-xxx-x1-xx01.xxxxx.xx:50470'] Could you please help us solve this issue? Thank you

Hi All, I'm new person is a hadoop world. Can not find answer on this question. How to gracefully stop YARN role on a data node and  save or pause running jobs So the jobs will not exit with killer or crash status. I know that in ClouderaManager you can decommission yarn role when you can stop it. Is this a safe way to keep the jobs running and not failed state? Is this a gracefull yarn role shutdown or where is other way to do this?   Please advice me how to do this. Thank you.  

OS WINDOWS11Java: d:\Java\jdk1.8.0_202hadoop: d:\hadoop-3.1.3datanodes : D:\hadoop-3.1.3\data\datanode & D:\hadoop-3.1.3\data\namenodedone "gsudo chwon 777 -R data" %PATH%I have overwritten bin files from here apache-hadoop-3.1.3-winutils/bin at master · s911415/apache-hadoop-3.1.3-winutils (github.com) hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.disk.balancer.enabled</name> <value>false</value></property> <property> <name>dfs.data.transfer.protection</name> <value>false</value> </property> <property> <name>dfs.encrypt.data.transfer</name> <value>true</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/D:/hadoop-3.1.3/data/namenode</va

Hi, I would like to know where can I download a CDH open-source for tests : for Ubuntu (or Centos)?   Many thanks!   BR, jludac  

Hello,I need to move one big table (16 GB)  from one oracle database to another. When I try to move table with more then 1 GB data, my nifi GUI crashes. Can you suggest me how to do it? Thanks' in advance Lejla 

Hello,When trying to access Hbase through Hue i have this error in the log WARN org.eclipse.jetty.server.HttpChannel: handleException / org.apache.hadoop.security.authorize.AuthorizationException: User: hbase is not allowed to impersonate hueknowing that i had set a Thrift Server instance on Hbase and it is running okay. 

 Now I used SSL with Kafka nodes that made from Self cert but it cannot pass audit by infra team. I need to adapt company cert without using self cert to use with syntax look likes as follow : openssl req -new -sha256 -nodes -newkey rsa:2048 -config <(cat <<-EOF[req]default_bits = 2048prompt = nodefault_md = sha256req_extensions = req_extdistinguished_name = dnSamplePassword123[ dn ]C=THST=BangkokL=ChaxxxO=xtac Public Company LimitedOU=Enterprise Service SupportemailAddress=john.doe@xtac.co.thCN = rs-xxx-hmb-201.xtac.dev [ req_ext ]subjectAltName = @alt_names [ alt_names ]DNS.1 = rs-xxx-hmb-201DNS.2 = rs-xxx-hmb-201.xtac.dev EOF) It cannot use all of these syntax, please help me how should I do or what I did it wrong??