Q&A

Hi Everybody,I'm attempting to download and use for evaluation CDH 6.3.2.Gathering information I see that is possible to use it without any license using Cloudera express edition. The point is that if I attempt to downloade it I need username and password that I need to generate after a valid subscription is performed.  Can somebody confirm me how to proceed? is it possible that to download a license free version I need anyway to contact the sales Team. kind Regards

Hi Everyone, I collect data files with NiFi using ListSFTP AND FetchSFTP processors. In these data files I have several different lines. I want to know if there is a processor that can allow me to count the number of lines in the files as I collect them to have a total number of lines. Someone can help me.

What is the difference between using the PublishKafka processor versus the PublishKafkaRecord processor when sending data from Apache NiFi to Kafka?

Hello,I am trying to migrate my Nifi registry from one server to another.Used niFi toolkit CLI for the same.Below are the commands used -On source registry server :registry list-bucketsregistry list-flowsOn target registr server:registry create-flowregistry sync-flow-versionsFlows with versions got migrated properly, but one issues noticed is the version date on target server.Instead of actual date of version it shows migrated date for all versions in target server after migration.Find the attached screnn shot for ref.Is there any way to fix this? ThanksMahendra

Hi Team, Is there any command to find CM agent and Server version from the command line? I know we can find out using the rpm -qa version but any other way I can get to find the java -version?   

Hi, I am extracting image width and image height using ExtractImageMetadata processor and comparing the Image Width and Image Height with threshold value. If it is greater than the threshold value than I will resize the image. Attribute names, Image Width and Image Height is different for different image formats.   e.g. For .png PNG-IHDR.Image HeightPNG-IHDR.Image Width   For .jpg JPEG.Image Height  JPEG.Image Width I want to have a regular expression something like below. So that irrespective of image format, I will be able to compare the Image Width and Image Height. ${"*Image Width":ge(1000):and(${"*Image Height":ge(1000)})}   Above regular expression is not working as expected. Any input will be helpful.   Thanks and Regards, Biswa    

Hi Team, How to find active name node from cli and also want to know where to check automatic namenode failover triggered in the cluster?

In log4j2 official we can see below content.Official says check debug level can use the second way. if applicableto above debug level?

Hello,Receiving from an embedded micro over Ethernet a JSON file of the form:{"id" : 123456,"ip" : "*","t" : -12.9,"T" : -23.8,"variables": ["user1",0,12.97,23]}and knowing what each variable represents in the "variables" array as:"username" : "user1","active" : 0,"temperature" : 12.97,"age" : 23How can I update the JSON file with the key name of the variables such that the end result is:{"id" : 123456,"ip" : "*","t" : -12.9,"T" : -23.8,"variables": ["username" : "user1","active" : 0,"temperature" : 12.97,"age" : 23]}and then be able to access these variables with their name?Thank you.

Hi, I am trying to create query with attribute value in getMongo. My attribute is itemType, i want to create query using this. {"itemType": ${itemType}} if i put hard code value there instead of ${itemType} it works

How does it scale? If I have 100 k templates, how will Flow XML load in memory? How does it manage individual processors in memory? 

I restarted some service via Cloudera Manager UI and roles were restarted too. I want to inspect the role log, but "Log search is not supported for this role" error is shown when clicking on "Full log file" link belowIs there a way to enable log for role/service? 

When I turn off the computer and start HBase again.The following problems arise.How can I do? 2021-05-20 17:19:00,425 WARN [master/rhel145-2-fgos:16000:becomeActiveMaster] regionserver.HRegion: Failed initialize of region= master:store,,1.1595e783b53d99cd5eef43b6debb2682., starting to roll back memstorejava.io.EOFException: Cannot seek after EOF at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1648) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168) at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323) at org.apache.hadoop.hbase.wal.WALFactory.cre

I have a situation where my namespace system table is not online and because of that I’m seeing these messages in HBase master log: 2021-03-17 20:29:54,614 WARN  [Thread-18] master.HMaster: hbase:namespace,,1575575842296.0c72d4be7e562a2ec8a86c3ec830bdc5. is NOT online; state={0c72d4be7e562a2ec8a86c3ec830bdc5 state=OPEN, ts=1616010947554, server=itk-phx-prod-compute-6.datalake.phx,16020,1615483461273}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined. I came across this article for fixing this problem: https://docs.cloudera.com/runtime/7.2.7/troubleshooting-hbase/topics/hbase_running_hbck2.html But while following the article and running suggested command, running into following problem: getting “Failed to specify server's Kerberos principal name” error. I need clarification on following two points: Do we need any specific format to run hbck2 utility if the cluster is kerberized? I.e if the principal needs

We are trying to pull metrics into Promtheus andfor the life of me I spent most of lastnight trying to get CDH 6.2 to work with a javaagent.  I read a previous post where someone was have a heck of a time with jmx exporter and Ambari, so I'm assuming this is also an issue in CDH? If I goto:HBase Service Environment Advanced Configuration Snippet (Safety Valve)and add this:HBASE_MASTER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10110 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9214:/run/cloudera-scm-agent/hbasemaster.yml"HBASE_REGIONSERVER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10111 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9215:/run/cloudera-scm-a

KUDU version:  1.9.0+cdh6.2.01、memory_limit_hard_bytes: 100G2、memory.soft_limit_in_bytes: -13、memory.limit_in_bytes: -14、maintenance_manager_num_threads: 45、block_cache_capacity_mb : 2GThe cluster has 4 tablet-server，and three yarn nodemanager are in the same node with tablet-server.When I running a MR job in Yarn，just hive sql， and kudu tablet server will quit random, with fowlling logs:  T 5e5cdb8cf25d4c93aeaf013781419109 P ac586d8c49f84c4c82770ae079256893 -> Peer ac44fc76284d4b959eca897309e465b0 (ch4.360kad.com:7050): Couldn't send request to peer ac44fc76284d4b959eca897309e465b0. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 10.0.57.16:26274 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.    I dont know how sole this problem 

Dear Experts,I am interested in accessing all the data in a hbase table once, so it will be cached.I am using an application that reads from hbase and I notice it is a lot faster when the data was read before, so I would like to "preheat" the data by warming it up: accessing it all once, so it is stored in the cache.I am quite familiar with the cluster configuration necessary to enable caching but wonder what is the easiest way to do this.Is there a command that accesses all the data in all regions 1x? 

Hi, I am new to Nifi and three words to say. "I love it". To move forward, one needs to make a serious decision and trying to solve one puzzle.  The flow I need to design is reading from Kafka messages from 1000 different topics (each topic is customer, need to keep them separate) and each topic has 100 different types of command-type or event-type. Every event type has JSON data with different fields and values which need to transform.  My worry is what is the best way to handle this?How to route and handle 100 different command/events along with 1000 diff topics?

Hi,  I am facing an issue while running ranger user sync. It's able to sync groups but not users. I am not getting any errors also. 

Hello all, We are facing an issue in viewing YARN logs from Resource Manager UI. We have two HDP clusters. Cluster1 and Cluster2 have similar configurations. Cluster1 has no problem but getting the unauthorized issue from Cluster2 for viewing the YARN logs.   Cluster1 (the good one) has the below configurations.yarn.acl.enable=trueyarn.admin.acl=yarn,user1 admingroup1,admingroup2 Cluser2 (the bad one) has the below configurations.yarn.acl.enable=trueyarn.admin.acl=dr.who,user1,yarn admingroup2 Please note there are users in admingroup2 who want to view the YARN logs but not able to do so on Cluster2. It shows as below. Please share if there are more things to check? Thank you for the help. 

Under Nifi UI,  component toolbar is inactive . I have given all the permission under ranger for process groups . I don't have login issues but cant select any processers once I login .

 I am trying to learn flume and for this I am creating some rows to ingest. I have a row like"ted";"27";"hi, i am good; you?";"ok"With the delimiter as ; it separates me also the hi, i am good you? I want only to change field when the ; has a " next to it. Is this possible?

Hello,I am trying to use nifi toolkit cli to list the bucket in Nifi registry using below command -/usr/hdf/3.4.1.1-4/nifi-toolkit/bin/cli.sh registry list-buckets -u http://<ip>:<port>But getting below error :/usr/hdf/3.4.1.1-4/nifi-toolkit$ /usr/hdf/3.4.1.1-4/nifi-toolkit/bin/cli.sh registry list-buckets -u http://<ip>:<port>cli.sh: JAVA_HOME not set; results may varyException in thread "main" java.lang.NoClassDefFoundError: javax/annotation/Priority        at org.glassfish.jersey.model.internal.ComponentBag.modelFor(ComponentBag.java:551)        at org.glassfish.jersey.model.internal.ComponentBag.lambda$registerModel$10(ComponentBag.java:477)        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)        at org.glassfish.jersey.internal.Errors.process(Errors.java:229)&n

I am trying to load data from the below input to db.   My input is as shown below i, John, $100 ii, Kevin, $150 iii, Steve, $200   Created getfile, ReplaceAttributes, Extracttext, Route to attributes, putsql flow   I have managed to parse fixed width file records and replaced with comma delimited values.  ExtractText processor with Enable Multiline Mode=true, Enable DOTALL Mode=true. After execution I see below in provenance event in attributes tab   line i line.0 John line.1 $100   Please advise how to load other two records which was not getting displayed in the attributes. 

Hello,I struggle on a dependency issues in spark, now. Being new in spark, I hope there is a simple remedy.The question is, is there any mechanism how to separate dependencies of the spark engine and dependencies of a application. Example: The latest version of spark-core_2.12 (3.1.1, March 2021) depends onhadoop-client (3.3.0, March 2020) which itself depends on hadoop-common (3.3.0, July 2020)which finally depends on an antient version of gson (2.2.4, May 2013).You can easily find many other examples, e.g. commons-codec, protobuf-java ... So, what if your application, basically a library developed outside spark, depends on the latest (no longer compatible) version of gson 2.8.6?My obviously naive approach to start a spark application ends in runtime incompatibility clashes (e.g. with gson) Best regardsJaro