Solved

Hi guys, How to get user specified configuration data in hive udf? what is your solution?i think there are basically three ways to achieve this goal:1. use a hdfs file to store the configuration data in xml/json/txt format, and then read the hdfs file in udf, users need to change the hdfs file when they want to change specific configuration parms;2. use a local xml/json/txt file and pack it into the udf jar, then read the local file in udf, users need to change the local file and repack it into the udf jar when they want to change specific configuration parms;3. use a hive table to store the configuration data, then read the table using hive sql dml statements in the udf (I know this sounds strange, normally we don't issue sql queries to hiveserver2 in udf, but this should also be possble),  users can use hive sql dml to change specific configuration parms when they need to.4. use a hive table to store the configuration data, then read the underneath hdfs file to get configur

Hi,  I got a problem while sending a Zero Byte flow file through Nifi PutSFTP processor. Wondering if there is some configuration I have to set? thanks. Use case: get a file from a directory (GetFile Processor) send the file through SFTP (PutSFTP Processor)if success, change the filename to new filename (UpdateAttribute Processor)Remove the all of the content (ReplaceText Processor). Zero Byte flow file send the new filename through SFTP (PutSFTP Processor) I gotten this error while sending Zero Byte flow file over using PutSFTP ProcessorPutSFTP[id=1a091db4-1f9e-14cd-5206-1514485febc8] Rejecting StandardFlowFileRecord[uuid=247457a8-6608-4271-a626-3c016a1bbe7c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1618805512561-40, container=default, section=40], offset=15043, length=0],offset=0,name=AS_MPS_20200927_010010.xml.done,size=0] because it is zero bytes It will be great if someone could help me.  Thanks in adv

Hello, I am taking a big data class and am new to this whole field, so I apologize for any incorrect use of terminology. I am trying to use sandbox. I was able to create a virtual machine, login to port 4200 with the username "root" and change the admin password using "ambari-admin-password-reset". The password works when I log into port 8080. But when I try to go back to port 4200 and login as "admin", instead of "root", the password doesn't work. Is there a different password? What am I missing? Any suggestions would be great. Thanks

Is there a processor that can detect hidden characters in NiFi? In our test environment I am using an API invoke to deliver a JSON payload, for added security we want to filter out any hidden and potentially malicious text. I have been trying to use RouteText, RouteOnContent and ExtractText processor to only allow alphanumeric and punctuation characters through but can't seem to get the regex to work when looking for 'uncommon' text or characters? any help would be appreciated. Andy 

Hello,  I have followed this guide:https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_ag_change_hostnames.html In order to change the hostname on my Cloudera single node CMS system. However, now none of my services will start. Everything is set to localhost (because its all on this one host). I notice that the old hostname is still referenced for the server... Even though Hive, Hue, YARN, etc are all set to "localhost" for their databases my ZooKeeper server keeps thinking it's still the previous hostname.How do I get rid of the hostname that is stuck in Cloudera Manager / ZooKeeper? The only other thing I can think to do is remove the "Server Name override" for the ZooKeeper myid and then restart the service to see if it picks up the new hostname.

Hi, According to the official Nifi documentation, the state allows Nifi processors to "resume from the place where it left off after NiFi is restarted. Additionally, it allows for a Processor to store some piece of information so that the Processor can access that information from all of the different nodes in the cluster". If my understanding is good, when we configure a zookeeper Provider, the state will not be persisted locally, instead, the data will be sent to zookeeper. I've explored the zookeeper znodes and could not find any data related to the state, all I can find are the informations about the Coordinator and Primary nodes. However, the local state directory is still filled. The configuration is very simple, I've 3 external ZK nodes and 3 Nifi instances. Here is an exerpt of the nifi.properties file:nifi.cluster.is.node=truenifi.zookeeper.connect.string=zk-node1:2181,zk-node2:2181,zk-node3:2181nifi.state.management.embedded.zookeeper.start=falsenifi.

File 'hdfs://xxx:8020/data/hive/warehouse/xxxx.db/test_bak/8743abe2968ecbe7-3f9c13700000000_2053286657_data.0.parq' has an incompatible Parquet schema for column 'xxxx.test_bak.ordered_revenue'. Column type: STRING, Parquet schema: optional double ordered_revenue [i:20 d:1 r:0] I made this change in Hue.

Hi, One of our customer willing to migrate to CDP 7.x, however customer wants to use code base of Flume developed in earlier version of CDP.Customer is not willing for NiFi migration and want to check if we can copy the open source Flume binaries to CDP 7.x and setup the runtime of Flume. Can this be achievable? Any suggestion are appreciated. ThanksVenkata Madhu 

Hi Community I am trying to copy HDFS from an HDP 2.6.x cluster (kerberized) to a CDP Private Cloud Base 7.1.5 cluster (not kerberized) and using other ports as well, it gives me an error So I write the command in consolehadoop distcp hdfs://svr2.localdomain:1019/tmp/distcp_test.txt hdfs://svr1.local:9866/tmp/ what could be the origin of the fault?  Thank you ***********ERROR*************************Java config name: nullNative config name: /etc/krb5.confLoaded from native config>>>KinitOptions cache name is /tmp/krb5cc_11259>>>DEBUG <CCacheInputStream> client principal is useradm/admin@LOCALDOMAIN>>>DEBUG <CCacheInputStream> server principal is krbtgt/LOCALDOMAIN@LOCALDOMAIN>>>DEBUG <CCacheInputStream> key type: 18>>>DEBUG <CCacheInputStream> auth time: Sun May 09 18:39:04 VET 2021>>>DEBUG <CCacheInputStream> start time: Sun May 09 18:39:04 VET 2021>>>DEBUG <CC

Hello, I'd like to implement a lock mechanism in a data flow that prevents a flow file from progressing.  Similar in a way to  the wait processor, holding a flowfile in a queue while another flowfile with a corresponding signal attribute passes through a 'gate' further down the flow, similar to a notify. In my specific case I load a number of files received together in to target tables. I have a single flow which does the following: PutSql - issues a truncate table command that empties the target staging table.  The table name is taken from an attribute.PutDatabaseRecord - Inserts the records into the staging table based on the same attribute.PutSql - executes a stored procedure passing the relevant tables name as parameter from the attribute.  This SP merges data from the staging table into a 'main' table.A number of these files, share the same staging and target tables and on occasions I have a race condition where the the staging table is trun

Hi, I am new to NIFI, I wanted to Get total count of number of events per minute from consumer kafka.After searching on internet, I get 'UpdateCounter' , but information was very less. Can someone suggest me how to add counter . I added ${filename} in UpdateCounter>properities>counter name, It is giving individual file name with value 1 in hamburger menu>counter.Or is there any other processor that can give the event count from the consumer kafka.Thank you in advance.

Hello,   We are currently using CDH 6.3 with Kafka as one of the components.  We have Kafka and Zookeeper services installed and running.   Want to install the Kafka Connect service to install and run a few source and sink connectors.  Is it possible to do so with this distribution.

I am really new to Hive/Impala. I am looking to convert some SQL queries into Hive. I have  a column called date (Ex:2011-09-22 05:01:38) and I would like to convert it into the following format: 09-22-2011-05.01.38 AM Any help is appreciated.

To be honest right at the beginning of this post: I am new at Cloudera and would like to download CDH6.1.1 for a customer as a trial (no production from my perspective). However, I got stuck in the paywall, which requires credentials, which I cannot request as the request page requires the credentials I am asking for. So can someone give me credentials, a download-link to be used without credentials or a hint for any third option?Likely this was answered somewhere in the community but most threads I found did not work out for me.

Hi,I have many flow file which has multiple field like mentioned belowcustomefields_12345 : nullcustomefields_12346: { "welcome to this issue"} I have to route these flow files on the condition :if customfield is null --> navigate to other processorif customfield is notnull or having some value other than null should be navigate to other processor.Please let me know how to achieve this. early response will be appreciated. Note: customfield should be used rather than customfield_12345. thanks!  

Hello experts, I have nifi registry with MySQL db as metadata store and FileSystem as Persistence provider.And this registry has 10+ flows with multiple versions. Now I am trying to integrate registry to GIT, that is trying to change persistence provider from FileSystem to GIT. I referred some docs (below urls) and steps to switch persistence provider etc, those all help for fresh GIT integration set and they wont carry/migrate already created buckets/flows/versions from FileSystem to GIT https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#switching-from-other-flow-persistence-provider https://community.cloudera.com/t5/Community-Articles/Storing-Apache-NiFi-Versioned-Flows-in-a-Git-Repository/ta-p/248713 Any help on this would be much appreciated. ThanksMahendra

Hey everyone. I'm in the process of migrating data from a CDH 6.3.2 cluster to CDP 7.1.6.   I'm using distcp to migrate data as it seems to be impossible to perform in-place migrations from CDH 6.3.2 and CDP 7.1.6 (according to cloudera documentation -- https://docs.cloudera.com/cdp-private-cloud/latest/upgrade/topics/cdpdc-upgrade-paths.html).     I migrated all the resources that I need from HDFS but now I would like to also migrate the yarn application logs (the ones that we can access in resource manager UI or using yarn logs command).  I tried to use distcp to copy the logs from the old cluster to the new, to the directory that is configurable using yarn-site.xml. However, the migrated logs don't appear in resource manager UI neither using yarn logs -applicationId <app-id>. There is any way to make the logs from the old cluster available in the new cluster and accessible via resource manager or using the yarn logs command?   Thanks in advance. C

Hi Everybody,I'm attempting to download and use for evaluation CDH 6.3.2.Gathering information I see that is possible to use it without any license using Cloudera express edition. The point is that if I attempt to downloade it I need username and password that I need to generate after a valid subscription is performed.  Can somebody confirm me how to proceed? is it possible that to download a license free version I need anyway to contact the sales Team. kind Regards

Hi Everyone, I collect data files with NiFi using ListSFTP AND FetchSFTP processors. In these data files I have several different lines. I want to know if there is a processor that can allow me to count the number of lines in the files as I collect them to have a total number of lines. Someone can help me.

What is the difference between using the PublishKafka processor versus the PublishKafkaRecord processor when sending data from Apache NiFi to Kafka?

Hello,I am trying to migrate my Nifi registry from one server to another.Used niFi toolkit CLI for the same.Below are the commands used -On source registry server :registry list-bucketsregistry list-flowsOn target registr server:registry create-flowregistry sync-flow-versionsFlows with versions got migrated properly, but one issues noticed is the version date on target server.Instead of actual date of version it shows migrated date for all versions in target server after migration.Find the attached screnn shot for ref.Is there any way to fix this? ThanksMahendra

Hi, I am extracting image width and image height using ExtractImageMetadata processor and comparing the Image Width and Image Height with threshold value. If it is greater than the threshold value than I will resize the image. Attribute names, Image Width and Image Height is different for different image formats.   e.g. For .png PNG-IHDR.Image HeightPNG-IHDR.Image Width   For .jpg JPEG.Image Height  JPEG.Image Width I want to have a regular expression something like below. So that irrespective of image format, I will be able to compare the Image Width and Image Height. ${"*Image Width":ge(1000):and(${"*Image Height":ge(1000)})}   Above regular expression is not working as expected. Any input will be helpful.   Thanks and Regards, Biswa    

Hello,Receiving from an embedded micro over Ethernet a JSON file of the form:{"id" : 123456,"ip" : "*","t" : -12.9,"T" : -23.8,"variables": ["user1",0,12.97,23]}and knowing what each variable represents in the "variables" array as:"username" : "user1","active" : 0,"temperature" : 12.97,"age" : 23How can I update the JSON file with the key name of the variables such that the end result is:{"id" : 123456,"ip" : "*","t" : -12.9,"T" : -23.8,"variables": ["username" : "user1","active" : 0,"temperature" : 12.97,"age" : 23]}and then be able to access these variables with their name?Thank you.

Hi, I am trying to create query with attribute value in getMongo. My attribute is itemType, i want to create query using this. {"itemType": ${itemType}} if i put hard code value there instead of ${itemType} it works

How does it scale? If I have 100 k templates, how will Flow XML load in memory? How does it manage individual processors in memory?