Q&A

Hi All,I have created a profile with organization email but i have selected the wrong company. Now I would like to change the company.Any help would be appreciated. Thanks,Rajan

I am new to the Cloudera CDP and got stuck at the CMS level 

My previous processor group is for update and insert a table. After that, I want to use ExecuteSQL processor.My question is: How to clear all flowfiles and proceed to the ExecuteSQL processor, so the ExecuteSQL processor only run once.For example, my pic above, there are 3 flowfiles queued, it means that the ExecuteSQL processor will run 3 times? (I think). how to clear them and only run once

As an admin of a CDH cluster , some query has submitted  to the hiveserver2 ,  but the query still  in the hiveserver make the explain stage , which does not submit to the yarn cluster and has no application id on yarn . And the query make the hiveserver2 wrong  which  has a too long query such as select * from aaa where code in ('xxx','xxx1','xxx3',.......'xxx2000000') ;A sql has more than milion row  may make the hiveserver2 corrupt .  The 10002 web page  seems does not have some action button like yarn web the deal with the query . The cdh version is 6.3.2 . hive version 2.1.1 .When this situation occur , I have to restart the hiveserver2 .I want to know if there is some way to kill a query through hive queryid or hive sessionid o instead of yarn applicationid . It is also usefull when someone query the metastore which multithread such as "use hadoop " sql with 30 or more active connect  ,  The adminastritor

Greetings,I'm currently grabbing data from a website, "splitting" it into its separate components and trying to route them so I can save them into a single zipped file.My workflow currently works as follows:InvokeHTTP --> SplitText --> RouteText ...The original Dataset is a bunch of satellite data (example below).  I’m then splitting it into separate 3 line records. I then run RouteText and try and grab specific records.  For example, if I wanted TDRS 3 and AMSC 1 and SKYNET 4C, how do I pull them out with its respected 2 lines?  I tried adding a Property:  SatelliteData1 = ${literal('TDRS 3')}.  This only gives me the first line of the Split record (I need all 3).  I tried OR-Statement, but don't think I'm using it right. ${literal('TDRS 3''):or(${literal('AMSC 1')}):or(${literal('SKYNET 4C')})}Any suggestions will be greatly appreciated.  Thanks in advance.Original Data:Splitting:  

Hi, I have a Hadoop 3.1.1 cluster and recently I found that the NameNode heap memory usage in the cluster is very high. I saw 629973631 file objects in the cluster through WebUi, so according to my calculations, it should occupy no more than 90GB of memory, right? Why is the current memory usage consistently above 140GB? Is this related to me enabling erasure codes? 

Hi All,I am facing the issue of adding the hue service in CDP 7.1.9, after using the correct DB credentials, the DB test is failing for the hue service only.Facing the following issues, can anyone suggest any solution?2024-01-17 11:15:08,066 INFO scm-web-5531:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/testConnection2024-01-17 11:15:08,067 INFO scm-web-5531:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HueTestDatabaseConnection with sensitive arguments. Host: DbHost{id=1546335515, hostId=41e041ff-13f3-47c0-b9a3-8b8e23e8cea7, hostName=NVMBDPRP005704CDPNN1}2024-01-17 11:15:08,172 INFO scm-web-5531:com.cloudera.cmf.service.ServiceHandlerRegistry: Host command HueTestDatabaseConnection launched with id=15463512302024-01-17 11:15:08,180 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:15463512302024-01-17 11:15:08,182 INFO scm-web-5531:com.cloudera.enterprise.Java

Hello everyone,in the execution of a Spark script, which invokes several applications, we have different timings in each of the runs performed, in fact it went from about 8h on 18/11/2023, to about 17h on 09/12/2023 to about 33h on 02/12/2023.Since this is a very heavy process, all daily schedules were disabled to give the application a chance to allocate all resources.The application is run sporadically during weekends, and we do not know exactly when the next run will be.We have noticed that when running Spark applications that are part of the script, a delay of about 3 to 4 hours is encountered between the execution of one job and its next.Do you have any suggestions regarding a possible cause of the anomaly encountered? Thank you

Hello Community,I am getting below error on my HBase table. The table has been fine for close to a year until 2 days ago. What's the cause?org.apache.nifi.hbase.PutHBaseRecord PutHBaseRecord[id=ee1b356c-aa58-13f5-ac63-84a796230216] Failed to put records to HBase.org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested row out of range for batchMutate on HRegion USAGE:DATA_TRANSPARENCY,a11ced068cb301a6c6c502932d686a05_20200621142500_189290_366,1676533563461.bbbebf59c6e4e9750e9f5825850d6f4f., startKey='a11ced068cb301a6c6c502932d686a05_20200621142500_189290_366', getEndKey()='a11f2100d30ecc142b4c0466c01797ad_20230202125000_129718_339', row='a1200e570c1a24fee95b1004dbff50eb_20240108193317_2079_8316'at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1041)at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)

Hi, I have email archived in .MSG format files in folder location which is mentioned in SQL database table.I need to get the file from folder location , read the file , check if there is any attachment in msg file , if yes then need to extract that attachment and make the entry of that file into database tableI am new to NIFI , So anyone can help me with some sample flow for this requirement? Thanks In advance , Prats 

As seen from the image above. My requirement is for the "ExecuteSQL" processor to run only once after all the queued files are processed via "PutDatabaseRecord" processor.Right now it will run 190+57 times but I want to run after all the queued files are processed through. Is there any way to do this in Nifi ?

Hello Everyone,We are facing below issue in our CFM.Background: We have NiFi Flow which uses GetHDFS processor to fetch all the HDFS files and performs few transformation and put the data back to HDFS using putHDFS processor. The CRON used for this was: 0 0/45 06-19,21-05 1/1 * ? *Now: Now based on the requirement we have changed the CRON schedule to "0 30 10-19 1/1 * ? *" (0 30 10-19 * * ? *) , the getHDFS processor runs at CRON schedule and also process the data on the day of change. But from next day/tomorrow onwards the processor is running/threads are intiated on all nodes based on CRON schedule however it does not process the data and gives a below warning.Warning: Error while retrieving list of files due to DestHost:desport <master_hostname>:<port>     LocalHost:localport <worker_node>:<port>Failed on Local Exception: java.io.IOException: Couldn't setup connection for <nifi_user>@principal to <master_hostname> Doe

Hello,it's not clear to me how users are managed in CDP Public Cloud / Machine Learning / Cloudera Cloudera Data Visualization.It's ok I can create a new user from the the DataViz menu, but in this case, how are they linked to users that I manage from Management Console / User Management in Control Pane?In DataViz Users & Groups, do I have to create a new user with same workload username and password assigned to the user in the User Management section?Are these users synchronized somehow?I'm asking because once a user has been created in DataViz, I am not able to change its password, so I'm wondering if somehow they are synchronized and can be managed from the User Management section in Management Console.Thank you,Andrea 

Hi all,My nifi crash for some reasons that I could not figure out, can anyone helps?1. A chain of lookups that I have, the table being looked up contains around 50,000,000 rows2. After running some node, it crashed like this:3. After this, I relogged into NiFi to see what error happened:My Lookup service setup:I can provide more details upon request.Please help!!!Thanks  

Hello about log4j2 failure (CVE-2021-44228) in cloudera there is the KB: https://my.cloudera.com/knowledge/Resolution-for-TSB-2021-545---Critical-vulnerability-in-log4j2?id=332012 instructs to run the script to remove a class in the jar etc. our doubt is, do we need to run this script in any version of CM / CDH or just the version from 6.x.x?

Hi, I just started using Nifi and i would like to know if it's possible to perform a POST with a content-type set to `x-www-form-urlencoded`. I need to pass these key/values to my request :  grant_type refresh_tokenclient_id myClientIdclient_secret myClientSecretrefresh_token myRefreshToken  

My environment is testing on Windows 10. I installed version 1.14 and started. The chrome 91.0.4472.124 says it is not secure and I confirm to go to the site "https://localhost:8443/" not secure. The address changes to "https://localhost:8443/nifi/login" and asks for username and password. What username and password should I use? Where can I generate this username and password?

Hi: From google : If you are using NiFi 1.0 you can use the GenerateTableFetch processor. It allows you to choose the "page" (aka partition) size, and will generate SQL statements, each of which will grab one "page" of data. You can route those into ExecuteSQL and it will retrieve smaller sets of results at a time. If you have a NiFi cluster, you can route GenerateTableFetch into a Remote Process Group that points at an Input Port on the same cluster, (which will distribute the SQL statements across the cluster), then the Input Port can be connected to the ExecuteSQL. This allows you to fetch rows from a table in parallel I have three nodes on my nifi cluster, I follow the post, put GenerateTableFetch on primary node(test01) to execute, and then send to remote process group on the same cluster. then output port to ExecuteSQL. But the actual behavior is ExecuteSQL was execute just in one node (test02, or test 03, or tes01). My question is how to fetch rows from a table in parallel(test0

Hi I got a random message on telegram asking if I was available wen I asked for wot I got told I’d receive a WhatsApp message , I did from a man saying he wanted to give me details on a job anyway wen I didn’t rely he kept midering so I asked for details  he told me I needed to register see first photo wen I clicked on it and looked at the terms it says (see second photo) so I told him it looked like a scam and he told me to just register so I looked into it and joined u to find out wot is going on   

Hi All, I am loading the data into one table, once data  loaded, the store procedure processor should be executed automatically. which processor can be used for this scenarios.  do we have any options in WAIT/Notify processor's?

Hello! I am trying to install Nifi 2.0.0 M1 on Ubuntu 20.04. I downloaded Nifi from the official website and removed all HTTPS-related configurations. When I run Nifi with Python disabled, it works fine. However, when I enable Python, the following error occurs, and I'm not sure how to fix it. 😟 Does anyone have any knowledge or experience with this? (Python 3.10 is installed.) 

When upgrading our docker container from 1.19 to 1.22 with the same nifi.properties and flow.xml.gz and repo's we get the following error. Also the same if we tried to migrate to 1.23. Any one any idea's? Google and Chatgpt can't seem to help out 😅 Thank you in advanced. 2024-01-10 09:54:49,115 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFiJan 10 09:54:49 awsinstancename docker[21712]: java.lang.VerifyError: Bad return typeJan 10 09:54:49 awsinstancename docker[21712]: Exception Details:Jan 10 09:54:49 awsinstancename docker[21712]: Location:Jan 10 09:54:49 awsinstancename docker[21712]: com/fasterxml/jackson/databind/cfg/MapperBuilder.streamFactory()Lcom/fasterxml/jackson/core/TokenStreamFactory; @7: areturnJan 10 09:54:49 awsinstancename docker[21712]: Reason:Jan 10 09:54:49 awsinstancename docker[21712]: Type 'com/fasterxml/jackson/core/JsonFactory' (current frame, stack[0]) is not assignable to 'com/fasterxml/jackson/core/TokenStreamFactory' (from met

Is this the correct way of exposing a nifi property as env variable(in start.sh): prop_replace 'nifi.web.http.port' "${NIFI_WEB_HTTP_PORT:-8080}" I expect it to expose nifi.web.http.port as NIFI_WEB_HTTP_PORT, and also setting it to default value of 8080.

Hi,Im trying to connect LDAP from Nifi.I've done most of the instructions in the official documentation and read most posts on the Cloudera Community. However, I couldn't connect to my company's LDAP system. I configured login-entity-providers and authorizers for LDAP.When I start Nifi, it starts successfully and gives no error. When I try to log in with my initial admin it gives me "Unable to validate the supplied credentials. Please contact the system administrator." error. My questions are:1- When we are setting up initial admin, we are not setting password of that user. How do we log in without password?2- I'm using USE_DN for identity strategy for ldap-provider. My initial admin name and ldap user are same. So I'm trying to login with my user like: Username: CN=K015576,CN=Users,DC=tcmb,DC=gov,DC=trPassword: "my_ldap_password" What path should I follow? Is there a mistake in my configure files?I have added my conf files below: nifi.properties, login-entity-

The sqoop import action job in oozie is still running.What should I check?oozie version:5.2.1 hadoop version:3.3.4 sqoop version:1.4.7help Me....  >>> Invoking Sqoop command line now >>>2022-10-26 08:26:56,510 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.2022-10-26 08:26:56,543 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.72022-10-26 08:26:56,556 [main] WARN org.apache.sqoop.tool.BaseSqoopTool - Setting your password on the command-line is insecure. Consider using -P instead.2022-10-26 08:26:56,566 [main] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.2022-10-26 08:26:56,607 [main] INFO org.apache.sqoop.manager.MySQLManager - Preparing to use a MySQL streaming resultset.2022-10-26 08:26:56,607 [main] INFO org.apache.sqoop.tool.CodeGenTool - Beginning