Q&A

Hi all, I'm a newbie. I want to read a local .csv file that contains only an ID column, then use that list of ID in a dynamic query to get data from an oracle table. My flow is as below:I configure processors as following:+ReplaceText(1): is to replace "\n" with ","+ ReplaceText(2): Remove the last empty row+ ExtractText: assign ID list to attribute named list_id+UpdateAttribute: to specify name and value of property. The list contains 10 ID+ ExecuteSQL:There is no task In for UpdateAttribute, the flow stop at ExtractText, the detail tab in view data governance is below, it said File.Size is 0 bytes. However it doesn't return any error so I don't know where to fix. Could you please help.Thank you 

All, I created a new processor using python3.12 in NiFi 2.0.0 M4 release of NiFi.  My directory structure looks like this based on the documentation - I don't include anything under bundled dependencies and my MANIFEST.MF looks like this.Manifest-Version: 1.0Build-Timestamp: 2024-10-07T16:22:20ZNar-Id: processors-narNar-Group: processorsNar-Version: 0.0.2I zipped everything and created a .nar called TransformOpenskyStates.nar. Once created, I added it to my NiFi_home/extensions directory.After looking in the logs, I see this - 2024-10-07 13:44:34,481 INFO [NAR Auto-Loader] org.apache.nifi.nar.NarAutoLoaderTask Found ./extensions/TransformOpenskyStates.nar in auto-load directory2024-10-07 13:44:39,487 INFO [NAR Auto-Loader] org.apache.nifi.nar.StandardNarLoader Starting load process for 1 NARs...2024-10-07 13:44:39,494 INFO [NAR Auto-Loader] org.apache.nifi.nar.StandardNarLoader Creating class loaders for 1 NARs...2024-10-07 13:44:39,496 INFO [NAR Auto-Loader] o

Hi Experts,Please let us know if we can use any NiFi processors for truncating hive table data. Thanks!

When we try to enable the user as Admin we get the following message:Modifying user role only allowed when local authentication is enabled.BRMichele

There are two different examples of SQL in SQL Server1. DECLARE @Max_date_id varchar(10)SELECT @Max_date_id = MAX(date_id) FROM t_my_tableSELECT @Max_date_id2. DECLARE @Max_date_id varchar(10) = (SELECT MAX(date_id) FROM t_my_table)SELECT @Max_date_idHow can I do the same thing in "Impala Query Manager"?   

I made a docker compose to deploy nifi in cluster on my server.I get the error Authentication Proxy Server not trustedHere are my configurationsdocker-composeversion: '3.8'x-common-nifi: environment: &nifi-env NIFI_WEB_HTTPS_PORT: 8443 NIFI_CLUSTER_IS_NODE: "true" NIFI_CLUSTER_NODE_PROTOCOL_PORT: 11444 NIFI_ZK_CONNECT_STRING: zookeeper:2181 NIFI_ELECTION_MAX_WAIT: 1 min NIFI_ZK_ROOT_NODE: /nifi NIFI_SENSITIVE_PROPS_KEY: ${KEY_SENSITIVE} NIFI_VARIABLE_REGISTRY_PROPERTIES: http://nifi-registry:18080 AUTH: ldap INITIAL_ADMIN_IDENTITY: 'uid=admin,ou=admins,dc=myhost,dc=dev' #LDAP_AUTHENTICATION_STRATEGY: 'LDAPS' LDAP_AUTHENTICATION_STRATEGY: 'SIMPLE' LDAP_MANAGER_DN: 'cn=admin,dc=myhost,dc=dev' LDAP_MANAGER_PASSWORD: ${ADMIN_PASSWORD} LDAP_URL: 'ldap://openldap' LDAP_USER_SEARCH_BASE: 'dc=myhost,dc=dev' LDAP_USER_SEARCH_FILTER: 'uid={0}' LDAP_IDENTITY_STRATEGY: 'USE_DN' KEYSTORE_TYPE: JKS KEYSTORE_PASSWORD: ${KEY_STORE} KE

HI @MattWho We installed the nifi in cluster of two nodes in Azure vm, But we unable to get the below users icon in the NIFI Page which was attached. i read few articles related to these user setup and found that we need to use LDAP process to get the user screen,  currently it's for testing purpose and we don't have idea on LDAP thing, can you please guide us what we need to change in the NIFI .properties page to get the user icons. Attaching our nifi properties as well for your reference, Request you to please check and update me.   # Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http:/

Hi team,Thanks for your time in advance.I have an issue bothering me a long time. Can somebody point me the right direction?I created a hbase instance in my local - no distribution, with phoenix coprocessors, then go to phoenix, run a sample DDL and upsert. For a simply select * from thetable, It shows the ids but null values for all of non-id columns. Settings:1: hbase-2.6.0-hadoop3: I copied phoenix-server-hbase-2.6.jar to hbase_home/lib; then add coprocessors to hbase-site.xml:  <property>    <name>hbase.coprocessor.region.classes</name>    <value>      org.apache.phoenix.index.GlobalIndexChecker,      org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver,      org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver,      org.apache.phoenix.coprocessor.ScanRegionObserver,        org.apache.phoenix.coprocessor.ServerCachingEndpoint

Hello everyone anybody setup garfana for cdp private cloud Need to stup any refrence docs for how to setup grafana Thanks

Hello everyone, My name is Emmanuel Katto. Can anyone share their experiences with Impala for querying large datasets in Cloudera? What are some tips for optimizing query performance?Looking forward to your tips/suggestions.Thanks in advance!Best,Emmanuel Katto

For a hive2 cluster, I do not have to escape a double quote inside a  regexp_replace  where the  single quotes to start and end the regex.But on hive3 in order for the code to work, the double quote needs to be escaped (Example to follow).Why the difference?  -- Works in Hive 2 but not hive3select id , case when url rlike '(?i)^https://www.linkedin.com/("?in|pub|company|profile)' then url else regexp_replace(url,'^https://www.linkedin.com/','https://www.linkedin.com/in/') end as social_urlfrom db.tablelimit 10-- Works in hive3select id , case when url rlike '(?i)^https://www.linkedin.com/(\"?in|pub|company|profile)' then url else regexp_replace(url,'^https://www.linkedin.com/','https://www.linkedin.com/in/') end as social_urlfrom db.tablelimit 10  

File "ooziepython.mod/oozie/action/OozieAction.py", line 47, in main    status = self.run(args)  File "StreamingAction.py", line 107, in run    success = subprocess.check_call(command.split())  File "/opt/hadoop3/yarn/local/filecache/178/jython-standalone-2.7.3.jar/Lib/subprocess$py.class", line 548, in check_call    retcode = call(*popenargs, **kwargs)  File "/opt/hadoop3/yarn/local/filecache/178/jython-standalone-2.7.3.jar/Lib/subprocess$py.class", line 535, in call    return Popen(*popenargs, **kwargs).wait()  File "/opt/hadoop3/yarn/local/filecache/178/jython-standalone-2.7.3.jar/Lib/subprocess$py.class", line 892, in __init__    self._execute_child(args, executable, preexec_fn, close_fds,  File "/opt/hadoop3/yarn/local/filecache/178/jython-standalone-2.7.3.jar/Lib/subprocess$py.class", line 1402, in _execute_child    raise OSError(errno.ENOENT, os.strerror(errno.ENOEN

getting this error while starting id can't update anythind in adding snn or remove nn same shows anyone help me

Hi Team,Currently i am using 5 node cluster of NIFI -  1.27 version.node have 47 Gb RAMMy NIFI heap memory level keep on increasing and crashing by reaching 150 hours.Frequency of data is around 100 Gb in a day.I am using variety of  processors (Kafka, split and merge , distributed cache processors,Attribute level processors, hive3streaming processor,(select and put)Execute SQL (postgre,SSIS),Kudu processor etc).Please find  nifi properties below:# JVM memory settingsjava.arg.2=-Xms16gjava.arg.3=-Xmx20gjava.arg.7=-XX:ReservedCodeCacheSize=512mjava.arg.9=-XX:+UseCodeCacheFlushingjava.arg.23=-XX:ParallelGCThreads=8java.arg.24=-XX:ConcGCThreads=4java.arg.25=-XX:G1ReservePercent=10java.arg.26=-XX:+UseStringDeduplicationjava.arg.27=-XX:InitiatingHeapOccupancyPercent=25java.arg.28=-XX:MaxGCPauseMillis=200java.arg.40=-XX:SurvivorRatio=8java.arg.41=-XX:NewRatio=3java.arg.42=-Xmn6gjava.arg.13=-XX:+UseG1GCnifi.content.repository.archive.max.retention.period=7 hoursnifi.content.rep

Eg: Update table table_name emp_name=? where empid=?   But My column name is emp_id

hi alli have an CDP and CDS Private Cloud,so i trying to explore more about Data Catalog for Sensitivity Profiler.i try to make an custom rule.but the rules still not working.did you guys have try to make some custome rules ?and how its works ?i need some reference thanks

Hello Dear all,I am encountering an issue with an Oozie workflow that involves moving data from an external table to HDFS. The Oozie job successfully completes the process of bringing data from the external table to HDFS. After that, Sqoop should transfer this data into an ORC table.The issue occurs at a specific point in the Oozie workflow, where it tries to read two files from an HDFS directory: one containing the YARN application_id and the other with the instructions for database insertion. The job fails, and the directory remains empty. In the logs, I get the following error:org.apache.hadoop.ipc.Client - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby.Context:This error began after an incident where directories in HDFS were deleted. This may have affected the failover process and high availability (HA) of HDFS.The error seems to be relate

Im on CM 7.10.1, cloudera runtime 7.1.8. I am trying to enable auto tls after creating the cluster, after i input ssh credentials (which are correct), I am getting error "java.lang.IllegalStateException: Failed to run CMCA command, return code: 2" More info: I have enabled kerberos but have not enabled auto tls yet Could someone please guide me on how to troubleshoot this issue?

Hi,  Has anyone come across such a problem when enabling Auto-TLS?Thanks appreciate your comments  java.lang.IllegalStateException: Failed to run CMCA command, return code: 2 at com.cloudera.cmf.command.CertmanagerRunner.runCmcaCommand(CertmanagerRunner.java:55) at com.cloudera.cmf.command.GenerateCmcaCmdWork.runSetupCmcaCommand(GenerateCmcaCmdWork.java:65) at com.cloudera.cmf.command.GenerateCmcaCmdWork.doWork(GenerateCmcaCmdWork.java:202) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:178) at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:178) at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117) at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:107) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeGlobalCommandHelper(ServiceHandlerRegistry.java:865) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeGlobalCommand(Ser

I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files. Is there any alternate flow I can build to avoid such backlogs?Can I completely avoid fork record processor and choose some other flow logic to solve this ? @mburgess @MattWho   Please help here

HelloI am struggling to set the time zone for the sessions on Spark Sql while submitting the query over the HUE.Maybe somebody already has this problem and know how to solve it - I would be grateful for the help. So far I checked that:on pyspak session running on linux the time zone is the one i want :spark.conf.get("spark.sql.session.timeZone")'Europe/Warsaw'However then i submit the TIMESTAMP on HUE running query against SparkSql I got the UTC Time Stamp.SELECT TIMESTAMP 'now';    TIMESTAMP '2024-07-02 08:01:06.157849' My current time  12024-07-02T06:01:06Z the UTC TIME I have also added the variable to the spark-defaults.conf file like spark.sql.session.timeZone=Europe/WarsawBut after restarting Spark3 and Hue nothing has changed. Thank you for your help in advance.Regards,Bart 

Hi! Is there a way to create an Application on Cloudera Cloudera Data Visualization and expose it to users that don't have credentials to login? What I intend to do is to create an app with dashboards that can be accessed in a read-only way without the need to login, because this users won't have credentials to login. Thanks!

Hi folks, I tried to submit a query with params via ODBC to Impala. Thi is my code:   import pyodbc# connectcrsr = pyodbc.connect('DSN=impala', autocommit=True).cursor()param1 = 1000000000param2 = '2023-01-01'# long queryquery = """select *from target_test.test d left outer join (select cli, fsc, nom_rag_soc, dat_ini_val, dat_ccz_inl, dat_csz_efv, dta_cessata, sta, prg_var, cod_cau_csz_dl, rn from (select c2.cod_cli as cli, trim(c2.cod_fsc) as fsc, trim(c2.nom_rag_soc) as nom_rag_soc, c2.dat_ini_val, dat_ccz_inl, dat_csz_efv, (case when c2.dat_csz_efv= '1900-01-01 00:00:00' then '2999-01-01 00:00:00' else c2.dat_csz_efv end) as dta_cessata, sta, prg_var, cod_cau_csz_dl, row_number() over (partition by cod_fsc order by (case when c2.dat_csz_efv='1900-01-01 00:00:00' then '2999-01-01 00:00:00' else c2.dat_csz_efv end) desc) as rn from target_test.test2 c2 where c2.sta='c' and c2.dat_

Hi,I want pass from csv file to InvokeHTTP processor for changing begin and end timestamp of URL.This is my flow and how to do update this flow?

Hi all,I need a help 😉I am trying to install and launch Apache Hive meta store with PG database  on Kubernetes.The version of Hive is : apache-hive-4.0.0-bin.I configured hive-site.xml and all volume needed for Apache Hive.The init scheme was done succsufly by executing : ./schematool -dbType postgres -initSchemaSince, when try to launch hive, I obtained this error that nput name is null. @Former Member-hive-v2-78d588-gf8rv:/opt/apache-hive-bin/bin$ hiveWARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/apache-hive-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.loggin