Q&A

Apache NiFi 1.25.0I'm trying to connect SQL Server 2008 R2 using ConvertJSONToSQL processor but got this error :Using this driver :jar : mssql-jdbc-12.6.0.jre11.jarclass : com.microsoft.sqlserver.jdbc.SQLServerDriverI have no problem connect SQL Server 2008 R2 using SSMS. Am I missing some configuration? I did using default https configuration on nifi.properties provided by Apache NiFi when installed.

Hello Team, I have an issue with setting the Knox authentication with PAM. I have the default login in /etc/pam.d/   $ cat /etc/pam.d/login#%PAM-1.0auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.soauth substack system-authauth include postloginaccount required pam_nologin.soaccount include system-authpassword include system-auth# pam_selinux.so close should be the first session rulesession required pam_selinux.so closesession required pam_loginuid.sosession optional pam_console.so# pam_selinux.so open should only be followed by sessions to be executed in the user contextsession required pam_selinux.so opensession required pam_namespace.sosession optional pam_keyinit.so force revokesession include system-authsession include postlogin-session optional pam_ck_connector.so  Knox-sso looks as following (the default one) 

Good evening everyone, I have a problem on ranger, the users have access to the databases and everything works, but on the ranger web ui if I try to search for users belonging to the groups the following screen is shown. Even if checking in Unix the group contains several users:I saw that there is a KB for the same problem after upgrade but I already upgraded to 7.1.7sp1. Can anyone help me? Thank you

I have a parquet table named 'cats' holding 2 columns (day STRING, number INT). Statistics are generated both for the table and the 2 columns (e.g. max, min, distinct can be seen in DESCRIBE FORMATTED default.cats number).Still, when running an explain command for a select max, it looks like the statistics aren't being used. Why is this? How can Hive be configured to use the stats?All the following settings are true:set hive.cbo.enable;set hive.stats.autogather;set hive.compute.query.using.stats;set hive.stats.fetch.column.stats;set hive.stats.fetch.partition.stats;explain select max(number) from default.cats; 

Hi All, I would like to replace first element of an array value  i.e would like to replace with first net val with nifi attribute input{"iH": [{"DN": "711449","iLineList": [{"DNnumber": "711449","DNLineNumber": 2},{"DNNumber": "711449","DNLineNumber": 3}],"netval": "22.09"},{"DN": "711450","iLineList": [{"DNnumber": "711450","DNLineNumber": 2},{"DNNumber": "711450","DNLineNumber": 3}],"netval": "11.09"}]}defined SPEC like :[{"operation": "shift","spec": {"*": {"0": {"netVal": "iH[&1].netVal[0]","*": "iH[&1].&"},"*": "&1"}}},{"operation": "modify-overwrite-beta","spec": {"*": {// "invoiceHeader": {"0": {"netVal": "${netVal}"}// }}}}]Expected output:{"iH" : [ {"DN" : "711449","iLineList" : [ {"DNnumber" : "711449","DNLineNumber" : 2}, {"DNNumber" : "711449","DNLineNumber" : 3} ],"netval" : "${netval}"}, {"DN" : "711450","iLineList" : [ {"DNnumber" : "711450","DNLineNumber" : 2}, {"DNNumber" : "711450","DNLineNumber" : 3} ],"netval" : "11.09"} ]} 

Hi everybody,Testing CDP 7.1.9, looks like we hit a bug in Hive . The "UNBOUNDED FOLLOWING" actually behaves as "CURENT ROW". But only when the other side of the window is "UNBOUNDED PRECEEDING". Did someone hit this one ?If we change the PRECEEDING, it's okThanks in advance

>>[21/Feb/2024 16:06:55 +0800] 2712551 MainThread agent ERROR Heartbeating to master:7182 failedI am getting this error, so I cant continue installation.I am using RHEL 8 btw.

Hi Team, We have a usescase wherein we want to import the templates directly into nifi canvas using the rest apis provided by nifi we are hitting with the below error. If anyone faced this issue previously please assisst.we are able to upload, download but import is not working To get the tokenbearer=$(curl 'https://nifi-host:nifi-port/nifi-api/access/token' --data 'username=username&password=password' --compressed --insecure)To import the template curl -i -X -k POST -H "Content-Type: application/xml" -H "Authorization: Bearer $bearer" -F "template=@/root/tmp/template.xml" https://nifi-host:nifi-port/nifi-api/process-groups/processor-group-id/templates/import Nifi version - 1.8.0.3.3.1.0-10error - HTTP 400 Bad RequestThanks 

Our team is working on a client solution involving Cloudera, and we need to develop a Proof of Concept (POC). Is there a way to access Cloudera for free to set up a development environment? Specifically, we need HDFS and Impala integration; later, we will connect Impala to Tableau for visualization. Any guidance on accessing these services without cost would be greatly appreciated.

Hi all,I tried to connect to Impala in Windows Server 2012 through ODBC v2.7.0.1002 and get error FAILED![Cloudera][DriverSupport] (1170) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct.This is log error from odbcERROR 4196 Simba::ThriftExtension::TETSaslClientTransport::SaslLoggingCallback: ::LogUsername: GetUsernameEx(NameUserPrincipal) failed: 1332But I tried in my local laptop with same settings and ODBC version it can connect.I check from windows server can telnet to Impala IP and port.Is there any certificates that need to install in windows server? when I check in certificates console my laptop got much certificate than serverMy laptopWindows server that can't connect to ImpalaPlease suggest.  

Hello,I am a beginner starting to learn Kafka and Cloudera. I am running a simple java code for producer,and am using Cloudera SMM through Docker Desktop. On running my java code,I keep getting this error-"Disconnecting from node 1001 due to socket connection setup timeout". There are no issues in my code or pom. I am using Java Maven in VSCode. Please help me to fix this issue

It was observed that our application generates an error message that includes sensitive information about its environment, files, or associated data such as internal system path.is there a way to put a custome error page in nifiBanner Disclosureit was observed that the application is disclosing information about internal components details through response.

Hi. I want to change the LDAP user  as currently we are using a Domain Admin account. We want to use the service account for LDAP configuration. I  have found below properties in hdfs configuration to change.hadoop.security.group.mapping.ldap.bind.userhadoop.security.group.mapping.ldap.bind.passwordhadoop.security.group.mapping.ldap.basehadoop.security.group.mapping.ldap.search.attr.group.nameCould you please guide me how can I change these and is there any document for this?I am using cloudera 5.10 version.

I am trying build a security work around Cloudera platform for log4j. I know for log4j cloudera has different jars built. But question is are these present only in /opt/cloudera/cm and /opt/cloudera/parcels//CDH-7.1.7-1.cdh7.1.7.p1000.24102687/, or it can be anywhere under /opt/clouderaIs /opt/cloudera/ default directory installation path for any cloudera product?1.2.17-cloudera6 jar only cdh7 or they are applicable to cdh7 and below

How we can use  AttributesToJSON process to reformat the n1,n2,n3 parameters to json string as below ?{  "n1": 20,  "n2": {    "n3": "1400000474"  }}      we try to do that but the output of json string coming with \" , we want to pass the output of AttributesToJSON process to postHttp process{"n1" : "20""n2" : "{\"n3\":\"8001001922\"}",}

when try to create topic after enable TLS kafka-topics --create --topic demo --bootstrap-server 192.168.1.235:9093 --partitions 3 --replication-factor 3 this command give me this error  Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1709199819559) timed out at 9223372036854775807 after 1 attempt(s)24/02/29 11:42:40 INFO internals.AdminMetadataManager: [AdminClient clientId=adminclient-1] Metadata update failedorg.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1709199789530) timed out at 9223372036854775807 after 1 attempt(s)Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.24/02/29 11:42:40 ERROR admin.TopicCommand$: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1709199819559) timed out at 9223372036854775807 after 1 attempt(s)at

Hi there,I am experiencing really slow ingestion with putDatabaseRecord process.The flow is simple - query data from mySQL 5.6 and ingesting it into redshift.Currently im just testing it so I list only specific tables from those 2 databases and then I query 'select * from $db.tablename' - but the ingestion using putDatabaseRecord is terribly slow (100 rows 5min).I am using AvroReader inside putDatabaseRecord but didnt make any changes to the avroreader - I just make sure flow files be small (100 rows).Does AvroReader split the 100 rows into 100 SQL statements?How do I fix this to have better performance? 

Following a rolling restart of the cluster, one CDSW Docker Daemon is stuck in "busy" state. I can't start or stop it. I checked in the server and docker or any other related process are not running. Due to the fact that this Docker Daemon runs on the server where the master and application services run, I cant start docker and k8, so the whole CDSW application is down. When I try to start all the services on that server: CDSW Docker Daemon, Docker master, Docker Application at the same time, I see in the logs that the Daemon starts but very soon received a signal to be terminated. This is something that I cant explain. Again, I checked for any background process which could be responsible for this but didn't find any. Most probably, the fact that the Daemon service is stuck in the "busy" state creates this problem.

I'm looking to administer HDFS and Hive replication via API. The replication jobs have been set up. My question is, can these replication jobs be viewed in the Cloudera Manager UI under Replication Policies?

Hello experts,We are using Nifi for data ingestion via http - using 'HandleHttpRequest' processor.Data is getting ingested continuously but once in a while when incoming volume is little high we get 'Service Unavailable - 503' error from nifi end point.Below is the log we see in nifi app logs -  2024-03-04 11:20:25,558 WARN [StandardHttpContextMap-c2ffb1fd-2c09-3a90-0e5e-18384499605f] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=c2ffb1fd-2c09-3a90-0e5e-18384499605f] Request from 10.13.2.204 timed out; responding with SERVICE_UNAVAILABLE Imp configs :[HandleHttpRequest processor]Container Queue Size : 2000   [StandardHttpContextMap controller service]Maximum Outstanding Requests : 8000Request Expiration : 3min nifi.web.jetty.threads=200 (in nifi.properties)Maximum Timer Driven Thread Count = 200 Total number of running processors on this env : 4200+This nifi is running on 8 core machine (ec2 instance) When we look at the CPU

Where are all the cache entries for the DistributedMapCacheServer stored in the NiFi file system, what is the maximum number of entries it can hold, and if I want to increase the default value from 10,000 to 100,000, will this require any infrastructure changes on the server or file system?

Hi all, I am practicing Kafak on my CDP 7.1.8 with Kerberos enabled. I can create topics under Kerberos authentication. However, when I test producing and consuming message, the consumer side never receive a message. Here are some screenshot: Consumer: kafka-console-consumer --bootstrap-server host2.my.cloudera.lab:9092 --topic topic001 --from-beginning --cons umer.config /root/kafka/krb-client.properties23/04/03 04:37:00 INFO utils.Log4jControllerRegistration$: [main]: Registered kafka:type=kafka.Log4jController MBean23/04/03 04:37:01 INFO consumer.ConsumerConfig: [main]: ConsumerConfig values: allow.auto.create.topics = true auto.commit.interval.ms = 5000 auto.offset.reset = earliest bootstrap.servers = [host2.my.cloudera.lab:9092] check.crcs = true client.dns.lookup = use_all_dns_ips client.id = console-consumer client.rack =

I'm new to Apache Nifi Community and I got stuck at one point - please help me for the below problem :In 'Validate CSV' processor I'm giving below schema(schema_used) to validate the CSV input. But, processor is showing error(err_screenshot).Validate CSV -Schema_Used - StrNotNullOrEmpty(), ParseDate("MM/dd/yyyy"), StrNotNullOrEmpty(), StrNotNullOrEmpty(), Optional(), StrNotNullOrEmpty(), StrNotNullOrEmpty(), StrNotNullOrEmpty(), StrNotNullOrEmpty(), Optional(), Optional(), StrNotNullOrEmpty()The above schema is showing below error -err_screenshot -Note: The Schema is working totally fine, if I remove 'Optional()'.Problem - Please tell me how can I use 'Validate CSV' processor in Apache Nifi and validate Mandatory/Optional Fields of CSV. Also, suggest/share any script(groovy/python) if that can validate the CSV.

Hello experts, We have setup our nifi to consume message from kafka broker which has SSL as security protocol. It worked well with pre-prod brokers, but facing below error in prod broker.(we have configured ssl context service with respective ssl certs) Preprod nifi <-> Preprod kafka : workingProd nifi <-> Prod kafka : Not  working telnet is working from Prod nifi to Prod kafka broker. 2023-07-17 15:30:58,814 WARN [Timer-Driven Process Thread-190] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdOnHandR3NiFiConsumer-28, groupId=ProdOnHandR3NiFiConsumer] Connection to node -3 (kafka3.xxx.pivotal.emea.kaas.3stripes.net/xxx.128.xx.150:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.2023-07-

I have a simple question, Is CDP (CDH 7.x) open source?   when I try to download the cm7 from this URL https://archive.cloudera.com/p/cm7/ Its ask for password.   I am following this URL https://docs.cloudera.com/cdpdc/7.0/installation/topics/cdpdc-configure-repository.html   Thanks