Ask the Community

Hello everyone. I want to create a table to support ute-8 in an string column, however, I don´t know the value I have to pass when I create the table:create table test(id int,message string ENCODING codec)as kudu;Which are all differents values I can use to codec parameter for utf-8Thanks in advance.

Hello,We are using a Nifi cluster with 5 nodes. Each node is a machine with 48 cores and 280-300 Gi of available memory. The issue we currently have is Nifi having trouble keeping up when a high number of tasks is required, 1,000,000+ tasks. The files are eventually transferred but errors do pop up in the mean time that seem to be irrelevant and more related to the flow trying to keep up. It's a simple flow with a GetFile > UpdateAttribute > PutSFTP > LogMesssage. I have increased concurrent tasks and batch sizes but this has little to no effect. 

i am facing challenge to deploy the apache nifi on single node k8 cluster in docker desktop. Here is my nifi-config.yml and nifi-deployment.yml file  with following error message. nifi-config.yml apiVersion: v1kind: ConfigMapmetadata:  name: nifi-configdata:  nifi.properties: |    nifi.web.http.host=0.0.0.0    nifi.web.http.port=8080    # Site to Site properties (if needed)    # nifi.remote.input.host=...    # nifi.remote.input.secure=...    # nifi.remote.input.socket.port=...    # nifi.remote.input.http.enabled=...    # nifi.remote.input.http.transaction.ttl=30 sec    # HTTPS properties (if applicable)    # nifi.web.https.host=...    # nifi.web.https.port=...    # nifi.web.https.network.interface.default=...    # nifi.web.https.application.protocols=h2 http/1.1    # TLS Cipher Suites (if applicable)    # nifi.web

I configured as  blockManager.driver.port 21750  and blockManager.port as 21700 , maxRetries  as 30025/02/05 17:16:08 WARN TransportChannelHandler: Exception in connection from /172.21.2X0.XXX:59698java.io.IOException: Connection reset by peer        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)        at sun.nio.ch.IOUtil.read(IOUtil.java:192)        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)        at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254)        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)   &nb

Hi i have json input like that:{ "messageId": 1234, "markerId": "T", "dateFrom": 6436058131202690000, "dateTo": -3840351829778683400, "records": [ { "recordId": 1, "account": "152739203233" }, { "recordId": 2, "data": { "email": "jsmith@gmail.com", "firstName": "John", "lastName": "Smith" } }, { "recordId": 3, "city": "Los Angeles" }, { "recordId": 4, "idNumber": "12345" }, { "recordId": 5, "accountNumber": "55671" }, { "recordId": 6, "account": "6789189790191" }, { "recordId": 7, "city": "San Fransisco" } ]} And I would like to have output like that:[ { "messageId" : 1234, "markerId" : "T" "dateFrom" : 6436058131202690000, "dateTo" : -3840351829778683400, "recordId" : 1, "account": "152739203233"}, { "messageId" : 1234, "markerId" : "T" "dateFrom" : 6436058131202690000, "dateTo" : -3840351829778683400, "recordId" : 2, "email":

Hi Team@cravani @pvishnu I am performing POST operation on existing entity in Atlas, which does add classification to the entity.It is succeeding with 200 status where as not reflecting in Atlas. Please find below request and response json.URL : http://<host_name>:21000/api/atlas/v2/entityPost Request : {    "entity" :    {            "typeName": "hive_table",            "attributes": {              "qualifiedName": "prod_de9.test123",              "name": "test123"            },            "status": "ACTIVE",            "classifications": [            {                "typeName": "Country",                "enti

Hi,We have 3 servers VerticaServer, DataLakeNameNode, and DataStageServer. In a DataStage job, we are trying to COPY data into a Vertica table by getting it from external table in DataLake using WEBHDFS. The DataStage job on DR is a copy of production job  which is running fine on a daily basis.I have run following checks to ensure that there are no connectivity issues1. Ping is working fine between all three servers2. Telnet from VerticaServer to DataLakeNameNode on port 50070 is working3. On VerticaServer 'curl -i "http://<IP>:50070/wehdfs/v1/?op=LISTSTATUS"' is returning '200 OK'4. Webhdfs is up on DataLakeNameNodeIn DataStage job, we are getting following error when we run command 'COPY STG.PARTY FROM 'webhdfs://<IP>:50070/warehouse/tablespace/external/hive/STG/PARTY/*' DELIMETER ',' ENCLOSED BY "" NO ESCAPE ABORT ON ERROR'':"The reported error is: [SQLSTATE 58030] java.sql.SQLException: [Vertica] [VJDBC] (6776) ERROR: Failed to glob [webhdfs://<IP>:5007

I want to implement file checking with NiFi. I have a directory where files appear once a day. NiFi parses data from this file and deletes it. I would like to receive a notification if the file does not appear in the directory within 24 hours. Is there a way to do this check with NiFi?

@MattWho  Caused by: org.apache.nifi.web.client.api.WebClientServiceException: Request execution failed HTTP Method [GET] URI [http://localhost:8080/realms/master/protocol/saml/descriptor]using these settings:nifi.security.user.saml.idp.metadata.url=http://localhost:8080/realms/master/protocol/saml/descriptornifi.security.user.saml.sp.entity.id=org:apache:nifi:saml:spnifi.security.user.saml.identity.attribute.name=nifi.security.user.saml.group.attribute.name=nifi.security.user.saml.request.signing.enabled=falsenifi.security.user.saml.want.assertions.signed=truenifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256nifi.security.user.saml.authentication.expiration=12 hoursnifi.security.user.saml.single.logout.enabled=falsenifi.security.user.saml.http.client.truststore.strategy=JDKnifi.security.user.saml.http.client.connect.timeout=30 secsnifi.security.user.saml.http.client.read.timeout=30 secsFollowing instructions from here, except i am u

I run the database "ASK_SELL" with 43 millions of records with size of 122 GB in Hadoop. Its run for almost 1 hour and 40 minutes and stop it with the error message bellow.ERROR: Execute error:[Cloudera][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code:0, SQL state: null, Query: SELECT 1 -- /* keep alive */, Error message from Server: Invalid SessionHandle: SessionHandleWARNING: File deletion failed for DESTINO2.BENEF_CONC.DATA.Whats going on ?

I have been struggling with this for a week now.Here are my files:Authorizers.xml<?xml version="1.0" encoding="UTF-8" standalone="yes"?><authorizers><userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">cn=localhost,ou=ngc</property><property name="Initial User Identity 2">CN=localhost,O=ngc,L=San Diego,ST=California,C=US</property></userGroupProvider><userGroupProvider><identifier>ldap-user-group-provider</identifier><class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class><property name="Authentication Strategy">SIMPLE</property><property name="Manager DN">cn=localhost,ou=ngc</property><property n

Hello,I'm trying to upload file to S3 compatible IBM Cloud Object Storage by PutS3Object processor.It fails with the attached error only when target ICOS bucket is configured to protect object.When we upload file with s3cmd, it works.Is there any solution for this problem?Thanks, 2025-02-07 10:17:00,892 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=b4359d36-0193-1000-4303-1924a76d8f5a] Failed to put StandardFlowFileRecord[uuid=3f891ca3-d3b9-4ffa-b446-306386f637db,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1738891020007-580, container=default, section=580], offset=0, length=894211],offset=0,name=89e42cbe-172f-4882-8b9e-695b27c7a6e5,size=894211] to Amazon S3 due to com.amazonaws.services.s3.model.AmazonS3Exception: Missing required content hash for this request: Content-MD5 or x-amz-content-sha256 (Service: Amazon S3; Status Code: 400; Error Code: MissingDigest; Request ID: db2b96e4-d540-4ca9-9dd1-0605ad4b8658; S3

Team,Need help in setting up NiFi Cluster with 3 nodes, while i am trying to setup with below parameters, i am getting error. Please suggest where i am going wrong NiFi Version: nifi-2.2.0state-management.xml<cluster-provider><id>cluster-provider</id><class>org.apache.nifi.controller.state.providers.raft.RaftStateProvider</class><property name="Connect String">Host1_IPAddress:2181,Host2_IPAddress:2181,Host3_IPAddress:2181</property><property name="Root Node">/nifi</property><property name="Session Timeout">10 seconds</property><property name="Access Control">Open</property></cluster-provider>NiFi Properties:nifi.cluster.is.node=truenifi.cluster.node.address=Host1_IPAddressnifi.cluster.node.protocol.port=9999nifi.cluster.flow.election.max.candidates=3nifi.sensitive.props.key=KeyValuenifi.state.management.provider.cluster=cluster-providernifi.web.https.host=localhostnifi.web.https.port=8444Error

I have set up Apache NiFi in a Docker container and am using Nginx as a reverse proxy to handle SSL termination. However, when I try to access the NiFi UI through the custom domain configured in Nginx, I receive an "HTTP ERROR 400 Invalid SNI" message.  Below is my Docker Compose configuration: version: '3'services: nifi: build: context: . dockerfile: Dockerfile ports: - "8443:8443" volumes: - nifi-data:/opt/nifi/nifi-current nginx: image: nginx:latest volumes: - ./nginx.conf:/etc/nginx/nginx.conf - ./sslcert:/etc/nginx/sslcert ports: - "80:80" - "443:443" depends_on: - nifivolumes: nifi-data: And here is the relevant part of my nginx.conf:   events {}http { server { listen 80; server_name nifi.xxx-xxx-python-mps; return 301 https://$host$request_uri; } server { listen 443 ssl; server_name nifi.xxx-xxx-python-mps; ssl_certificate /et

Hi Community,I recently upgraded Apache NiFi from 1.27.0 to 2.2.0 in my Kubernetes environment. After the upgrade, I noticed that the web server is not getting the correct IP address in the logs, and my NiFi UI is not accessible.Setup Details:Previous Version: NiFi 1.27.0Current Version: NiFi 2.2.0Deployment: Running in KubernetesLogs Behavior:In 1.27.0, the NiFi web server correctly binds to the expected IP.In 2.2.0, the web server logs do not show the correct IP, and the pod eventually restarts after a few minutes.Troubleshooting Done:Verified nifi.properties settings, including  nifi.web.https.host.Checked kubectl logs --previous but did not find any fatal errors before the restart.Ensured the container has enough memory and CPU.Checked for readiness/liveness probe failures in Kubernetes but found no failures.Questions:Has anyone faced a similar issue after upgrading to NiFi 2.2.0?Are there any new configuration changes in 2.2.0 that could affect the web server binding/IP resol

@MattWho Is there anywhere I can find what types of filters I can use in the "User Search Filter" or the Group one?  I am trying to widdle down the amount of objects that are populated from LDAP into my Users. In my AD Server, users are in 2 different OU's.  So, for example, I'll have users in ou=Admin Accounts,ou=Domain Admins,dc=my,dc=network,dc=com but I'll also have users in ou=Regular Users,ou=Testers,dc=my,dc=network,dc=com.  And the only way to really pull both are to have my base search set to dc=my,dc=network,dc=com.  Doing this pulls computer accounts and all users, even users that would never need access (like service type accounts). I can filter out the computer accounts by using this filter: (objectCategory=cn=Person,cn=Schema,cn=Configuration,dc=my,dc=network,dc=com) But I'd like to also filter out the accounts, or OU's that aren't needed.  Or, if possible just use the 2 NiFi groups I have in AD (Nifi Admins/Nifi Users) wi

Hi All,I want to calculate data from a CSV file as the data source, then retrieve the formula from the database table. For example, the CSV file contains columns for "id" and "amount," while the database has columns for "id" and "formula." When the "id" from the CSV file is found in the database, the "amount" will be calculated using the corresponding formula based on that "id". How can I create this using NiFi? Thanks

The error from my Spark job is++++Failing this attempt.Diagnostics: Application application_1738011234567_0014 initialization failed (exitCode=255) with output: main : command provided 0main : run as user is xxxxmain : requested yarn user is xxxxUser xxxx not found++++I read this post <https://community.cloudera.com/t5/Support-Questions/MapReduce-job-failing-after-kerberos/td-p/160273>. My group mapping configuration is hadoop.security.group.mapping = org.apache.hadoop.security.LdapGroupsMapping. I kinited xxxx before the job run. I added the AD user xxxx to an AD group hadoop. But I still got the same error.This online doc might be appliable <https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/security-authorization/topics/cm-security-authorization-ldap-group-mappings.html#ariaid-title3>I might need to add the flag -Dcom.cloudera.cmf.service.config.emitLdapBindPasswordInClientConfig=true to the variable CMF_JAVA_OPTS flag. But the documentation is for CDP 7.1.8 and does

When trying to insert using hive on tez, the insert operations hangs. With mapreduce, it works fine. How can i resolve this problem?

Hi Team,I’m seeking your support in converting a string to a timestamp with microseconds in Hive. When using unixtimestamp, I notice that the milliseconds are being dropped. Could you please advise on how to capture milliseconds (and microseconds) correctly, and which data type I should use to store the value in Hive?SELECT from_unixtime(unix_timestamp('2024-11-05-10.03.17.872195', 'yyyy-MM-dd-HH.mm.ss.SSSSSS')) as times,'2024-11-05-10.03.17.872195' as valeFROM  

Hi, I have a json in the below format[{"order_id": 1,"category": 123},{"order_id": 2,"category": 123 },{"order_id": 3,"category": 456},{"order_id": 4,"category": 321}] I want to create flowfiles seperatly for all the categories,flowfile1 - orders of category 123:[{"order_id": 1,"category": 123},{"order_id": 2,"category": 123 }]flowfile2 - orders of category 456[{"order_id": 3,"category": 456}]flowfile3 - orders of category 321:[{"order_id": 4,"category": 321}] I don't want to use splitjson, because this will be applied on millions of records. I don't want millions of flowfiles.I tried EvaluateJsonPath and RouteOnAttributeall_category : $.[*].category - this creates on attribute with all category valuesRouteOnAttribute category_123 : ${all_category:contains(123)}category_321 : ${all_category:contains(321)}category_456 : ${all_category:contains(456)}But all the 3 flowfiles routed has all the order details.Please guide me here, I am new to NIFI 

i am new at nifi . i want to use listftp with tracking entities with rides cache . i use those configs but no files listed i donot now whats is wrong here 

I need to configure nifi for log rotation. I have seen this article: Understanding how the logback.xml configuration in... - Cloudera Community - 248662Here I have few doubts regarding the property.<maxHistory>30</maxHistory> It's mentioned in comments as<!-- keep 30 log files worth of history -->But on checking I have come across a document logback.xml Example - Mkyong.com where its mentioned as     <!-- each archived file, size max 10MB --> <maxFileSize>10MB</maxFileSize> <!-- total size of all archive files, if total size > 20GB, it will delete old archived file --> <totalSizeCap>20GB</totalSizeCap> <!-- 60 days to keep --> <maxHistory>60</maxHistory>   So, my doubt is does maxhistory refers to number of file or number of days. Also do we really need to add totalSizeCap because anyway

Hi Team,I am upgrading my old nifi 1.14 to 2.0 M2 version. But i want to support of both flow xml and flow json. for this reason what should my approach to migrate it as soon as possible.. So my nifi can work either with xml or with json as per my requirement.Thank you for your time. 

Hello Cloudera Community,I was wondering if is it a good practice to install a CDP in a server where I installed CDH previously?Is it something achievable?If yes, which would be the considerations that I need to take care of? Any advice? Thanks!!