Ask the Community

Hi      I have integrated Secured Nifi with NGINX and is integrated with Onelogin SAML 2.0 Custom Connector (Advanced)After entering credentials of one login it shows user logged in on onelogin portal,But on Browser it redirects to /nifi-api/access/saml/login/consumer this url and it shows HTTP ERROR 401 UnauthorizedURI:STATUS:MESSAGE:/nifi-api/access/saml/login/consumer401UnauthorizedFew findings are- Recipient value in SAML payload is empty  and Cookie value is not matching with InResponseTo value in SAML PayloadAlso not sure how to match itfrom nifi.user.log file i can see error:SAML Authentication Request Identifier Cookie not foundCan anyone please guide here?

Hi Team,how to sort this below output in Ascending order  by maintaining same space in both header values. I've tried using query record but couldn't able to add spaces using SPACE function. but space which i've given is considering as SPACE(numeric) value instead of INT.input:NO CUSTOM.;NO INCSUPB;NO ALLBPBM ;IMPORTE BRUTO;DESCUENTO;IMPORTE NETO0000000023;123548    ;999888     ;875,92       ;176,88   ;699,040000000012;123547    ;777666     ;2347,04      ;591,84   ;1755,2 Expected Output:NO CUSTOM.;NO INCSUPB;NO ALLBPBM ;IMPORTE BRUTO;DESCUENTO;IMPORTE NETO0000000012;123547    ;777666     ;2347,04      ;591,84   ;1755,20000000023;123548    ;999888     ;875,92       ;176,88   ;699,04

I am facing issue with upserting any record in phoenix table thru mapreduce jobs. When using upsert command inside sqlline.py, records are getting inserted but when trying to load/BulkLoad records thru mapreduce or psql.pygetting below error:Exception in thread "main" org.apache.phoenix.schema.MaxPhoenixColumnSizeExceededException: ERROR 732 (LIM03): The Phoenix Column size is bigger than maximum HBase client key value allowed size for ONE_CELL_PER_COLUMN table, try upserting column in smaller value. Upsert data to table SYSTEM.CATALOG on Column TENANT_ID exceed max HBase client keyvalue size allowance, the rowkey is TENANT_ID= AND TABLE_SCHEM=SYSTEM AND TABLE_NAME=CATALOG AND COLUMN_NAME=TENANT_ID AND COLUMN_FAMILY= at org.apache.phoenix.compile.UpsertCompiler.setValues(UpsertCompiler.java:150) at org.apache.phoenix.compile.UpsertCompiler.access$500(UpsertCompiler.java:124) at org.apache.phoenix.compile.UpsertCompiler$UpsertValuesMutationPlan.execute(UpsertCompiler.java:1309) at org.a

Hello all,I'm trying to convert many records from json to parquet with ConvertRecord processor.Most succeed, but convert fails for some files with this error. org.apache.parquet.schema.InvalidSchemaException: Cannot write a schema with an empty group: optional group pop_pools  I assume this is because some json files contain the following field. "pop_pools": {} As source record is log data, we can't modify it.Is there any way to avoid this error and convert the records to parquet format?Thanks,

Hi AllI have installed Apache hive 3.1.3 with postgres as remote metastore. schematool initiation successfully happened. Able to create table also but not able to insert any record. Can anyone please help. Hadoop version is 3.3.4. Error is as under:ERROR exec.TaskRunner: Error in executeTask java.lang.NoSuchMethodError: org.apache.hadoop.yarn.util.Apps.setEnvFromInputProperty(Ljava/util/Map;Ljava/lang/String;Ljava/lang/String;Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)V at org.apache.hadoop.mapreduce.v2.util.MRApps.setEnvFromInputProperty(MRApps.java:717) at org.apache.hadoop.mapred.YARNRunner.setupContainerLaunchContextForAM(YARNRunner.java:537) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:584) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:326) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:251) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at org.apache.hadoop.map

Hello,I' trying to merge over 20000 parquet files into 1 file with MergeRecord processor.But it generates 7 files.It doesn't seem to be caused by Bin size, Bin age or number of records limitation.Looking into the generated file, it looks that similar log records are in the same file. So I assume MergeRecord processor looks into source record content and determine the target file to be merged into.Is this expected behavior?I don't set value in Correlation Attribute Name field.Thanks,

Hi All, This could be a silly question, Apologies for that. How do i serialize the headers sent from PublishKafka_2_6 processor ?Currently, the processor i am using is sending headers along with the payload as a string   ID:N1,GRANTED-ID:[root,UI1] {...<Payload>...}    we have other applications communicating with Kafka , which sends out headers along with payload as   GRANTED-ID: ��♣sr‼java.util.ArrayListx��↔��a�♥☺I♦sizexp☻w♦☻t♦roott♦UI1xID: ��♣t♣N1{...<payload>...}  Is it possible for NiFi also to send across as the above format? 

We have created new java service for establishing connection with hive of Cloudera kerborized cluster, but stuck because of below error. Error remains the same even we generate kerberos ticket for the hive user through its keytab at remote machine. Tried establishing manual connection using the hive beeline utility as well after generating Kerberos ticket but facing the same issue after several hit and trail .If any one having authentic document or steps to establish jdbc or beeline connection with remote hive for the kerboriszd cluster please share.Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)Detail error message is below (with masked host and server deatils>> -bin]$ ./beelineSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/maxiq/apache-hive-3.0.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/ma

I am using NiFi ListSFTP 2.1.0 to scan files in the root directory of an SFTP server. The Listing Strategy is set to Tracking Timestamps, and the Remote Poll Batch Size is set to 5000. There are over 10,000 files on the server, and each file has a timestamp. The processor only outputs 1000 files during its first run, but subsequent runs do not list the remaining files. However, with the same configuration, ListSFTP 1.9.1 works as expected. Why is this happening? 

Hello,I am new with Apache nifi.i need support to which processors i should use to read an files from server i know it's privilege  

Hi, I have a secured Nifi cluster. Im trying to call a nifi api to start\stop processor using postman. I followed the instruction for the api "PUT /processors/{id}/run-status". Provided the Bearer token and the Json Body. However I keep getting 403 Forbidden message. Does anybody know why? I'm able to run other APIs successfully such as getting processor info "GET /processors/{id}"! Im guessing its because Im using SSL secured nifi with jks keystore and truststore, but not sure how to provide this information to postman. Can anyone help please? 

Hello, I am trying to implement file processing in batched manner using NiFi processors. My use case is, there are 70-80K files coming daily having size of 200-300 MB each.Taking input those files from S3 store and sending those to spark execution by Livy processor. Plan is to not sending each file location to spark, instead we can batch few files and sending to spark by Livy, so livy connections to spark will get reduced. Below are consideration while creating batch.1. Batch Size: Batch will be based on size e.g. 1000MB2. Wait duration: If there are not enough file to complete batch size then, batch will start after specific wait durationI am trying to implement this using wait, notity and updateAttribute(using stateful variables) based on batch size and wait time, but its not working fully. Any leads/suggestions how to implement this would be much appreciable. Thanks.  

hi i am trying to use this {"hostNames": ["...","..."],"sshPort": 12345,"userName": "...","password": "...","privateKey": "...","passphrase": "...","parallelInstallCount": 12345,"cmRepoUrl": "...","gpgKeyCustomUrl": "...","javaInstallStrategy": "...","unlimitedJCE": true,"gpgKeyOverrideBundle": "...","agentReportedHostnames": [{"key": "...","value": "..."},{"key": "...","value": "..."}],"subjectAltNames": ["...","..."]} anybody help how fill this ??

This issue occurred right after I enabled TLS on my CDP Private Cloud Base 7.1.7. The client call to HBASE Thrift API failed at TLS hanshake.Below is the connection test output with the handshake failure.++$ openssl s_client -connect mycompany.com:9191CONNECTED(00000003)write:errno=0---no peer certificate available---No client certificate CA names sent---SSL handshake has read 0 bytes and written 287 bytesVerification: OK---New, (NONE), Cipher is (NONE)Secure Renegotiation IS NOT supportedCompression: NONEExpansion: NONENo ALPN negotiatedEarly data was not sentVerify return code: 0 (ok)---++My Thrift API port is 9191 (not the default 9090). This port worked well before TLS was enabled.There should be no certificate/ca issue because the Thrift (on the same node) UI over TLS works just fine. Below is the connection test output showing a successful handshake.++$ openssl s_client -connect mycompany.com:9095CONNECTED(00000003)depth=2 CN = MYROOTCA...---Certificate chain...---Server cer

Unable to connect to Hortonworks Sandbox please help.The login is stuck and I'm not able to login to hadoop. Tried all of the networking suggestions but none worked. Please help.

Facing issue while Run a set of services for the first time.2025-01-10 10:59:40,491 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (1 skipped) AgentAvroServlet: heartbeat processing stats: average=1310ms, min=17ms, max=73273ms.2025-01-10 10:59:40,499 ERROR agentServer-57:com.cloudera.server.common.AgentAvroServlet: Error processing Avro requestorg.eclipse.jetty.io.EofExceptionat org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:810)at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)at org.eclipse.jetty.server.HttpConnection.send(HttpConnection

Why is this happening? The installation went perfectly, but when I try to do anything else, it disconnects because the job is waiting too long. It's not like I'm running out of memory or anything. What could be the cause?

I have a CDH instance which is comes bundled from Oracle Big Data Appliance. Its on CDH CM 6.3.4. I know its already EOS. License expired is soon at 30 March 2024.Can CDH license renewal still applicable or does subscription to CDP license would work on CDH instance?

I'm trying to deploy CDP Data Services on Dedicated OCP Platform and facing problem with Vault permission. I've already using root token to deploy.Installation's logs:2024/11/12 15:45:34 Vault created at the end point: 'vault-cdp-vault.apps.poc.xplat2024/11/12 15:45:34 Trying to reach OpenShift API server :2024/11/12 15:45:34 => 200 OK2024/11/12 15:45:34 Get Vault Status.2024/11/12 15:45:35 Vault is unintialized. Trying to initalize.2024/11/12 15:45:35 Get Vault Status.2024/11/12 15:45:35 Checking vault server health ...2024/11/12 15:45:35 Get Vault Status.2024/11/12 15:45:35 Vault server is initialized.2024/11/12 15:45:35 Unseal vault server with Seal Type : shamir.2024/11/12 15:45:35 Unseal vault server with Seal Type : shamir Completed.2024/11/12 15:45:35 Get Vault Status.2024/11/12 15:45:35 {    "initialized": true,    "sealed": false,    "standby": false,    "performance_standby": false,    "replication_performance_mode": "disab

Hi, I am trying to set up a workflow in Nifi, where I need to process Flowfiles in a specific order. To my understand, this is what the EnforceOrder processor was designed for.The part I am having problems with looks like this:In Set Merge Priority I am setting a Merge Priority attribute (1 for the left node, 2 for the right node). I am also setting an Enforce property to group the EnforceOrder by:My EnforceOrder processor is configured as followed:Only flowfiles with merge_priority = 1 will be successfully routed, while the flowfiles with merge_priority = 2 are stuck in "Wait". Files with merge_priority = 2 to arrive at the EnforceOrder processor before the flowfiles with merge_priority = 1. To my understanding, the flowfiles should be re-processed as soon as a file with the current merge_priority has been processed?

Hello I'm running apache zookeeper 3.8.4.File zookeeper.log does not get created.Log files are empty except the following message:SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".SLF4J: Defaulting to no-operation MDCAdapter implementation.SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details.How can I configure zookeeper with logs working ?According to the links, this is due to slf4j not detecting logback.The process runs with the following options in the commandline (extract)The classpath does seem to include slf4j and logback jars, as well as the conf directory containing logback.xml/apps/dis/java/jdk/bin/java-Dzookeeper.log.dir=/apps/zookeeper/log-Dzookeeper.log.file=zookeeper.log-cp.../apps/zookeeper/bin/zkcurrent/bin/../lib/sl

The repositoryhttps://repo.hortonworks.com/content/repositories/releases/is not working for 17 days now.  What is happening?Related to https://community.cloudera.com/t5/Support-Questions/Hortonworks-maven-repo-broken/m-p/398655

Hi All, We are in process of migrating HDP cluster to CDP. As we would like to retain Atlas lineage in CDP from HDP, we have exported them from HDP cluster which gave us a .zip file. Extracted it and remediated the contents of .json files to match the CDP cluster name and server names, then zipped it back to a new_name.zip file. Now when running Atlas import API command to import remediated Atlas data to CDP Atlas, we are getting below error for most of the Atlas imports. {"errorCode":"ATLAS-404-00-007","errorMessage":"Invalid instance creation/updation parameters passed : type_name.entity_name: mandatory attribute value missing in type type_name"} Also, have some more errors, which certainly need to be looked into but not super priority.{"errorCode":"ATLAS-400-00-08A","errorMessage":"Attempting to import empty ZIP file."}:{"errorCode":"ATLAS-500-00-001","errorMessage":"java.lang.NullPointerException","errorCause":null} Having tried to figure this out since a w

Hello,I am trying to delete a custom Type, but I receive the error "ATLAS-409-00-002: Given type InformationAssets has references".  This type has no entities and no relationships associated with it. I created a second custom Type (which I can delete) and compared the API results from: /api/atlas/v2/types/typedef/name/But both look the same.  I'm not sure what the dependency is?  Is there anyway to find out what references are out there for this custom Type?

Hello,I have a custom Type that has an attribute that references another Type by mistake.  I have tried to delete this attribute using a PUT api call to: /api/atlas/v2/types/typedefsBut got the error:ATLAS-400-00-00D: attribute delete not supported.I even tried to to update the attribute to not longer reference the Type but instead have "typeName": "string", but I get the error:ATLAS-400-00-029: Data type update for attribute is not supportedIs there a way to delete an attribute from a custom Type?