Ask the Community

Hello, I've tried to build an application using http://repo.hortonworks.com/content/repositories/releases/ and its unable to find them. When using the browser I've notice that if I try to browse the site with http I get a 404 if I do it with https I got the nexus UI.Is something wrong in the nginx of the http site?Last week this worked properly.Thank you in advance.Best regards

Hi guys,I want to convert Nested JSON to Flattened JSON using JOLT transform.Nested JSON:{"data": {"getBusinessRelationListing": {"edges": [                {                    "node": {                        "id": "1964730",                        "name": null,                        "code": "CN02",                        "bejoCompany": true,                        "legacyVendorId": [],                        "locatonType": [                            "Commercial"  &n

Hi, repo.hortonworks.com is not giving any response. When will it be back online?

Hi all,When running a Sqoop import , the job gets stuck at INFO mapreduce.Job: map 0% reduce 0%, and the process does not progress further. Additionally, the following error message appears in the logs: INFO conf.Configuration: resource-types.xml not found.INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.Despite this error, there are still plenty of available resources in the cluster, so resource allocation should not be the issue.How can I resolve the "resource-types.xml not found" issue?Where should the resource-types.xml file be placed in CDP? 

Hello Team, We recently upgraded from CDP 7.1.6 to CR 7.1.8. After the upgrade, we are having issues with starting the HDFS NFS Gateway role. Earlier, on CDP 7.1.6, we would run the command - hdfs --daemon start portmap and then start the HDFS NFS Gateway role from the Cloudera Manager, however after the upgrade this is not working (that is the HDFS NFS Gateway role doesn't start after running the above command and starting the role from the Cloudera Manager). We are getting the following issueDue to some security issues, we are not able to start rpcbind service. Is there a workaround to start the NFS Gateway? Note, the mount service does not start when even though the portmap service looks to have a running process. Best RegardsSayed Anisul Hoque  

Hi Everyone,Struggling with transforming the JSON flow file, would appreciate help from the community!input json flow file:[{"firstName": "John","LastName": "Malkovich","MiddleName": "J","nameType": {"type1": "Other""type2": "Primary"}}, {"firstName": "Johnny","LastName": "Cage","MiddleName": "M","nameType": {"type1": "Basic""type2": "Primary"}}]expected output json flow file:{"fileName": "file.xml","count": "2","individuals": [{"firstName": "John","LastName": "Malkovich","MiddleName": "J","nameType": {"type1": "Other""type2": "Primary"}}, {"firstName": "Johnny","LastName": "Cage","MiddleName": "M","nameType": {"type1": "Basic""type2": "Primary"}}]​}Jolt  specification :[ { "operation": "default", "spec": { "fileName": "file.xml"", "count": "=2" } }, { "operation": "shift", "spec": { "*": { "@": "individuals[]" } } } ] gettingError:TransformException: The Spec provided can not handle input that is a top level Json Array. 

Hi , My Nifi loads all the AD groups from my LDAP server , but it doensn't load any member information of it . As per the image , one can see that the Members list is empty.How is it possible to get the users in the group too ? I can add it manually , but I would like the ldap to be able to retrieve the users as per the group. 

hello,I'm trying to read a parquet file using the ConvertRecord processor and I'm getting the error:ConvertRecord[id=e599cd8f-9a1d-3134-4daf-af7bc91cdd57] Failed to process FlowFile[filename=objectTable_tract_5074_DC2_2_2i_runs_DP0_2_v23_0_1_PREOPS-905_step3_31_20220314T212509Z-part0_output.parquet]; will route to failure: org.apache.avro.SchemaParseException: Illegal initial character: 0In my file the columns are numeric and the first one starts with 0 (zero).

Hi Team, I am trying to implement the CSFLE logic using the AWS KMS  service. I am using ScriptiedTransformRecord processor with Groovy script. For testing purpose , i have used the sample encryption key which created by openssl and tested the script , it is working fine. However,  i would like to use the AWS KMS for the encryption keys. Is the a way to interact with KMS to get the key .Here is my flow

Hi All,So I have a flow that at the end changes particular status through API. The direct upstream processor is putEmail, which is supposed to notify dynamically generated contacts with data.After this e-mail is sent, the requirement is to wait 24 hours before using the API call. I have had a hard time understanding the Wait processor and its relation to signal release. ControlRate does not really work (or I cannot set it up properly). What would be a suggested way to deal with such a scenario?

 insert into temp.string_Test values ("아이오에우");=> insert into temp.string_Test values ("아이오에우")insert into temp.string_Test values ('아이오에우');=> INSERT INTO `temp`.`string_Test` VALUES (CAST('아이오에우' AS CHAR(5))) Like above, when I use single quotation with utf8 words, it casts the word into CHAR and the word is broken.Could you fix this problem for impala jdbc connector?  

Please help troubleshoot the following error encountered during insert operation in Hive.Command executed: INSERT INTO example (C1, C2) VALUES(1, "|VALUE ONE"),(1, "|VALUE TWO"),(1, "NULL"),(1, "|VALUE FOUR"),(1, "|VALUE FIVE"); Result of execution:INFO  : Compiling command(queryId=hive_20241125083349_27568c92-ab64-4322-81fd-3bddb8bdbbe3): INSERT INTO example (C1, C2) VALUES(1, "|VALUE ONE"),(1, "|VALUE TWO"),(1, "NULL"),(1, "|VALUE FOUR"),(1, "|VALUE FIVE")INFO  : Semantic Analysis Completed (retrial = false)INFO  : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), FieldSchema(name:_col1, type:varchar(20), comment:null)], properties:null)INFO  : Completed compiling command(queryId=hive_20241125083349_27568c92-ab64-4322-81fd-3bddb8bdbbe3); Time taken: 0.81 secondsINFO  : Executing command(queryId=hive_20241125083349_27568c92-ab64-4322-81fd-3bddb8bdbbe3): INSERT INTO example (C1, C2) VALUES(1, "|VALUE ONE"),(1,

Hello,I'm triyng to sort record in parquet file like below in order of timestamp.{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:58.440995\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message AAA\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:58.429579\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message BBB\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:08.441709\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message CCC\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:08.428501\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message DDD\"}{"Src_obj__event_metadata":"{\"timestamp\":\"2024-11-01T00:23:48.440624\",\"severity\":\"Info\"}","Src_obj__user_data":"{\"message\":\"Message EEE\"}I confugured the following query in QueryRecord processor and it passed validation in the processor.SELECT * from flowfile ORDER BY JSON_EXTRACT(Src_obj__event_metadata, "$.t

I am facing an issue with Notify and Wait starting from version 2.0.0. In order for them to listen to each other in version 1.28, I need to configure them with the MapCacheClientService 2.0.0 and MapCacheServer 2.0.0 services. However, in version 2.0.0, although I can still find both services in the documentation, in practice, the configuration only includes the MapCacheClientService 2.0.0 service, and I can no longer find the MapCacheServer 2.0.0 service.Using Wait and Notify in version 2.0.0 seems much more difficult compared to previous versions. I spent two days trying to figure it out, but I haven't made any progress. If possible, could you please guide me on the correct configuration so that Wait and Notify can work properly, like they did in the older versions?

I'm trying to configure NiFi for OIDC but when I start the nifi process it dies with the error below.  A curl shows that at least basic connectivity is working to the IDP.   Any ideas what could be causing this?  I can provide my nifi.properties and/or authorizers.xml if needed.2024-11-20 00:22:09,786 ERROR [main] o.s.web.context.ContextLoader Context initialization failedorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependencyexpressed through method 'setFilterChains' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'securityFilterChain' defined in org.apache.nifi.web.security.configuration.WebSecurityConfiguration: Unsatisfied dependency expressed through method 'securityFilterChain' parameter 7; nested exception is org.springframewo

HI All, Im getting below error  after new fileds are being added to the payload from KafkaOpenTX, datatype: double precisioncharges, Datatype: stringnote:before these 2 fields added to the payload its working fine with existing json reader and writerError:05:25:40 UTCERRORConsumeKafkaRecord_2_6[id=8f8pq352c-0rcb] Failed to parse message from Kafka using the configured Record Reader. Will route message as its own FlowFile to the 'parse.failure' relationship: org.apache.nifi.serialization.MalformedRecordException: Successfully parsed a JSON object from input but failed to convert into a Record object with the given schema- Caused by: org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Ljava.lang.Object;@6b9b8a0c] of type class [Ljava.lang.Object; for field invoiceLineList.ecoTaxValues to any of the following available Sub-Types for a Choice: [ARRAY[STRING], ARRAY[DOUBLE]]input payload:{"Identifier":"123456","creationDate":"2024-11-22

I created the following project:and I want to connect 2 files by field "id"Processor name "GenerateRecord" has the following settings:Processor name "GenerateRecord2" has the following settings:Processor name "JoinEnrichment" has the following settings:The error occurs when "JoinEnrichment" works, I can’t understand why there, I tried everything I could

I'm trying to get a set of file .csv from a directory in a server sftp. My nifi is deployed in a cluster with only one node for nifi. If I try to download a file .csv through sftp> from the same node on which nify is deployed, I can get it without problem. However the processor SFTP in nifi is not able to do it, but I continuosly get the following stacktrace  ERROR org.apache.nifi.processors.standard.FetchSFTP: FetchSFTP[id=1f5b3ddd-2d4c-1889-ffff-ffffa3d58254] Failed to fetch content for StandardFlowFileRecord[uuid=ccf2a3ae-51cc-470c-80e0-6477b045a536,claim=,offset=0,name=file.csv,size=0] from filename /test/nifi on remote host 10.100.201.248:22 due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=1f5b3ddd-2d4c-1889-ffff-ffffa3d58254]: java.io.IOException: error; routing to comms.failure: org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=1f5b3ddd-2d4c-1889-ffff-ffffa3d58254]: java.io.IOE

Hi ,I haven been playing with nifi on docker lately and its been quite the challenge and the learning experience. To best  understand how to utilize docker for nifi, Im hoping the community can help me with addressing the following observations\questions: 1- Most of the examples I found on the internet including the nifi official docker page seem to be suitable for single host deployment ! Im finding this is strange - unless Im missing something - but doesnt that defeat the purpose of having cluster with no single point of failure ? What are the the scenarios where someone wants to deploy single host multiple container cluster vs multiple host single container ? 2. Getting to understand docker networking I found that if I want to create multi host cluster and have the cluster to have visibility  to our work network then the ideal way to do it is using "host" networking,  is this correct or is there a better way (maybe using overlay networking with swarm? if I d

Hi everyoneI try to split to records an Excel sheet, using ExcelReader. But get this error:Spoiler (Highlight to read)SplitRecord[id=01921045-9bc9-1353-b2a2-96dfb4f1e8af] Failed to split FlowFile[filename=test.xlsx]: org.apache.nifi.processor.exception.ProcessException: IOException thrown from SplitRecord[id=01921045-9bc9-1353-b2a2-96dfb4f1e8af]: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Read next Record from Excel XLSX failed on row 546 in sheet reportdata1- Caused by: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Read next Record from Excel XLSX failed on row 546 in sheet reportdata1- Caused by: org.apache.nifi.serialization.MalformedRecordException: Read next Record from Excel XLSX failed on row 546 in sheet reportdata1- Caused by: java.lang.NumberFormatException: For input string: ""SplitRecord[id=01921045-9bc9-1353-b2a2-96dfb4f1e8af] Failed to split FlowFile[filename=test.xlsx]: org.apache.nifi.processor.exception.P

How to configure OIDC authentication in nifi-registry 1.19.1?When configuring secure access ssl + oidc does not display the UI when trying to access. In the log I don't see errors just warnings, one of them is this:o.s.l.core.support.AbstractContextSource Property 'userDn' not set - anonymous context will be used for read-write operationsIs it possible to access this version through OIDC?

I have a text file which needs to be fetched from remote server through SFTP. File will be having 26k+ rows with | separated values like below:I have used below processor's to do the data manipulation. But facing “Processing halted.Out of memory heap” error after splitJSON.Please help with the better approach to handle this case.

 I have create two partitioned tables  with same ddl,  except for that one is internal table  , another is external table 。Both has same file location (oss://xxxx/xxx )。  create table test_analyze( id int, name varchar(20)) partitioned by (dt string) ; create table test_analyze_external( id int, name varchar(20)) partitioned by (dt string) ;msck repair table test_analyze;analyze table test_analyze partition(dt='20241121') compute statistics; msck repair table test_analyze_external;analyze table test_analyze_external partition(dt='20241121') compute statistics;   when I query  count() for table 'test_analyze'， then quickly get  the result ,  it seems fetch result from stats。 but when I send the same query to table 'test_analyze_external',  it can't return stats result , and active a MR job。And  it's  sql-explain below：  +----------------------------------------------------+|

Hello, I am having an issue on cleanning my flow file. I have tried everything. The message that NiFi returns is Failed to drop FlowFiles due to java.lang.IllegalStateException: Cannot update journal file /flowrepo/flowfile_repository/journals/13211407.journal because no header has been written yetI do not know nothing about this issue. My processors are already stopped and all other services are disabled. Thank you  in advance.

Hello everyone. Hope you are doing well.I am trying to achieve to run simple query on hive, using nifi processors. I Have tested couple scenarios, with different versions (also tried apache kyuubi 1.8.0 with jdbc drivers).CDP private cloud 7.1.7EnvironmentApache NiFi - Version 1.18.0 (java 11)Hive - Version 3.1.3jdbc driver is downloaded from cloudera, version 2.6.25 and using HiveJDBC41.jar1. scenariousing Apache NiFi DBCPConnectionPool 1.24.0URL : jdbc:hive2://example.com:10000/default;KrbRealm=EXAMPLE.COM;KrbHostFQDN=example.com;KrbServiceName=hive;AuthMech=1;KrbHostFQDN=_HOST;principal=test@EXAMPLE.COMDriver Class : com.cloudera.hive.jdbc41.HS2DriverDriver Location : /opt/nifi/nifi-current/lib/DRIVER.jarand using kerberos credential service with principal/keytabNifi error log same as on screenshot.2. scenariousing HiveConnectionPool 1.19.1url : jdbc:hive2://example.com:10000/defaultHive Configuration Resources : /opt/nifi/nifi-curre