Find answers, ask questions, and share your expertise
Recently active
My data set is as below."IM43163","SOUTH,OFC","10-Jan-23""IM41763","John:comment added","12-Jan-23"CREATE EXTERNAL TABLE database.table1( `col_1` string, `col_2` string, `col_3` string)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ( "separatorChar"="," , "quoteChar"="\"") Since the record is splitted amont two rows therefore it is not loading properly and gives null values. The output I am getting is mentioned below.IM43163SOUTH,OFC10-Jan-23IM41763?????
Is there capability to perform a reverse DNS lookup in NiFi without executing an external script, process, or code? (i.e. without execute code permission)
I have a flow that fetches all files from a given directory—they could be gzip, zip or csv, with the gzip and zip files holding a single csv file. I then route on MIME type, decompress the gzip files, unpack the zip files, and then bring what are now ALL csv files back together. This is working. I then want to create a zip archive of all of these csv files, and MergeContent seemed like a good candidate (outside of using ExecuteStreamCommand to run zip from the OS). But no matter what I do, the results are inconsistent:The first time I ran it with default properties in MergeContent (and setting Merge Format to ZIP), it created a single zip file — I thought I was done!But on the second time, it created six zip files, so I realized I wasn't done, and started playing with propertiesChanged Maximum number of Bins to 1 - created five zip filesChanged Correlation Attribute Name to absolute.path, which is the same for all flow files - created four zip filesChanged Maximum Group Size to 50
Hi,We have huge and complex XML files. For example: 15-20 levels in XML tree structure, approximately 180 basic types and 200 complex types, 1 to many relations between nodes in XML tree structure.As the output we want to have tables in Hive or Impala and to use SQL to query this tables. Could you please advise how to that in the most effective way?Effective - that is reducing manual coding works. Best regards
Nifi UI is showing blank screen. Processors are not visible. This is happening for all users/Administrators
Hi, I'm trying to connect nifi instance to HashiCorp Vault. As per instructions, I added HashiCorpVaultParameterProvider and StandardHashiCorpVaultClientService to my instance. I configured both of them and managed to fetch secrets from Vault to nifi. Right after I fetched secrets I was asked to create respective nifi context to work with fetched secrets. I created one and used in a couple of my processor groups simply by selecting context in "Process Group Parameter Context" dropdown. Secrets were accessable inside processor groups and worked fine. Then I updated secrets on the Vault side, and according to instructions I should have refetched them again using "HashiCorpVaultParameterProvider" but fetch icon did nothing. So new secrets were not come. In order to achieve this I had to drop created context and all related references in process groups. Only after these actions I managed to fetch new secrets again. So, it looks wierd because every time when I update something on
I read data from Kafka with ConsumerKafka processor and send them to MergeContent processor because I want to merge each hundred thousand data in a file. Then send them to UpdateAttribute processor cause I want to change filename, after that send them to PutSFTP processor cause I want to send data to an external server. My problem is that the number of data coming from Kafka is 100 thousand per second. But after going through this route, the output sends about 28 thousand data per second to the server and the lag value in the Kafka consumer increases over time. My Vm has 14 cores CPU and 21g RAM. The number of topic partitions that read from Kafka is 30, so I set thread to 30 in Consumer Kafka processor. Do you have any suggestions and ideas how I can increase the speed to reduce and eliminate the lag?
Hey there, for the sake of simplicity, I try to break down my problem to the core. I have flowfiles and their content is plain text (without any schema) representing data read from files on the filesystem. Via site-to-site I send them on a remote NiFi instance and place them there. During that ETL pipeline I need to query some additional metadata, sometimes via SQL queries, sometimes just some logic derived from the data. Most of this metadata is simply added as attribute as desired. The ExecuteSQL processor however, overwrites the flowfile content, for example. This is why I need to duplicate/fork the flowfiles at some point. One flow keeps the original data content, the other queries the additional data which gets written to the flowfile content. How can I merge these corresponding 2 flowfiles each back together? I know the ForkEnrichment and JoinEnrichment pattern, but they only merge content, additionally the content needs to be in some kind of sche
Dear Experts,We want to learn how to perform batch processing on semi-structured JSON flow files in NiFi. We know that we can use a Jolt Transform Record, an Update Record processor, and a Query Record processor, for which we need to set reader and writer controller services. However, we are unsure how to set up reader and writer controllers for semi-structured data schemas.In the text below, we have a multi-line JSON flow file in which the "code" field is static and the "other" field is an unstructured nested JSON. We want to query the "code" field in batch mode and include the "other" field information in the resulting flow files for processing in subsequent processors. How should we set the reader and writer controllers to achieve this?Regards,{"code": "6", "other": {"feild1": "data1"}}{"code": "7", "other": {"feild2": "data2"}}{"code": "8", "other": {"feild3": "data3"}}
Hi,I'm using the Cloudera CDP 7.1.7 Private Cloud Base version and would like to have a confirmation if there is a possibility to be able to set, in the capacity scheduler of Yarn, time rules as I did previously in the old CDH.It would be interesting to be able to use that functionality also in CDP, but in the Cloudera documentation I've searched on the web there's no mention of it.Could you give me some hints if this feature is still present in CDP? Thanks
Groovy Script Crashing: Unexpected Character '(' in groovy script in apache nifi?
I have this data and I need to split VisitList's content into different flowfile one by one.What should I write in JsonPath Expression so that data got split, I tried $.* but then it is considering this given data into one file only while I want to split on the basis of VisitList.Data[ { "employer": "98765", "loc_id": "312", "topId": "Management", "VisitList": [ { "S1": "HR", "S2": "Accountant" }, { "S1": "Manager", "S2": "Sr. Manager" } ] }] I want to split Data into files like this 1st flowfile[ { "employer": "98765", "loc_id": "312", "topId": "Management", "VisitList": [ { "S1": "HR", "S2": "Accountant" } ] }] 2nd flowfile[ { "employer": "98765", "loc_id": "312", "topId": "Management", "VisitList": [ { "S1": "Manager", "S2": "Sr. Manager" } ] }] I tried putting $.* in JsonPath Expres
EnvironmentOS: Windowsmain batch Script: sample_Import_Load.bat@Former Member off && pushd "%~dp0" && setlocalcall "C:\Program Files\CARIS\BASE Editor\5.5\system\caris_env.bat"call "C:\sample_Testing\1_XYZ\import-points-loop.bat"set pythonpath=c:\Program Files\CARIS\BASE Editor\5.5\python\3.5python "C:\sample_Testing\2_Coverages_Upload\coverage_loader_sample.py" -L dba/sql@localhost -D sample_TEST -o surfac -t .csar -f "C:\sample_Testing\2_Coverages_Upload" -m "C:\sample_Testing\2_Coverages_Upload\Sample_Meta.csv" -vpopd && endlocalpauseNested batch script: Import-points-loop.bat@Former Member off && pushd "%~dp0" && setlocalcall "C:\Program Files\CARIS\BASE Editor\5.5\system\caris_env.bat"FOR %%f in (*.xyz) do (carisbatch --run ImportPoints --input-format ASCII --input-crs EPSG:3395 --output-crs EPSG:32659 --info-file "C:\sample_Testing\1_XYZ\Example.info" "%%~nf.xyz" "C:\sample_Testing\2_Coverages_Upload\%%~nf.csar")popd && endlocalREM
Hi,This is a reproducible, simple issue where the performance is surprisingly bad. It is a follow-up to the case under this link, where initially a stoackoverflow issue occurred. The script below ran for 26 hours in over 8 cores at full calculation as is seen in the hardware statistics. Of course the object is "quite large"- but in similar operations with such an object it does not take so long. The generated size was 42.5GB in the 8 parquet files on HDFS. Here is the code:from pyspark.sql import SparkSessionfrom pyspark.sql.functions import randss = SparkSession.builder.appName("test_replication") \.config("spark.kryoserializer.buffer.max.mb", "2047") \.config('spark.sql.execution.arrow.pyspark.enabled', "true") \.config("spark.driver.maxResultSize", "16G") \.config("spark.driver.memory", "4G") \.config("spark.executor.memory", "16G") \.config("spark.dynamicAllocation.maxExecutors","8") \.config("spark.executor.instances", "2") \.config("spark.executor.cores
Hello,I am running since approx 4 weeks into unsolvable OOM issues, using CDSW, yarn cluster, pyspark 2.4.7 and python 3.6. It seems that I am making generally something fundamentally wrong. The following is a reproducible code example to mimic one issue in its simplest form: A large spark dataframe is created and then it is supposed to be written in a parquet file on HDFS. The HDFS-disk has more than enough space. Below I took partitioning out. However, coalesce as also partitionBy lead to same issues. Generating only a small file works (to rule out other possible issues). The reproducible code is the following. from pyspark.sql import SparkSessionfrom pyspark.sql.functions import randss = SparkSession.builder.appName("test_replication") \.config("spark.kryoserializer.buffer.max.mb", "2047") \.config('spark.sql.execution.arrow.pyspark.enabled', "true") \.config("spark.driver.maxResultSize", "2G") \.config("spark.driver.memory", "3G") \.config("spark.executor.memory
Hi,I'm using a PutDatabaseRecord with UPSERT statement for a PostgreSQL database, but the ON CONFLICT keys are not being quoted event with the setting "Quote Column Identifiers" set to true.ERROR:Batch entry 0 INSERT INTO "eventstorage"."2062d999b1f26bcb32c3f39d22bd34"(..., "FDZ_UUID", ...) VALUES (...) ON CONFLICT (FDZUUID) DO UPDATE SET (...) = (..., EXCLUDED."FDZ_UUID", ...) was aborted: ERROR: column "fdzuuid" does not existI've seen an JIRA issue regarding this behavior and it is as Resolved: https://issues.apache.org/jira/browse/NIFI-8043.I'm using nifi 1.17.0@MattWhosorry for pinging you directly, but you were the assignee of the jira ticket and I've seen you commenting other similar posts. Thanks
Hi,I am using CDP PC trial version with AWS for learning CDP PC and am trying to register an environment on CDP PC and repeatedly getting the below error. I am following https://www.cloudera.com/campaign/videos/cdp-setup.html (pdf booklet) Datalake creation failed. 1. Data Access Role (arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role) is not set up correctly. Please follow the official documentation on required policies for Data Access Role. Missing policies (chunked): arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir/* arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/ranger/audit arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn
I want to fetch the files based on wildcard(*xyz*.txt) in nifi and if files are not available then it will keep on looping again check whether the files are exist or not. Can you please suggest the solution in NIFI. Files are present in server.
now I am use Ambari2.7.5 and HDP3.1.5 to install my cluster, but I get some vulnerabilities. so where can I download the source code or jars of HDP3.1.5. please help me.
Hello, Recently we have noticed several ETL jobs receiving the following error: errorCode:10280, errorMessage:Error while compiling statement: FAILED: LockException [Error 10280]: Error communicating with the metastore), Query: create temporary external table `databasenamehere`.`tablenamehere_dmx_20230607074828418241275540` (TableAttributesHere TBLPROPERTIES ('transactional'='false').', vendor contact information: support.precisely.comConnect : (TBLCOUNT) 179 records were committed to target database "jdbc:hive2://servernamehere:10000;AuthMech=1;SSL=1;KrbHostFQDN=servernamehere;KrbRealm=domainhere;KrbServiceName=hive" table "tablenamehere", outof which 0 records were loaded successfully and 179 records were rejected We are on Hive 3.1.3000.7.1.7.1000-141 and the metastore is MySQL version mysql Ver 14.14 Distrib 5.7.37, for Linux (x86_64). We have also noticed recently that queries are taking much longer than normal to do basic tasks, and wonder if there is somethin
I am on a kerberized CM 7.7.1 private cloud and after watching the "Cloudera Search Training" tutorial, I'd like to create a collection with SolrCloud.I created a configuration and was able to upload it successfully to Zookeeper: kinit <my_username>solrctl instancedir --create testcollection <my_local_path> This looks alright to me: solrctl instancedir --list testcollectionlocalFSTemplatemanagedTemplatemanagedTemplateSecureschemalessTemplateschemalessTemplateSecure Calling: solrctl collection --create testcollection -s 1 Gives the error:Error: A call to SolrCloud WEB APIs failed:(there's nothing more) I suspect it is because no secure connection is being used. I therefore try it with: solrctl --solr https://<node with solr>:8995/solr collection --create testcollection -s 1 However, Solr cannot find the configuration:content-security-policy: default-src 'none'; base-uri 'none'; connect-src 'self'; form-action 'self'; fo
Unable to up the Nifi Cluster Mode with OIDC IntegrationHave Existing Cert-Manager v1.5.3 (Let's Encrypt)Currently using OIDC with Cluster ModeNifi Image version 1.19.1Pull nifi release v1.1.3Kong gateway/proxy version 3.2My value.yaml (Note: I only put necessary info value not a full value.yaml) replicaCount: 2properties: externalSecure: false isNode: true httpsPort: 8443 webProxyHost: nifi-cluster.domain.name:443 service: type: ClusterIP httpsPort: 8443 # nodePort: 30236 annotations: kubernetes.io/ingress.class: kong-nginx konghq.com/protocol: "https"ingress: enabled: true className: kong-nginx annotations: cert-manager.io/cluster-issuer: letsencrypt-xxx konghq.com/strip-path: "false" konghq.com/plugins: session konghq.com/session-affinity-cookie: my_session_cookie konghq.com/protocols: "http,https" # Enable both HTTP and HTTPS konghq.com/session-affinity: "true" # Enable session affinity tls: - secretName: nifi-certificate hosts
Hello,I'm trying to read messages from a IBM MQ. for that I have set up a JMSConnectionFactoryProvider Service, and configured a ConsumeJMS proccesor.The JMSConnectionFactoryProvider is connecting to the IBM Queue successfully, and even the ConsumeJMS outputs a file from the queue when I run it, but the file's content is corrupted. it is suppose to be readable, but it comes out as gibbirish.I noticed that the output flow file has 2 interesting header properties:JMS_IBM_Character_Set: IBM424JMS_IBM_Encoding: 785 Maybe it has something to do with the corrupted output? I tried to specify IBM424 in the 'Chracter Set' property on ConsumeJMS proccessor, but it had no effect.Thanks a lot!
Hi,kindly need your support in below, I have Text file, I have to convert it to byteArray and send it through REST API, I tried many ways to convert the file to byteArray, last one is:- ExecuteScript with below code:import base64import codecswith codecs.open('/path/to/input/file.txt', 'r', encoding='utf-8') as file:file_content = file.read()file_bytes = file_content.encode('utf-8')encoded_bytes = base64.b64encode(file_bytes)encoded_string = encoded_bytes.decode('utf-8')flowFile = session.putAttribute(flowFile, 'fileContent', encoded_string)session.transfer(flowFile, REL_SUCCESS) but still get error: utf-8' codec can't decode byte 0xff in position 0: unexpected code byte in <script> at line number 6 REST API Config:HTTP Method: POSTAPI URL:.....Request Content-Type: application/octet-streamauthentication User: username authentication password: password we applied it in VB.net as below:Function CallApi(APIURL As String,filepath As String) As StringDim req A
Hello NiFi team, I'm new to NiFi, I'm trying to get data from Elasticsearch using QueryElasticsearchHttp however I have a self_signed certificate I'm not sure how to use that if you can give some examples or just some basic steps I can try to set that up on my end. Thank you in advance for the help.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.