Member since
06-08-2017
39
Posts
2
Kudos Received
0
Solutions
02-01-2019
09:25 AM
thanks @shu its working like charm @Shu
... View more
01-16-2019
12:06 PM
if possible please share processor configuration sreen-shot
... View more
01-16-2019
12:04 PM
Hey still getting error, I am also using same nifi-1.8.0 version
... View more
01-16-2019
09:24 AM
Hi Shu, I download the above template and imported into nifi-1.8.0. when try to drag that template into nifi canvas it gives error. Thanks in advance
... View more
01-15-2019
05:23 AM
1 Kudo
Hello, I am trying to convert multiple json file into csv file in that, In source folder i am getting different json file with different schema.
need to convert json file to csv file with dynamic schema mapping and put that file into destination folder. I have tried bellow flow, 1) getFile processor- to get file from source. 2) UpdateAttribute - set attribute schema.name : record 3) inferAvroSchema -configure
Schema Output Destination : flowfile-attribute
Input Content Type : json 4)convertRecord processor -
Record Reader :JsonTreeReader
Record Writer :CSVRecordSetWriter 5) putFile : put file into destination folder.
But in above flow getting error in convertRecord Processor,
Error:
Failed to process StandardFlowFileRecord[uuid=69603ac0-ac43-44b5-82f3-5c1c89e4abaf,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547526092906-1, container=default, section=1], offset=878, length=878],offset=0,name=orders1.json.avro,size=878]; will route to failure: Failed to create schema from the Schema Text after evaluating FlowFile Attributes. Please find the attached screen-shot of configuration of convert record processor. Thanks
... View more
Labels:
- Labels:
-
Schema Registry
11-15-2018
06:32 AM
Hi Matt, Are you listing and fetching the new files or just listing them? >>I want only the list of file names(new and updated) Are you running this listing using a Cron scheduling strategy once a day? >>Yes, once in a day (9AM). what is the range on number of files that may get listed each day? >>The file range in not fixed (stuck here only) Can you suggest something, if range on number of files is not fixed then how to list out all file names. Thanks
... View more
11-14-2018
05:29 AM
Hello, In file watcher scenario, everyday i am getting new and updated file in the local folder. The incoming file count is varies in everyday. Need to send email notification with updated and new file name list with specific schedule time. Using Listfile processor, I am getting all new and updated file name. How to list out all filename in one file and send that file using email notification. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
10-11-2018
05:05 AM
Trying the same using curl command curl -v -X POST \
--insecure \
--cert /home/test/certs/laven-ie-prod-laven-integration.crt \
--key /home/test/certs/laven-ie-prod-laven-integration.key \
--header "Content-Type: application/json" \
--data @/tmp/input/input.json \
https://integration-laven-ie-prod.laven.com/lm/test/ file is posted successfully trying in Invokehttp processor but getting same error...
... View more
10-11-2018
04:37 AM
Hi, H have created flow, 1) Get Json file using getfile processor and post that file using Invokehttp processor(https://integration-laven-ie-prod.laven.com/lm/test/). 2) I have 3 files laven-ie-prod-laven-integration.crt laven-ie-prod-laven-integration.key laven-ie-prod-laven-integration.p12 3) Can someone help me how to use that file in Invokehttp processor, a)When i am using .crt and .key file that time StandardSSLContextService show keystores properties is invalid b)When using .p12 file that time show error Routing to Failure due to exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target can someone help me to figure out this issue. Thanks in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
09-19-2018
12:05 PM
Hi, I want to split that file name('_') and store split values in variable for further operation. The file name is combination of various parameter. file Name = BBAS_2018091873_COEP_20180918141337_Cleaned_txt_scorable.csv Stores the values in variable like, like
a=BBAS ,b=2018091873 , c=COEP, d=20180918141337 , e=Cleaned, f=txt , g=scorable.csv Thanks in advance!!!
... View more
Labels:
- Labels:
-
Apache NiFi
09-07-2018
07:37 AM
Hello, I have a two flow, from every flow get one flow file. May be there some delay between getting file from different flow How to compare one flow file attribute with other flow file attribute. After that send mail depending on success or failure condition Is there any easy way to do that in Apache NIFI. Thanks in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
08-30-2018
04:40 AM
Hello, I have Fetch Open source API data using GetHTTP processor and save as text file. Is there any way to auto increment page because size of data is more. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
08-23-2018
01:04 PM
Hello, I am new in Apache NIFI, I have to import CRM data into hive. what is possible way to import data into hive. Thanks in advance
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
07-16-2018
07:44 AM
Hello, I have designed data flow to fetch data from twitter and put in kafka. Getting data in json format but not user specific. 1) That processor stream live Twitter data or old data? 2) If processor stream live data then how to stream old data ? 3) How to fetch user specific incremental data using gettwitter processor ? 4) If my flow failed some times then how to stream that specific time and live data using gettwitter processor ? Thanks.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
05-14-2018
07:05 AM
Hello, I have created a flow like ExecuteSQL->putfile on success relation if ExecuteSQL processor failed I have to capture the error of ExecuteSQL processor and send email notification to user with error file Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
02-23-2018
05:34 AM
thanks shu, it's working and thats helpfull for me.
... View more
02-22-2018
05:21 AM
thanks shu, I tried your approach it is working fine with success relationship that time I received the mail but if any query failed that time not receive any mail.
... View more
02-21-2018
01:48 AM
Hello, I have design the flow in that I executed the query like CTAS,Insert into, truncate table in PutHiveQL Processor. Depending on query failed or success I have to send email but PutHiveQL not generated any metadata. I have set rollback property true but that data goes on queue. How to send email depending upon PutHiveQL Processor success or failure.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Pig
02-18-2018
11:59 AM
1 Kudo
Hello, I have design the nifi flow like this selectHiveQL->convertAvroToJson->SplitJson->EvaluateJsonPath->RoutOnAttribute(depends on condition the query goes on) ->presto or ->ReplaceText->putHiveQL hive table is query_id job_type query 1 presto create table hive.db.table as select * from mysql.db.table 2 hive ALTER table db.table set tblproperties ("orc.create.index"="true") 3 hive ALTER table db.table set tblproperties ("transactional"="true")
4 presto create table hive.db.table_stage as select * from mysql.db.table selectHiveQL-select query on hive table contain query_id, job_type and query convertAvroToJson->convert avro format into json SplitJson-split the record EvaluateJsonPath-set the query_id, job_type and query attribute
RoutOnAttribute-depending on condition of job_type that query is executed in presto or hive presto-custom processor executed the query ReplaceText-set the putHiveQL query putHiveQL-execute hive query complete flow-flow.png problems: 1) How to stop the flow if any query failed from any processor. 2) I have tried to connect the presto failure to presto processor that time if query failed the next query not goes to presto processor but failed query is continuously executed in presto processor. 3) if the query is alternate and presto query failed that time hive query goes to hive processor(backpressure of hive failure also set to 1). 4) How to send email notification if the flow is successfully completed or failed? Please provide your suggestions . Thanks in advance .
... View more
Labels:
- Labels:
-
Apache NiFi
09-27-2017
08:31 AM
I have custom processor like ExecuteSql . after executing query i get resultset. now i want add new values like queryid , starttime , and end time in resultset and send in flow file like normal result set. How i can do this. please advice thank you in advance
... View more
Labels:
- Labels:
-
Apache NiFi
09-25-2017
08:52 AM
after executing query i get resultset. now i want add some value into resultset like query id , query start time , query end time in resultset, so i can get in nifi flow.
... View more
09-25-2017
05:57 AM
Hello, I have created custom processor same like ExecuteSQL. after executing query all data get in resultset . I have to add one external string value . String queryID="123456" -it may be static or dynamic. Where I have to add that value in resultSet or flowfile?? and how to add that value?? using convert AvroToJson I have take that queryID using EvaluateJsonPath processor and use further operation. the code is same like below, Can someone point me at an example to get me going -------------------------------------------------------------------------------------------------------------------- fileToProcess = session.write(fileToProcess, new OutputStreamCallback() { @Overrid public void process(final OutputStream out) throws IOException { try { logger.debug("Executing query {}", new Object[]{selectQuery}); final ResultSet resultSet = st.executeQuery(selectQuery); final JdbcCommon.AvroConversionOptions options = JdbcCommon.AvroConversionOptions.builder() .convertNames(convertNamesForAvro) .useLogicalTypes(useAvroLogicalTypes) .defaultPrecision(defaultPrecision) .defaultScale(defaultScale) .build(); nrOfRows.set(JdbcCommon.convertToAvroStream(resultSet, out, options, null)); } catch (final SQLException e) { throw new ProcessException(e); } } }); long duration = stopWatch.getElapsed(TimeUnit.MILLISECONDS); // set attribute how many rows were selected fileToProcess = session.putAttribute(fileToProcess, RESULT_ROW_COUNT, String.valueOf(nrOfRows.get())); fileToProcess = session.putAttribute(fileToProcess, RESULT_QUERY_DURATION, String.valueOf(duration)); fileToProcess = session.putAttribute(fileToProcess, CoreAttributes.MIME_TYPE.key(), JdbcCommon.MIME_TYPE_AVRO_BINARY); logger.info("{} contains {} Avro records; transferring to 'success'", new Object[]{fileToProcess, nrOfRows.get()}); session.getProvenanceReporter().modifyContent(fileToProcess, "Retrieved " + nrOfRows.get() + " rows", duration); session.transfer(fileToProcess, REL_SUCCESS); -----------------------------------------------------------------------------------------------------------------------------------------------
... View more
Labels:
- Labels:
-
Apache NiFi
09-14-2017
01:25 PM
Hello, My NIFI flow is successfully run After that I have seen that the In,Read/Write,Out in processor view But after 5 min the all information is flush from processor view I check the processor code, there is no code for flush information after 5 min that is happen in every processor I have to change that time, how can I do that
... View more
- Tags:
- Data Ingestion & Streaming
- nifi-controller-service
- nifi-processor
- nifi-state
- nifi-streaming
- processor
Labels:
- Labels:
-
Apache NiFi
08-23-2017
11:25 AM
Hi Matt, I read your blog, I understand your groovy script. but in my situation which processor I use,what is sequence of processor when i used the Executescript processor. what is the changes in the script that provided in blog.
... View more
08-22-2017
06:22 PM
thanks Matt for the reply, can I used the attribute to store the query_startTime= query_endTime after the first query execution completed. can I insert the attribute query_startTime and query_endTime after the second query execution and vice versa. Please kindly guide me for the same.
... View more
08-22-2017
02:09 PM
Hello, I have Designed the bellow data flow 1 - -> 2 - ConvertAvroToJSON -> 3 - splitJSon -> 4 - EvaluateJsonPath -> 5 - UpdateAttribute -> 6 ExecuteSQL -> 7 - replaceText -> 8- putSQL. 1 - 2 - 3 : query your table and split by row. You will have one flowfile by row. Use $ for the JsonPathExpression in the split processor. 4 - Retrieve the query_id and the query_execute in two attributes of each flowfile (values of your properties are $.query_id and $.query_execute) 5 - Create a new attribute query_startTime with the value ${now():format('HH:mm:ss') 6 - Execute the query ${query_execute} 7 - Write your insert query : INSERT INTO queryExecutionInfo VALUES (${query_id}, ${query_startTime},{${now():format('HH:mm:ss')); 8 - Execute your insert problem: In 5th step query_startTime for all query is same ExecuteSQL processor is required some time to execute the query. after the first query execute the value are inserted into queryExecutionInfo table sucessfully. but when second query execution completed the query_startTime same as first query_startTime and end time is different. I have to insert second query_startTime as first query_endTime and vice versa. Can someone point me at an example to get me going
... View more
Labels:
- Labels:
-
Apache NiFi
08-20-2017
04:18 AM
Thanks Quentin for reply, I designed the flow as you said,after the replace text the query_id,query_startTime,query_endTime is not inserting into queryExecutionInfo table . I attached the screen shot of replaceText and ExecuteSql processor. Please kindly guide me for the same.
... View more