Member since
05-06-2019
5
Posts
0
Kudos Received
0
Solutions
06-05-2019
05:32 PM
Problem solved.....the processor was set to "Primary Node Only", once I set it to use "All nodes", the flow started processing files again. Used this link to solve my issue: https://community.hortonworks.com/questions/189649/nifi-flow-files-stopped-and-the-processors-are-run.html
... View more
06-05-2019
04:04 PM
Have a process that I started yesterday. It was working and processing flow files. It is pulling from SQL and pushing to SQL and the data pull has roughly 15 million rows. When I started yesterday it had 29 files in the queue (where is says 27 now). It did process the initial 2 flow files. This morning I come in and it stopped processing the files right before the ExecuteSQL processor. I read on another thread about number of threads, it was set to 20 and I increased to 30 and then 50 with no change. I did stop the processor and restart, no luck. The server is clustered but I am only using it on a single node. Attached are images of the processing and how it essentially stopped, the queued up files, the schedule and properties of the executesql processor. Thanks in advance!
... View more
Labels:
05-21-2019
07:49 PM
Trying to make a generic template that uses some input variables (as what columns to pull) and then put them into a sql destination table. I have it working.....if it's strings but recently was using my template and pulling in Maximo data (don't get me started).....well it will have INT columns and then have nulls in them which messes up everything. So plan was to convert the incoming Avro schema to an all string element avro schema (which I can do)......but my issue is that within the controller, I have to specify each column name in the AvroRecordSetWriter. Is there a way to say field1 is "Field1" AND copy over the value from the original avro schema index[0] element and put it as Field1 in the new all strings Avro schema? In the end....I'm trying to convert incoming schema to have ALL strings and to have the schema/flowfile has element names as "field1", "field2", etc. Thanks in advance!
... View more
Labels:
05-07-2019
02:03 PM
I have thought about it but using the execute sql as it's a glorified Up-Sert statement comparing on a business key hash value that is created during the flow. The person that initially created this flow did a good job but there are many manual entries that must be completed when doing a copy and paste for the template of this flow. Was looking to remove as many of the manual entries that must be changed.
... View more
05-06-2019
10:04 PM
New to Nifi but here is my thought, I am trying to make a template for a flow. I would prefer to create as many of the processors to read everything from the variables at the flow level. Essentially....the template would be used to copy data from a sql db to another sql db....only changes would be what server, what copy from/copy to tables by upserting, table columns etc. Will also be adding additional fields for a business and content hashes. Because of the sql and dynamic-ness, I have to use sql.args.$.value to use in the insert statement. So thought is to grab the values and store in a value called "Field##" where ## is the number from the "SourceField" variable (must be in order)....then use this "Field##" to populate the sql args value and then use another variable to populate the destination table (most likely a different column name). So from my thoughts, in the attributes of the flow file, I would have the following....setting from the initial source db query. Field1 "test1" Field2 "test2" In a groovy script I would get the number of "source variables" and then do a for loop and assign all of the sql args from the Field## attributes. Something like: for (int i = 1; i < countargs; i++) { def myType = "sql.args." + i + ".type" ff = session.putAttribute(ff, myType, '-9') //setting it to use a string type def myValue = "sql.args." + i + ".value" ff = session.putAttribute(ff, myValue, '${Field" + i + '}') } The creating of the sql.arg with a # works as it's just a concat string. But I would like the for loop to set the following: sql.args.1.value = test1 sql.args.2.value = test2 Any thoughts appreciated.
... View more
Labels: