Can you provide more information on your dataflow ? let's say you are using GenerateFlowFile to create the json Kafka output, what happens next? How are you enriching the data and what kind of processor where you are using the json reader\writer service that is causing the error? I need to see the full picture here because When I use same  json you provided in GenerateFlowFile processor and then passed it to QueryRecord with the same Json reader\writer service configuration, it seems to be working! ... View more

Hi @PradNiFi1236 , How are you adding the new fields? You Json appears to be invalid as provided. ... View more

Hi , Can you provide more explanation\screenshot of your dataflow and the configuration set on each processor\controller service? Also if you can provide sample data that can be converted to parquet which can then reproduce the error that would be helpful as well. Thanks ... View more

Hi, Can you elaborate more about the API call and what method its using? if you also can provide how the API works using postman or curl that would be helpful too. Just FYI, if you are trying to send json body with Get method its not going to work using this processor and you have to write custom code using ExecuteScript to work around it. ... View more

Hi, Can you provide sample of this data highlighting what the problem and how do you expect to solve? Also what kind of processor\service are you trying to use to parse this data? ... View more

Hi @Syed0000 , Sorry for the delay. You json is quite complex and very nested which make jolt very hard to write. You probably know your data very well and my recommendation before writing jolt for such Json is that you try to simplify it first by stripping un need blocks and\or flattening the structure where the transformation is not going to be on the same nested level. I honestly tried but when the jolt got so nested it got harder to reference the upper fields and upper array indexes to maintain the same array position and then maintain the required grouping of fields, and may be that is the draw back of using jolt spec in these scenarios (without prior simplification of course :). This leads me to the second option, which I think  I have mentioned to you in prior post regarding using JSLT Transformation instead.  JSLT is better option in these cases because you can traverse the structure much easier , and since you have some condition on how to set fields and values this also would be easier to achieve with some expression language like if-else  . For example if we take the transformation to create  catalogCondition strucutre which seems to the most complex, here is how the jstl  looks like: let paths ={ "paths":[for(.tiers) { for(.conditions) "catalogCondition": { "category": [for(.conditionsMerch) { "include": if(.excludeInd==0) "yes" else "no", "categoryList":[.webCategoryId] } if(.merchandiseLevel!=5 and not(.keyItemNum))] } + { "products" : [for(.conditionsMerch) { "include": if(.excludeInd==0) "yes" else "no", "productList":[.keyItemNum] } if(not(.webCategoryId))] } } ] } { "pricePaths": $paths } I know this might look a little intimidating at beginning but its not near as bad as when you try to do it with jolt. I understand that jstl is bit of learning curve but I believe it would save you tons of time long term specially when dealing with long, complex and nested Json transformation. I know this is not the answer you were looking for but hopefully this can help you when facing other json transformation challenges in the future. Good luck.     ... View more

Hi @fisblack , Welcome to the community. Your transformation is kind of odd. Usually transformation is driven by existing values or structure  even when generating new fields with defaults. I have never seen a case where newly generated values drive transformation of existing ones. I assume you are getting the list of courseSubCategory from somewhere and the  script is rather dynamic utilizing some expression language & attributes. if you provide some context to how the list is coming about and what are you trying to achieve , maybe we can provide you with more accurate solution. There are multiple ways of solving your problem as is - assuming you always want to replicate an existing field(s) against  newly generated 3 elements array with default  values of 1,2&3- , here is one of them: [ { "operation": "shift", "spec": { "*": "existing.&", "#1|#2|#3": "courseSubCategory" } }, { "operation": "shift", "spec": { "courseSubCategory": { "*": { "@(2,existing)": "[&1]", "@": "[&1].courseSubCategory" } } } } ] If you want to make this more dynamic in terms of number of "courseSubCatgory", you can construct the array assignment string "#1|#2|#3" in some upstream processors and store the value as flow attribute (let says its called the same) , then you can make it more dynamic by using the flowfile attribute where the assignment of the courseSubCategory in the first shift will look like this: "${courseSubCategory}": "courseSubCategory" Hope that helps. If it does help, please accept the solution. Thanks       ... View more