About JOLTEnjoyer

JOLTEnjoyer · ‎08-06-2024

This guide provides a comprehensive overview of using JOLT within Apache NiFi. It is designed to help developers and data engineers efficiently transform JSON data, offering clear instructions and practical examples. This is an unofficial resource, created out of a personal initiative to fill the knowledge gap. The guide includes: Introduction to JOLT JOLT (JSON-to-JSON Transformation Language) enables flexible and powerful JSON data transformations. It allows you to restructure, filter, and modify JSON objects to fit specific requirements. Getting Started Begin with the JoltTransformJSON processor in NiFi. The guide provides setup instructions and tips for using the JOLT editor window to streamline your transformation tasks. Using the JOLT Editor The JOLT editor in NiFi includes features for creating and debugging transformation specifications. Our guide explains how to use these features effectively. Writing JOLT Specifications We explain the structure of JOLT specifications, focusing on the operation and spec fields. Various operations, including shift, default, remove, sort, and cardinality, are illustrated with practical examples. Advanced JOLT Operations Advanced operations like modify-default-beta and modify-overwrite-beta are covered in detail, providing insights into complex JSON modifications. Utilizing Wildcards Learn to use wildcards in JOLT for dynamic transformations. The guide covers the use of asterisks, at signs, hash signs, and ampersands with examples. Contributing to the Guide Contributions are well welcomed to enhance this guide. If you have suggestions, corrections, or additional examples, please submit them via the GitHub repository. Your input can help improve the resource for the entire community. Conclusion By following this guide, you will gain a comprehensive understanding of using JOLT in NiFi for efficient JSON transformations, enhancing your data processing workflows. For full details, examples, and more, read the complete guide here. DISCLAIMER: An external user contributed to this article. Cloudera may not verify that the steps may be applicable for all use cases and may be very specific to a particular distribution. Please follow with caution and at your own risk. If needed, raise a support case to get confirmation.

JOLTEnjoyer · ‎06-22-2024

Hi @Thar11027 , the spec here seems really simple: [ { "operation": "shift", "spec": { "*": { "PortfolioSharing": { "*": "[]" } } } } ] Tell me if you have any doubts

JOLTEnjoyer · ‎06-06-2024

@Thar11027 yes, but it will be more complex: you can put a shift operation and using the "*" wildcard. So if for example you do this: [ { "operation": "shift", "spec": { "*": { "*/*": "[&1].&", //with a '/' in the middle somewhere "/*":"[&1].&", //with a '/' at the start followed by something "*/":"[&1].&" //with a '/' at the end following something } } } ] You will obtain all the desired fields, so it will be more easy to do the manual substitution(try it on the jolt demo site). Know that if you want to make it full automated, it will be a little more difficult, because then you would have to manipulate the string of the field. If you are interested in that I really suggest you to look at the last example on the guide I already sent you (My Guide). (I don't know if maybe it will be more appropriate to open another question about this other problem, because the topic changed and maybe if someone with the same problem is searching for it, it can be found)

JOLTEnjoyer · ‎06-06-2024

@Thar11027 to do that, you will need to add a modify-default-beta and a remove operations. I'll give you an example here: [ { "operation": "modify-default-beta", "spec": { "*": { "ResidentNonresident": "@(1,Resident/Non-resident)", "NationalityCountryofIncorporation": "@(1,Nationality/CountryofIncorporation)" } } }, { "operation": "remove", "spec": { "*": { "Resident/Non-resident": "", "Nationality/CountryofIncorporation": "" } } } ] Here you can see I've used only a few fields just to show you the logic: you have to manually assign (in the first spec) the value of the "wrong" field to the corrected field, and then at the end, remove all the old fields that now are useless. You can test your jolt on this site that allows you to play with the specifications all you want: jolt-demo.appspot.com/ and if you have any doubts, check the guide i already mentioed in my first reply.

JOLTEnjoyer · ‎06-05-2024

Hi @mstfo , Firstly it seems that the output you have after the replacetext processor isn't really a json, because of the missing commas (' , ') between the fields. If your output is exactly like this, it will be necessary to add those missing commas so then you can get a json like this: [ { "Name": "Gta V", "Type": "Xyz", "content": "{\"Game\":{\"Subject\":[{\"Time\":{\"@value\":\"201511021057\" }}]}}" } ] To obtain a json like this you can either: try to change the logic when you get the data from the database; try to add the commas manually. The first option is the quicker solution, but there is the possibility that maybe you can't change the input you receive from the db, so perhaps you have to manually change it yourself. Anyway, in case you manage to obtain a correct json from your input, then you can manipulate the string using the functions in the beta operations in jolt (like the =split()) and then reassemble the wanted json. I wrote a guide and in the very last example i did, there is a similar case, so if you want go there and give it a look -> JOLT guide .

JOLTEnjoyer · ‎06-05-2024

Hi, I've seen your jolt and i made some correction to make it work, here is what i did: [ { "operation": "shift", "spec": { "*": { "CustomFields": { "*": { "@(0,Value)": "[&3].@(0,ViewName)" } }, "*": "[&1].&" } } } ] I basically put everything in an array, and I removed the "value" part, because it wan't necessary. If you have any doubts on how to use the Jolt, I have written a simple guide on how to use it here: My guide Hope i helped

Online	Offline
Last Visited	‎09-07-2024 05:15 AM

Member Since	‎06-05-2024 01:16 AM
Last Visited	‎09-07-2024 05:15 AM
Posts	6
Kudos received	4

Cloudera Community

Re: JOLT TRANSFORMATION returns unexpected result

Re: JOLT TRANSFORMATION returns unexpected result

JOLT Guide for Apache NiFi

Re: JOLT returns UNexpected array

Re: JOLT TRANSFORMATION returns unexpected result

Re: JOLT TRANSFORMATION returns unexpected result

Re: How can i parse string nested json with Nifi?

Re: JOLT TRANSFORMATION returns unexpected result