Nifi JoltTransformationJson for Operation : modify-overwrite-beta to use concat function

srinivaspadala_ — Sat, 05 Nov 2016 12:22:42 GMT

The below input and script is working in http://jolt-demo.appspot.com/#andrewkcarter2 , But it is not working in Nifi JolttransformationJson . This example is concatenation of firstname and lastname .

I understood it is becuase of not having "Operation : modify-overwrite-beta" in existing Nifi - JoltTransformation plugin . Is it possible to get this function by custom ? If yes, Please provide the process to implement . or let us know the process to ultilize it . Thanks .

@Yolanda M. Davis - Please through some light on this .Thanks .

Input :
{
	"data": [{
		"IDF": "123",
		"FirstName": "George",
		"LastName": "Kosher",
		"PaymentInfo": [{
			"Type": "ABC",
			"Text": "Soft",
			"Amount": 3
		}, {
			"Type": "ABC",
			"Text": "Text",
			"Amount": 5
		}],
		"PaymentCard": [{
			"CardNumber": "12345",
			"CardType": "Credit"
		}, {
			"CardNumber": "56789",
			"CardType": "Credit"
		}]
	}, {
		"IDF": "456",
		"FirstName": "Mill",
		"LastName": "King",
		"PaymentInfo": [{
			"Type": "ABC",
			"InstructionText": "Hard",
			"Amount": 6
		}, {
			"Type": "ABC",
			"InstructionText": "Text",
			"Amount": 8
		}],
		"PaymentCard": [{
			"CardNumber": "12345",
			"CardType": "Credit"
		}, {
			"CardNumber": "56789",
			"CardType": "Credit"
		}]
	}]
}


Script : 
[
  {
    "operation": "shift",
    "spec": {
      "data": {
        "*": { // data arrayf
          "*": "data[&1].&", // pass thru stuff
          "PaymentInfo": {
            "*": {
              "Amount": "data[&3].Amount[]",
              "Text": "data[&3].PaymentText[]",
              "InstructionText": "data[&3].PaymentText[]"
            }
          },
          "PaymentCard": {
            "0": {
              "CardType": "data[&3].CardType"
            }
          }
        }
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "data": {
        "*": { // data array
          "Name": "=concat(@(1,FirstName),' ',@(1,LastName))",
          "Amount": "=sum" // should work on Jolt 0.0.24
        }
      }
    }
  },
  {
    "operation": "remove",
    "spec": {
      "data": {
        "*": { // data array
          "FirstName": "",
          "LastName": ""
        }
      }
    }
  }
]

Output : 


{
  "data" : [ {
    "IDF" : "123",
    "Amount" : [ 3, 5 ],
    "PaymentText" : [ "Soft", "Text" ],
    "CardType" : "Credit",
    "Name" : "George Kosher"
  }, {
    "IDF" : "456",
    "Amount" : [ 6, 8 ],
    "PaymentText" : [ "Hard", "Text" ],
    "CardType" : "Credit",
    "Name" : "Mill King"
  } ]
}

Re: Nifi JoltTransformationJson for Operation : modify-overwrite-beta to use concat function

ydavis — Mon, 07 Nov 2016 10:09:08 GMT

Hi @srini,

NiFi does support the addition of custom transformations which can be referenced with a "drop in" jar. You could create your own transformation class, which extends the jolt library, that provides the functionality you need and be available to NiFi on the file system. Given that this is a new function in a later version of Jolt I would be careful using that the as a custom jar; that may cause a conflict but I haven't tested this myself to be sure.

Below are some use cases on how to apply custom functions in NiFi

Case 1 – Custom Transform Selected

In this case if the Custom option of Transform is selected then 1) a Custom Transform Class Name should be entered and 2) one or more module paths should be provided. The Custom Transform Class Name should be a fully qualified classname (e.g. eu.zacheusz.jolt.date.Dater). The Module Path can take a comma delimited list of directory locations or one or more jar files. Once these fields are populated the Advanced view will support validation and saving of the specification. A user can switch between transformation types in the UI but not custom class names & module paths.

Case 2 – Chain Transformation Selected with Custom Transformations embedded

In this case if you wants to use one or more transforms (that include custom transformations) in your specification then the Chainr spec option can be used with one or more module path provides. In this case the Custom Transform Class Name property is not required and would be ignored (since one or more custom transformations could be invoked in the spec). As in Case 1 the Advanced view will support validation and saving of specification if the required fields are populated for this case.

I hope this helps! Please let me know if you have any questions.

question Re: Nifi JoltTransformationJson for Operation : modify-overwrite-beta to use concat function in Archives of Support Questions (Read Only)

Nifi JoltTransformationJson for Operation : modify-overwrite-beta to use concat function

Re: Nifi JoltTransformationJson for Operation : modify-overwrite-beta to use concat function