@ANMAR    Try with this regex     (?:\"key2\"\\s*:\\s*)(.*?),   This will extract only the value of "key2" key i.e "value2" - if you don't need quotes to be extracted then use this regex     (?:\"key2\"\\s*:\\s*)"(.*?)",   This will extract only the value of "key2" key i.e value2 ... View more

@ANMAR    Try with this regex in ExtractText processor. (?:"x":.\w+?)(\d+) This regex will extract only the digit in "x" key and adds that value for "y" key in ReplaceText processor. ... View more

@ANMAR  You need to use ExtractText processor and matching regex to extract only the integer value. --------------------------------------------- Add new property in ExtractText processor  val     (\d+)      - Then use ReplaceText processor with below configs: Search Value } Replacement Value ,"y":"${val}"} Character Set UTF-8 Maximum Buffer Size 1 MB Replacement Strategy Literal Replace Evaluation Mode Entire text - By using Replacetext processor we are extracting the value and adding "y" key with the extracted value. -------------------------------------------- Input data:     {"x":"avc123.abc.com"}      Output:     {"x":"avc123.abc.com","y":"123"}       ... View more

@RandomT  You can check compression on .avro files using avro-tools bash$ avro-tools getmeta <file_path>  For more details refer to this link - sqlContext.setConf //sets global config and every write will be snappy compressed if you are writing all your data as snappy compressed then you should use this method. - In case if you are compressing only the selected data then use  exampleDF.write.option("compression", "snappy").avro("output path") for better control over on compression. ... View more

@Erkan ŞİRİN Did you try using yarn-client (or) yarn-client instead of yarn in .master. If error still exists then add spark-yarn.jar to the build path, then try to submit the job again. Refer to this link for more details about similar issue. ... View more