@Julià Delos, The issue is as we are having same file name for the 2 output files to resolve this issue you need to change the file name in Update Attribute processor By adding property as filename ${UUID()} Configs:- So we are changing the filename of the flowfile to UUID as uuid is unique, by using uuid as your filename you wont get any issues in PUTFile processor. ... View more

@Julià Delos You need to split the json array first by using SplitJSON processor:- JsonPath Expression property as $.* Configs:- Input:- [ { "id": "6935895746", "type": "PushEvent", "actor": { "id": 32568916, "login": "bigajwiktoria", "display_login": "bigajwiktoria", "gravatar_id": "", "url": "https://api.github.com/users/bigajwiktoria", "avatar_url": "https://avatars.githubusercontent.com/u/32568916?" } }, { "id": "6935895745", "type": "PushEvent", "actor": { "id": 463230, "login": "taylorotwell", "display_login": "taylorotwell", "gravatar_id": "", "url": "https://api.github.com/users/taylorotwell", "avatar_url": "https://avatars.githubusercontent.com/u/463230?" } } ] Output:- As in the above array we are having 2 messages so split json processor results 2 flowfiles ff1:- { "id": "6935895746", "type": "PushEvent", "actor": { "id": 32568916, "login": "bigajwiktoria", "display_login": "bigajwiktoria", "gravatar_id": "", "url": "https://api.github.com/users/bigajwiktoria", "avatar_url": "https://avatars.githubusercontent.com/u/32568916?" } } ff2:- { "id": "6935895745", "type": "PushEvent", "actor": { "id": 463230, "login": "taylorotwell", "display_login": "taylorotwell", "gravatar_id": "", "url": "https://api.github.com/users/taylorotwell", "avatar_url": "https://avatars.githubusercontent.com/u/463230?" } } Then use Evaluate Json Path processor:- in this processor we are extracting the json message values to attributes. Change property Destination to flowfile-attribute Add properties actor-login $.actor.login id $.id type $.type Configs:- Replace Text processor:- Now we are replacing the entire json message with our extracted attributes in Replace Text processor. Search Value (?s)(^.*$) Replacement Value "${id}","${type}","${actor-login}" Character Set UTF-8 Replacement Strategy Always Replace Evaluation Mode Entire text Configs:- now your flowfile will have your required output as content. Flow:- SplitJson(split relation) //splitting Json Array to individual messages-->EvaluateJSONPath(Match Relation) //extracting the required values from json message and adding them as flowfile attributes --> Replace Text (Success) //Replacing flowfile contents with the required values If the answer addressed your question, Then Click on Accept button below, that would be great help to community users if they are facing similar kind of issues. ... View more

@VINAYAK DORNALA If you don't want to pass last value on every run, then create sqoop job as sqoop job will store the last value in metastore and pass the last value when we run the job again. --incremental lastmodified:- Creating sqoop job:- sqoop job --create test -- import --connect 'jdbc:mysql://quickstart:3306/retail_db' --username retail_dba --password cloudera --table orders --split-by order_id --target-dir /user/sqoop/orders --check-column order_date --merge-key order_id --incremental lastmodified --as-textfile sqoop job --list //list out created sqoop jobs sqoop job --exec test //execute sqoop job sqoop job --delete test //delete sqoop job last modified mode works with merge-key, this mode compares the existing old data in the directory with the new data, If same merge key appears in the old data then sqoop merges that old data with the new data in reducer phase, then the new data after merge will be written to the target directory. when we creates sqoop job then it will imports the data as incrementally for the first time run sqoop imports all the data and stores the last value in metastore and for the next run sqoop make use of metastore last value only imports newly added data only. Lastmodified with merge-key argument will work with the existing target directory, If we won't mention merger-key argument then sqoop will throw an error if the target directory already exists. --incremental append:- sqoop job --create test -- import --connect 'jdbc:mysql://quickstart:3306/retail_db' --username retail_dba --password cloudera --table orders --split-by order_id --target-dir /user/sqoop/orders --check-column order_date --incremental append --as-textfile Append mode works with the existing directory(if directory not exists then creates directory), Stores the last state value in metastore. For the second run of sqoop job only imports the newly added data after the last state value and creates new file in the target directory. In this mode mapper is going to be initialized because we are not mentioning any merge key. You can decide which mode will be best fit for your use case. Note:- "--" and a space before the import creating sqoop job. Use Options file to store credentials (user name, password and connect string) and pass it as as a parameter to --options-file. Creating options file in sqoop ... View more

@Pallavi Ab As per your Logs Caused by: java.io.IOException: PropertyDescriptor PropertyDescriptor[Directory] has invalid value /user/cmor/kinetica/files/sample_b.csv.The directory does not exist. Can you check is the above directory exists in HDFS by using below command bash# hdfs dfs -test -d /user/cmor/kinetica/files bash# echo $? bash# hdfs dfs -test -e /user/cmor/kinetica/files/sample_b.csv bash# echo $? //if echo returns 0 file or directory exists //if echo returns 1 file or directory exists Make sure the path in the Directory property is correct and run the processor again. Usage of hdfs test command bash# hdfs dfs -test -[defsz] <hdfs-path> Options:<br>-d: f the path is a directory, return 0.<br>-e: if the path exists, return 0.<br>-f: if the path is a file, return 0.<br>-s: if the path is not empty, return 0.<br>-z: if the file is zero length, return 0. ... View more

@Pallavi Ab once make sure your file is in the directory and Nifi has permissions to the directory. I am not sure about your get hdfs configurations, take a look on the below configs and configure your processor same configs in the screenshot shown below. Configs:- Important Property is Keep source file configure this property as per your needs. Keep Source File false true false Determines whether to delete the file from HDFS after it has been successfully transferred. If true, the file will be fetched repeatedly. This is intended for testing only. ... View more

@Mohamed Hossam I think you are missing space in search value property. Use the below regex in search value property ^(.*?) (.*?) IP (.*?) > (.*?) .*$ (or) ([^\s]+)\s([^\s]+)\sIP\s(.*)\s>\s([^\s]+).* Use any of the above regex's. Config:- ... View more