@RAUI It depends on how your bashrc file is configured, if your path variable in bashrc file having the directory path that your script is in. Then you don't need to specify bash in command path argument ... View more

@RAUI Once you ingest the files into HDFS then only you need to trigger the spark application then using ExecuteStreamCommand processor would be the correct approach as this processor accepts incoming connection and triggers spark applications. if you are having more than one file that you are storing into HDFS then it would be better to use MergeContent processor after PutHDFS processor. Configure the MergeContent processor to wait for minimum number of entries (or) by using Max Bin Age..etc property because if you connect Success relation from PutHDFS to ExecuteStreamCommand processor as soon as first file written to HDFS then application is going to be triggered from NiFi we are not waiting for all the files stored into HDFS directory. Triggering shell script using ExecuteStreamCommand processor configs: ... View more

@Raja M Probably the issue with the Region Configuration that you have used in PutSNS processor configs, Check Topic/Target arn is pointing to(region) then change the Region property value according to it. ... View more

@Yu-An Chen The issue is relationships that are feeding from FetchHDFS to mergeContent processors are commas.failure,failure, please use Success relation to feed MergeContent processor Flow: please change the feeding relationships as per the above screenshot. In addition save and upload this template to your instance for more reference and configure MergeContent/PutSFTP processors order-files-merge-content-194166.xml ... View more

@Yu-An Chen Keep FetchHDFS processor configs as below. Give core-site.xml,hdfs-site.xml path in Hadoop Configuration Resources property. Keep HDFS Filename property value as ${path}/${filename} ${path},${filename}are the attributes that needed by FetchHDFS processor to fetch the file from hdfs directory. These attributes are going to be associated with the each flowfile, added by ListHDFS processor. ... View more

@JAy PaTel Let me explain how to do your case using Shell script 1.Using whileloop: Create a input file with all the required tables i.e bash$ vi required_tables.txt mssql_table1 mssql_table2 Then the shell script will read the above file data line by line and executes sqoop import for each table bash$ vi sqoop_import_while.sh while read line; do tableName=`echo $line | cut -d'.' -f2` sqoop import --connect jdbc:sqlserver://<HOST>:<port>;databasename=<mssql_database_name> --username xxxxx - -password xxxx --table $tableName --hive-import --hive-database <hive_database_name> --fields-terminated-by ',' -m 1 done </home/required_table.txt(give your required_tables.txt file path) Now the script reads each line from required_table.txt file and assigns the value to tableName variable then we are using tableName variable in our sqoop import staetment. Once the import is finished for the first table then script reads the next line and performs import again for the second table. 2.Using Forloop: bash$ vi sqoop_import_for.sh declare -a req_tables=("mssql_table1" "mssql_table2") for t in "${req_tables[@]}" do sqoop import --connect jdbc:sqlserver://<HOST>:<port>;databasename=<mssql_database_name> --username xxxxx --password xxxx --table $t --hive-import --hive-database <hive_database_name> --fields-terminated-by ',' -m 1 done By using this script we are not reading the table names from the file instead we are defining an array variable and iterating through all the array elements and performing sqoop import. We have defined req_tables array and given all the required tables then performing for loop each table in req_tables array and passing the name to --table argument in sqoop import. give the permissions to the shell script file then execute the script by using ./sqoop_import_for.sh (or) ./sqoop_import_while.sh You can choose either of the above ways to import only the required tables using sqoop. Let us know if you are facing issues ..!! ... View more