Sqoop can only insert into a single Hive partition at one time. To accomplish what you are trying to do, you can have two separate sqoop commands: sqoop with --query ... where year(EventTime)=2019 (remove year(EventTime)=2020) and set --hive-partition-value 2019 (not 2020) sqoop with --query ... where year(EventTime)=2020 (remove year(EventTime)=2019) and set --hive-partition-value 2020 (not 2019) This way sqoop will write into the one partition you want. Since this is one-time import, the solution should work just fine. Let me know if this works and accept the answer if it makes sense. ... View more

Glad things are moving forward for you, Heri.   Examining your sqoop command, I notice the following: --check-column EventTime tells sqoop to check this column as the timestamp column for select logic --incremental lastmodified tells sqoop that your source SQL table can have both records added to it AND records updated in it. Sqoop assumes that when a record is updated or added its EventTime is set to current timestamp. When you run this job for the first time, sqoop will pickup ALL records available (initial load). It will then print out a --last-value timestampX.  This timestamp is the cutoff point for the next run of the job (i.e. next time you run the job with --exec incjob, it will set --last-value timestampX) So, to answer your question, it looks like sqoop is treating your job as an incremental load on the first run: [EventTime] < '2020-04-17 08:51:00.54'. When this job is kicked off again, it should pickup records from where it left off automatically.   If you want, you can provide a manual --last-value timestamp for the initial load, but make sure you don't use it on subsequent incremental loads.   For more details, please review sections 7.2.7 and 11.4 of Sqoop Documentation     If this is helpful, don't forget to give kudos and accept the solution. Thank you! ... View more

You will need to get both the year and the month of the original date (because of the leap year considerations). In terms of how to accomplish this in NiFi one way is to use ExecuteScript processor and Python's calendar library with the function calendar.monthrange(year,  month) that returns the last day of month as the second argument.  ... View more

You can try the ReplaceText NiFi processor withe the approached described here. That will be a clean way of doing what you want without much scripting. ... View more

This is the offending code: new = year+month+day+hour # Considering date is in mm/dd/yyyy format #converting the appendd list to strings instead of ints b=[str(x) for x in new] #joining all the data without adding b = '/'.join(b) #convert to unix dt_object2 = datetime.strptime(b, "%Y/%m/%d/%H") It looks like at some point the values of year, month, day, hour are set to strings "Year", "month", "DOY", "Hour".  Then when new = year+month+day+hour is called, the strings get concatenated into "YearmonthDOYHour".  You then split and join that string so that's why you see a '/' character between each character in the Python error message. I'll leave it you to debug this, as I've lost track of all the code changes at this point.    Also note that the incoming data may be providing you with Day of Year (DOY) instead of day of month, which is what %d. You may need to use %j to parse that out with zero padding (see documentation here).   If this is helpful, don't forget to give kudos or accept solution.  ... View more