This problem is typically solved by either (a) clearing cookies and restarting your browser; and/or (b) logging out and back into CDSW. Let me know if that works for you. ... View more

Hi Guy,   Please try adjusting your command to the following: ozone sh volume create --quota=1TB --user=hdfs o3://ozone1/tests Note the documentation states that the last parameter is a URI in the format <prefix>://<Service ID>/<path>. Service Id is what you found in ozone-site.xml. ... View more

Hi @Maria_pl , generally speaking the approach is as follows: 1. Generate a dummy flow file that will trigger (GenerateFlowFile processor) 2. Next step is UpdateAttribute processor that sets the start date and end date as attributes in the flow file 3. ExecuteScript is next. This can be a python script, or whichever language you prefer, that will use the start and end attributes to list out all the dates in between. 4. If your script produces single file output of dates, you can then use SplitText processor to cut each row into its own flow file and from there each file will have its own unique date in your range.   Hope that makes sense.  ... View more

Sqoop can only insert into a single Hive partition at one time. To accomplish what you are trying to do, you can have two separate sqoop commands: sqoop with --query ... where  year(EventTime)=2019 (remove year(EventTime)=2020) and set --hive-partition-value 2019 (not 2020) sqoop with --query ... where  year(EventTime)=2020 (remove year(EventTime)=2019) and set --hive-partition-value 2020 (not 2019) This way sqoop will write into the one partition you want. Since this is one-time import, the solution should work just fine. Let me know if this works and accept the answer if it makes sense. ... View more

Glad things are moving forward for you, Heri.   Examining your sqoop command, I notice the following: -- check-column EventTime tells sqoop to check this column as the timestamp column for select logic --incremental lastmodified tells sqoop that your source SQL table can have both records added to it AND records updated in it. Sqoop assumes that when a record is updated or added its EventTime is set to current timestamp. When you run this job for the first time, sqoop will pickup ALL records available (initial load). It will then print out a --last-value timestampX.  This timestamp is the cutoff point for the next run of the job (i.e. next time you run the job with --exec incjob, it will set --last-value timestampX) So, to answer your question, it looks like sqoop is treating your job as an incremental load on the first run:  [EventTime] < '2020-04-17 08:51:00.54'. When this job is kicked off again, it should pickup records from where it left off automatically.   If you want, you can provide a manual --last-value timestamp for the initial load, but make sure you don't use it on subsequent incremental loads.   For more details, please review sections 7.2.7 and 11.4 of Sqoop Documentation     If this is helpful, don't forget to give kudos and accept the solution. Thank you! ... View more

You will need to get both the year and the month of the original date (because of the leap year considerations). In terms of how to accomplish this in NiFi one way is to use ExecuteScript processor and Python's calendar library with the function calendar.monthrange(year,  month) that returns the last day of month as the second argument.  ... View more

You can try the ReplaceText NiFi processor withe the approached described here. That will be a clean way of doing what you want without much scripting. ... View more

This is the offending code: new = year+month+day+hour # Considering date is in mm/dd/yyyy format #converting the appendd list to strings instead of ints b=[str(x) for x in new] #joining all the data without adding b = '/'.join(b) #convert to unix dt_object2 = datetime.strptime(b, "%Y/%m/%d/%H") It looks like at some point the values of year, month, day, hour are set to strings "Year", "month", "DOY", "Hour".  Then when new = year+month+day+hour is called, the strings get concatenated into "YearmonthDOYHour".  You then split and join that string so that's why you see a '/' character between each character in the Python error message. I'll leave it you to debug this, as I've lost track of all the code changes at this point.    Also note that the incoming data may be providing you with Day of Year (DOY) instead of day of month, which is what %d. You may need to use %j to parse that out with zero padding (see documentation here).   If this is helpful, don't forget to give kudos or accept solution.  ... View more