Same problem here, on my Mac OS 10.11. First error was "Connection timed out lost connection".
I found that the ~/Downloads/ path was referring to a path on the VM in Azure, not my local path.
So I jump the command ssh <username>@<host>-p 22; and retry. Now the path seems correct but I got the error "Permission denied"
How did you resolve it, @Cameron Warren ?
Thanks