Something to never forget again:

1

rsync --recursive me@remote:'"~/Very complicated path/"' ~/Downloads/

So: single quote, double quote, path, double quote, single quote