Luckily, there is a way in which you can define this behaviour and it's called the kernel "swappiness" value. The value has a range of 0 to 100, with zero roughly meaning that process memory will never get swapped out for the sake of disk caching and a value of 100 means that process memory is very aggressively swapped out, in favour of disk caching. A more in depth explanation of how the kernel manages swappiness, can be found here.
By default, this value is set to 60, which is configured more for server throughput, rather than desktop responsiveness.On my desktops I usually set the value to 10, which seems to be a good fit for desktop responsiveness.
The way to set it on Ubuntu is:
$ sudo sysctl vm.swappiness=10
and to make the changes permanent, just add the following line to /etc/sysctl.conf: