My 2cents is that in a lot of cases swap is being used for unimportant stuff leave more RAM for your app. Do a "ps aux" and look at all the RAM used by weird stuff. Good news is those things will be swapped out.

Example on my personal VPS

   $ free -m
                  total        used        free      shared  buff/cache   available
   Mem:            3923        1225         328         217        2369        2185
   Swap:           1535        1335         200