Drupal Community Portal

Hi everyone, I've got quick question about updating my drupal version. In the update text, it says to "Remove all old files and directories from the Drupal installation directory."; I'm a bit confused; if I remove those files aren't I going to lose all my modules and themes, apart from the ones that come with Drupal?