Fivestar module settings

I installed the Fivestar module on my photo site, but it doesn't seem to be working like I assume it should. I have rollover voting disabled (through Voting API), because I don't want people from the same IP voting more than once. It all seems to work fine when a user is logged in, but for Anonymous user, the behavior is weird: if you click the stars to vote again, the number of votes stays the same (good), but the votes are apparentlt still figured into the average (bad). Here's a sample node:
http://holgaiowa.com/content/leave-these-songs-unsung
Currently it says:
"Your rating: None Average: 5 (2 votes)"
But if you click the first star, the average goes down to 3. You can return it to 5 by clicking the fifth star.
Which brings to attention the other issue. Even if I am an anonymous user, and I vote, it still says "Your rating: none."
Is this bugged, or have I just failed somewhere in the configuration?