The whole point of rep is to reflect the way one user views another. If somebody genuinely doesn't like a user's post then they do have the right to -rep. After all, it is reputation.