Fat or not?

I know I'm not fat, but why do people think that I have a mental or emotional problem if I want to lose some weight? Not a lot, just a few pounds. I don't really care about the number, though I would like it to be less. I'm running out of close that fit and my "fat" clothes are now my everyday clothes. So yes, my current weight is a problem. And why do people think that if I'm not heavily overweight it should be easier? Proportionally, I have more weight to lose than a lot of people. Wish me luck.