Why are we resistant to the idea that we might have privilege?

Reading Scalzi's Lowest Difficulty Setting and the follow-up got me thinking. People are generally quite resistant to the idea that they have any sort of privilege. Their (and my) automatic, knee-jerk response tends to be "What? No I don't!"

But why is this?

I can tell you why I'm resistant to it. I'm resistant to it because for the vast majority of my life I was being given the message that I'm lucky about and should be thankful for things I didn't care about, many of which I didn't even like. For example, my parents would take us on stressfully long family vacations - whole summers lost to fighting off carsickness while having zero privacy - and tell me that I should be grateful that I get to travel. When we were travelling, my parents tried to save money by never eating at restaurants, instead taking us to a supermarket and telling us to pick out what we wanted to eat for dinner. But we never had a fridge or a stove or a microwave (and often not even a kettle), or even dishes or utensils. I'd ask if we can go to a restaurant because I'd been yearning for days for a nice big salad and a steaming plate of pasta, and they'd tell me I should be thankful we have food at all. My father went through this phase where he calculated that if they hadn't had kids they could drive a Mercedes instead of a Honda so he told us that we should be thankful they made that sacrifice and decided to have us. But, on top of the fact that I'm intrinsically nihilistic, this was during the worst of my bullying; I, and everyone else involved, would have been far happier if they'd gone for the Mercedes instead. (Even now, if I hadn't been born I obviously wouldn't be around to care, and I seriously doubt my parents would be postmenopausally regretting not having an overly-introverted, socially-awkward daughter with a non-lucrative career path and a lifestyle that rejects their values.)

So, because of all this, any sort of hint or insinuation that I have some sort of privilege or advantage or some other thing I should be thankful for evokes this feeling of all this stressful shit that I didn't even want to deal with in the first place piling up my tetris blocks and if they'd just left me alone I could go be alone in my room with a book and be much happier.

But these are all my own personal neuroses, stemming directly from specific feelings and experiences in my own life. None of this is broadly applicable to the general population.

So where's it coming from for everyone else?