All it means when people say “you’re speaking from a place of privilege” is that you’re likely to underestimate how bad the problem is by default because you are never personally exposed to that problem. It’s not a moral judgement of how difficult your life is.