As social animals, we care what people think.
It's nearly impossible to remove all caring this way. People that claim they don't care what people think are lying to themselves and others.
Recognizing this bias is the first step in mitigating it. You will never completely remove it, but you can make progress in caring less.