The problem is not even really in the outfit itself, but in where it is used. You won't believe it, but the tracksuit is designed for sports! Not for walks along the sea, not for sitting on the sofa at home, not for a café, and not for a random date with a girl.
And you don't need to say that "it's comfortable for me and who sees me at home?". You can also be comfortable in jeans and a regular t-shirt, but at home your partner also sees you. If you don't have a partner, throw away the flip-flops, and the wife-beater shirt, start going to the gym in your tracksuit, and who knows, maybe everything will change!