Would it upset you if your friend basically called you fat?

No, some people just have different body types, it doesn't mean you're fat. They made a size large for a reason, because people fit in it. A lot of people wear size large, otherwise, you wouldn't see that size in every single store. You're absolutely perfect just the way you are.