Everyone is guilty of this....even most of us here.
Occasionally, people on this forum make threads about their friend or relative with diabetes. These threads often cite that the person manages their condition poorly and mentions seeing them eat something they probably shouldn't. The fact of the matter is, you're not usually seeing/hearing the full story.
Your situation isn't terribly different. A person seemingly close to you saw you skipping lunch and may have jumped to the wrong conclusions. You may have excellent control, but they don't see you take every shot and every test. They only see you skipping lunch.
I used to get annoyed by these sorts of comments, and to a lesser extent still do. However, I try to remember that when it's coming from someone who cares about me, they almost always mean well. As @Novorapidboi mentioned, I try to educate or explain to them that I have things under control and there's a reason I'm doing what I'm doing.