In my experience, nobody questions things we do, when we do it with confidence and our heads held high. The questions arise when one's body language is that of discomfort and embarrassment; which in turn makes others feel uncomfortable, and they enquire, mainly to ease their own discomfort.
Mainly others aren't bothered what anyone else eats.
The few times I have been asked, I have responded that I have found I'm not dealing with carbs very well these days, so I'm leaving them alone, for now. Of course, "for now" could just as easily be "forever".