Hmm, it's unusual to need the same amount of insulin for a banana or for nothing at all! But there are various factors that may contribute.
The first thing is that many people's basal needs are not consistent throughout the day, so just because your levels are good in the mornings doesn't mean your basal rate is ideal over the whole 24 hour period.
People on a pump do "basal testing" where they fast for particular portions of the day to work out what basal levels are appropriate at that time of day.
The second thing is that one's liver stores glucose (in the form of glycogen) that can be released to fuel our body when necessary. All sorts of factors contribute to this, like adrenaline, stress, caffeine, etc. This varies considerably from person to person. Perhaps when you skip a meal your body produces a liver dump for some reason.
It could even by psychological... twice I have seen a big sugar spike after drinking a sugar-free soft drink (and confirmed that it really was sugar-free using my meter!). I think it was a psychological response!