All sounds perfectly normal to me and not disimilar to my own experience. I found the same with eating porridge too.
Your body will always add extra glucose to your system when you get roughly below 4.0 via a process called
gluconeogenesis. It is one of a number of homeostatic processes that keep various things like body temperature, hydration levels, glucose, iron, calcium leves etc etc within strict ranges. It is perfectly normal. Even if you didn't eat anything for a week, your BG would go up and down, say between 4.0 and 6.0. Certain medications though can interfere with this process so people who take insulin have to be careful, but it doesn't apply to you.
I take some carbs now but keep them restricted to things like brown rice, barley, wholewheat pasta etc. I still avoid white bread, mashed potatoes etc. I eat things like lentils, peas, beans which provide enough carbs for me but which also make my body have to work a bit harder to digest them, so I have a sort of balance that way.
If you still have weight to lose, limiting your carb intake is still a good idea but you might like to start enjoying things like strawberries as they will shortly be in season, raspberries, blackberries, cherries, pears, apples, pears and plums and, personally, I eat loads of tomatoes. In fact, I have loads of fruit and veg but still avoid things like melons, grapes and other very sweet fruits.
But, as Sally and James write, your levels sound very good. Don't worry.