afaiu it's more to do with what sort of exercise you do - if the exercise is hard (or hard for you) then it will produce stress hormones which cause the liver to release glucose, which often results in an increase in BG level (this is typically seen for activities such as weight lifting, sprinting, etc.). If you do gentle exercise (walking, gardening), then you produce fewer exercise related stress hormones, so get less glucose produced which usually results in a drop in BG level. Other activities sit somewhere in the middle and often move between the two states - e.g. when cycling, going up hills produces a rise, while riding on the flat produces a drop.
As a given exercise session proceeds your muscles also become more sensitive to insulin, so even hard exercise will eventually produce a drop in BG level (probably caused by a tapering off of the stress hormones as well as muscle insulin sensitivity rising.)
Re the OP's question, I wonder if part of it might be that his/her BG is already quite low at the start of the exercise, so even if it were gentle he/she may be getting additional glucose produced due to glucagon secretion (i.e. the body trying to avoid low blood sugar) and a subsequent overshoot.