Thing is you have two changes in play there - swapping milk and ditching veggies.
Of course, both may be issues, but right now, you have no idea if only one is.
It can take a little longer than a few days for these things to become clear. WShen I try these elimination phases, where I believe I might have an issue, I make only one change each minimum of 2 weeks. If you were to think you had an issue with gluten, that required several weeks of exclusion to make any judgement.
I do hope you find your answer, or answers.