I practically live on coffee. Cafetiere at work (with semi-skimmed milk), pour-over at home (with skimmed), and lattes everywhere in between (sometimes full fat, sometimes skinny) and I've never noticed any variation of milk type affecting my bg. If milk is really hitting your numbers, have you ever tried almond or hazelnut milk? The unsweetened varieties taste great and have nigh on 0 carbs. I personally love it and often buy a container for a change of pace.