It is a genetic and age related thing. The carbs in milk are the complex sugar lactose. Although lactose is a sugar your body cannot absorb it unless it is broken down into simpler sugars, but unlike childrem many adults have lost the ability manufacture the enzyne which breaks down lactose so that particular sugar cannot affect blood glucose level of many adults.
This where genetics play a part if your genetic make up is northern european there is only a 30% chance that you don't breakdown lactose, but if you genes come from anywhere else in the world where dairy has not been an important part of the diet ithe figure rises to 70%.
The higher sugars in semi-skimmed milk while not a myth is not quite a true picture. If you remove fat from milk the perecentage of everthing else in the milk that remains including sugar increases, and the increased difference in percentage sugar content is tiny !!!!!!! The numbers are:
Sainsburys standard milk contains 9g of sugars ( Lactose) per 200 ml glass .
Sainsburys semi-skimmed contains 9.2 g of sugars ( Lactose) per 200 ml glass.
According to my maths that is only a trivial 2 to 3 % increase.
The higher fat content of standard milk however has an advantage in that is considered to moderate our bodies absobtion of carbs.