Whilst they should not affect your BG, the usual advice is to ween yourself off sweets so you no longer have the desire for sweet things.
It is better to have an occasional nice tasting sweet or chocolate than lots of sugar free sweets or diabetes chocolate.