I do consider myself lucky there as Ive heard a few horror stories about sugar alcohols and other artificial sweeteners. There is so much conflicting information out there, it makes your head spin.
I believe that xylitol and erythritol are both good to use for baking. I preferentially use erythritol in a "golden" granulated form (Sukrin Gold) - mainly because of the danger to dogs issue with xylitol, and also because erythritol's also lower in calories, and together with stevia it's the one that seems to be most recommended for diabetic use.