You are not obliged too get tested and there's no evidence that mask wearing prevents infection but as a fellow non mask wearer I agree that socially it can be uncomfortable if you haven't got that magic bit of cloth over your face for whatever reason. Witness grown men who've forgotten theirs popping in to a shop and holding up their tee shirts to cover their faces! You are braver than them in knowing that you can't cope with a mask.
Bit of common sense would be to NOT go and see your mum if you have any kind of cold, flu etc. and to wash your hands often in any case. And perhaps ask your brother to leave the decision of whether she wants to see you, up to her whilst saying that you understand his genuine concerns etc. etc.