I put up with a doctor for a couple years that I didn't really like and it turns out my care was lacking ( I was a T1 versus a T2 and had gotten a misdiagnosis) The best thing I ever did was switch doctors. I ended up with an internist I loved and I was properly diagnosed. Then I moved and decided if I didn't like them it wasn't worth having a doctor I didn't like.
It turns out I really like my doctors I have now too, I was a little unsure at the first visit for my new GP/internist. She kept pushing for me to take a statin even though my LDL is low, my HDL is high and my Trig are low because "every diabetic should be on one". I finally said if it's going to be a problem I can switch doctors if she prefers and she backed off right away. We came to an understanding I know she is going to ask each time and she knows I am going to say no.
She has been a go getter, even to the point of calling the visiting endo directly to get me into her in days because she knew nothing about insulin pumps and I had one.
The only problem with switching, which I really believe if you don't like your doctor you should do, is when you switch you still don't know what you will get either. So personally only switch if you think you are lacking care and /or you really just don't like them. You might wait for a few visits to decide.