Metformin doesn't make you go hypo, so you have no worries there. A normal person without diabetes will have levels in the 4's. Have you tested what your levels are two hours after breakfast? This could be an indicator of how well your pancreas is working.
If it were me, then I would continue to take the metformin, but to make an appointment to see your doc as you may need your dosage adjusted. BTW my fasting level was 4.7 this morning and I will take my meds as usual.