Keep the faith!
If you are swimming 4 times a week, then you will be building lean muscle mass whilst losing fat.
I swim and cycle myself, and despite having lost nearly 20kg since May, my weight loss has slowed over the last couple of months. However - I am still losing fat, just replacing it...