I was a massive swimmer at one time, but was always told, and believed, we should never swim on a full tummy for fear of cramp.
Old wives' tale? No idea.
By the way, exercise does not necessarily bring your levels down immediately, or at least it doesn't with me. I find my levels increase after exercise for possibly up to 40 or 50 minutes later. If you swim in the sea after an overnight fast, you are likely to find you get a large liver dump. Your liver will be under the impression your body is short of glucose and try to look after you by dumping glucose. Best to exercise after eating, an hour or so after.