Unfortunately close grappling might pull the cannula out, particularly if you were sweating. Any other contact sport except wrestling, and maybe rugby, you would be fine.
However putting a new cannula in is a few minutes work at most.
I have taken my pump off for karate for the last three years. For up to two hours at a time. I am liable to be got at anywhere except my middle and groin where I have my cannula site. The site has never been disturbed though it has been kicked and punched. Medtronic used to sell a between the shoulder blades harness. I believe the stitching was poor and so no longer for sale.