The worst you can do if you've been very low carbing and in ketosis, is kicking yourself out of it with a big binge, and needing to start over again.
I think if, as you say, your diet has been "brutal" you definitely need to ease up now to where you can cope long term with how you're eating. IMO it's better to do this than end up on a huge binge because you've been denying yourself
too much. A low carb diet is often going to be a long term/permanent lifestyle change, so it really does need to be sustainable. Little steps are much better at getting you where you aim to be than trying to take flying leaps and falling flat on your face!
I've been very lucky so far in that most of my falls from grace are a one off scoff of the "I think I'll just have a taste of that", and they've not been enough to get my body shouting our for more carbs.
But I do remember when I was getting almost permanent migraines many years ago when I was so desperate for sugar that I'd sit down and eat great spoonfuls out of the sugar bowl - I actually mentally found it totally revolting but my body was completely overruling my brain. I've never wanted to end up doing that again.
It is quite possible to have an enjoyable long term very low carb diet, but you
do need to spend some time exploring acceptable alternatives for the high sugar/carbs foods that you are (should be!) no longer eating. High cocoa (75% or more) dark chocolate is fine to eat in small pieces; cakes, muffins, biscuits can be baked with ground almonds or other flour alternatives, and you can investigate suitable sweeteners such as Stevia or erythritol to use; many fruits
can be enjoyed if you keep portions small - and use your meter to test what that portion size might safely be.
Robbity