I think that fear's universal honestly. If you haven't yet, check out the maintenance-focused community and the one for people who've come off the medication entirely, worth reading through both. They serve different purposes, the first is for people continuing some maintenance amount (sometimes unchanged, sometimes gradually reduced). The second exists for people no longer taking it, whether they walked away voluntarily or had to.
Personally I'm in maintenance and tapering down (peaked at 7.5, ran 5 for two months, now down to 2.5). Haven't decided whether I'll eventually stop entirely, but I'm content easing into maintenance slowly. Worth mentioning though, my starting point was more extreme than yours, more than half my body weight needed to come off, and I've stuck with this for eighteen months, so my perspective's shaped differently. Good luck figuring out your own maintenance approach, and congrats on what you've already accomplished!