No worries.
Yeah, that is a side-effect of using that tool, so a little discipline is needed.
One trick I use, is I magic select all the area's I need to take the colour out of, then 'Grow' the selection by 1px (so it slightly intrudes on the adjacent pixels) then use the Colour to Alpha tool. That way, if there's any AA on the image, it quite nicely makes it AA into Transparency rather than into the left-overs of the colour, then a hard cut into transparency
GIMP is quite different to use than other popular image tools, but once you get the basics down, and learn some techniques, you'll never go back to the others