Just came back from vacation from sunny Turkey and in half a day fixed a bug that I spend a couple of days trying to fix before I left. Funny how that works some time.
The bug had to do with several brushes having a surface that lie on the same plane and where sometimes the wrong surface would be categorized as being visible, such as a surface that should actually not exist anymore because it was removed by another brush.
Now I can finally focus on merging brushes on the edges where they intersect, this should fix all the problems I'm having with gaps between brushes. I already have a good idea on how to do this, so hopefully this shouldn't take too long.