So yesterday I had a little bit of time to try a couple of small things.
I changed the page sorting, first sorting the pages from high mipmap level to low mipmap level, then from more to less visible (by counting how many samples I find for each page in the readback buffer).
I also decreased the amount of pages I store in my GPU side page-cache, because I suspect that OpenGL updates the entire texture even when I only update a small part of it, and a smaller texture would upload faster.
Next to that i fixed a small bug where a buffer wasn't reset properly.
Oh, and I defragmented my system :)
And lo and behold... my virtual texture app worked flawlessly!?
No popping at all!
So naturally I wanted to figure out which of my changes was responsible for this!
.. and then my wife pulled me away from my computer ;)
I guess I'll have to look at it next time I have time, but I'm really happy at the current results!