Could you maybe add radio boxes to the image folder list so rather than remove the folder, it could be unselected for display?
Sometimes I want to add a folder of photos that aren't mine into the mix.
It would be much easier to just click 'enable/disable' than remove it and have to traverse the tree to add it again.
No rush. Just a thought ;)