Call the library's shutdown procedure before you dlclose. A dynamically linked library is a resource like any other. Ya gotta close it properly.
Of course, if the library doesn't probably attempt to close libraries it's responsible for during its shutdown procedure, that's another can of worms.