Look at musl, dlclose marks a library as invalid, so when reallocations happened next the pointers get reused. It's garbage collected, but things get unloaded.
Though, you do seem to be correct that POSIX allows things to remain in the address space:
> The use of dlclose() reflects a statement of intent on the part of the process, but does not create any requirement upon the implementation, such as removal of the code or symbols referenced by handle. Once an object has been closed using dlclose() an application should assume that its symbols are no longer available to dlsym(). All objects loaded automatically as a result of invoking dlopen() on the referenced object shall also be closed if this is the last reference to it.
> Although a dlclose() operation is not required to remove structures from an address space, neither is an implementation prohibited from doing so. The only restriction on such a removal is that no object shall be removed to which references have been relocated, until or unless all such references are removed. For instance, an object that had been loaded with a dlopen() operation specifying the RTLD_GLOBAL flag might provide a target for dynamic relocations performed in the processing of other objects-in such environments, an application may assume that no relocation, once made, shall be undone or remade unless the object requiring the relocation has itself been removed. [0]
[0] https://pubs.opengroup.org/onlinepubs/009696799/functions/dl...