The only safe, consistent, reliable approach is not to close DLLs.
That doesn't address the need of some DLLs to malloc() resources in the context of the applications linking to them.
This problem _cannot_ be solved _generically_. Any solutions are extremely API-specific and impose restrictions on their users (the linking applications) which, if violated, will lead to Undefined Behavior.
Edit: as an example of cases which must bind resources in the application's context: see the Classloading in C++ paper at <https://wanderinghorse.net/computing/papers/index.html#class...> (disclosure: i wrote that article).
Of course, C/C++ applications written in the traditional model with static data everwhere would have difficulty not leaking tokens and holding the DLL open, but it's still far from impossible to write such a safe API.
> That doesn't address the need of some DLLs to malloc() resources in the context of the applications linking to them.
If there is a context boundary and really such a need, then the DLL can keep a list of all such resources, and destroy all those resources once closed. Access to them would similarly have to be protected by a token.
That's true, but those approaches are only viable if you trust the DLL in question. External libraries are fundamentally opaque/could contain anything, and if you're in a tinfoil-hat mood, it's quite easy to make new libraries that emulate the ABI of the intended library but do different (maybe malicious, maybe just LD_PRELOAD tricksy) things.
Consider: an evil wrapper library could put the thinnest possible shim around the "real" version of the library and just not properly account for resources, exposing library (un)loaders to use-after-free without much work, even if the library loaders relied upon the approaches proposed.
Since there aren't good cross-platform and race-condition-free ways of saying "authenticate this external library via checksum/codesigning, then load it", there are some situations where the proposed approaches aren't good enough.
Sure, most situations probably don't need that paranoia level (or control the code/provenance of the target library implicitly). But the number of situations where that security risk does come up is larger than you'd think, especially given automatic look-up-library-by-name-via-LD_LIBRARY_PATH-ish behavior.
The measures I suggested before were all in the context of buggy users that can't resist the urge to keep references to the library's resources lying around all over the place. But untrusted code can never be made safe with anything short of a strong sandbox.