Yes, you are responsible for all the code you ship to your users. Not pinning dependencies is asking for trouble. It is literally, "download random code from the Internet and hope for the best."
Pinning dependencies also means you're missing any security fixes that come in after your pinned versions. That's asking for trouble too, so you need a mechanism by which you become aware of these fixes and either backport them or upgrade to versions containing them.
All code is fundamentally not ever secure.