Hard disagree. There are a lot of implementation "details" that, if you want to do properly, are a lot of hard work and very much nontrivial. For example, do try to write a compiler with efficient incremental compilation, and especially one that does so while also having optimization passes. And that's just one example, most things in compiler implementations actually turn out to be fairly complex. And lots of features that modern languages support e.g. more powerful typesystems, trait/typeclass systems, etc. are also very very tricky.
While designing a language is by no means trivial, it generally really occupies just a very small fraction of the language/compiler developer's time. And, in most cases, the two things (language design + implementation details) have to walk hand-in-hand, since small changes to the language design can vastly improve the implementation end.