In an early chapter it outlines, rather eloquently, the distinctions between the Frequentist and Bayesian paradigms and in particular the power of well-designed Frequentist or likelihood-based models. With few exceptions, an analyst should get the same answer using a Bayesian vs. Frequentist model if the Bayesian is actually using uninformative priors. In the worlds I work in, 99% of the time I see researchers using Bayesian methods they are also claiming to use uninformative priors, which makes me wonder if they are just using Bayesian methods to sound cool and skip through peer review.
One potential problem with Bayesian statistics lies in the fact that for complicated models (100s or even 1000s of parameters) it can be extremely difficult to know if the priors are truly uninformative in the context of a particular dataset. One has to wait for models to run, and when systematically changing priors this can take an extraordinary amount of time, even when using high powered computing resources. Additionally, in the Bayesian setting it becomes easy to accidentally "glue" a model together with a prior or set of priors that would simply bomb out and give a non-positive definite hessian in the Frequentist world (read: a diagnostic telling you that your model is likely bogus and/or too complex for a given dataset). One might scoff at models of this complexity, but that is the reality in many applied settings, for example spatio-temporal models facing the "big n" problem or for stuff like integrated fisheries assessment models used to assess status and provide information on stock sustainability.
So my primary beef with Bayesian statistics (and I say this as someone who teaches graduate level courses on the Bayesian inference) is that it can very easily be misused by non-statisticians and beginners, particularly given the extremely flexible software programs that currently are available to non-statisticians like biologists etc. In general though, both paradigms are subjective and Gelman's argument that it is turtles (i.e., subjectivity) all the way down is spot on and really resonates with me.