Dbt – Incremental but Incomplete

Basically people are constantly calculating metrics based on existing tables. Think something as simple as a moving average or the sum of two separate columns in a table. Once upon a time you would set up a cronjob and populate these every day as a SQL query in some python or Perl script.

Dbt introduced a language for managing these “metrics” at scale including the ability to use variables and more complex templates (Jinja.)

Then you do dbt run (https://docs.getdbt.com/reference/commands/run) and kapow the metric is populated in your database.

More broadly dbt did two other things: 1. It pushed the paradigm from ETL to ELT (so stick all the data in your warehouse and then transform it rather than transform it at extraction time.) 2. It created the concept of an “analytics engineer” (previously know as guy who knows SQL or business analyst.)