Why You Should Probably be Using Elementary with your dbt Project
If you’ve read my previous articles, you’ll know that I’m a dbt evangelist and a huge advocate of the importance of proper testing infrastructures in a data pipeline. Recently, my team at Tempus adopted a tool called Elementary, an open-source dbt package with an incredibly powerful out-of-the-box testing toolkit. This article outlines what Elementary is, and why you should probably be using it. Did I mention it’s free?
What is Elementary
Elementary is a powerful dbt package that allows you to supercharge your tests and the infrastructure that makes a test impactful. While dbt tests allow you to catch errors in theory, you can never really do much with them — the alerting and observability capabilities just aren’t there.
Using Elementary, you can seamlessly integrate Slack with your dbt project, conjure up a powerful reporting/analytics dashboard with the click of a button, and introduce an anomaly detection layer that may put at least one analyst on your team out of a job. If all of this sounds a bit too good to be true, then you probably have a sufficient amount of skepticism! However, I hope that by the end of the article, I’ll convince you why you should add Elementary into whatever data mart you’re building.
Slack Alerts
Slack alerts with dbt can be tricky to set up. On my last team, in order to create live reporting we added regex to our entrypoint to parse our dbt logs and created a Slack app that ran off the entrypoint during our daily runs. Although the end result worked, it was certainly not as easy as Elementary, and definitely not as advanced. With Elementary, Slack integration takes less than an hour by following their guide here. After that hour, here’s a glimpse of what you can expect —
What are the important components here? Above, we see a dbt model that is failing, as well as a data anomaly detection test that is erroring out (more on that later). We can also see a test description, any relevant tags, and owners/subscribers. With Elementary, tests no longer fail silently! Instead, they actually @ a user on Slack, and continue to do so until they’re turned off (brilliant).
Before Elementary, our team’s test alerting relied on a dashboard in Data Studio built on top of a log sink, and the “alerts” were a daily email that you actually had to open. Safe to say this is a lot better.
Elementary Reporting Dashboard
The next best part of Elementary is the out-of-the-box dashboard that allows you to visualize any failing tests. Here are a few views of our dashboard —
This dashboard is available out of the box, totally code-free. It runs on a simple index.html file and is generated by running the command edr report. Although serving this to a secure website can be tricky, Elementary also offers a paid version where they handle hosting for you.
Here’s another view that shows an anomaly detection test failure —
Above, we can visualize the number of rows flowing into one of our prediction tables. We expect a certain amount of data flowing in — too much or too little and it’ll throw an error. Once again I want to emphasize that this visual is entirely no-code. Elementary just does this by itself.
Elementary Tests and Anomaly Detection
The last visual showed off the power of Elementary’s anomaly detection. What’s more impressive is how easy it is to implement that detection —
This small bit of .yaml gives Elementary enough to run with and identify when anomalies have occurred. We don’t even need to modify our model — Elementary picks up data behind the scenes, calculates the standard deviation of num_rows being written in, and flags whenever the volume for a single day is out of that range.
There are more tests than just anomaly detection too, and each test that Elementary offers is highly predictable. With volume anomaly detection, for instance, we can tell Elementary to be hyper-sensitive (one standard deviation), bucket data by the week, only look at positive or negative deviations, or only calculate standard deviations using a week’s worth of data!
Conclusion
Elementary is an extremely potent open-source package to use with dbt. The Slack alerts alone are worth it, and when you add on the ease of use, and high degree of customizability, I think it’s a no-brainer to use with any dbt project. Although dbt is extremely powerful and has changed the data engineering landscape permanently, it definitely is limited in its scope of what it can accomplish. Elementary solves a lot of the common dbt problems I have on a regular basis, and I’m excited to keep exploring its capabilities.
