On this release we are introducing 2 new anomaly detection tests configurations. The goal is to improve the accuracy and unlock more detection use cases. 🎯💪
If enabled, daily values will be compared only to the same week day previous values.
This configuration would improve accuracy and detection of datasets that have different volume and metrics on weekends and different weekdays.
anomaly_direction: [ both | spike | drop ] ↗️↘️
By default, elementary alerts on both spikes and drops of a metric. Using this configuration, you can limit anomaly tests to alert only on the wanted direction.
For example - If you want volume anomalies to fail only on drop in row count, and not on a spike, just add anomaly_direction: drop to the test config.
on-run-end performance improvements ⏳
We fixed problems in artifacts caching, and implemented an incremental upload. This means that upon a change to models / tests / sources - elementary will now only update the changed resources.
This should improve the performance of the hooks, and we have additional ideas and planned tasks around further improvements.
Special thanks to @panapost for helping with this effort 🤩
Added support to unsupported time buckets on Bigquery (week, month), and made buckets full and consistent to calendar times - Thank you @Kenneth Roos and @Julian Schulte for reporting!
Added log rotation to edr.log to prevent large log files that won’t open - Thank you @Stéphane Burwash and @Grant Mehrer for reporting!
Fixed elementary tests custom names issue - Thank you @Fabien Traventhal for reporting!
Fixed subscribers parsing issue - Thank you @Muthu for reporting!
Added target-path to upload_source_freshness - Requested by @Ange Melhuish!
