> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementary-data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Anomaly Tests Troubleshooting

> [card] [Generate your anomaly test with Elementary AI](https://elementary-data.com/community)
>
> Let our Slack chatbot create the anomaly test you need.

> [accordion] Verify the data collection for your anomaly test
>
> First, check if your test uses a timestamp column:
>
> ```yaml theme={null}
> # In your YAML configuration
> data_tests:
>   - elementary.volume_anomalies:
>       arguments:
>         timestamp_column: created_at# If this is configured, you have a timestamp-based test
> ```
>
> > [accordion] If you have a timestamp-based test (recommended)
> >
> > * Metrics are calculated by grouping data into time buckets (default: 'day')
> > * Detection period (default: 2 days) determines how many buckets are being tested
> > * Training period data (default: 14 days) comes from historical buckets, allowing immediate anomaly detection with sufficient history
> >
> > Verify data collection:
> >
> > ```sql theme={null}
> > -- Check if metrics are being collected in time buckets
> > SELECT
> > bucket_end,
> > metric_value,
> > COUNT(*) as metrics_per_bucket
> > FROM your_schema.data_monitoring_metrics
> > WHERE full_table_name = 'your_table'
> > GROUP BY bucket_end, metric_value
> > ORDER BY bucket_end DESC;
> >
> > ```
> >
> > * Each bucket should represent one time bucket (e.g., daily metrics)
> > * Gaps in `metric_timestamp` might indicate data collection issues
> > * Training uses historical buckets for anomaly detection
> > * The format for full\_table\_name is DATABASE.SCHEMA.TABLE\_NAME
> >
> > **Common collection issues:**
> >
> > * Missing or null values in timestamp column
> > * Timestamp column not in expected format
> > * No data in specified training period
>
> > [accordion] If you don't have a timestamp configured
> >
> > * Training period data builds up over multiple test runs, using the test run time as its timestamp column. This requires time to collect enough points; for a 14 day training period, the test would need 14 different runs on different days to have a full training set.
> > * Metrics are calculated for the entire table in each test run
> > * Detection period (default: 2 days) determines how many buckets are being tested
> >
> > Check metric collection across test runs:
> >
> > ```sql theme={null}
> > -- Check metrics from different test runs
> > SELECT
> > updated_at,
> > metric_value
> > FROM your_schema.data_monitoring_metrics
> > WHERE full_table_name = 'your_table'
> > ORDER BY updated_at DESC;
> > ```
> >
> > * Should see one metric per test run and per dimension
> > * Training requires multiple test runs over time
> > * Each new test run creates the training point for a time bucket. A second test run within the same bucket will override the first one.
> > * The format for full\_table\_name is DATABASE.SCHEMA.TABLE\_NAME
> >
> > **Common collection issues:**
> >
> > * Test hasn't run enough times
> > * Previous test runs failed
> > * Metrics not being saved between runs

> [accordion] Verify anomaly calculations
>
> Anomaly detection is influenced by:
>
> * Detection period (default: 2 days) - the time window being tested
> * Sensitivity (default: 3.0) - how many standard deviations from normal before flagging
> * Training data from previous periods/runs
> * `metrics_anomaly_score` calculates the anomaly based on the data in `data_monitoring metrics`.
>
> Check calculations in `metrics_anomaly_score`:
>
> ```sql theme={null}
> -- Check how anomalies are being calculated
> SELECT
>     metric_name,
>     latest_metric_value,
>     training_avg,
>     training_stddev,
>     anomaly_score,
>     is_anomaly
> FROM your_schema.metrics_anomaly_score
> WHERE full_table_name = 'your_table'
> ORDER BY bucket_end DESC;
> ```
>
> * `anomaly_score`: The standardized score that measures how many standard deviations a data point is from the mean
> * `is_anomaly`: A boolean field that indicates whether the anomaly score exceeds the configured threshold

> [accordion] 'Not enough data to calculate anomaly' error
>
> This occurs when there are fewer than 7 training data points. To resolve:
>
> ### For timestamp-based tests:
>
> * Check if your timestamp column has enough historical data
> * Verify time buckets are being created correctly in `data_monitoring_metrics`
> * Look for gaps in your data that might affect bucket creation
>
> ### For non-timestamp tests:
>
> * Run your tests multiple times to build up training data.
> * Check `data_monitoring_metrics` to verify the data collection. The test will need data for at least 7 time buckets (e.g 7 days) to calculate the anomaly.

> [accordion] Missing data in data\_monitoring\_metrics
>
> If your test isn't appearing in `data_monitoring_metrics`:
>
> Verify test configuration:
>
> ```yaml theme={null}
> data_tests:
>   - elementary.volume_anomalies:
>       arguments:
>         timestamp_column: created_at# Check if specified correctly
> ```
>
> ### Common causes:
>
> * Incorrect timestamp column name
> * Timestamp column contains null values or is not of type timestamp or date
> * For non-timestamp tests: Test hasn't run successfully
> * Incorrect test syntax

> [accordion] Training period changed, but results are the same
>
> If you change it after executing elementary tests, you will need to run a full refresh to the metrics collected. This will make the next tests collect data for the new **`training_period`** timeframe. The steps are:
>
> 1. Change var **`training_period`** in your **`dbt_project.yml`**.
> 2. Full refresh of the model ‘data\_monitoring\_metrics’ by running **`dbt run --select data_monitoring_metrics --full-refresh`**.
> 3. Running the elementary tests again.
>
> If you want the Elementary UI to show data for a longer period of time, use the days-back option of the CLI: **`edr report --days-back 45`**

> [accordion] BigQuery: anomaly tests scan the whole table or fail on require\_partition\_filter
>
> dbt Fusion and dbt Core 1.11 record run timings (such as `execute_started_at`) with nanosecond precision, which
> BigQuery timestamps don't support. Starting with version 0.26.1, Elementary truncates these values to microseconds
> when it uploads them. Rows uploaded by earlier versions may still contain nanosecond values, so by default Elementary
> also truncates timestamps whenever it casts them on BigQuery.
>
> The truncation converts each timestamp to a string and back, so BigQuery can't prune partitions on it. As a result,
> anomaly tests on partitioned tables scan the whole table, and tables with `require_partition_filter` reject the query.
>
> To use plain timestamp casts again:
>
> 1. If needed, fix the nanosecond values that earlier versions already uploaded:
>
>    ```shell theme={null}
>    dbt run-operation elementary.fix_nanosecond_timing_values
>    ```
>
>    This updates the timing columns of `dbt_run_results` and `dbt_source_freshness_results` in place, truncating them
>    to microseconds. It scans both tables, and only needs to run once.
>
>    You can skip this step if no rows with nanosecond values are within the time range that your reports and alerts
>    read (`days_back`). For example, if you upgraded to 0.26.1 longer ago than that time range, the older rows are
>    never read. The same is true if you didn't run an earlier Elementary version with dbt Fusion or dbt Core 1.11.
>
> 2. Disable the truncation:
>
>    ```yaml dbt_project.yml theme={null}
>    vars:
>      bigquery_truncate_nanosecond_timestamps: false
>    ```
>
> > [!WARNING]
> >
> > If rows with nanosecond values are within that time range, run `fix_nanosecond_timing_values` before you set
> > `bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, queries that read those rows fail.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.