Skip to main content
This guide contains the necessary steps to connect a Databricks environment to your Elementary account.

​
Create service principal

  1. Open your Databricks console, and then open your relevant workspace.
  2. Click on your Profile icon on the right and choose Settings.
Choose settings
  1. On the sidebar, click on Identity and access, and then under the Service Principals row click on Manage.
Choose settings
  1. Click on the Add service principal button, choose “Add new” and give a name to the service principal. This will be used by Elementary Cloud to access your Databricks instance.
Add service principal
  1. Click on your newly created service principal, add the “Databricks SQL access” entitlement, and click Update. Also, please copy the “Application ID” field as it will be used later in the permissions section.
Add databricks SQL access
  1. Next, you may also need to allow Token Usage for this service principal (if it is not allowed for all users). To do so, under the settings menu choose Advanced -> Personal Access Tokens -> Permission Settings. Then add the service principal there.
Add databricks SQL access
  1. Create a personal access token for your service principal. For more details, please click here
  2. Finally, in order to enable Elementary’s automated monitors feature, please ensure predictive optimization is enabled in your account. This is required for table statistics to be updated (Elementary relies on this to obtain up-to-date row counts)

​
Permissions and security

​
Required permissions

Elementary cloud requires the following permissions:
  • Elementary schema read-only access - This is required by Elementary to read dbt metadata & test results collected by the Elementary dbt package as a part of your pipeline runs. This permission does not give access to your data.
  • Information schema metadata access - Elementary needs access to the system.information_schema.tables and system.information_schema.columns system tables, to get metadata about existing tables and columns in your data warehouse. This is used to power features such as column-level lineage and automated volume & freshness monitors.
  • Read access needed for some metadata operations (optional) - In order to enable Elementary’s automated volume & freshness monitors, Elementary needs access to query history, as well as Databricks APIs to obtain table statistics. These operations require granting SELECT access on your tables. This is a Databricks limitation - Elementary never reads any data from your tables, only metadata. However, there isn’t today any table-level metadata-only permission available in Databricks, so SELECT is required.

​
Grants SQL template

Please use the following SQL statements to grant the permissions specified above (you should replace the placeholders with the correct values): 
-- Grant read access on the elementary schema (usually [your dbt target schema]_elementary)
GRANT USE CATALOG ON CATALOG <catalog> TO `<service_principal_app_id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA <elementary_schema> TO `<service_principal_app_id>`;

-- Grant access to information schema tables
GRANT USE CATALOG ON CATALOG system TO `<service_principal_app_id>`;
GRANT USE SCHEMA ON SCHEMA system.information_schema TO `<service_principal_app_id>`;
GRANT SELECT ON TABLE system.information_schema.tables TO `<service_principal_app_id>`;
GRANT SELECT ON TABLE system.information_schema.columns TO `<service_principal_app_id>`;

-- Grant select on tables for history & statistics access
-- (Optional, required for automated volume & freshness tests - see explanation above. You can also limit to specific schemas used by dbt instead of granting on the full catalog)
GRANT USE CATALOG, USE SCHEMA, SELECT ON catalog <catalog> to `<service_principal_app_id>`;

​
Add an environment in Elementary (requires an admin user)

In the Elementary platform, go to Environments in the left menu, and click on the “Create Environment” button. Choose a name for your environment, and then choose Databricks as your data warehouse type. Provide the following fields in the form:
  • Server Host: The hostname of your Databricks account to connect to.
  • Http path: The path to the Databricks cluster or SQL warehouse.
  • Access token: The token you generated for the Elementary service principal (see step 7 under “Create service principal” above)
  • Catalog (optional): The name of the Databricks Catalog.
  • Elementary schema: The name of your Elementary schema. Usually [your dbt target schema]_elementary.

​
Add the Elementary IP to allowlist

Elementary IP for allowlist: 3.126.156.226

​
Need help with onboarding?

We can provide support on Slack or hop on an onboarding call.