Administer Your Databricks Account with Terraform

 

Managing your Databricks environment can be accomplished through several methods: the UI (user interface), Databricks API, Databricks CLI (Command Line Interface), and Terraform. Each method offers unique benefits, but for robust administration and version control, Terraform stands out.

Why Choose Terraform Over UI or SQL Commands?

While the Databricks UI and SQL commands offer simplicity and ease of use, they have limitations, especially when it comes to scalability, consistency, and version control. Here’s why Terraform is the superior choice:

1. Version Control and Auditability

UI and SQL Commands: Changes made through the UI or SQL commands lack inherent version control, making it difficult to track who made what changes and when. This can lead to challenges in maintaining a clear audit trail and managing rollbacks.

Terraform: Terraform configurations are stored as code, which means they can be versioned using tools like Git. This ensures that every change is documented, auditable, and reversible. You can easily track modifications over time and collaborate with your team more effectively.

2. Consistency and Repeatability

UI and SQL Commands: Manual changes through the UI or SQL commands are prone to human error and inconsistencies. Repeating the same setup across multiple environments can be time-consuming and error-prone.

Terraform: Terraform allows you to define your infrastructure declaratively. Once written, the same configuration can be applied consistently across multiple environments, ensuring that your setup is repeatable and error-free.

3. Automation and Efficiency

UI and SQL Commands: Performing administrative tasks manually can be tedious and time-consuming, especially for large-scale environments. Automation is limited and often requires custom scripting.

Terraform: Terraform automates the provisioning and management of your infrastructure. You can script complex setups and changes, reducing the manual effort required and speeding up deployment times. Terraform integrates well with CI/CD pipelines, enabling fully automated infrastructure deployments.

4. Scalability

UI and SQL Commands: As your environment grows, managing it through the UI or SQL commands becomes increasingly cumbersome and less efficient.

Terraform: Terraform is designed to handle infrastructure at scale. It can manage complex environments with numerous resources efficiently, providing a scalable solution that grows with your organization.

Example: Granting Privileges with Terraform

Let’s dive into a practical example to see how Terraform can simplify and enhance the management of your Databricks privileges.

Simple Privilege Grant

Suppose you need to grant the USE_CATALOG privilege to the data_engineer group on a development catalog. Here’s how you can do it with Terraform:

// Grants on dev catalog
resource "databricks_grants" "dev_catalog" {
  catalog = databricks_catalog.dev.name
  grant {
    principal  = "data_engineer"
    privileges = ["USE_CATALOG"]
  }
}

This configuration is concise, version-controlled, and can be reused across different environments.

Advanced Example: Managing Schema Access

Assume you have additional groups, such as data_scientist and data_analyst, and you want to manage their access to the gold schema. You might want to restrict access for these groups while granting more extensive privileges to the data_engineer group. Here’s how you can achieve this with Terraform:

// Grants on gold schema
resource "databricks_grants" "gold" {
  schema = databricks_schema.gold.id
  grant {
    principal  = "data_engineer"
    privileges = ["USE_SCHEMA",
                  "CREATE_TABLE", 
                  "EXECUTE", "MODIFY", "SELECT"]
  }
  grant {
    principal  = "data_scientist"
    privileges = ["USE_SCHEMA", "SELECT"]
  }
  grant {
    principal  = "data_analyst"
    privileges = ["USE_SCHEMA", "SELECT"]
  }
}

Best Practices

Databricks recommends grouping users and then granting privileges to these groups. This strategy simplifies management and ensures that permissions are applied consistently across your organisation. Using Terraform for this purpose brings additional benefits:

• Version Control: Track changes over time and roll back if necessary.
• Collaboration: Work together with your team on infrastructure code using version control systems like Git.
• Automation: Automate the deployment of infrastructure changes, reducing the risk of human error.

By integrating Terraform into your Databricks administration workflow, you can achieve a more scalable, maintainable, and collaborative approach to managing your data platform.