Domain 4 of 4 · Chapter 3 of 4

Development Lifecycle in Azure Databricks

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Implementing Data Engineering Solutions Using Azure Databricks premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Version control with Git folders
  • Promote from a pinned Git reference
  • A layered testing strategy
  • Databricks Asset Bundles: the project as code
  • Deploy, promote, and automate across environments
  • Exam-pattern recognition

Choosing a CI/CD deployment approach on Azure Databricks

AspectDatabricks Asset BundlesProduction Git folderGit-with-jobsTerraform provider
What it version-controlsCode plus resources in databricks.ymlCode files and notebooks onlyCode files only; job config not in GitWorkspaces and infrastructure
Deploys jobs and pipelines as code?YesNo, code onlyNo, job pulls code at run timeYes
Best whenFull CI/CD across dev, staging, prodYou deploy only code with external CI/CDA job runs a pinned Git ref, limited task typesProvisioning infra or many workspaces
Databricks recommendationRecommended defaultWhen bundles are more than neededRapid iteration to pinned production runsInfrastructure as code

Decision tree

Deploy resources as code?(jobs/pipelines, not just notebooks)External CI/CD available?e.g. GitHub ActionsProvisioning infra ormany workspaces?Production Git folderpull code on mergeGit-with-jobspull pinned ref at runTerraform providerinfra as codeDatabricks AssetBundles (default)NoYesYesNoYesNo
Cheat sheet

Cheat sheet

  • Databricks Git folders connect the workspace to a remote Git provider
  • Production jobs should reference a specific branch or tag, not a mutable copy
  • Never commit secrets to a Git folder
  • Branch, commit, and push from the Git folder UI; open PRs in the provider
  • Resolve merge conflicts by pulling main and reconciling text diffs
  • A merge to main can trigger CI that deploys the change
  • A layered strategy spans unit, integration, end-to-end, and UAT tests
  • Unit tests run with pytest; integration and E2E tests exercise real jobs on test data
  • A bundle can define a test job so CI runs tests on every deploy
  • A bundle is defined in databricks.yml with resources and targets
  • Development mode isolates deployments; production mode enforces production settings
  • Variables and target overrides parameterize a bundle per environment
  • databricks bundle validate checks the configuration before deploying
  • databricks bundle deploy -t promotes a bundle to a target environment
  • The REST API deploys resources programmatically using a service principal token
  • databricks bundle destroy removes deployed resources

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Azure Databricks Git folders concepts
  2. CI/CD with Databricks Git folders
  3. Secret management
  4. Use Git with Lakeflow Jobs
  5. Unit testing for Databricks notebooks
  6. What are Declarative Automation Bundles?
  7. Substitutions and variables in Declarative Automation Bundles
  8. bundle command group
  9. Declarative Automation Bundles deployment modes
  10. Service principals for CI/CD