2 — Databricks Workspace

Pre-class Checklist — 2 weeks before

TRAINER ONLY

2 weeks before Step 2 of 9

Databricks Workspace (mhpdeworkshop_databricks)

Pre-provisioned — workspace already exists in resource group 1000_data_engineering_workshop; do not create a new one.

Item Value
Workspace mhpdeworkshop_databricks (ID 3359135813781456)
Resource group 1000_data_engineering_workshop (same as ADLS2)

    • Where: Azure Portal → workspace Overview → copy URL (e.g. https://adb-…azuredatabricks.net)
    • Configure: Add to credentials sheet / cohort portal for Module 2
    • Doc: Connect to your workspace

    • Where: Databricks → click your username (top-right) → Settings → Admin settings (or Admin console icon in sidebar)
    • Verify: Identity and access section is visible; you can open Users and Groups
    • Doc: Workspace admin settings

    • Where: Databricks → your username → Settings → Linked accounts → Git integration
    • Configure: Provider GitHub → paste a GitHub Personal Access Token (repo scope) → Save
    • Verify: Status shows connected
    • Doc: Connect your Git provider

    • Where: Databricks → Home → Git folder → open 00_setup.py → edit ATTENDEE_ID → attach cluster → Run all
    • Verify: Catalog → mhpdeworkshop_databricks_2026 shows tr_01_juewei_bronze, _silver, _gold
    • Doc: Git folders (Repos)

    • Verify: Schemas tr_02_alissa_bronze, _silver, _gold appear in Catalog

    • Where: Databricks → Home (your user folder) → ⋮ → Create → Git folder
    • Configure: URL https://github.com/jinjuewei/MHPDataEngineerWorkshop · Name MHP-DE-Workshop-2026 · Branch main · Sparse checkout → one cone pattern per line: databricks, streaming/databricks, ml/databricks
    • Doc: Create a Git folder

    • Where: Open Git folder → Git dropdown (top bar) → Pull
    • Note: Workspace does not auto-sync with GitHub — pull after every repo push before class
    • Doc: Update a Git folder

    • Where: Same as above — each trainer pulls independently

    • Where: Admin settings → Identity and access → Groups (or Account console → User management → Groups if account-level groups)
    • Configure: Add group for each name → add members (trainers to trainer group; all attendees to trainee group)
    • Doc: Manage groups

    • Where: Settings → Workspace admin → Identity and access → Groups → workshop_trainees_2026 → Entitlements tab (legacy UI: Admin settings → Identity and access)
    • Configure: Trainees need workspace, SQL, and Admin access for notebook labs. Keep unrestricted cluster creation off so they use the Workshop policy only. Set entitlements as follows:
    Entitlement Trainee group
    Workspace access On
    Databricks SQL access On
    Allow unrestricted cluster creation Off
    Admin access On

    Why Admin access is on: In practice, trainees hit permission errors in 00_setup.py (Unity Catalog schema/config steps) when Admin access is off — even after ALL PRIVILEGES on mhpdeworkshop_databricks_2026. Enabling Admin access on workshop_trainees_2026 avoids those failures for the workshop.

    Workshop trade-off: Trainees can open Settings → Workspace admin. That is acceptable for this time-boxed sandbox. Still keep Allow unrestricted cluster creation off so cluster create is limited to the Workshop compute policy (cost and isolation control).

    Do not enable Allow unrestricted cluster creation on workshop_trainees_2026. Trainers configure policies and catalog grants; individual trainer accounts stay in Settings → Workspace admin as needed.

    • Verify:
      1. Log in as a test trainee (member of workshop_trainees_2026 only) → run 00_setup.py on a Workshop cluster → schemas appear in mhpdeworkshop_databricks_2026
      2. Compute → Create compute → only policy Workshop is available (no unrestricted / personal compute outside policy)
      3. Cluster create succeeds under Workshop; 01_bronze_ingestion.py reads ADLS2 after facilitator shares the storage key
    • Doc: Default entitlements · Compute policies

    • Where: Workspace → Users → <trainer-email> → MHP-DE-Workshop-2026 → Share (top-right)
    • Configure: Add group workshop_trainees_2026 → permission Can Run
    • Verify: Log in as test trainee → open trainer folder → notebooks open and run (read-only)
    • Doc: Share workspace objects

    • Where: Admin settings → Identity and access → Users → Add user (or Invite user)
    • Configure: Enter attendee work emails → assign to workshop_trainees_2026 group
    • Verify: Each attendee accepted invite email and can log in
    • Doc: Add users to your workspace

    • Where: Databricks sidebar → Catalog → Create catalog (or + next to Metastore)
    • Configure: Name mhpdeworkshop_databricks_2026 · leave mhpdeworkshop_databricks_2025 unchanged
    • Doc: Create catalogs

    • Where: Catalog → mhpdeworkshop_databricks_2026 → Permissions tab → Grant
    • Configure:
      1. Principal workshop_trainees_2026 → ALL PRIVILEGES → OK
      2. Principal workshop_trainer_2026 → ALL PRIVILEGES → OK (or set trainers as Owner on Overview tab)
    • Verify: Permissions tab lists both groups — no per-attendee schema grants needed after 00_setup.py
    • Doc: Manage privileges in Unity Catalog > Workshop trade-off: shared sandbox — everyone can read/write all schemas in this catalog.

    Optional — same grants via SQL

     
    GRANT ALL PRIVILEGES ON CATALOG mhpdeworkshop_databricks_2026 TO `workshop_trainees_2026`;
GRANT ALL PRIVILEGES ON CATALOG mhpdeworkshop_databricks_2026 TO `workshop_trainer_2026`;

    • Where: Sidebar Compute → Policies tab → Create policy (or open existing Workshop → Edit)
    • Not this: Settings → Compute → Serverless usage policies is for serverless cost tags only — it does not create classic cluster policies (Module 2 needs compute policies on the Compute page)
    • Configure: Clone Shared Compute → Workshop · override runtime, 1 worker, node size, auto-terminate, runtime engine = Standard (no Photon) · 1 cluster per user (details below)
    • Permissions: Compute → Policies → Workshop → See all permissions → grant workshop_trainees_2026 + workshop_trainer_2026 → Can use · set Max compute resources per user = 1 (in policy Advanced options or permissions)
    • Verify: Policies tab shows Workshop; test trainee sees Workshop in Compute → Create compute → Policy; Photon acceleration is not selectable (policy locks Standard engine); can run 00_setup.py on UC catalog mhpdeworkshop_databricks_2026
    • If Policies tab is missing: Confirm Premium workspace, workspace admin role, and classic compute enabled — Settings → Compute → Serverless usage policies is the wrong screen
    • Doc: Create compute policies · Policy families (Shared Compute) · Compute policy reference · Cluster access modes · Serverless usage policies (not used here) · Trainee cluster guide

    Step-by-step — create the Workshop compute policy (UI)

    Purpose: Each trainee creates their own cluster (de-workshop-{ATTENDEE_ID}) under this policy. Trainers configure the policy and dry-run — they do not bulk-create roster clusters.

    Fast path: Compute → Policies → Shared Compute → ⋮ → Clone → rename Workshop. Shared Compute already gives Standard access (Unity Catalog) and multi-node — do not use Personal Compute (single-node) or Job Compute (policy families).

    What to set (essentials only)

    Setting Value Why it matters
    Max compute resources per user 1 One cluster per trainee
    Databricks Runtime 16.4 LTS (default); allow 15.4 LTS if 16.4 missing Matches lead-trainer dry-run
    Workers 1 Medallion labs need a worker, not single-node
    Node type Standard_DS3_v2 (driver + worker) Enough for PySpark labs; predictable cost
    Auto-terminate 30 min Idle cost control
    Runtime engine Standard (fixed; Photon disabled) Oct 2024 lab (~3.65M Bronze rows) does not need Photon; avoids higher DBU cost when Shared Compute defaults leave Photon acceleration checked

    Everything else can stay on Shared Compute defaults (access mode, all-purpose type, no pools, etc.). Skip spot/on-demand, tags, and libraries unless a dry-run shows a problem.

    Lock runtime engine (required): Shared Compute does not disable Photon by default. Add this definition (Definitions → Edit definition as JSON or append to the policy JSON):

    "runtime_engine": {
  "type": "fixed",
  "value": "STANDARD",
  "hidden": true
}

    Trainees then cannot enable Photon on Workshop clusters. Module 2 still teaches Photon on Slide 11 (SQL warehouses, serverless, production ETL) — the lab cluster deliberately uses standard Spark for cost and simplicity (disable Photon via policy).

    Definitions UI: Open inherited rows → Edit the six settings above. If a row is missing, Add definition. Labels vary by workspace (Databricks runtime version, Runtime engine, Node type, …) — match by meaning, not exact name.

    Permissions

    1. Create / Save the policy
    2. See all permissions → grant workshop_trainees_2026 + workshop_trainer_2026 → Can use

    Verify (5 minutes)

    1. Compute → Create compute → Policy Workshop → confirm Standard UC access, 16.4 LTS, 1 worker, Standard runtime engine (no Photon acceleration toggle)
    2. Create de-workshop-tr_01_juewei → run 00_setup.py → schemas in mhpdeworkshop_databricks_2026
    3. Log in as test trainee → Workshop appears → create succeeds

    Optional: Workshop-ML (Module 9 only)

    Clone Power User Compute → Workshop-ML · set runtime to 16.4 LTS ML (CPU) · same groups · 1 cluster per user. Trainees use Workshop for Modules 2–6 and Workshop-ML only for Module 9 (ML setup).

    Advanced: full attribute list and JSON examples — compute policy reference.

    • Where: Compute → attach cluster with DBR 16.4 → run notebooks 00_setup → 03_gold_kpis
    • Verify: Bronze/Silver/Gold tables populated; optional Module 8/9 if offered
    • Doc: Create a cluster

    • Who creates clusters: Each trainee — Compute → Create compute under policy Workshop, name de-workshop-{ATTENDEE_ID} (during pre-class or at start of Module 2)
    • Trainer pre-class: Policy + Can use on workshop_trainees_2026 (above) · lead-trainer dry-run on a trainer cluster only
    • Trainer verify: Log in as a test trainee (or dedicated test user in workshop_trainees_2026) → Create compute → policy Workshop appears → create succeeds · point trainees to Databricks setup § Create cluster
    • Doc: Create compute policies · Trainee cluster guide

    • Where: Pre-class email + Databricks setup
    • Configure: Trainees open trainer shared folder and clone 00_setup.py to Home (Option A Git folder or Option B import)

    • Where: Databricks sidebar → SQL → SQL Warehouses → Create SQL warehouse
    • Configure: Name de-workshop-wh · Type Pro or Serverless · Size 2X-Small · Auto-stop 5 min
    • Permissions: Warehouse → Permissions → workshop_trainees_2026 + workshop_trainer_2026 → Can use
    • Doc: Create a SQL warehouse

    • Where: Account console → Settings → Partner-powered AI (account admin) + workspace Admin settings → AI/BI if shown
    • Configure: Partner-powered AI On for the workspace
    • Verify: Trainer dry-run Assistant, ai_query() in SQL warehouse, Genie space — see facilitator guide § Module rhythm
    • Doc: Databricks AI assistive features

Next 3 — Snowflake (trainer) Needs ADLS2 SAS token (page 1)