DreamLake managed Experiments

Python SDK 0.10.0 provides this managed transport. It requires the @dreamlake/ml-dash administrative CLI 0.2.0 and matching enabled DreamLake and Dash servers. Verify the installed client versions and server capabilities before setup; a successful health check alone does not establish support.

Managed mode uses your DreamLake organization and project permissions. You do not create a second Dash organization or invite researchers twice. The backend creates an internal experiment workspace and records immutable project and instance identities. Independent Dash projects and their existing login remain available through the standalone commands.

Sign in and choose a project

Use the existing DreamLake login, or the new administrative CLI's managed login:

ml-dash auth login
ml-dash auth status --json
ml-dash orgs list --json
ml-dash projects list --namespace my-org --json
ml-dash projects access --namespace my-org --project research --json

An organization member's default project access is READ. An owner or an explicit WRITE/ADMIN project grant is required for experiment creation and logging. Belonging to an organization alone does not imply write permission.

An administrator or authorized writer can ask an agent to inspect setup, review the exact resources, apply the same plan idempotently, and verify the result. Planning does not create resources. Retain the review, intent ID, and receipt so an interrupted apply can resume without guessing a new destination.

Review, apply and verify setup

ml-dash instance capabilities --json
ml-dash setup plan --namespace my-org --project research --out setup-review.json --json
# Review the exact stable namespace/project/instance IDs, revision and plan hash.
ml-dash setup apply --namespace my-org --project research --apply --review setup-review.json --idempotency-key SETUP_INTENT_ID --out setup-receipt.json --json
ml-dash setup status --namespace my-org --project research --json

Use a stable intent ID and retain the review and receipt. If the apply response is lost, inspect status and use setup resume with the same review and intent; do not substitute a new target or a current revision silently. A conflict calls for a new explicit review. Review files never contain signing keys or tokens.

Log with Python

The SDK 0.10.0 managed transport reuses the current DreamLake login. It does not read a Dash token or put workload credentials in the experiment plan or training source.

from uuid import uuid4
from ml_dash import Experiment, ManagedConnection

connection = ManagedConnection.from_login("my-org", "research")
connection.setup()  # authorized, idempotent setup; no duplicate Dash organization

with Experiment(
    prefix=f"my-org/research/train-{uuid4().hex}",
    managed=connection,
).run as exp:
    exp.params.set(learning_rate=0.001, batch_size=32)
    for step in range(60):
        exp.metrics("train").log(step=step, loss=1.0 / (step + 1))
    exp.flush()

The prefix names the DreamLake project in this mode. The SDK obtains a project-scoped workload handle, keeps its short-lived credential in memory, and renews using current DreamLake project permission. Changing the configured environment cannot send that environment's login to an existing connection. The server supplies the web page URL; the SDK does not infer it from an API host.

Call exp.flush() to send pending data. exp.metrics.flush() is retained as a fluent no-op. The existing ingestion pipeline queues metrics before Firehose and Arrow commit; a successful flush is not a claim that storage has committed. Verify numeric data through the DreamLake page or committed metric-row API, retrying the read while ingestion completes. Errors must remain visible rather than appear as an empty successful experiment.

Read the committed result

Use the experiment page URL supplied by the SDK/server, or inspect the same run through the CLI. Select the stable run ID from the returned authorized inventory:

ml-dash runs list --namespace my-org --project research --json
ml-dash metrics list --namespace my-org --project research --run RUN_ID --json
ml-dash metrics read --namespace my-org --project research --run RUN_ID --metric train --limit 100 --json
ml-dash logs read --namespace my-org --project research --run RUN_ID --json
ml-dash files list --namespace my-org --project research --run RUN_ID --json

Cursor pages retain nextCursor; repeat the same query with --cursor to continue. A bounded first page is not a complete inventory. For a full administrative and workload walkthrough—including review/apply, scoped writer rotation/retirement, file transfer, legacy adoption and external bindings—see the CLI reference.

Permissions and credential recovery

• READ permits viewing runs, metrics, logs, and authorized artifacts.
• WRITE/ADMIN permits setup, creating runs, parameters, metrics, logs, artifacts, and run status changes. A normal writer credential does not permit deletion.
• Deleting an experiment requires an explicit delete session and retains its stored data. It does not delete or recreate the project.
• Workload credentials do not manage organization membership or other users' credentials. An optional run scope also prevents listing sibling runs.
• Renewal checks live project permission. Previously issued capabilities expire within 60 seconds plus at most 5 seconds of clock tolerance after revocation. Retiring a workload or changing its generation rejects its old credentials on the next request. Rotation verifies a candidate before retiring the old one.

If permission is removed, obtain the correct project grant. Do not add a second Dash membership to bypass managed authorization. If a service or mapping changes, inspect the saved receipt; retries never silently redirect to a different instance.

Managed compute jobs (unreleased follow-up)

This workflow is under development and has no newly assigned release version. Published SDK 0.10.0 and administrative CLI 0.2.0 do not include the job broker. Installing them or seeing a healthy Dash service does not enable job reporting.

First verify managed setup for the existing organization project. Its binding must be ready, with stable namespace, project and service instance IDs. The submitter needs current project WRITE/ADMIN permission to report experiment data. They separately need existing machine/host and execution authorization to launch the task. Project WRITE grants logging; it grants no host access, enrollment, shell execution or cluster provisioning rights.

A matching control plane must preserve the version 1 reporting envelope and select a Nymph worker advertising experiment_reporting: 1. The worker must have reporting explicitly enabled with a fixed trusted DreamLake API origin; the task envelope must match that origin. Arbitrary job-provided API URLs and older workers without reporting support are refused. The BFF reporting feature must also be enabled and use the same trusted origin. Production origins use HTTPS; loopback exceptions require explicitly enabled local testing.

The platform ensures one stable experiment run for the tracked job and creates an exact job/run-scoped credential, owned by the original submitter. The job, run, attempt and instance remain fixed. Retries reuse verified identities; they do not create an empty replacement project or adopt a name-matched run. The worker brokers short-lived read/write leases and supplies local broker coordinates to the child process. New SDK source can use Experiment() in that environment, or explicitly select ManagedConnection.from_job(), to log to the platform-created run. It creates no second run and does not fall back to standalone login if reporting is unavailable. These are development APIs, not instructions for the released 0.10.0 package.

The worker supplies DREAMLAKE_EXPERIMENT_BROKER and DREAMLAKE_EXPERIMENT_BROKER_TOKEN. Keep these out of training source, saved configuration, review files, logs and durable journals. User login tokens and persistent Dash credentials are not injected into jobs. The broker is restricted to literal HTTP loopback and an explicit port. Each renewal checks the live task attempt and current project authorization. Leases last at most 60 seconds with at most 5 seconds clock tolerance. Retired or superseded generations are rejected on the next Dash request.

Interactive credentials cannot renew, rotate or retire a job-owned handle, even for the same user. Job management is a separate internal capability; ordinary read/write leases cannot provision projects, delete runs or manage credentials. Cancellation, terminal jobs, revoked access, stale attempts and changed destinations fail closed. Only the BFF's observation of a confirmed control-plane terminal result may synchronize the job run to COMPLETED, FAILED or CANCELLED. A cancellation request is not completed cancellation. SDK close only flushes; worker finish and credential retirement do not decide the run's terminal status. Repeating the same terminal result is safe; a terminal run is never revived. Credential retirement still cleans up the exact handle after run soft deletion. Inspect authorized job/experiment links and committed metrics to verify results; process exit alone establishes neither an application outcome nor committed metric storage.

Existing data and independent services

Adopting an existing populated Dash project requires a complete membership/grant inventory, explicit authority on both systems, reviewed plan, verified backup, writer drain, and signed operator checkpoint. Adoption preserves IDs and data; it does not union the old grants into DreamLake permissions. A later setup retry must reuse the adopted project, not create an empty replacement. Recovery retains the inventory and receipts and refuses to recreate revoked grants.

External integration is a separate mode with two authorities. An operator adds compatible HTTPS endpoints to a server allowlist, and users select a verified alias. The service negotiates identity, delegation, capability, and instance contracts. An arbitrary URL plus a personal token is not a multiuser connection. Neither mode changes compute launch authority or provisions a cluster.

The unreleased pagination follow-up lists external workspaces with an opaque cursor and stable namespace IDs. Each page filters active organization namespaces and current Dash OWNER membership before the limit. Exact selection traverses bounded pages and rejects repeated cursors, duplicate IDs and instance changes. Equal names never establish a binding. See the CLI reference for the new cursor contract and its release status.

Development limits

The managed SDK currently accepts a single run segment under the project, parameters, numeric metrics, logs, lifecycle status, and root attachments up to 8 MiB. Nested folders, run/metric metadata, SDK write protection, and metadata on attachments are not yet supported by this managed transport. It rejects these options rather than discarding them. Standalone SDK behavior is unchanged. Source-backed storage and Source migration are excluded from this work.