# Folder (Files)

Upload and manage experiment artifacts - models, plots, configs, and results. Files are automatically checksummed and organized with metadata.

## Fluent Interface Overview

The folder API uses a fluent interface that supports multiple styles:

```python

from ml_dash import Experiment

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Upload file from disk
    exp.files("checkpoints").upload("./model.pt")
    exp.files("checkpoints").upload("./model.pt", to="checkpoint.pt")

    # Save objects as files
    exp.files("models").save_torch(model, to="model.pt")
    exp.files("configs").save_json(config, to="config.json")

    # List files in a location
    files = exp.files("/models").list()

    # Download a file
    exp.files("some.text").download()
    exp.files("some.text").download(to="./local_copy.text")

    # Download using glob patterns
    file_paths = exp.files("images").list("*.png")
    exp.files("images").download("*.png", to="local_images")

    # Delete files
    exp.files("some.text").delete()
    exp.files.delete("some.text")
    exp.files.delete("images/*.png")

    # Specific file types
    exp.files.save_text("content", to="view.yaml")
    exp.files.save_json(dict(hey="yo"), to="config.json")
    exp.files.save_blob(b"xxx", to="data.bin")
```

## Basic Upload

### Upload Existing File

```python

from ml_dash import Experiment

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Upload a file to a prefix
    result = exp.files("models").upload("./model.pth")
    print(result)  # {'id': 'pending', 'status': 'queued'}
```

Uploads are [buffered](/guides/buffering.md) by default, so `upload()` and the
`save*()` methods return `{"id": "pending", "status": "queued"}` and write the
file in the background. The file can't be listed or downloaded until the buffer
flushes (at the latest when the `with` block exits). To get the stored record
(`filename`, `sizeBytes`, `checksum`, `id`, ...) back from the call itself,
set `ML_DASH_BUFFER_ENABLED=false` before creating the experiment, or read it
later with `exp.files("models").list()`.

### Save Objects as Files

Save Python objects directly without creating intermediate files:

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Save dict/list as JSON
    config = {"model": "resnet50", "lr": 0.001}
    exp.files("configs").save_json(config, to="config.json")

    # Save bytes directly
    exp.files("data").save_blob(b"binary data", to="data.bin")

    # Save PyTorch model
    import torch
    model = torch.nn.Linear(10, 5)
    exp.files("checkpoints").save_torch(model, to="checkpoint.pt")
    exp.files("checkpoints").save_torch(model.state_dict(), to="weights.pt")
```

### Direct Method Style

You can also use the direct method style without specifying a prefix:

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Upload file directly
    exp.files.upload("./model.pt", to="models/model.pt")

    # Save objects directly
    exp.files.save_text("yaml content", to="configs/view.yaml")
    exp.files.save_json({"key": "value"}, to="data/config.json")
    exp.files.save_blob(b"\x00\x01\x02", to="binary/data.bin")
```

## Organizing Files

Use paths to organize files logically:

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Models
    exp.files("models").upload("model.pth")
    exp.files("models/checkpoints").upload("best_model.pth")

    # Visualizations
    exp.files("visualizations").upload("loss_curve.png")

    # Configuration
    exp.files("config").save_json(config, to="config.json")

    # Results
    exp.files("results").upload("results.csv")
```

## Listing Files

### List All Files

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # List all files
    files = exp.files().list()

    for file_info in files:
        print(f"File: {file_info['filename']}")
        print(f"  Path: {file_info['path']}")
        print(f"  Size: {file_info['sizeBytes']} bytes")
        print(f"  ID: {file_info['id']}")
```

### List by Prefix

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # List files in specific prefix
    model_files = exp.files("/models").list()
    config_files = exp.files("/configs").list()
```

### List with Glob Pattern

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # List files matching pattern
    png_files = exp.files("images").list("*.png")
    model_files = exp.files().list("*.pt")
    all_configs = exp.files().list("**/*.json")
```

## Downloading Files

### Download Single File

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Download by filename/path
    exp.files("model.pt").download()  # Downloads to current directory
    exp.files("model.pt").download(to="./local_model.pt")  # Custom destination

    # Download from specific prefix
    exp.files("models/best.pt").download(to="./best_model.pt")
```

### Download with Glob Pattern

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Download all PNG files from images prefix
    paths = exp.files("images").download("*.png", to="./local_images")

    # Direct style with path/pattern
    paths = exp.files.download("images/*.png", to="local_images")

    print(f"Downloaded {len(paths)} files")
```

### Download by File ID (Legacy)

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Look up the ID of a file uploaded in an earlier run
    file_id = exp.files("models").list()[0]["id"]

    # Download by ID
    downloaded_path = exp.files(file_id=file_id).download()
    print(f"Downloaded to: {downloaded_path}")
```

### Checksum Verification

Downloads automatically verify checksums to ensure file integrity. Download
from a later run, after the upload has been flushed:

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # The stored SHA-256 checksum
    print(exp.files("models").list()[0]["checksum"])

    # Download (checksum verified automatically)
    downloaded = exp.files("model.pth").download(to="./verified_model.pth")
    print(f"Download verified and saved to: {downloaded}")
```

## Deleting Files

### Delete Single File

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Delete by filename/path
    result = exp.files("some.text").delete()

    # Direct style
    result = exp.files.delete("some.text")
```

### Delete with Glob Pattern

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Delete all PNG files from images prefix
    results = exp.files("images").delete("*.png")

    # Direct style
    results = exp.files.delete("images/*.png")

    print(f"Deleted {len(results)} files")
```

## File Metadata

Add description, tags, and custom metadata:

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    result = exp.files("models",
        description="Best model from epoch 50",
        tags=["checkpoint", "best"],
        metadata={"epoch": 50, "val_accuracy": 0.95, "optimizer_state": True},
    ).save_torch(model, to="best_model.pth")
```

## Saving Specific File Types

### Save Text

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Save text content
    yaml_content = """
    model:
      architecture: resnet50
      pretrained: true
    """
    exp.files("configs").save_text(yaml_content, to="model.yaml")

    # Or using direct style
    exp.files.save_text(yaml_content, to="configs/model.yaml")
```

### Save JSON

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    config = {"model": "resnet50", "lr": 0.001}

    # Save JSON
    exp.files("configs").save_json(config, to="config.json")

    # Or direct style
    exp.files.save_json(config, to="configs/training.json")
```

### Save Binary Data

```python

with Experiment(prefix="alice/project/my-experiment").run as exp:

    binary_data = b"\x00\x01\x02\x03"

    # Save blob
    exp.files("data").save_blob(binary_data, to="weights.bin")

    # Or direct style
    exp.files.save_blob(binary_data, to="data/embeddings.bin")
```

## Training with Checkpoints

Save models during training:

```python

import torch
from ml_dash import Experiment

with Experiment(prefix="alice/cv/resnet-training").run as exp:

    exp.params.set(model="resnet50", epochs=100)
    exp.log("Starting training")

    best_accuracy = 0.0

    for epoch in range(100):
        train_loss = train_one_epoch(model, train_loader)
        val_loss, val_accuracy = validate(model, val_loader)

        # Log metrics (single call with nested dict)
        exp.metrics.log(
            epoch=epoch,
            train=dict(loss=train_loss, accuracy=val_accuracy),
            eval=dict(loss=val_loss, accuracy=val_accuracy)
        )

        # Alternative: prefix-based logging
        # exp.metrics("train").log(loss=train_loss, accuracy=val_accuracy, epoch=epoch)
        # exp.metrics("eval").log(loss=val_loss, accuracy=val_accuracy, epoch=epoch)

        # Save checkpoint every 10 epochs
        if (epoch + 1) % 10 == 0:
            exp.files("checkpoints",
                tags=["checkpoint"],
                metadata={"epoch": epoch + 1, "val_accuracy": val_accuracy},
            ).save_torch(model.state_dict(), to=f"checkpoint_epoch_{epoch + 1}.pt")

        # Save best model
        if val_accuracy > best_accuracy:
            best_accuracy = val_accuracy

            exp.files("models",
                description=f"Best model (accuracy: {best_accuracy:.4f})",
                tags=["best"],
                metadata={"epoch": epoch + 1, "accuracy": best_accuracy},
            ).save_torch(model.state_dict(), to="best_model.pt")

            exp.log(f"New best model saved (accuracy: {best_accuracy:.4f})")

    exp.log("Training complete")
```

## Saving Visualizations

Upload matplotlib plots using the convenient `save_fig()` method:

```python

import matplotlib.pyplot as plt
import numpy as np
from ml_dash import Experiment

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Generate plot
    losses = [0.5, 0.4, 0.3, 0.25, 0.2]
    plt.plot(losses)
    plt.title("Training Loss")
    plt.xlabel("Epoch")
    plt.ylabel("Loss")

    # Save directly (auto-closes figure)
    exp.files("visualizations").save_fig(to="loss_curve.png")

    # Save as PDF with custom DPI
    xs = np.linspace(-5, 5, 100)
    plt.plot(xs, np.cos(xs), label='Cosine')
    plt.legend()
    exp.files("visualizations").save_fig(
        to="cosine_function.pdf",
        dpi=150,
        transparent=True,
        bbox_inches='tight'
    )
```

**Note:** `save_fig()` automatically closes the figure after saving to prevent memory leaks.

## Saving Videos

Upload video frame stacks using the `save_video()` method. This is useful for saving training visualizations, agent rollouts, or any sequence of images.

`save_video()` needs the video extra, which installs imageio and ffmpeg:

```bash
pip install "ml-dash[video]"
```

```python

import numpy as np
from ml_dash import Experiment

with Experiment(prefix="alice/project/my-experiment").run as exp:

    # Generate frame stack
    frames = [np.random.rand(200, 200) for _ in range(20)]

    # Save as MP4 (default 20 FPS)
    exp.files("videos").save_video(frames, to="animation.mp4")

    # Save with custom FPS
    exp.files("videos").save_video(frames, to="animation.mp4", fps=30)

    # Save as GIF
    exp.files("videos").save_video(frames, to="animation.gif")
```

### Practical Example: Agent Rollout

Record an agent's trajectory or any animated visualization:

```python

import numpy as np
from ml_dash import Experiment

def render_frame(x, y):
    """Render a frame with a marker at position (x, y)."""
    canvas = np.zeros((200, 200))
    # Draw a 10x10 square at position
    canvas[max(0, x-5):x+5, max(0, y-5):y+5] = 1.0
    return canvas

with Experiment(prefix="alice/rl/agent-rollout").run as exp:

    # Simulate agent moving across the canvas
    frames = [render_frame(100 + i, 80) for i in range(20)]

    exp.files("videos").save_video(frames, to="rollout.mp4")
```

### Video Encoding Options

Control video quality and encoding with additional parameters:

```python

# High quality MP4
exp.files("videos").save_video(frames, to="high_quality.mp4", fps=30, quality=8)

# Lower quality for smaller file size
exp.files("videos").save_video(frames, to="compressed.mp4", fps=30, quality=5)
```

Additional keyword arguments are passed to imageio's writer (e.g., `quality`, `codec`, `bitrate`).

### Frame Format Support

`save_video()` automatically handles various frame formats:

```python

# Grayscale frames (H×W)
frames = [np.random.rand(480, 640) for _ in range(30)]
exp.files("videos").save_video(frames, to="grayscale.mp4")

# RGB frames (H×W×3)
frames = [np.random.rand(480, 640, 3) for _ in range(30)]
exp.files("videos").save_video(frames, to="rgb.mp4")

# Stacked array (N×H×W or N×H×W×C)
frames = np.random.rand(30, 480, 640, 3)
exp.files("videos").save_video(frames, to="stacked.mp4")
```

**Frame value ranges:**
- Float values (0.0 to 1.0) - automatically scaled to 0-255
- Uint8 values (0 to 255) - used directly
- Float32 values - automatically converted

**Note:** An empty frame list raises `ValueError: frame_stack is empty`.

## Storage Format

**Local mode** - Files stored with prefix-based organization:

```
.dash/
└── alice/                              # owner
    └── project/                        # project
        └── my-experiment/              # experiment name
            └── files/
                ├── models/
                │   ├── 7218065541365719/
                │   │   └── model.pth
                │   └── 7218065541366823/
                │       └── best_model.pth
                ├── visualizations/
                │   └── 7218065541367921/
                │       └── loss_curve.png
                └── config/
                    └── 7218065541368015/
                        └── config.json
```

Each file is stored as: `files/{prefix}/{snowflake_id}/{filename}`
- **prefix**: Logical organization path (e.g., "models", "configs", "visualizations")
- **snowflake_id**: Unique identifier generated for each file
- **filename**: Original filename

**Remote mode** - Files uploaded to S3, metadata in MongoDB:
- Files stored: `s3://bucket/files/{namespace}/{project}/{experiment}/{prefix}/{file_id}/filename`
- Metadata: path, size, SHA256 checksum, tags, description

**File size limit:** the ML-Dash server accepts up to 256MB per file and
rejects larger uploads with `413 Payload Too Large`. The SDK itself refuses
files over 100GB, which is the only limit in local mode.

---

**That's it!** You've completed all the core ML-Dash tutorials. Check out the API Reference for detailed method documentation.