Optimization Modes

A mode is the single highest-level knob in MemScale. It is set through the OptimizationMode enum and tells the decision engine how aggressively to trade throughput for memory.

from memscale import Config, OptimizationMode
 
config = Config(mode=OptimizationMode.BALANCED)  # the default

The three modes

BALANCED is the default when no mode is given.

How a mode maps to techniques

A mode does not apply techniques directly — it sets the defaults the decision engine starts from, and the engine then decides which layers actually receive each technique. The Config defaults reflect this:

• enable_checkpointing — on by default
• enable_offloading — on by default
• enable_tiling — off by default (more experimental)
• use_mixed_precision — off by default
• use_8bit_optimizer — off by default

Choosing AGGRESSIVE raises the engine’s willingness to apply the heavier techniques (offloading and tiling) to more layers; CONSERVATIVE restricts it to the cheapest ones.

Predicted vs. measured reduction

Mode names describe intent, not a guaranteed percentage — the actual reduction depends on the model, batch size, and sequence length. The benchmark suite measured 68–76% peak-VRAM reduction across BERT and GPT-2 models on an RTX 3090 with the default (balanced) configuration.

Choosing a mode

• Model nearly fits, you need a little headroom → CONSERVATIVE
• Typical medium model, want a good default → BALANCED
• Model OOMs and must fit → AGGRESSIVE

The Memory Budget guide walks through this choice by model size.