Experiment Runner

This module ties everything together and provides both CLI and programmatic interfaces for running experiments.

Experiment Configuration

class structured_stochasticity.experiment.ExperimentConfig(model_name='meta-llama/Llama-3.2-1B', device='cuda', torch_dtype='float16', injection_layers=<factory>, noise_scale=0.1, noise_strategy='gaussian', injection_mode='continuous', task_name='tower_of_hanoi', complexity_range=(3, 7), trials_per_complexity=10, k_values=<factory>, selection_method='majority_vote', max_new_tokens=1024, output_dir='experiments', experiment_name=None)

Bases: object

Configuration for an experiment run.

Configuration Sections:

• Model: model_name, device, torch_dtype

• Injection: injection_layers, noise_scale, noise_strategy, injection_mode

• Task: task_name, complexity_range, trials_per_complexity

• Evaluation: k_values, selection_method, max_new_tokens

• Output: output_dir, experiment_name

Parameters:

• model_name (str)

• device (str)

• torch_dtype (str)

• injection_layers (list[int])

• noise_scale (float)

• noise_strategy (str)

• injection_mode (str)

• task_name (str)

• complexity_range (tuple[int, int])

• trials_per_complexity (int)

• k_values (list[int])

• selection_method (str)

• max_new_tokens (int)

• output_dir (str)

• experiment_name (str | None)

model_name: str = 'meta-llama/Llama-3.2-1B'

device: str = 'cuda'

torch_dtype: str = 'float16'

injection_layers: list[int]

noise_scale: float = 0.1

noise_strategy: str = 'gaussian'

injection_mode: str = 'continuous'

task_name: str = 'tower_of_hanoi'

complexity_range: tuple[int, int] = (3, 7)

trials_per_complexity: int = 10

k_values: list[int]

selection_method: str = 'majority_vote'

max_new_tokens: int = 1024

output_dir: str = 'experiments'

experiment_name: str | None = None

classmethod from_yaml(path)

Load config from YAML file.

Parameters:

path (str)

Return type:

ExperimentConfig

to_dict()

Convert to dictionary.

Return type:

dict

__init__(model_name='meta-llama/Llama-3.2-1B', device='cuda', torch_dtype='float16', injection_layers=<factory>, noise_scale=0.1, noise_strategy='gaussian', injection_mode='continuous', task_name='tower_of_hanoi', complexity_range=(3, 7), trials_per_complexity=10, k_values=<factory>, selection_method='majority_vote', max_new_tokens=1024, output_dir='experiments', experiment_name=None)

Parameters:

• model_name (str)

• device (str)

• torch_dtype (str)

• injection_layers (list[int])

• noise_scale (float)

• noise_strategy (str)

• injection_mode (str)

• task_name (str)

• complexity_range (tuple[int, int])

• trials_per_complexity (int)

• k_values (list[int])

• selection_method (str)

• max_new_tokens (int)

• output_dir (str)

• experiment_name (str | None)

Return type:

None

Experiment Class

class structured_stochasticity.experiment.Experiment(config)

Bases: object

Main experiment class.

Handles: - Loading model and tokenizer - Setting up noise injection - Running evaluation - Saving results

Example Usage:

from structured_stochasticity.experiment import Experiment, ExperimentConfig

config = ExperimentConfig(
    model_name="meta-llama/Llama-3.2-1B",
    noise_scale=0.1,
    complexity_range=(3, 6),
    k_values=[1, 5, 10]
)

experiment = Experiment(config)
experiment.run()
experiment.print_summary()
experiment.save_results()

Parameters:

config (ExperimentConfig)

__init__(config)

Parameters:

config (ExperimentConfig)

setup()

Load model, tokenizer, and task.

generate_responses(prompt, k)

Generate K responses for a prompt.

Parameters:

• prompt (str)

• k (int)

Return type:

list[str]

run()

Run the full experiment.

Return type:

dict[int, EvaluationResult]

save_results(output_path=None)

Save experiment results to JSON.

Parameters:

output_path (str | None)

print_summary()

Print summary of results.

Quick Experiment

structured_stochasticity.experiment.run_quick_experiment(model_name='meta-llama/Llama-3.2-1B', noise_scale=0.1, complexity_range=(3, 5), k_values=[1, 5], trials=5)

Quick experiment runner for notebooks/scripts.

Returns dict with results summary.

Convenience function for notebooks and quick tests:

from structured_stochasticity import run_quick_experiment

results = run_quick_experiment(
    model_name="meta-llama/Llama-3.2-1B",
    noise_scale=0.1,
    complexity_range=(3, 5),
    k_values=[1, 5],
    trials=5
)

Parameters:

• model_name (str)

• noise_scale (float)

• complexity_range (tuple[int, int])

• k_values (list[int])

• trials (int)

Return type:

dict

CLI Entry Point

structured_stochasticity.experiment.main()

CLI entry point.

Run experiments from the command line:

# With config file
ss-experiment --config configs/default.yaml

# Override parameters
ss-experiment --model meta-llama/Llama-3.2-1B --scale 0.15