Security & Reproducibility

Two cross-cutting concerns that apply to every layer covered so far — not a new stage in the pipeline, but a set of requirements the whole pipeline has to satisfy.

General Security

• IAM: identity and access management — who, and what service, can do what (see Cloud Computing for ML). Over-permissioned roles are the single most common source of real security incidents, not exotic exploits.
• Secrets management: API keys, database credentials, and model-provider tokens belong in a secrets manager (AWS Secrets Manager, HashiCorp Vault) injected at runtime — never in a config file, a Dockerfile, or a git-committed .env.

On Kubernetes, that runtime injection is a Secret resource mounted as an environment variable — the value never touches the image or the Deployment YAML in plaintext:

apiVersion: v1
kind: Secret
metadata:
  name: model-api-secrets
type: Opaque
stringData:
  LLM_PROVIDER_API_KEY: "..."   # in practice, populated by a tool like External Secrets Operator from Vault/Secrets Manager, not hand-typed
---
# in the Deployment's container spec:
envFrom:
  - secretRef:
      name: model-api-secrets

• API auth/RBAC: every serving endpoint (see APIs & Model Serving, LLM Hosting & Serving Patterns) needs authentication and role-based access control — not every caller should have access to every model or every capability.
• TLS: encrypt traffic in transit, always — between users and the API gateway, and ideally between internal services too.
• Container/dependency scanning: scan images and dependency trees for known CVEs as part of CI (see Containers, CI/CD & ML CI/CD) — catching a vulnerable base image or package before it ships, not after.

LLM-Specific Security

Generative systems introduce attack surfaces that don't exist in classical ML serving:

• Prompt injection: malicious instructions embedded in user input (or in retrieved RAG content) attempt to override the system prompt or intended behavior — e.g. a document retrieved by RAG contains hidden text instructing the model to ignore its original task. Mitigation is layered (input/output filtering, strict system-prompt design, treating retrieved content as untrusted data rather than instructions) rather than fully solved by any single technique.

• Data leakage: a model trained or fine-tuned on sensitive data can memorize and later regurgitate it; a RAG system with insufficiently scoped retrieval can surface documents a given user shouldn't see — both require access controls enforced before retrieval/generation, not just filtering the output after.
• Jailbreaks: adversarial prompts designed to bypass a model's safety training — an ongoing arms race, addressed with a combination of model-level safety tuning and system-level guardrails (input/output classifiers) rather than either alone.
• Excessive agency: giving an LLM-based agent (see Agents) more real-world capability (API calls, file writes, financial transactions) than a given task actually requires — the fix is the same principle as least-privilege IAM, applied to what tools an agent is allowed to invoke and under what confirmation requirements.

• PII protection: detecting and redacting personally identifiable information both in data used for training/fine-tuning and in logs/traces captured during serving (see Observability) — logging full prompts and completions by default is a common, easily-avoided way to accidentally create a PII exposure.

Reproducibility Checklist

Given the same inputs below, a training run should produce the same result — the foundation everything else in this section (experiment tracking, CI/CD gating, incident debugging) depends on:

• Code: the exact git commit.
• Data: the exact dataset version (see Data Engineering & Versioning).
• Model: the exact architecture and starting weights (for fine-tuning).
• Params: every hyperparameter, logged (see Experiment Tracking).
• Environment: exact package versions, ideally captured as a container image (see Containers), not just a requirements.txt.
• Seeds: random seeds for data shuffling, weight initialization, and any stochastic training component.
• Hardware: GPU type and count — some operations (certain CUDA kernels, some forms of mixed precision) are not bit-for-bit deterministic across different hardware or GPU counts, so "same result" sometimes means "same result within a documented, acceptable tolerance," not literal bit-identical output.

The seeds and determinism knobs a training script actually needs to set, and the one environment variable PyTorch requires for its own deterministic-algorithm guarantee to hold:

import torch, random, numpy as np

def set_seed(seed: int):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.use_deterministic_algorithms(True)

CUBLAS_WORKSPACE_CONFIG=:4096:8 python train.py --seed 42

A pipeline that can't satisfy this checklist can't be debugged when something goes wrong in production — "the model started behaving differently" is unanswerable without knowing exactly what changed.

Next: The Full MLOps Architecture, Priority Stack & Learning Path — putting every page in this section together into one end-to-end picture.