Linux, Git & Developer Tooling

The tools this page covers aren't ML-specific, but every ML job assumes fluency in them — training jobs run on Linux servers, every serious codebase is versioned with Git, and debugging a stuck training run at 2am means being comfortable at a shell prompt with no IDE in sight.

The Shell

• Pipes (|): chain commands together, feeding one command's output as the next's input — cat train.log | grep "loss" | tail -20 to see the last 20 logged loss values without opening the file.

• Redirection (>, >>, 2>): send a command's output to a file instead of the terminal (> overwrites, >> appends), and 2> specifically redirects error output — separating a training script's real output from its warnings.
• grep: search text for a pattern — grep -n "CUDA out of memory" *.log to find exactly which log file and line hit an OOM.
• awk: pattern-scan and process structured text, especially column-based data — extracting the 3rd whitespace-separated column from every line of a metrics dump.
• sed: stream-edit text — find-and-replace across files without opening an editor.
• Process management: ps (list running processes), top/htop (live resource usage — the first thing to check when a training run "feels slow"), kill/kill -9 (terminate a process gracefully vs. forcibly), nvidia-smi (the GPU equivalent of top, showing GPU utilization, memory, and which processes are using them).

A real 2am-debugging session using all of the above together — a training run is stuck, and this is the actual sequence of commands to find out why:

tail -f train.log | grep -i "error\|oom\|nan"          # watch for trouble live, filtered
grep -n "CUDA out of memory" *.log                       # search every log file for a known failure signature
awk '{print $3}' metrics.log | sort -n | tail -5          # the 5 highest values in a column of a metrics dump
sed -i 's/batch_size: 64/batch_size: 32/' config.yaml     # edit the config in place, no editor needed

ps aux | grep train.py                                    # find the stuck process
nvidia-smi                                                 # is the GPU actually busy, or idle while "running"?
kill -9 12345                                              # nothing else worked -- force-kill by PID

The Linux Filesystem and Permissions

• Filesystem hierarchy: /home (user files), /tmp (scratch space, often cleared on reboot — never store anything you need to keep here), /var/log (system and service logs), /etc (system configuration).
• Permissions: every file has read/write/execute permissions for its owner, its group, and everyone else (rwxr-xr-x style notation, or the equivalent octal 755). A common real gotcha: a script that "isn't running" because it's missing execute permission (chmod +x script.sh), or a training job failing to write checkpoints because it lacks write permission on the output directory.

ls -l train.sh                    # -rw-r--r--  1 alice  alice  842 train.sh -- not executable yet
chmod +x train.sh                 # -rwxr-xr-x -- now it is
chmod 755 checkpoints/             # owner: rwx, group/other: r-x -- the standard "shared, read-only to others" mode
chown alice:ml-team checkpoints/   # change owner/group -- fixes "permission denied" on a shared output dir

Git Internals (Not Just Commands)

Knowing git add/commit/push gets you through daily work; understanding what's actually happening underneath makes merge conflicts, rebases, and "how do I get back a commit I thought I lost" far less scary:

• Blob: the compressed content of a single file's contents, identified by the SHA-1 hash of that content — identical file content anywhere in a repo's history is stored exactly once.
• Tree: a snapshot of a directory — a list of blobs (files) and other trees (subdirectories), each with a name and mode.
• Commit: a pointer to one tree (the full project state at that point), plus a pointer to its parent commit(s), an author, and a message. A commit is a snapshot of everything, not a diff — Git computes diffs on the fly for display, but what's actually stored is a chain of full snapshots (deduplicated via content-addressed blobs, so this is cheaper than it sounds).
• Branch: nothing more than a movable pointer to a commit — git branch creates a new named pointer, git commit on that branch moves the pointer forward. This is why Git branches are so cheap to create compared to some other version control systems' branches.

The plumbing commands that make blobs/trees/commits directly visible, not just conceptual:

git cat-file -p HEAD                    # the commit object: tree pointer, parent, author, message
git cat-file -p HEAD^{tree}              # the tree object: filenames pointing at blob hashes
git cat-file -p <blob-hash>              # the blob object: the raw file content itself
git rev-parse HEAD                       # the current commit's actual SHA-1
git log --graph --oneline --all          # the commit-pointer chain, drawn as a graph

• Why this matters for rebase vs. merge: a merge creates a new commit with two parents, preserving the actual history of both branches. A rebase rewrites commits — it replays your branch's commits one by one on top of a new base, producing new commits with new hashes. Understanding that a rebased commit is a genuinely different object (not the "same" commit relocated) explains why force-pushing after a rebase is necessary, and why rebasing commits that have already been pushed and pulled by someone else causes real problems (their history now disagrees with yours about what those commits even are).

git checkout main && git pull
git checkout -b add-mixed-precision
# ...commits...

git merge main                    # preserves both branches' real history, creates a merge commit

# vs.

git rebase main                   # replays add-mixed-precision's commits on top of main's tip -- new hashes
git push --force-with-lease       # required after rebase; --force-with-lease refuses if someone else pushed first

Beyond add/commit/push, the commands that come up once a repo has real history and real conflicts:

# interactive rebase: squash 3 messy WIP commits into one clean one before pushing
git rebase -i HEAD~3
# opens an editor listing the 3 commits -- change "pick" to "squash" on the last 2

# a merge conflict, resolved by hand
git merge main
# CONFLICT (content): Merge conflict in train.py
# <<<<<<< HEAD / ======= / >>>>>>> main markers show both versions inline
git add train.py                  # after manually editing to resolve the conflict
git commit                        # completes the merge

# bisect: find which of the last 50 commits introduced a regression, via binary search
git bisect start
git bisect bad HEAD                       # current commit is broken
git bisect good v1.2.0                    # this old tag was known-good
# git checks out a commit halfway between -- test it, then:
git bisect good   # or: git bisect bad     # repeat until git names the exact culprit commit
git bisect reset

# worktrees: check out a second branch into its own directory, no stashing needed
git worktree add ../hotfix main           # a real second working copy, same repo, same .git

Linux & Git Essentials

A handful of specific tools and conventions that come up constantly once you're actually operating servers and shipping code as a team, rather than just running scripts locally.

• SSH: remote login to a Linux server (a training box, an inference server) over an encrypted connection. Key-based auth (a public/private keypair) replaces passwords — the server holds your public key, you prove ownership of the matching private key, no secret ever crosses the network. ~/.ssh/config lets you name a host once instead of retyping its IP and key path every time, and -L port-forwarding tunnels a port on a remote box (say, a Jupyter server or TensorBoard bound to localhost on the GPU machine) to your own laptop as if it were running locally.

ssh-keygen -t ed25519 -C "you@example.com"        # generates ~/.ssh/id_ed25519 (private) and .pub (public)
ssh-copy-id alice@gpu-box-1.internal               # installs your public key on the remote server

# ~/.ssh/config -- name the host once
# Host gpu-box-1
#   HostName gpu-box-1.internal
#   User alice
#   IdentityFile ~/.ssh/id_ed25519

ssh gpu-box-1                                       # now just this, instead of the full user@host every time
ssh -L 8888:localhost:8888 gpu-box-1                # tunnel: remote Jupyter on :8888 becomes localhost:8888 for you

• systemd: Linux's standard init system and service manager — the thing that keeps a long-running process (an inference server, a background worker) alive across crashes and reboots, instead of it dying the moment an SSH session disconnects. A unit file declares how to start it, when to restart it, and what it depends on; systemctl controls it, journalctl reads its logs.

# /etc/systemd/system/inference-server.service
[Unit]
Description=Model inference server
After=network.target

[Service]
ExecStart=/usr/bin/python3 /opt/app/serve.py
Restart=on-failure          # auto-restart if the process crashes
User=alice

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now inference-server   # start it now, and on every future boot
sudo systemctl status inference-server         # is it running? when did it last restart?
journalctl -u inference-server -f              # follow its logs live, same idea as `tail -f`

• cron: schedules a command to run automatically at fixed times — the standard way to run something like a nightly retraining job or log rotation without a human triggering it. crontab -e opens your personal schedule; each line is minute hour day month weekday command.

crontab -e
# 0 2 * * *  /usr/bin/python3 /opt/app/retrain.py >> /var/log/retrain.log 2>&1
# ^ runs retrain.py at 2:00 AM every day, appending both stdout and stderr to a log file

• Environment variables: key-value pairs available to every process in a shell session — the standard place for config and secrets that shouldn't be hardcoded (API keys, database URLs, which environment a service is running in). export sets one for the current shell and anything it launches; a .env file (loaded by a library like python-dotenv, or docker run --env-file) is the common way to keep a whole set of them out of source control.

export OPENAI_API_KEY="sk-..."          # available to this shell and any process it starts
printenv OPENAI_API_KEY                  # check what a variable is currently set to
echo 'export OPENAI_API_KEY="sk-..."' >> ~/.bashrc   # persist it across future shell sessions

# .env (gitignored -- never commit real secrets)
# DATABASE_URL=postgres://localhost/mydb
# OPENAI_API_KEY=sk-...

• Git hooks: scripts Git runs automatically at specific points in the commit/push workflow — living in .git/hooks/, or managed more reliably across a team via the pre-commit framework, which turns them into a versioned config file everyone's clone actually uses (raw .git/hooks/ scripts aren't tracked by Git itself, so they don't travel with a git clone). The most common use: block a commit or push that fails linting or tests, so broken code never reaches the shared history in the first place.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.6.8
    hooks:
      - id: ruff        # lints staged Python files before the commit is allowed to complete
      - id: ruff-format

pre-commit install              # wires the hooks into this clone's .git/hooks/
git commit -m "add feature"     # ruff runs first -- the commit is blocked if it fails

• Semantic versioning (SemVer): the MAJOR.MINOR.PATCH convention (v2.4.1) for naming releases so anyone depending on your package or API can tell at a glance what a version bump means — increment PATCH for a backward-compatible bug fix, MINOR for a backward-compatible new feature, MAJOR for a breaking change. A Git tag marks the exact commit a given version was cut from, and is what CI/CD pipelines (see CI/CD & ML CI/CD) typically key a release build off of.

git tag -a v1.3.0 -m "Add batch inference endpoint"   # backward-compatible new feature -> MINOR bump
git push origin v1.3.0                                 # pushes the tag, often triggering a release workflow
git tag                                                 # list all tags in this repo

Compilers vs. Interpreters

• Compiled languages (C++, Rust) translate source code into machine code ahead of time, producing a standalone executable — faster at runtime, but a build step is required before anything runs.
• Interpreted languages (pure Python) execute source code line-by-line at runtime via an interpreter — no separate build step, slower per-operation.
• What Python/PyTorch actually are: Python itself is technically compiled to bytecode (not machine code) which the CPython interpreter then executes — a hybrid, not purely interpreted in the strictest sense. The reason PyTorch code isn't nearly as slow as "pure Python would suggest" is that the actual numeric work (matrix multiplies, convolutions) happens in pre-compiled C++/CUDA kernels — Python is just the orchestration layer calling into compiled code, which is also exactly why a Python for loop over tensor elements is catastrophically slower than a single vectorized tensor operation: the loop version pays Python's interpretation overhead per element, the vectorized version pays it once.

Next: Software Engineering Practice — turning this tooling fluency into code that survives contact with a team and a production system.