systems

My crash course in production, as the volunteer SRE of a large community site

The site was broken into twice, and my health checks stayed green both times.

Versions. The runs below use the official php:8.2-apache image pinned at sha256:1309a229…, and debian:trixie-slim pinned at sha256:a99cfc51…, both in Docker, so nothing is installed on the host. The image comparison near the end uses the newest 8.2, 8.3 and 8.4 tags on 22 October 2026.

The site went down, and the chat called my name

Between July and October, a volunteer-run library site I was moving to my own machine was broken into twice, and lost most of its images. Both times, the checks I trusted said everything was fine.

On a Sunday morning in August, the owner of a site I was helping sent a screenshot to our group chat. The page was white, with one line on it: there has been a critical error on this website.

Two minutes later, the next message was just my name.

I remember the jolt. Wait, what did I do? I was startled, and a bit embarrassed to be named in front of everyone, even though I had done nothing. My first reply was that I had not touched anything for days, and my copy of the site was still working fine. Both were true, and neither said anything about the real site.

That was the first of many checks in this story that came back fine. None of them was wrong. Each one answered a question next to the one I needed to ask.

A large site on a $68 server

To explain why it was my name, I have to go back a month. A friend had added me to the group chat of the volunteers who run a library site. It is not a small site: close to 200,000 articles and about 38,000 images, collected since 2014. In July, the old server’s logs showed about 150,000 requests and 25 GB of traffic a day, around a third of the requests from bots. At that rate it serves about 750 GB a month.

I am a backend developer with a machine at home, so I offered to move the site there, for free.

About a week later the old server expired, and I paid the bill with my own card to bring the site back: about $68. One of the volunteers asked whether that was the price for a year. It was for a month, and they had been paying it for years.

Everyone in the chat looked at the price, me included. Nobody asked what it was paying for. I logged in and found out:

one rented machine, one Plesk panel, about $68 a month the library site site site site site site back door site site site site one broken site could reach into the others; one had carried a back door since February 2024
What the $68 paid for. Red is the attacker.

My plan was simple: copy everything, run each site in its own container at home, and put it behind a tunnel instead of a public IP. The migration only read from the old server. The copy went onto a test domain that nobody else knew, for the volunteers to check for a month. Until then, the real site stayed on the old server.

So by that Sunday I had the passwords, and I was the one moving the site. When it broke, mine was the obvious name.

Proving it was not me

Back to that Sunday. It took about an hour to track down what had happened. On 17 July, WordPress had released a fix for a flaw in its core that let a stranger run code without logging in. The old server was still on an affected version, and it was broken into on 20 July, three days later. A suspicious file was sitting in the web folder: a web shell, a small script that lets whoever knows its address run commands on the server. More joined it over the following week. Whoever put them there had been using the server for almost two weeks, and on that Sunday their tools used up every PHP worker it had, which is what took the site down.

Proving I had not done it was harder than fixing the server. Everyone in the chat knew I had the passwords, so it was my credibility at stake. The first thing anyone looks at is the date on the file, and that date said the file was as old as its neighbours. Here is why that answer meant nothing:

#!/bin/sh
# One file, two clocks: the one ls shows, and the one the kernel keeps.
set -eu
docker run --rm debian:trixie-slim@sha256:a99cfc517144bc59b1978475ec53b46ecabec7e43635402ee5b77cc54cd1b20a sh -c '
  cd /tmp
  touch -d "2019-03-01 10:00" old-neighbour    # a file that has been here for years
  echo "<?php // new" > new.php                # a file written just now
  touch -r old-neighbour new.php               # copy the neighbour'"'"'s mtime onto it

  echo "## ls -l"
  ls -l --time-style=+%F new.php | awk "{print \$6, \$7}"
  echo
  echo "## stat"
  stat -c "mtime %y" new.php | cut -c1-25
  stat -c "ctime %z" new.php | cut -c1-25
'
## ls -l
2019-03-01 new.php

## stat
mtime 2019-03-01 10:00:00
ctime 2026-10-22 10:43:42

The date ls -l shows is the mtime, and any program that can write the file can also set it. The ctime is different: the kernel sets it to the current time whenever the file changes, and there is no call to set it directly. Faking it needs root, to move the system clock or edit the disk, and the attacker ran as the web server’s user. That is what settled it on the old server: the first suspicious file had an old mtime, but its ctime said 03:07 on 20 July, almost two weeks before that Sunday.

That was one piece of evidence. Three more did not depend on it: the new files belonged to the web server’s user, not to root and not to my account; there was no SSH session on 20 July; and of the four sites I was moving, only one was touched.

A domain nobody knew

Just in case, I checked my own copy next. Someone had found it too. An admin account that none of us had created showed up on the test domain at 00:14 on 1 August, 19 days after I set the domain up. There was no web shell on it yet.

So the hidden domain was not hidden. Its name was public from the first day: every HTTPS certificate is written to public Certificate Transparency logs, and the certificate for my test domain went in on 12 July, the day I set it up. I cannot prove that is how it was found, but the name was never a secret.

That Sunday afternoon I upgraded WordPress on my copy to 7.0.2, added a startup step that removes files that should not be in the web folder, and changed every password. Then the owner and I moved the domain to my machine. By four in the afternoon the real site was running at home. I was relieved.

Looking back, I fixed the layer I had seen fail and did not review the layers below it. I did not rotate the WordPress salts, so old login sessions stayed valid, and I did not go through the list of admin accounts. Both waited until the second fix.

Seven and a half weeks later, one layer down

For seven and a half weeks I assumed the site was fine. I had patched it, after all.

Then, on 23 September at 23:43, a new file appeared in the web folder of the live site. I only found it later. This time the way in was not WordPress at all.

a request goes down CDN every request passes through, the attack too reverse proxy nginx in front of the containers WordPress, the application core files, plugins, admin accounts August: upgraded here, after break-in one PHP and Apache, the official image no php.ini · register_argc_argv On · PEAR September: break-in two came in here volume mounted into the container site files and all the images September: images deleted here
The fix was one layer above the way in. Red is the attacker, teal is a fix.

The way in was one layer below WordPress, in the infrastructure under the application. The official PHP image runs with no php.ini at all, so PHP uses its built-in defaults. One of those defaults, register_argc_argv, should be off on a web server: it lets PHP read parts of a web request as if they were command-line arguments. The image also carries the PEAR command-line tool, which has nothing to do with serving pages and does what its arguments tell it.

The CDN logs show a request carrying arguments for that tool, and nine seconds later a new PHP file in the web folder answering requests. No login was involved, and neither of the two lives in WordPress.

The image does ship a production config file with the option off. Its own documentation says it is “strongly recommended” to copy that file into place for production. I skipped that step.

So I checked the option, and my first diagnostic check gave me the wrong answer. I ran php -i inside the container, and it said on. Then I ran it on a container I had already fixed, and it still said on. The tool was not lying. The same PHP runs in two modes, and they do not read the same settings:

#!/bin/sh
# Ask one official image the same question twice: once as the command line, once as Apache.
set -eu
image=php:8.2-apache@sha256:1309a2299d4dfc1d5cdf62291469b84ec4626021d818cef4cb30e79e7e10e6cc
page='<?php echo ini_get("register_argc_argv") ? "On" : "Off", "\n";'

run_both() {
  docker run -d --name two-phps "$@" "$image" >/dev/null
  docker exec two-phps sh -c "echo '$page' > /var/www/html/argv.php"
  for _ in $(seq 50); do   # wait until apache answers, at most 50 tries
    docker exec two-phps php -r 'exit(@file_get_contents("http://localhost/argv.php") === false ? 1 : 0);' && break
  done
  printf 'command line : '
  docker exec two-phps php -r 'echo ini_get("register_argc_argv") ? "On" : "Off", "\n";'
  printf 'apache       : '
  docker exec two-phps php -r 'echo file_get_contents("http://localhost/argv.php");'
  docker rm -f two-phps >/dev/null
}

echo "## the image as shipped"
docker run --rm "$image" php --ini | grep 'Loaded Configuration File'
printf 'php.ini-production, shipped but not in place: '
docker run --rm "$image" sed -n 's/^register_argc_argv = //p' /usr/local/etc/php/php.ini-production
run_both

echo
echo "## with register_argc_argv = Off in a mounted ini file"
printf 'register_argc_argv = Off\n' > off.ini
run_both -v "$PWD/off.ini:/usr/local/etc/php/conf.d/zz-off.ini:ro"
rm off.ini
## the image as shipped
Loaded Configuration File:         (none)
php.ini-production, shipped but not in place: Off
command line : On
apache       : On

## with register_argc_argv = Off in a mounted ini file
command line : On
apache       : Off

The command-line mode always turns that option on, whatever the file says, because command-line scripts need their arguments. Only the apache line is the PHP that serves traffic.

The day the images went

On 24 September I deleted a 77 GB copy of the images from a spare disk to free some space. I counted the images on the server as the other copy. They were not a copy: they were the originals, on the same volume the web server could write to.

On 25 September, at around 14:51, someone opened the admin area of the live site in a normal browser, through a VPN, as one of the site’s old admin accounts. With a web shell on the server, they did not need its password to get a session. The login page was blocked; the admin area behind it was not.

At around 15:14 they deleted the whole image folder, about 38,000 images.

38,421 images the database knew about 6,704 31,717 in the 2021 backup not in it: almost everything from 2021 to 2025
The only backup left held less than a fifth of the images.

The owner had a hard drive with an older backup, from around May 2021. It is the only copy that survived.

Two things made one session enough to delete everything. The web server’s user owned the whole image folder, with no snapshots and no versions behind it, so a single delete was final. And the only copy outside that folder was the one I had just removed. When I first offered my disk in July, I had added “no backup yet”. I never said who would make the backup, or when.

The monitoring said nothing. Every container was running, and every health check was green. After the images were deleted, the site still answered every page with a 200, because the database still had a row for every image.

Closing each layer, then freezing writes

The second fix went layer by layer, from PHP down to the network.

Everything the second fix changed
  • PHP: the option off in a config file that every container mounts, and the PEAR tool removed at every start.
  • Files: 847 .htaccess files cleaned.
  • Accounts: the attacker’s accounts removed and, at the owner’s request, every old admin account too. All sessions ended, salts and the database password replaced with new ones.
  • The public domain: the admin area closed to the internet, as well as the login page.

The worst find was the network. On 14 October I tested what the container could actually reach. There was no network policy, so it could reach almost every service on my home network, and it had a Kubernetes token mounted that it never needed. I closed both that day, long overdue. Nothing showed the attacker used either, but I cannot show they could not have.

On 15 October the owner made a decision I had not expected: nobody needs to log in anymore, just freeze it. So we did, and the site lost its write access. I checked it by asking for pages, images and the API, which all still worked, and by sending the kinds of request that write, which were all refused.

The container helped, but only partly. It kept the attacker in one box: one user, one database, and the other two sites still matched their checksums. But the images were in the same box. And with different attackers and different ways in, “it would have been worse without it” is plausible, but it is a counterfactual I cannot measure.

The pattern

Here is the whole thing on one line.

12 Jul copy made 17 Jul WordPress fix released 20 Jul old server broken into 1 Aug my test copy found 2 Aug noticed, fix one 23 to 25 Sep break-in two, images deleted 14 to 15 Oct network closed, site frozen July August September October
Red is the attacker, teal is a fix. Seven and a half quiet weeks sit in the middle.

Every check I relied on along that line was correct. Each one measured something real, and each time it was the thing next to what I needed.

The check I trusted What it really measured What I needed to know
“$68 a month” what it cost what it was paying for
“the test site still works” my copy, on my machine the real site, on the old server
the date ls -l shows a value any writer can set when the file was written: the ctime
“nobody knows this domain” whether people know the name whether it is listed anywhere: it was
“I patched it” the layer I patched the layer the request goes through next
php -i in the container the command-line mode the mode Apache runs
“the login page is blocked” the login form whether a session can reach the admin area
“there is a copy” that the images exist a copy the web server cannot touch
containers running, pages 200 that processes answer that the files are still there
“nothing showed it was used” what I saw what was reachable

Only one check in this story measured exactly what it claimed: after the freeze, every write was refused. It is also the only fix that did not depend on my finding the next hole first.

None of this is special to a volunteer site. At work it is the same shape: a health check that passes while the data behind it is missing, a config check run in a different process from the one serving traffic, a backup nobody has restored. The way out each time was to move the check closer to the thing I cared about: the ctime instead of the mtime, the page Apache serves instead of the command line, the files on disk instead of the rows in the database.

What I would build differently

I would give each site its own blast radius. A dozen sites on one rented server at $68 a month looked cheap. The real cost was that they shared one machine, one panel and nobody watching it: one broken site could reach into the others, and one carried a back door from an older break-in, there since February 2024, for two and a half years.

I would upgrade, and then follow the setup steps of what I upgraded to. The July break-in came through an old WordPress, and that 2024 back door sat on a site still running WordPress 5.4. Upgrading would have stopped the July one. It would not have stopped the September one, because that came from a setup step I skipped, and the newest images still leave that step to whoever runs them:

The three latest official PHP images, checked the same way
#!/bin/sh
# Would a newer PHP image have closed the second way in? Ask the three latest.
set -eu
for tag in 8.2-apache 8.3-apache 8.4-apache; do
  docker pull -q "php:$tag" >/dev/null
  docker run --rm "php:$tag" sh -c '
    printf "%-7s php.ini: %-7s PEAR tool: " "$(php -r "echo PHP_VERSION;")" \
      "$(php --ini | sed -n "s/Loaded Configuration File: *//p")"
    [ -f /usr/local/lib/php/pearcmd.php ] && echo present || echo absent'
done
8.2.35  php.ini: (none)  PEAR tool: present
8.3.36  php.ini: (none)  PEAR tool: present
8.4.27  php.ini: (none)  PEAR tool: present

I would pick a smaller stack where it can be smaller. This site no longer needs to change, so every part that can still write is a part that can still hurt it. If I could move it again, I would move it to something with fewer of those parts.

I would give the web server only the access it needs. It could delete the whole image library and reach my home network, and it never needed either to serve a page. The network was closed on 14 October, and the freeze took away the write access.

I would make the files that never change read-only from the start. Almost all of the site’s files were old images that never change once uploaded. If that folder had been read-only to the web server, a break-in as the web server’s user could not have deleted it. The deletion on 25 September went through the admin area, which runs as exactly that user, so it would have failed. This does not stop someone who gets root; that is what snapshots on another machine are for.

I would keep backups the web server cannot reach, each with an owner and a date. And I would not delete a copy again before another one has been read back.

I would make health checks test the content, as well as the process. The check that would have caught it is mundane: count the files on disk against the rows in the database. The same standard applies whether the job is paid or volunteer; I held this one to a lower standard because it was free, and that was my mistake.

I would not treat a community site as a small target. The first break-in was a bot that scans everything. The second was a person who chose to delete twelve years of images, and I do not know why. A site that some people care about may also have people who want it gone.

What sounded right and did not hold here

Every check I run now has to say what it measures, and what it leaves out.