Is everything really a file?

Versions. Source links are pinned: torvalds/linux at v7.1, apple-oss-distributions/xnu at xnu-11215.41.3, golang/go at 01534385, busybox at 1_36_1. Every program output below is from go1.25.12 linux/arm64 in a container rather than on bare metal, so the numbers come from a virtual machine.

TL;DR “Everything in Linux is a file” is the first thing anyone learns and the last thing anyone checks. It is truer than it sounds: for things that are obviously not files, sockets included, the kernel builds a real file object and attaches a table of functions saying what read and write mean for it. That is why two handles can share one read position, and why a number you close comes straight back. It is also why cmd > file 2>&1 and cmd 2>&1 > file become two sequences you can run in your head instead of two spellings you memorise.

The slogan, and the first thing that does not fit

Open a file and look at what came back:

# first.py
import os
print(os.open("/tmp/notes.txt", os.O_RDONLY))

$ echo hello > /tmp/notes.txt
$ python3 first.py
3

Three. Not an object, not a handle with methods, not a pointer. A number.

A number has to be a number of something, and you already know the first three entries of the thing it counts. Descriptor 0 is stdin, 1 is stdout, 2 is stderr. Those are not special globals with reserved names. They are the first three rows of a table your process carries, filled in before your code ran, and 2>&1 is an instruction about that table: make row 2 point wherever row 1 points.

So open() did not hand you a file. It put one in the next free row and told you the row number. Which is exactly why the number cannot be the file. Row 3 means nothing in another process, and nothing to the kernel either without knowing who is asking.

$ echo hello > /tmp/notes.txt   # something worth opening
$ exec 3< /tmp/notes.txt        # open it for reading, into slot 3
$ ls -l /proc/$$/fd             # show me this shell's table

# trimmed: the permission, owner and date columns are cut, and the notes on
# the right are mine, not the shell's
0 -> /dev/pts/0        the terminal, as stdin
1 -> /dev/pts/0        the terminal, as stdout
2 -> /dev/pts/0        the terminal, as stderr
3 -> /tmp/notes.txt    the one we just opened

docker run --rm -it alpine sh

/* setjobctl in busybox shell/ash.c */
fd = fcntl(fd, F_DUPFD_CLOEXEC, 10);
...
pgrp = tcgetpgrp(fd);
if (pgrp < 0) {
 out:
  ash_msg("can't access tty; job control turned off");

That answers what 3 is not. Every strange thing in this post follows from what sits behind it.

So what is behind the number?

Three levels, and nearly every surprise comes from collapsing two of them. This is not folklore. The standard names the levels in the sentence that defines open():

It shall create an open file description that refers to a file and a file descriptor that refers to that open file description.

POSIX.1-2024, open()

Read it slowly, because it introduces two things whose names are nearly identical, and keeping them apart is most of the work.

• The file descriptor is the number and nothing else. The standard defines it as “a per-process unique, non-negative integer”.
• The open file description is the object that number refers to: “a record of how a process or group of processes is accessing a file”. It holds the read and write position and the status flags.
• The object underneath is the level that sentence does not name. It is whatever the record is open on: an inode for a file on disk, a struct socket for a socket, a ring buffer for a pipe.

Which level a copy lands on turns out to be the whole game, and the next section is about that.

The question that makes the middle box worth knowing

The middle box is invisible. You never allocate one, never name one, never see one in a debugger. So why should you care that it is there?

Because of this situation, which you meet without going looking for it:

• a shell runs 2>&1, and now stderr and stdout are the same thing. It is why docker logs web 2>&1 | grep error finds anything at all, and it is the answer half of Stack Overflow gives when a pipe comes back empty.
• a process opens a log file and forks, and now parent and child both have it.

In both cases two descriptors refer to one open file. Do they share a read and write position, or does each one keep its own? The answer decides whether a forked child appending to a log corrupts its parent’s next write, and it is not something you can guess from the API.

The three-level model turns that into a question with one variable: which level got copied? Copy at level 1 and both numbers reach one middle box, so there is one position between them. Call open() twice and there are two middle boxes, so the positions are separate.

Which means it can be measured. dup(fd) copies at level 1, two open() calls do not, and the file holds ten letters so a short read shows where each one starts:

const path = "/tmp/fd-offset.txt"
must(os.WriteFile(path, []byte("ABCDEFGHIJ"), 0o644))

a := openRO(path) // syscall.Open(path, O_RDONLY, 0)
b, err := syscall.Dup(a)
must(err)
fmt.Printf("a := openRO(path)     a is fd %d\n", a)
fmt.Printf("b := dup(a)           b is fd %d\n", b)
fmt.Printf("read(a, 3)         -> %q\n", read(a, 3))
fmt.Printf("read(b, 3)         -> %q\n\n", read(b, 3))

c := openRO(path)
d := openRO(path)
fmt.Printf("c := openRO(path)     c is fd %d\n", c)
fmt.Printf("d := openRO(path)     d is fd %d\n", d)
fmt.Printf("read(c, 3)         -> %q\n", read(c, 3))
fmt.Printf("read(d, 3)         -> %q\n", read(d, 3))

a := openRO(path)     a is fd 3
b := dup(a)           b is fd 6
read(a, 3)         -> "ABC"
read(b, 3)         -> "DEF"

c := openRO(path)     c is fd 7
d := openRO(path)     d is fd 8
read(c, 3)         -> "ABC"
read(d, 3)         -> "ABC"

Nobody moved a position on b, because b is a number and numbers do not have positions. The position is one level down, and both numbers reach it. Two open calls build two of those instead, so the same bytes at level 3 sit under two positions at level 2 that know nothing of each other.

func listFDs(label string) {
  fmt.Printf("--- %s ---\n", label)
  entries, err := os.ReadDir("/proc/self/fd")
  if err != nil {
    fmt.Println("  no /proc (not Linux):", err)
    return
  }
  for _, e := range entries {
    target, _ := os.Readlink(filepath.Join("/proc/self/fd", e.Name()))
    fmt.Printf("  fd %-3s -> %s\n", e.Name(), target)
  }
}

--- at the top of main, before the three opens below ---
  fd 0   -> /dev/null
  fd 1   -> pipe:[51163285]
  fd 2   -> pipe:[51163286]
  fd 3   -> 
  fd 4   -> anon_inode:[eventpoll]
  fd 5   -> anon_inode:[eventfd]

$ strace -e trace=openat,epoll_create1,eventfd2,epoll_ctl,close ./probe   # the program above

openat(AT_FDCWD, "/tmp/fd-probe.txt", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 3
epoll_create1(EPOLL_CLOEXEC)            = 4
eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 5
epoll_ctl(4, EPOLL_CTL_ADD, 5, {events=EPOLLIN, ...}) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, ...}) = -1 EPERM
close(3)                                = 0
openat(AT_FDCWD, "/proc/self/fd", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 3

That is the answer to the question this section opened with. A forked child reading from an inherited descriptor moves the parent’s position, because fork copies the table and not the boxes. Two independent open() calls on one log file do not interfere, because each built its own box.

/* redirect in busybox shell/ash.c, running "2>&1" */
fd    = redir->nfile.fd;     /* N, left of the >     2 */
...
newfd = redir->ndup.dupfd;   /* M, after the &       1 */
...
dup2_or_raise(newfd, fd);    /* copy slot 1 into slot 2 */

/* ksys_dup3, then do_dup2 which it calls, in fs/file.c */
file = files_lookup_fd_locked(files, oldfd);   /* read the level 2 pointer out of the old slot */
...
get_file(file);                                /* say one more slot is about to hold it        */
rcu_assign_pointer(fdt->fd[fd], file);         /* write that same pointer into the new slot    */

Now try it on something that is definitely not a file

A file on disk fitting the file abstraction proves nothing. A socket is the real test. It has no bytes at rest, no length, no position you can seek to, and it did not exist a second ago.

So ask the kernel the same three questions about each of them, and compare the answers:

• the name, the one procfs shows for it
• the type, what fstat says it is
• the reopen, whether that name can be opened

Two results look plausible before running it. If a socket sits outside the file world, fstat refuses it and there is no inode to report. If it sits fully inside, all three answers come back looking like the file’s. Neither is what happens, and the question they split on is the reopen.

report below asks those three questions of whatever descriptor it is given, so it can be pointed at a regular file and a TCP socket in turn and the answers compared side by side. It reads /proc, so this one is Linux only too.

func report(label string, fd int) {
  path := fmt.Sprintf("/proc/self/fd/%d", fd)
  name, _ := os.Readlink(path) // 1. what is it called
  kind, ino := fstatKind(fd)   // 2. what is it

  fmt.Printf("%s: fd %d\n", label, fd)
  fmt.Printf("  the name procfs shows   %s\n", name)
  fmt.Printf("  fstat says it is        %s, inode %d\n", kind, ino)

  reopened, err := syscall.Open(path, syscall.O_RDONLY, 0) // 3. open that name
  if err != nil {
    fmt.Printf("  open() that name        %v\n", err)
  } else {
    fmt.Printf("  open() that name        worked, got fd %d\n", reopened)
    _ = syscall.Close(reopened)
  }
  fmt.Println()
}

const path = "/tmp/fd-regular.txt"
must(os.WriteFile(path, []byte("hello"), 0o644))
rf, err := syscall.Open(path, syscall.O_RDONLY, 0)
must(err)
report("regular file", rf)

sk, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, 0)
must(err)
report("tcp socket", sk)

regular file: fd 4
  the name procfs shows   /tmp/fd-regular.txt
  fstat says it is        regular file, inode 116947266
  open() that name        worked, got fd 7

tcp socket: fd 7
  the name procfs shows   socket:[53454226]
  fstat says it is        socket, inode 53454226
  open() that name        no such device or address

Two answers agree and one splits, and the split is the finding. That fstat answers at all, with a real inode rather than a zero, puts the socket inside the file world. The same number inside socket:[...] makes the procfs name the object itself, not a label for it. What the socket does not have is a path that reaches it. So the line between these two was never whether they are files. It is whether a name can be opened.

None of that is read off the outside. The call that builds a socket’s file hands it a real inode, a filesystem the kernel mounts for itself, and a table giving read, write and close a meaning for a socket. That is the whole of “everything is a file”. Not a metaphor and not a fake.

/* sock_alloc_file in net/socket.c */
file = alloc_file_pseudo(SOCK_INODE(sock), sock_mnt, dname,
                        O_RDWR | (flags & O_NONBLOCK),
                        &socket_file_ops);
...
file->private_data = sock;

/* sock_map_fd in net/socket.c */
int fd = get_unused_fd_flags(flags);   // the lowest free slot
...
newfile = sock_alloc_file(sock, flags, NULL);
if (!IS_ERR(newfile)) {
  fd_install(fd, newfile);
  return fd;
}

How long it lives

One rule sets the lifetime, and the rest is what counts as a reference.

It lives until nothing refers to it. That is the whole rule, and everything surprising about lifetimes comes from what counts.

A name counts. Closing a descriptor on a disk file does not delete the file, because the directory entry still names it, and that entry is a reference in its own right. The file outlives every process that had it open.

A socket has no name, so the descriptors are the entire count, and your table slot is the last thing holding it up. When the process exits the kernel closes every descriptor it held, each close is an fput, and the final one runs:

/* __fput in fs/file_table.c */
if (file->f_op->release)
  file->f_op->release(inode, file);

__fput

For a socket that release is sock_close, one entry in the same socket_file_ops table sock_alloc_file attached earlier. It calls __sock_release:

/* __sock_release in net/socket.c */
static void __sock_release(struct socket *sock, struct inode *inode)
{
  const struct proto_ops *ops = READ_ONCE(sock->ops);   // the protocol's table, below f_op

  if (ops) {
    struct module *owner = ops->owner;

    if (inode)
      inode_lock(inode);      // two closers must not race
    ops->release(sock);       // the real close: connection down, buffers freed
    sock->sk = NULL;          // drop the protocol's control block
    if (inode)
      inode_unlock(inode);
    sock->ops = NULL;
    module_put(owner);
  }

  ...
  if (!sock->file) {
    iput(SOCK_INODE(sock));   // no file wrapped this socket, so drop the inode here
    return;
  }
  WRITE_ONCE(sock->file, NULL);  // the file releases the inode itself, in __fput
}

__sock_release

The one thing the code cannot show is which function ops->release is. TCP and UDP share inet_release, which then calls sk->sk_prot->close; a Unix socket takes unix_release. So the file layer never learns what closing a connection means. It calls one pointer, and the protocol below it does the work.

Can anything outside your process hold it?

If the file has no usable name, your process looks like the only thing that could reach it. It is not, and the two exceptions are worth knowing because they are how the reference count outlives you.

fork(). The child gets the table, not copies of the objects in it (fork(2)). Same middle box, two tables pointing at it. The socket now survives either process on its own.

const path = "/tmp/fd-fork.txt"

func read(fd, n int) string {
  buf := make([]byte, n)
  m, _ := syscall.Read(fd, buf)
  return string(buf[:m])
}

func signal(fd int) { syscall.Write(fd, []byte{1}) }

func wait(fd int)   { syscall.Read(fd, make([]byte, 1)) }

if os.Getenv("FD_CHILD") == "1" {
  // 3 is the file, 4 is wait-for-parent, 5 is tell-parent
  fmt.Printf("child  read(3, 3)  -> %q\n", read(3, 3)) // read is syscall.Read
  signal(5)
  wait(4)
  fmt.Printf("child  read(3, 3)  -> %q\n", read(3, 3))
  signal(5)
  return
}

os.WriteFile(path, []byte("ABCDEFGHIJKL"), 0o644)
f, _ := os.Open(path)
fd := int(f.Fd())

fromParent, toChild, _ := os.Pipe() // parent writes toChild, child reads fromParent
fromChild, toParent, _ := os.Pipe() // child writes toParent, parent reads fromChild

fmt.Printf("parent read(%d, 3)  -> %q\n", fd, read(fd, 3))

child := exec.Command(os.Args[0]) // the fork is in here: clone, then execve
child.Env = append(os.Environ(), "FD_CHILD=1")
child.Stdout, child.Stderr = os.Stdout, os.Stderr
child.ExtraFiles = []*os.File{f, fromParent, toParent} // dup3'd into the child as 3, 4, 5
child.Start()

wait(int(fromChild.Fd()))
fmt.Printf("parent read(%d, 3)  -> %q\n", fd, read(fd, 3))
signal(int(toChild.Fd()))

wait(int(fromChild.Fd()))
child.Wait()

parent read(4, 3)  -> "ABC"
child  read(3, 3)  -> "DEF"
parent read(4, 3)  -> "GHI"
child  read(3, 3)  -> "JKL"

Passing it over a Unix socket. The one that surprised me: the same sharing, between processes with no relationship at all. The man page for SCM_RIGHTS is careful about the wording, because “passing a file descriptor” describes it badly. The number does not travel. The receiver gets a number of its own, out of its own table, pointing at the box the sender is using. A file with no usable name crosses a process boundary, and what crosses is a reference.

One machine only. AF_UNIX is for processes on the same host, and the limit is the mechanism rather than an omission: what crosses is a reference to an object in one kernel. Another machine has another kernel, and nothing to point at.

const (
  path = "/tmp/fd-scm.txt"
  sock = "/tmp/fd-scm.sock"
)

func sendFD(c *net.UnixConn, fd int) {
  c.WriteMsgUnix([]byte{0}, syscall.UnixRights(fd), nil)
}

func recvFD(c *net.UnixConn) int {
  oob := make([]byte, syscall.CmsgSpace(4))
  _, oobn, _, _, _ := c.ReadMsgUnix(make([]byte, 1), oob)
  scms, _ := syscall.ParseSocketControlMessage(oob[:oobn])
  fds, _ := syscall.ParseUnixRights(&scms[0])
  return fds[0]
}

func dial(path string) *net.UnixConn {
  addr := &net.UnixAddr{Name: path, Net: "unix"}
  for i := 0; i < 200; i++ {
    if c, err := net.DialUnix("unix", nil, addr); err == nil {
      return c
    }
    time.Sleep(5 * time.Millisecond)
  }
  panic("no receiver listening on " + path)
}

func send() {
  os.WriteFile(path, []byte("ABCDEFGHI"), 0o644)
  f, _ := os.Open(path)
  fd := int(f.Fd())
  fmt.Printf("sender   read(%d, 3)  -> %q\n", fd, read(fd, 3)) // read is syscall.Read

  c := dial(sock) // the same path receive() listens on
  sendFD(c, fd)

  c.Read(make([]byte, 1)) // wait until the receiver has read
  fmt.Printf("sender   read(%d, 3)  -> %q\n", fd, read(fd, 3))
}

func receive() {
  l, _ := net.ListenUnix("unix", &net.UnixAddr{Name: sock, Net: "unix"})

  c, _ := l.AcceptUnix()
  fd := recvFD(c)
  fmt.Printf("receiver read(%d, 3)  -> %q\n", fd, read(fd, 3))
  c.Write([]byte{0})
}

func main() {
  if len(os.Args) > 1 && os.Args[1] == "send" {
    send()
    return
  }
  receive()
}

$ ./passfd receive &
$ ./passfd send

sender   read(4, 3)  -> "ABC"
receiver read(8, 3)  -> "DEF"
sender   read(4, 3)  -> "GHI"

sender    sendmsg(...  SCM_RIGHTS, cmsg_data=[4])
receiver  recvmsg(...  SCM_RIGHTS, cmsg_data=[8])

And one that looks like an exception but is not. Linux does show the socket in /proc, which seems to contradict the whole “no name” claim. The run earlier in this post already settled it: the name is there, socket:[53454226], and opening it comes back ENXIO. So the socket does have a path, and you can read it. You just cannot open it. /proc gives you a name for looking, not a name for reaching. Which is a fair summary of the whole arrangement: the descriptor is the only handle that works, and it only works from inside the process that holds it.

So the descriptor you hold is three levels away from the connection, and it is the only one of the three that belongs to you.

Where reuse turns into a bug

It happens when a number changes hands.

Your event loop is handed an event that names fd 5. It looks up connection 5 and writes the response. By then fd 5 is a log file, and the reply for one user has gone somewhere no user will ever read it. Every step was correct. The number was not.

This is the bug the three levels make possible, and it does not need bad luck to appear. A dup makes it happen on every run, which turns it from something to argue about into something to watch.

epoll in one line: it hands back a tag you chose, not the file object.

One term first, because the bug lives inside it. epoll is the kernel half of an event loop: you register a set of connections, then ask it to block until some of them have something to read, and it hands back the list of which ones. Anything serving many connections on few threads is sitting on it, nginx and Node and the Go runtime included, usually far enough down that you never call it yourself. The part that matters here is what “which ones” means. You do not get an object back. You get a tag you chose at registration time, and almost everyone chooses the descriptor number.

So here is the reproduction. Wire the server end of a connection into epoll, dup it, close the original number, open a file so the number gets handed out again, then send a byte down the socket and see what the event says. describeFd reads /proc for what each number points at, so every line reports what it found. Linux only.

func main() {
  defer os.Remove("/tmp/fd-stale.log")

  epollFd := createEpollFd()
  serverFd, clientFd := openConnection()
  wireFdToEpoll(serverFd, epollFd)
  fmt.Printf("register serverFd        -> fd %d, %s\n", serverFd, describeFd(serverFd))

  keeperFd, _ := syscall.Dup(serverFd) // a second number for the same socket
  defer syscall.Close(keeperFd)
  syscall.Close(serverFd)
  fmt.Printf("dup, then close serverFd -> fd %d is %s\n", serverFd, describeFd(serverFd))

  logFd, _ := syscall.Open("/tmp/fd-stale.log", syscall.O_RDONLY|syscall.O_CREAT, 0o644)
  fmt.Printf("open a log file          -> fd %d, %s\n\n", logFd, describeFd(logFd))

  syscall.Write(clientFd, []byte("X"))
  fmt.Printf("client fd %d sends        -> to the socket, which is now only fd %d\n", clientFd, keeperFd)

  for _, fd := range waitForReadyFds(epollFd) {
    fmt.Printf("epoll_wait says          -> fd %d received an event\n", fd)
    fmt.Printf("the handler reads        -> fd %d, which is %s\n", fd, describeFd(fd))
  }
}

register serverFd        -> fd 5, socket:[54899389]
dup, then close serverFd -> fd 5 is nothing open at that number
open a log file          -> fd 5, /tmp/fd-stale.log

client fd 6 sends        -> to the socket, which is now only fd 7
epoll_wait says          -> fd 5 received an event
the handler reads        -> fd 5, which is /tmp/fd-stale.log

const (
  waitTimeoutMs    = 1000
  maxEventsPerWait = 4 // a ceiling on one call, not a quota it waits to fill
)

// the epoll instance is itself reached through a descriptor
func createEpollFd() int {
  epollFd, _ := syscall.EpollCreate1(0)
  return epollFd
}

func openConnection() (serverFd, clientFd int) {
  pair, _ := syscall.Socketpair(syscall.AF_UNIX, syscall.SOCK_STREAM, 0)
  return pair[0], pair[1]
}

func wireFdToEpoll(fd, epollFd int) {
  syscall.EpollCtl(epollFd, syscall.EPOLL_CTL_ADD, fd, &syscall.EpollEvent{
    Events: syscall.EPOLLIN, // report it when it has something to read
    Fd:     int32(fd),       // the tag handed back when it fires
  })
}

func waitForReadyFds(epollFd int) []int {
  events := make([]syscall.EpollEvent, maxEventsPerWait)
  n, _ := syscall.EpollWait(epollFd, events, waitTimeoutMs)
  var tags []int
  for i := 0; i < n; i++ {
    tags = append(tags, int(events[i].Fd))
  }
  return tags
}

func describeFd(fd int) string {
  name, err := os.Readlink(fmt.Sprintf("/proc/self/fd/%d", fd))
  if err != nil {
    return "nothing open at that number"
  }
  return name
}

The last three lines do not agree, and the disagreement is the bug. Nothing in the program is exotic: registering a connection, closing it, opening something else, and reading the ready list are the four things an event loop does all day.

Three ordinary rules line up to produce it, and none of them is a mistake on its own.

The number comes back at once. POSIX requires the lowest available one (§2.6), and on Linux the number you just released is the first one the next open() will try.

The entry outlives the number. Closing a descriptor does not take the registration out. It goes when the last reference to the socket goes, and keeperFd is still holding one. So the entry registered through fd 5 was still in the list when the byte arrived, and it is the one that fired.

The number in the event is one you handed over. The descriptor goes into the registration twice, and only one of the two is a descriptor to the kernel:

syscall.EpollCtl(epollFd, syscall.EPOLL_CTL_ADD, fd, &syscall.EpollEvent{
  Events: syscall.EPOLLIN, // report it when it has something to read
  Fd:     int32(fd),       // the tag handed back when it fires
})

The third argument is what to watch, and the kernel resolves it to that socket right there. Fd inside the struct is epoll_data, storage the kernel keeps and returns untouched, and almost everyone fills it with the descriptor. So the event is right about the object and wrong about the name.

Those three explain the wrong reply, and the same field is the fix. Store something that cannot be recycled behind your back: an id you allocate for the connection and never hand out twice, or the descriptor paired with a counter you bump on every close, checked before you act. Then a stale event fails a comparison instead of finding a stranger.

func main() {
  if len(os.Args) > 1 && os.Args[1] == "send" {
    send()
    return
  }
  receive()
}

// its own epoll instance, watching the descriptor this process holds
func watch(fd int) int {
  ep, _ := syscall.EpollCreate1(0)
  syscall.EpollCtl(ep, syscall.EPOLL_CTL_ADD, fd, &syscall.EpollEvent{
    Events: syscall.EPOLLIN,
    Fd:     int32(fd),
  })
  return ep
}

func send() {
  pair, _ := syscall.Socketpair(syscall.AF_UNIX, syscall.SOCK_STREAM, 0)
  serverFd, clientFd := pair[0], pair[1]
  ep := watch(serverFd)

  c := dial(sock)
  sendFD(c, serverFd) // the receiver gets its own number for this same socket
  c.Read(make([]byte, 1))

  fmt.Printf("one byte written to the other end\n")
  syscall.Write(clientFd, []byte("X"))
  fmt.Printf("sender   -> epoll told it: %v\n", told(ep))

  c.Write([]byte{0})      // your turn to report
  c.Read(make([]byte, 1)) // both have looked, nobody has read yet
  fmt.Printf("sender   -> read: %s\n", tryRead(serverFd))
  c.Write([]byte{0})
  c.Read(make([]byte, 1)) // wait for the receiver's line before exiting
}

func receive() {
  os.Remove(sock)
  l, _ := net.ListenUnix("unix", &net.UnixAddr{Name: sock, Net: "unix"})
  defer os.Remove(sock)

  c, _ := l.AcceptUnix()
  fd := recvFD(c)
  ep := watch(fd)
  c.Write([]byte{0}) // registered, safe to write the byte now

  saw := told(ep)         // look before anyone reads
  c.Read(make([]byte, 1)) // print only when the sender has had its turn
  fmt.Printf("receiver -> epoll told it: %v\n", saw)
  c.Write([]byte{0}) // looked, still have not read

  c.Read(make([]byte, 1)) // the sender has taken its read
  fmt.Printf("receiver -> read: %s\n", tryRead(fd))
  c.Write([]byte{0})
}

one byte written to the other end
sender   -> epoll told it: true
receiver -> epoll told it: true
sender   -> read: "X"
receiver -> read: nothing (resource temporarily unavailable)

What makes it worse

The surprise is here: close() is not a cancel.

The misdirected reply is the symptom you can see. Two more things follow from that surviving entry, and the run below produces both.

Register fd 5 with epoll, then register fd 5 a second time. The second call fails: it is already in there. Which makes epoll look like it is keeping track of the connection, and asking twice is simply a mistake on your side. fd 7 is that same connection under a different number, so it should fail the same way. The tag 4242 marks the first registration, so the rows can be told apart later.

func main() {
  epollFd, _ := syscall.EpollCreate1(0)
  pair, _ := syscall.Socketpair(syscall.AF_UNIX, syscall.SOCK_STREAM, 0)
  serverFd, clientFd := pair[0], pair[1]
  keeperFd, _ := syscall.Dup(serverFd)
  drain := make([]byte, 1)

  fmt.Printf("fd %d and fd %d name one socket\n\n", serverFd, keeperFd)
  fmt.Printf("register fd %d as tag 4242  -> %s\n", serverFd,
    outcome(wireFdToEpollAsTag(serverFd, epollFd, 4242), "accepted"))
  fmt.Printf("register fd %d once more    -> %s\n", serverFd,
    outcome(wireFdToEpollAsTag(serverFd, epollFd, 9999), "accepted"))
  fmt.Printf("register fd %d as tag %d     -> %s\n", keeperFd, keeperFd,
    outcome(wireFdToEpollAsTag(keeperFd, epollFd, int32(keeperFd)), "accepted"))
  fmt.Printf("the interest list          -> %s\n", interestList(epollFd))
  syscall.Write(clientFd, []byte("X"))
  fmt.Printf("one write                  -> tags %v\n\n", waitForReadyTags(epollFd))
  syscall.Read(keeperFd, drain)

  syscall.Close(serverFd)
  fmt.Printf("close fd %d                 -> DEL fd %d: %s\n", serverFd, serverFd,
    outcome(unwireFdFromEpoll(serverFd, epollFd), "removed"))

  var fresh [2]int
  syscall.Pipe(fresh[:])
  fmt.Printf("a pipe retakes fd %d        -> DEL fd %d: %s\n", fresh[0], fresh[0],
    outcome(unwireFdFromEpoll(fresh[0], epollFd), "removed"))

  fmt.Printf("the interest list          -> %s\n", interestList(epollFd))
  syscall.Write(clientFd, []byte("Y"))
  fmt.Printf("one write                  -> tags %v\n\n", waitForReadyTags(epollFd))
  syscall.Read(keeperFd, drain)

  fmt.Printf("DEL fd %d                   -> %s\n", keeperFd,
    outcome(unwireFdFromEpoll(keeperFd, epollFd), "removed"))
  fmt.Printf("the interest list          -> %s\n", interestList(epollFd))
  syscall.Write(clientFd, []byte("Z"))
  fmt.Printf("one write                  -> tags %v\n\n", waitForReadyTags(epollFd))
  syscall.Read(keeperFd, drain)

  syscall.Close(keeperFd)
  fmt.Printf("close fd %d                 -> the socket's last reference\n", keeperFd)
  fmt.Printf("the interest list          -> %s\n", interestList(epollFd))
}

fd 5 and fd 7 name one socket

register fd 5 as tag 4242  -> accepted
register fd 5 once more    -> file exists
register fd 7 as tag 7     -> accepted
the interest list          -> (fd 5, tag 4242) (fd 7, tag 7)
one write                  -> tags [7 4242]

close fd 5                 -> DEL fd 5: bad file descriptor
a pipe retakes fd 5        -> DEL fd 5: no such file or directory
the interest list          -> (fd 5, tag 4242) (fd 7, tag 7)
one write                  -> tags [7 4242]

DEL fd 7                   -> removed
the interest list          -> (fd 5, tag 4242)
one write                  -> tags [4242]

close fd 7                 -> the socket's last reference
the interest list          -> empty

func wireFdToEpollAsTag(fd, epollFd int, tag int32) error {
  return syscall.EpollCtl(epollFd, syscall.EPOLL_CTL_ADD, fd, &syscall.EpollEvent{
    Events: syscall.EPOLLIN,
    Fd:     tag,
  })
}

func unwireFdFromEpoll(fd, epollFd int) error {
  return syscall.EpollCtl(epollFd, syscall.EPOLL_CTL_DEL, fd, nil)
}

func waitForReadyTags(epollFd int) []int32 {
  events := make([]syscall.EpollEvent, 8)
  n, _ := syscall.EpollWait(epollFd, events, 300)
  tags := []int32{}
  for i := 0; i < n; i++ {
    tags = append(tags, events[i].Fd)
  }
  return tags
}

// procfs prints one line per interest-list entry, so the list can be read
// rather than inferred from what fires.
func interestList(epollFd int) string {
  raw, _ := os.ReadFile(fmt.Sprintf("/proc/self/fdinfo/%d", epollFd))
  var rows []string
  for _, line := range strings.Split(string(raw), "\n") {
    if !strings.HasPrefix(line, "tfd:") {
      continue
    }
    f := strings.Fields(line)
    tag, _ := strconv.ParseUint(f[5], 16, 64) // the kernel prints data in hex
    rows = append(rows, fmt.Sprintf("(fd %s, tag %d)", f[1], tag))
  }
  if rows == nil {
    return "empty"
  }
  return strings.Join(rows, " ")
}

func outcome(err error, ok string) string {
  if err != nil {
    return err.Error()
  }
  return ok
}

Two of the three answers are what turn a misdirected reply into something worse.

One write arrives as two events. The same descriptor twice is a duplicate and refused, the same socket under a second descriptor is not, so what the kernel matches on is the pair. One socket carries two rows, and one byte coming in is reported once per row.

Registering a descriptor is a promise to keep it. The row keyed on fd 5 keeps firing once fd 5 is closed, and that number can no longer reach it: removing by it fails while it is free, and finds no such row once a new file has taken it. Removing by fd 7 takes out only fd 7’s row. The key is still the pair, so there is one way back: put the same socket at the number again, with a dup2 from a reference you still hold, and the removal matches and succeeds. It works even if something else has taken the number, because dup2 closes whatever is there first. What it needs is the second handle, which is exactly what you do not have if your close was the last one.

The third answer is the one behind the original bug: the tag 4242 comes back exactly as it went in, so the kernel never looked a descriptor up on the way out. And the list itself is read from /proc/self/fdinfo rather than guessed at from what fires.

So cancel the registration before you close the descriptor, never after. Afterwards you are relying on a second handle you may not have, and on whether close() happened to be the last reference, which depends on something you may not control: whether anyone else is still holding the same file.

/* struct epoll_filefd in fs/eventpoll.c */
struct epoll_filefd {
  struct file *file;   // compared first
  int fd;              // only breaks ties
} __packed;

kqueue disagrees about all of this. I assumed it was the same idea with different spelling. On XNU, fd-based registrations hang off the descriptor table and are indexed by the number itself:

/* knote_fdfind in bsd/kern/kern_event.c */
if (is_fd) {
  /* fd-based knotes are linked off the fd table */
  if (kev->kei_ident < (u_int)fdp->fd_knlistsize) {
    list = &fdp->fd_knlist[kev->kei_ident];
  }
} else if (fdp->fd_knhashmask != 0) {
  /* hash non-fd knotes here too */
  list = &fdp->fd_knhash[KN_HASH((u_long)kev->kei_ident, fdp->fd_knhashmask)];
}

knote_fdfind

Matching is on the descriptor number plus the filter. Two operating systems, two different answers to “what is being watched”. That is why portable code over both has to choose semantics, not just syscalls.

So: is everything a file?

Yes, and more literally than the slogan suggests. It is not that the kernel treats things like files. It builds a real struct file for them and hangs a table of functions off it saying what the verbs mean. A socket is a file because someone wrote socket_file_ops.

Which is also why the abstraction has a visible edge. Ask epoll to watch a file on disk and it refuses:

/* do_epoll_ctl in fs/eventpoll.c */
if (!file_can_poll(fd_file(tf)))
  return -EPERM;

do_epoll_ctl

Regular files do not implement poll, so epoll_ctl rejects them with EPERM. Sockets and pipes have a readiness question worth asking. A file on disk is always “ready”, and the wait you actually care about is the one the disk imposes, which readiness has no way to express.

So the honest version of the slogan is narrower and more useful. Everything is a file in the sense that everything gets the same handle: a number, an entry in your table, and an object underneath with an ops table. Everything is not a file in the sense of behaving alike, and the sharp edges of this whole subject are exactly where that difference surfaces. A read position that two handles share. A number that comes back the moment you free it. An epoll registration that outlives the descriptor you closed. None of those are quirks. They are what it costs to put one interface over things that have nothing else in common.

What this buys you

A short list of what the runs above actually settled.

• The number is an index, and nothing more. It is a slot in a per-process table, so the same number means different things in two processes, and /proc/self/fd is where you look it up.
• dup copies at a different level than open. Two descriptors from dup share one read position; two open() calls on one path do not. That is the whole reason the middle box exists.
• Sharing crosses processes. fork and a descriptor passed over a Unix socket both land on the same open file description, so the shared position is not a parent-and-child special case.
• Sockets really are files. The kernel builds a struct file and hangs an ops table off it. /proc will even name the socket, and that name cannot be opened.
• The number you close is the next one handed out. Not eventually, immediately, and POSIX requires it.
• epoll reports a tag you chose, not an object. The entry is keyed on the file and the descriptor, it survives closing that descriptor, and once the number is gone you cannot cancel it by that number alone.
• Readiness is per-file, delivery is per-queue. Every watcher of a socket is told it is ready; only one of them gets the bytes.
• 2>&1 > file and > file 2>&1 differ for one reason. Redirections apply left to right and each copies whatever the slot holds at that moment, so a copy is not a link. Worth deriving once instead of memorising.

The last one generalises past shells. Anything handing out small reusable integers inherits it: connection ids, session ids, slots in an object pool. The fix has one shape, a counter beside the number that moves when the slot is recycled, so a stale reference can be recognised rather than prevented. Go’s runtime keeps exactly that counter beside every descriptor it watches (pollDesc.fdseq).

Sources

• POSIX.1-2024: §2.6 File Descriptor Allocation for the lowest-available rule, and open() for the definition of “open file description”.
• Linux man pages: socket(2), accept(2), epoll(7).
• Books: Kerrisk, The Linux Programming Interface. Stevens, UNIX Network Programming, Volume 1.

The source, pinned

Every line referenced above, at the revision it was read: Linux v7.1, XNU xnu-11215.41.3, Go 01534385, busybox 1_36_1.

• fs/file.c: alloc_fd, find_next_fd, __put_unused_fd
• fs/eventpoll.c: struct epoll_filefd, ep_cmp_ffd, eventpoll_release_file
• bsd/kern/kern_event.c: knote_fdfind (a source release, not the kernel Apple ships, so the line numbers are pinned to that release)