Every developer who has ever opened a file, printed a line, or sent a byte over a socket has used a Linux system call. Very few have ever watched one happen. Most explanations stop at “a system call is how a user program asks the kernel for a service”, which is true but tells you nothing about what the CPU actually does.

This article goes the other way. We take one single read() call and follow it, instruction by instruction, from your C source code, through the C library wrapper, across the syscall instruction on x86-64, into the kernel syscall table, down into the virtual filesystem layer, and all the way back into your variable. Along the way we look at real strace output and the exact registers involved.

Linux System Calls Explained in One Paragraph

A system call is a controlled entry point into the kernel. User space code runs with restricted CPU privileges and cannot touch hardware, page tables, or another process’s memory. When a program needs something privileged (reading a disk block, allocating memory, creating a process), it places a syscall number and arguments in specific CPU registers and executes a special instruction that switches the processor into kernel mode at a fixed, pre-registered address. The kernel validates the request, performs the work, places the result back in a register, and returns. The kernel never jumps to an address your program chooses, which is exactly what makes the boundary safe.

That is the short answer. Now let us prove it.

The Program We Will Trace

Here is the entire subject of our investigation:

/* readdemo.c */
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int main(void) {
    char buf[4096];
    int fd = open("hello.txt", O_RDONLY);
    ssize_t n = read(fd, buf, sizeof(buf));
    write(1, buf, n);
    close(fd);
    return 0;
}

Compile it with gcc -O0 -g readdemo.c -o readdemo and create hello.txt containing Hello, syscall!.

The Ten Stages of a System Call

Before diving in, here is the full route. Keep this map open mentally for the rest of the article.

1. Your code calls read(fd, buf, count) in user space (CPU ring 3).
2. The dynamic linker resolves read to the glibc wrapper in libc.so.6.
3. The wrapper loads the syscall number and arguments into registers.
4. The syscall instruction executes and the CPU switches to ring 0.
5. Hardware jumps to the address stored in the MSR_LSTAR register, which points to entry_SYSCALL_64.
6. The kernel swaps GS, switches to the kernel stack, and saves user registers into a pt_regs structure.
7. do_syscall_64() indexes the syscall table and calls __x64_sys_read().
8. The VFS layer resolves the file descriptor and copies data into your buffer with copy_to_user().
9. The return value lands in RAX, exit work runs (signals, scheduling, tracing), and sysretq or iretq returns to ring 3.
10. glibc inspects RAX, sets errno if needed, and returns to your line of C.

Stage 1 and 2: Your Call Is Not a System Call Yet

This is the single most common misconception. When you write read() in C, you are not issuing a system call. You are calling an ordinary function in the C library that happens to be named after one. It follows the normal System V AMD64 calling convention:

C function argument	Register used by the C ABI
1st (fd)	RDI
2nd (buf)	RSI
3rd (count)	RDX
4th	RCX
5th	R8
6th	R9

The glibc wrapper exists for several reasons that the kernel deliberately refuses to handle:

• It converts the kernel’s negative return codes into the familiar -1 plus errno convention.
• It handles cancellation points for POSIX threads.
• It can redirect some calls to the vDSO (more on that later) so no kernel transition happens at all.
• It papers over architecture differences so your code compiles on x86-64, ARM64 and RISC-V unchanged.

Stage 3 and 4: The syscall Instruction on x86-64

Here is the crucial detail. The kernel uses a different register convention than the C ABI. The fourth argument moves from RCX to R10, because the syscall instruction itself destroys RCX.

The x86-64 Linux Syscall ABI

Register	Role on entry	Role on exit
RAX	System call number (0 for read)	Return value or negative errno
RDI	Argument 1	Preserved
RSI	Argument 2	Preserved
RDX	Argument 3	Preserved
R10	Argument 4 (not RCX)	Preserved
R8	Argument 5	Preserved
R9	Argument 6	Preserved
RCX	Clobbered by hardware	Holds saved RIP
R11	Clobbered by hardware	Holds saved RFLAGS

So for our read(3, buf, 4096), the registers just before the transition look like this:

RAX = 0x0000000000000000   ; __NR_read
RDI = 0x0000000000000003   ; fd = 3
RSI = 0x00007ffd3b2a1c50   ; address of buf
RDX = 0x0000000000001000   ; 4096 bytes

syscall

The syscall instruction does four things in hardware, with no memory access and no stack push:

1. Copies RIP (the address of the next instruction) into RCX.
2. Copies RFLAGS into R11, then masks flags using the MSR_SYSCALL_MASK register.
3. Loads the kernel code segment and sets the privilege level to ring 0.
4. Loads RIP from the MSR_LSTAR model specific register.

That MSR_LSTAR value was written once by the kernel at boot in syscall_init(), and it points to a single assembly symbol: entry_SYSCALL_64. There is exactly one door into the kernel on x86-64, and user space cannot move it.

What About int 0x80?

Older 32-bit Linux used the software interrupt int 0x80 with arguments in EBX, ECX, EDX, ESI, EDI, EBP. It still works on x86-64 kernels for compatibility but it is slow (it goes through the interrupt descriptor table) and it truncates pointers to 32 bits. Never use int 0x80 in new 64-bit code. Note also that syscall numbers differ between the 32-bit and 64-bit tables: read is 3 on i386 and 0 on x86-64.

Stage 5 and 6: Inside entry_SYSCALL_64

The assembly in arch/x86/entry/entry_64.S is some of the most carefully reviewed code in the kernel. Simplified, it performs:

• swapgs: swaps the user GS base with the kernel per-CPU GS base, giving the kernel access to per-CPU data.
• Switches RSP from the user stack to the kernel stack for this thread (read from cpu_current_top_of_stack).
• With kernel page table isolation enabled (the Meltdown mitigation), switches CR3 to the full kernel page tables.
• Pushes all user registers onto the kernel stack in the exact layout of struct pt_regs.
• Calls the C function do_syscall_64(regs, nr).

From this point the kernel is running C code, on a separate stack, in your process context. It is still “your” task, just in kernel mode. That is why top can show time split between us and sy for the same process.

Stage 7: The Linux Syscall Table

The dispatch is almost anticlimactic. Conceptually:

if (likely(nr < NR_syscalls)) {
    nr = array_index_nospec(nr, NR_syscalls);
    regs->ax = sys_call_table[nr](regs);
}

Three things are worth noticing:

• It is a plain array of function pointers. The syscall number is just an index, which is why numbers can never be reused or reordered.
• array_index_nospec() is a Spectre v1 mitigation that clamps the index so speculative execution cannot read past the table.
• Modern handlers take a single struct pt_regs * argument. The wrapper __x64_sys_read() unpacks the individual arguments out of it. This was introduced to avoid leaking stale register contents into syscall handlers.

Where the Table Comes From

The table is generated at build time from a plain text file, arch/x86/entry/syscalls/syscall_64.tbl. You can inspect the numbers on your own machine:

ausyscall --dump | head -20
# or
grep -w __NR_read /usr/include/asm/unistd_64.h

Common x86-64 Syscall Numbers

Number	Name	Purpose
0	read	Read bytes from a file descriptor
1	write	Write bytes to a file descriptor
2	open	Legacy file open (glibc now uses openat)
3	close	Release a file descriptor
9	mmap	Map memory or files into the address space
60	exit	Terminate the calling thread
231	exit_group	Terminate all threads in the process
257	openat	Open relative to a directory fd
425	io_uring_setup	Create an asynchronous I/O ring

Stage 8: What read() Actually Does Inside the Kernel

The handler is declared with a macro, SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count), in fs/read_write.c. The __user annotation is not decoration: it is checked by the sparse static analyser to make sure the kernel never dereferences a user pointer directly.

The chain from there looks like this:

1. fdget_pos(fd) looks up the struct file in the process file descriptor table and takes a reference. If the fd is invalid you get -EBADF right here.
2. vfs_read() checks that the file was opened readable, calls rw_verify_area() to validate the offset and range, and handles mandatory locking.
3. The call dispatches through file->f_op->read_iter, the filesystem specific handler. This is polymorphism in C: ext4, XFS, procfs, a pipe and a socket all plug different functions in here.
4. For a regular file the page cache is consulted. On a cache hit the data is already in RAM. On a miss the task is put to sleep, an I/O request goes to the block layer, and the scheduler runs something else.
5. copy_to_user() moves the bytes into your buffer, with access checks and exception table entries so that a bad pointer returns -EFAULT instead of crashing the kernel.
6. The byte count is returned up the chain and stored into regs->ax.

Step 4 is the reason a system call is not merely “a slow function call”. A blocking syscall can cause a context switch, which is why your latency numbers for a cold read look nothing like a warm one.

Stage 9 and 10: The Return Trip and errno

Before going back to user space the kernel runs syscall_exit_to_user_mode(), which handles the pending work flags: deliver signals, reschedule if needed, notify ptrace or seccomp, handle rseq. Then it restores registers, switches CR3 and GS back, and returns with sysretq (the fast path) or iretq (the slow path, used when the saved state cannot be restored safely by sysret).

Back in glibc, the wrapper does something like:

if ((unsigned long) ret >= (unsigned long) -4095) {
    errno = -ret;
    return -1;
}
return ret;

That is the whole secret of errno. The kernel returns small negative values between -1 and -4095 to signal errors. The kernel never sets errno. It is a thread local variable in user space, written by the C library. This is why a raw syscall made without glibc leaves errno untouched.

Seeing It For Yourself: strace

Run strace ./readdemo and you get the whole conversation between your binary and the kernel:

execve("./readdemo", ["./readdemo"], 0x7ffc1d2e9a30 /* 42 vars */) = 0
brk(NULL)                               = 0x55e8f4c21000
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=78921, ...}) = 0
mmap(NULL, 78921, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3c2b1a0000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
...
openat(AT_FDCWD, "hello.txt", O_RDONLY) = 3
read(3, "Hello, syscall!\n", 4096)      = 16
write(1, "Hello, syscall!\n", 16)       = 16
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Three observations that teach more than any diagram:

• Your four lines of C produced dozens of syscalls. Most belong to the dynamic linker loading libc.so.6.
• Your open() became openat(AT_FDCWD, ...). glibc maps the old call onto the newer, race-free variant.
• The value after = is literally the content of RAX on return. read returned 16, not 4096, because the file is 16 bytes. Short reads are normal, not an error.

Useful strace Options

Command	What it gives you
strace -e trace=read,write ./app	Filter to specific calls only
strace -c ./app	Summary table with counts and time per syscall
strace -T -tt ./app	Timestamps plus time spent inside each call
strace -f -p 1234	Attach to a running process and follow threads
strace -k ./app	Print the user space stack trace for each call
strace -e trace=%file ./app	All filesystem related calls by category

Keep in mind that strace uses ptrace and stops the tracee twice per syscall, so it can slow a program down by an order of magnitude. For production use perf trace or a bpftrace one-liner such as bpftrace -e 'tracepoint:syscalls:sys_enter_read { @[comm] = count(); }'.

Register-Level View With GDB

If you want to see the registers rather than take my word for it:

$ gdb ./readdemo
(gdb) catch syscall read
(gdb) run
Catchpoint 1 (call to syscall read), 0x00007ffff7eab2a1 in read () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) info registers rax rdi rsi rdx
rax   0xffffffffffffffda  -38
rdi   0x3                 3
rsi   0x7fffffffdc30      140737488346160
rdx   0x1000              4096
(gdb) continue
(gdb) info registers rax
rax   0x10                16

Note the value -38 (ENOSYS) in RAX on entry. That is a deliberate kernel convention: the entry code stores -ENOSYS so a tracer can tell the difference between “about to run” and “finished”. After the call completes, RAX holds 16, the byte count, exactly matching the strace line.

Bypassing the C Library Entirely

You can issue the syscall yourself. The generic way uses the syscall() helper, which is itself a thin glibc wrapper:

#include <sys/syscall.h>
#include <unistd.h>

long n = syscall(SYS_read, fd, buf, 4096);

Or go all the way down with inline assembly:

static inline long raw_read(int fd, void *buf, unsigned long count) {
    long ret;
    __asm__ volatile (
        "syscall"
        : "=a" (ret)
        : "a" (0L), "D" (fd), "S" (buf), "d" (count)
        : "rcx", "r11", "memory"
    );
    return ret;   /* negative value IS the error code, errno is NOT set */
}

The clobber list is the important part. You must declare rcx, r11 and memory, otherwise the compiler will assume registers and memory survived the instruction, and you will get bugs that only appear at high optimisation levels. Use this for learning, not for production code. Real software should call the library wrapper so that cancellation points, vDSO acceleration and future ABI changes are handled for you.

The vDSO: System Calls That Are Not System Calls

Not every libc function that looks like a syscall crosses into the kernel. The virtual dynamic shared object is a small shared library that the kernel maps into every process. It contains real kernel code that is safe to execute in user mode, so calls that only need to read kernel-maintained data can skip the transition completely.

On x86-64 the vDSO typically accelerates:

• clock_gettime()
• gettimeofday()
• time()
• getcpu()

This is why you will almost never see clock_gettime in strace output even in a program that calls it in a tight loop. Check what your system exports with cat /proc/self/maps | grep vdso or by running LD_SHOW_AUXV=1 /bin/true and looking at AT_SYSINFO_EHDR.

How Expensive Is a System Call?

A plain function call costs a few nanoseconds. A round trip through the kernel for a trivial syscall like getpid() costs roughly 50 to 150 nanoseconds on current x86-64 hardware, and significantly more when speculative execution mitigations are active. The cost comes from:

• Privilege level change and register save and restore.
• CR3 reload and TLB pressure under kernel page table isolation.
• Indirect branch barriers inserted by Spectre mitigations.
• Cache and branch predictor pollution from running kernel code.

You can measure the mitigation cost on your own box with cat /sys/devices/system/cpu/vulnerabilities/* and by benchmarking with mitigations=off on a test machine only.

Reducing Syscall Count

The practical takeaway for application developers:

1. Batch your I/O. Reading 4 KB at a time instead of 1 byte at a time cuts syscalls by a factor of 4096. This is precisely what stdio buffering does for you.
2. Use vectored I/O. readv() and writev() handle multiple buffers in one transition.
3. Use epoll instead of polling. One epoll_wait() replaces thousands of blocking reads.
4. Consider io_uring. It uses shared memory ring buffers so thousands of operations can be submitted and completed with few or even zero syscalls.
5. Use sendfile() or splice() to move data between descriptors without bouncing it through user space.

Security at the Boundary

Because the syscall interface is the complete attack surface between an unprivileged process and the kernel, Linux offers several ways to restrict it:

• seccomp-bpf: a per-process BPF filter that inspects the syscall number and argument registers and can allow, deny, kill or trap. This is what container runtimes use to block calls like keyctl or ptrace.
• LSMs such as SELinux and AppArmor, which hook deeper inside the handlers rather than at the entry point.
• Audit, via auditctl -a always,exit -F arch=b64 -S read, for compliance logging.

Important subtlety: a seccomp filter can read argument registers but cannot dereference user pointers, because the memory could be changed by another thread between the check and the use. That is the classic time-of-check to time-of-use problem, and it is why seccomp filters match on file descriptors and flags rather than on path strings.

Putting the Whole Journey in One Table

Stage	Where it runs	Key symbol or file
Application call	Ring 3, user stack	your main()
Library wrapper	Ring 3, user stack	libc.so.6
Mode switch	CPU hardware	syscall + MSR_LSTAR
Entry stub	Ring 0, kernel stack	entry_SYSCALL_64
Dispatch	Ring 0	do_syscall_64, sys_call_table
Handler	Ring 0	__x64_sys_read, vfs_read
Data copy	Ring 0	copy_to_user
Exit work	Ring 0	syscall_exit_to_user_mode
Return	CPU hardware	sysretq or iretq
errno handling	Ring 3	glibc wrapper

Practical Gotchas Developers Hit

• Partial reads and writes. read() returning less than you asked for is normal on pipes, sockets and terminals. Always loop.
• EINTR. A blocking call interrupted by a signal returns -1 with errno == EINTR unless SA_RESTART is set. Retry.
• errno is only valid after a failure. It is not cleared on success, so checking it without first checking the return value will give you stale data.
• Syscall numbers are architecture specific. Hardcoding 0 for read breaks instantly on ARM64 or i386. Use SYS_read from <sys/syscall.h>.
• Not every libc function is a syscall. printf, malloc and fopen are library functions that call write, brk/mmap and openat underneath, often less frequently than you expect because of buffering.
• Unimplemented calls return ENOSYS. If you run a binary built against a newer kernel header on an older kernel, this is the error you will see.

Frequently Asked Questions

What is a system call in Linux, in simple terms?

It is a request from an unprivileged user space program to the kernel, asking it to perform an operation the program is not allowed to do itself, such as reading a file or creating a process. The request travels through specific CPU registers and a single hardware instruction, not through a normal function pointer.

Is read() a system call or a C library function?

Both, in a sense. read() in your C file is a glibc function. It wraps the kernel’s read system call, number 0 on x86-64. The library function handles errno conversion and thread cancellation; the kernel does the actual I/O.

How many system calls does Linux have?

On x86-64 there are over 350 entries in the syscall table, and the number grows with each kernel release as new interfaces such as io_uring and landlock are added. Check your own system with ausyscall --dump | wc -l.

Where is the Linux syscall table defined?

For x86-64 the source of truth is arch/x86/entry/syscalls/syscall_64.tbl in the kernel tree. A build script turns it into the sys_call_table array of function pointers and into the __NR_* constants exported in /usr/include/asm/unistd_64.h.

What is the difference between syscall and int 0x80?

int 0x80 is the legacy 32-bit software interrupt mechanism; it goes through the interrupt descriptor table and is noticeably slower. syscall is the dedicated 64-bit fast instruction that jumps directly to an address in the MSR_LSTAR register. The two also use different argument registers and different syscall numbers.

Why does strace show openat instead of open?

Modern glibc implements open() on top of the openat() system call with AT_FDCWD as the directory descriptor. openat avoids path resolution race conditions and is the preferred interface, so the older open syscall is rarely issued directly anymore.

Does a system call always cause a context switch?

No. It always causes a privilege level change (a mode switch), but a context switch to a different process only happens if the call blocks, for example waiting on disk I/O or an empty socket. A cached read() typically returns without the scheduler ever running another task.

How can I trace system calls without slowing down production?

Use eBPF based tooling rather than ptrace. perf trace, bpftrace and the syscalls tracepoints under /sys/kernel/debug/tracing/events/syscalls/ collect data in the kernel with far lower overhead than strace.

Conclusion

The user-kernel boundary stops being mysterious once you see that it is built from very ordinary parts: a number in a register, one hardware instruction, an array of function pointers, and a copy routine that checks its pointers. Everything else, from errno to seccomp filters to io_uring, is engineering built around that core.

The best way to make this stick is to do it yourself. Write the five line program from the top of this article, run it under strace, then set a catch syscall breakpoint in GDB and look at RAX, RDI, RSI and RDX with your own eyes. After that, every line of I/O code you write will look different.