A segmentation fault is the kernel telling you that your process touched memory it does not own. It is not a mystery, and it is not something you should chase with scattered printf calls. With three tools that are already on almost every Linux box (GDB, core dumps and AddressSanitizer), you can go from Segmentation fault (core dumped) to the exact line number in under five minutes.

This guide is deliberately hands-on. We build three programs that crash in the three classic ways, then run the same repeatable diagnostic checklist on each one so you can copy the workflow into your own project today.

The 5-minute segfault checklist (copy this)

1. Rebuild with debug info: gcc -g3 -O0 -fno-omit-frame-pointer prog.c -o prog
2. Run it under GDB: gdb -q ./prog then run
3. Get the stack: bt (or bt full for locals in every frame)
4. Inspect the faulting frame: frame 0, list, info locals, print ptr
5. Ask the kernel what happened: print $_siginfo and read si_addr
6. If it only crashes in production: enable core dumps and load the core: gdb ./prog core or coredumpctl debug prog
7. If GDB points at a line that looks innocent: rebuild with -fsanitize=address -fsanitize=undefined and run again

That is the whole method. The rest of this article shows what each step actually prints, and how to interpret it.

Step 0: build flags that make debugging possible

Most “GDB shows me nothing useful” complaints come from bad build flags. Before anything else, compile with:

gcc -g3 -O0 -fno-omit-frame-pointer -Wall -Wextra prog.c -o prog

Flag	Why it matters for segfault hunting
-g3	Embeds line numbers, variable names and macro definitions. Without it GDB shows raw addresses.
-O0	Stops the optimizer from inlining, reordering or deleting the code you are trying to read.
-fno-omit-frame-pointer	Keeps backtraces reliable, especially after a stack smash.
-Wall -Wextra	A surprising share of segfaults are already reported as warnings at compile time.

Note: if you cannot reproduce the crash at -O0, keep the optimization level of the failing build and just add -g. Debug info and optimization are independent; you will still get file and line, only variables may be <optimized out>.

Scenario 1: NULL pointer dereference

The most common segfault in C. A function returns NULL on failure, nobody checks it, and the next dereference kills the process.

The reproducer

/* null_deref.c */
#include <stdio.h>

struct config {
    char name[32];
    int  retries;
};

static struct config *load_config(const char *path)
{
    (void)path;
    return NULL;              /* pretend the file was missing */
}

static int get_retries(struct config *cfg)
{
    return cfg->retries;      /* boom */
}

int main(void)
{
    struct config *cfg = load_config("/etc/app.conf");
    printf("retries = %d\n", get_retries(cfg));
    return 0;
}

gcc -g3 -O0 -fno-omit-frame-pointer null_deref.c -o null_deref
./null_deref
Segmentation fault (core dumped)

Diagnosing it with GDB

$ gdb -q ./null_deref
Reading symbols from ./null_deref...
(gdb) run
Starting program: /home/dev/null_deref

Program received signal SIGSEGV, Segmentation fault.
0x0000555555555159 in get_retries (cfg=0x0) at null_deref.c:18
18          return cfg->retries;      /* boom */
(gdb) bt
#0  get_retries (cfg=0x0) at null_deref.c:18
#1  0x000055555555517f in main () at null_deref.c:24
(gdb) print cfg
$1 = (struct config *) 0x0
(gdb) print $_siginfo.si_addr
$2 = (void *) 0x0
(gdb) frame 1
#1  0x000055555555517f in main () at null_deref.c:24
24          printf("retries = %d\n", get_retries(cfg));
(gdb) info locals
cfg = 0x0

How to read this

• GDB stops exactly on the faulting instruction. Frame #0 is where the CPU died, not necessarily where the bug was introduced.
• cfg=0x0 in the frame header is the smoking gun. A parameter printed as 0x0 means NULL was passed in.
• si_addr = 0x0 confirms the kernel: the process tried to access address zero. Small addresses such as 0x8 or 0x20 mean a NULL pointer plus a struct member offset, which is still a NULL dereference.
• Walk up the stack with frame 1, frame 2, up, down until you find the function that should have checked the return value. That is the real bug site.

The fix

struct config *cfg = load_config("/etc/app.conf");
if (cfg == NULL) {
    fprintf(stderr, "cannot load config\n");
    return 1;
}

Scenario 2: stack overflow from deep recursion

This one confuses people because the code is often correct in logic and simply too deep. The thread’s stack (8 MB by default on most Linux distributions) runs into the guard page and the kernel raises SIGSEGV.

The reproducer

/* deep_recursion.c */
#include <stdio.h>

static long sum_to(long n)
{
    char padding[512];        /* make each frame fat */
    padding[0] = (char)n;
    if (n == 0)
        return 0;
    return padding[0] + n + sum_to(n - 1);
}

int main(void)
{
    printf("%ld\n", sum_to(1000000));
    return 0;
}

Diagnosing it with GDB

$ gdb -q ./deep_recursion
(gdb) run

Program received signal SIGSEGV, Segmentation fault.
0x0000555555555151 in sum_to (n=986733) at deep_recursion.c:6
6           char padding[512];
(gdb) bt 6
#0  sum_to (n=986733) at deep_recursion.c:6
#1  0x00005555555551b4 in sum_to (n=986734) at deep_recursion.c:10
#2  0x00005555555551b4 in sum_to (n=986735) at deep_recursion.c:10
#3  0x00005555555551b4 in sum_to (n=986736) at deep_recursion.c:10
#4  0x00005555555551b4 in sum_to (n=986737) at deep_recursion.c:10
#5  0x00005555555551b4 in sum_to (n=986738) at deep_recursion.c:10
(More stack frames follow...)
(gdb) print $_siginfo.si_addr
$1 = (void *) 0x7ffffd7fefd8
(gdb) print $sp
$2 = (void *) 0x7ffffd7ff000
(gdb) info frame
Stack level 0, frame at 0x7ffffd7ff210:
...

The three signals that identify a stack overflow

1. The backtrace is thousands of identical frames. Use bt 20 to see the top and bt -20 to see the bottom (where main lives). To count them: set confirm off then bt -1, or simply echo bt | gdb -batch -p PID | wc -l.
2. si_addr is very close to $sp and sits in the high 0x7ff... region. A fault address one page below the stack pointer means you hit the guard page, not a bad pointer.
3. The crash happens in the function prologue, on the line that declares locals, because that is where the stack pointer is decremented.

Check the limit with ulimit -s (usually 8192, meaning 8 MB). AddressSanitizer confirms it explicitly:

$ gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer deep_recursion.c -o dr_asan
$ ./dr_asan
==31427==ERROR: AddressSanitizer: stack-overflow on address 0x7ffc3a1fdff8 (pc 0x... bp 0x... sp 0x...)
    #0 0x55b0d in sum_to deep_recursion.c:6
    #1 0x55b7a in sum_to deep_recursion.c:10
    ...
SUMMARY: AddressSanitizer: stack-overflow deep_recursion.c:6 in sum_to

The fixes

• Convert the recursion to a loop, or add an explicit depth counter and bail out.
• Move large buffers off the stack: allocate with malloc instead of declaring char buf[1<<20].
• Raise the limit only as a last resort: ulimit -s 65536, or pthread_attr_setstacksize() for threads.
• Watch out for alloca() and variable length arrays inside loops; they are stack overflows waiting to happen.

Scenario 3: writing past an array bound

This is the nastiest of the three, because it frequently does not segfault at all. You corrupt the heap or the stack, and the crash appears somewhere else, minutes later, in code that is perfectly correct. This is exactly where GDB alone is not enough.

The reproducer

/* overflow.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void)
{
    int *scores = malloc(8 * sizeof *scores);
    if (!scores)
        return 1;

    for (int i = 0; i <= 8; i++)      /* off-by-one: writes scores[8] */
        scores[i] = i * i;

    char small[4];
    strcpy(small, "way too long");    /* stack buffer overflow */

    printf("%d %s\n", scores[7], small);
    free(scores);
    return 0;
}

What plain GDB shows you

$ gcc -g3 -O0 overflow.c -o overflow
$ gdb -q ./overflow
(gdb) run
49 way too long

Program received signal SIGSEGV, Segmentation fault.
__GI___libc_free (mem=0x6d20676e6f6c206f) at malloc.c:3368
3368    malloc.c: No such file or directory.
(gdb) bt
#0  __GI___libc_free (mem=0x6d20676e6f6c206f) at malloc.c:3368
#1  0x00005555555552a1 in main () at overflow.c:19

Notice the trap: the backtrace blames free() inside glibc at line 19, while the actual bug is on lines 11 and 14. The pointer 0x6d20676e6f6c206f is ASCII text (“o lon m”), a clear fingerprint of memory corruption by a string copy. When you see a crash inside malloc, free, memcpy or __stack_chk_fail, stop reading the backtrace and switch to AddressSanitizer. Related reading: Debugging Segmentation Faults using GEF and GDB.

AddressSanitizer pinpoints the exact line

$ gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer overflow.c -o overflow_asan
$ ./overflow_asan
=================================================================
==5120==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x502000000030 at pc 0x5610f4c0a2ef bp 0x7ffd0f1a26b0 sp 0x7ffd0f1a26a0
WRITE of size 4 at 0x502000000030 thread T0
    #0 0x5610f4c0a2ee in main /home/dev/overflow.c:12
    #1 0x7f2a3ce2a1c9 in __libc_start_call_main
    #2 0x5610f4c0a144 in _start

0x502000000030 is located 0 bytes after 32-byte region [0x502000000010,0x502000000030)
allocated by thread T0 here:
    #0 0x7f2a3d0b8887 in malloc
    #1 0x5610f4c0a26a in main /home/dev/overflow.c:8

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/dev/overflow.c:12 in main
Shadow bytes around the buggy address:
  0x502000000000: fa fa 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa
==5120==ABORTING

Everything you need is in the first ten lines:

• The error class: heap-buffer-overflow (others you will meet: stack-buffer-overflow, heap-use-after-free, global-buffer-overflow, stack-use-after-return, double-free).
• The operation: WRITE of size 4, so a 4-byte write, which matches an int.
• The faulting line: overflow.c:12, the loop body.
• The allocation site: overflow.c:8, so you immediately see the 32-byte buffer that was too small.
• The offset: 0 bytes after a 32-byte region screams off-by-one.

Useful ASAN_OPTIONS

ASAN_OPTIONS=abort_on_error=1:detect_leaks=1:strict_string_checks=1:detect_stack_use_after_return=1 ./overflow_asan

Option	Effect
abort_on_error=1	Raises SIGABRT so you also get a core dump you can open in GDB.
detect_leaks=1	Turns on LeakSanitizer at exit (default on Linux x86-64).
halt_on_error=0	Keeps running after a recoverable error to collect several reports in one run.
log_path=/tmp/asan	Writes reports to files, handy for daemons and CI.
fast_unwind_on_malloc=0	Better allocation stacks when the default unwinder gives you nothing.

Running ASan under GDB

You can combine both tools. Set abort_on_error=1, then break on the sanitizer reporting function to inspect live state at the moment of the error:

$ ASAN_OPTIONS=abort_on_error=1 gdb -q ./overflow_asan
(gdb) break __asan_report_error
(gdb) run
(gdb) bt

Add -fsanitize=undefined as well. UBSan catches signed overflow, misaligned loads, invalid shifts and NULL dereferences that ASan lets through, and it prints the same file:line format. The reasoning is set out in this piece.

Enabling and reading core dumps on Linux

Core dumps are how you debug a segfault that only happens on a customer machine, in a container or at 3 a.m. in production. The workflow is: let the crash write a core file, then open it in GDB later with the exact same binary.

1. Lift the core file size limit

# current shell only
ulimit -c unlimited
ulimit -c        # should print: unlimited

To make it permanent for interactive users, add to /etc/security/limits.d/99-core.conf:

*  soft  core  unlimited
*  hard  core  unlimited

For a systemd service, edit the unit with systemctl edit myservice and add:

[Service]
LimitCORE=infinity

2. Find out where the core goes

cat /proc/sys/kernel/core_pattern

Value you see	What it means	How to get the core
|/usr/lib/systemd/systemd-coredump ...	systemd captures and compresses cores (Fedora, RHEL 9+, Ubuntu 22.04+, Debian 12+, Arch)	coredumpctl
|/usr/share/apport/apport ...	Ubuntu Apport intercepts crashes	Look in /var/crash or disable apport
core	Plain file in the process working directory	ls -l core*

To force a predictable path, which is what I recommend on build servers and inside containers:

sudo sysctl -w kernel.core_pattern=/var/crash/core.%e.%p.%t
sudo mkdir -p /var/crash && sudo chmod 1777 /var/crash

# persist it
echo 'kernel.core_pattern=/var/crash/core.%e.%p.%t' | sudo tee /etc/sysctl.d/99-coredump.conf

Common specifiers: %e executable name, %p PID, %t timestamp, %s signal number, %u UID.

3. Open the core in GDB

$ gdb -q ./null_deref /var/crash/core.null_deref.7412.1786000000
Core was generated by `./null_deref'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  get_retries (cfg=0x0) at null_deref.c:18
18          return cfg->retries;
(gdb) bt full
(gdb) info registers
(gdb) info threads
(gdb) thread apply all bt

4. With systemd-coredump

coredumpctl list                 # every recent crash on the machine
coredumpctl list null_deref      # filter by program
coredumpctl info null_deref      # signal, command line, stack summary
coredumpctl debug null_deref     # opens GDB on the newest core, symbols included
coredumpctl dump null_deref > /tmp/core   # export the raw core file

If coredumpctl reports that the core was not stored, set Storage=external and a large ProcessSizeMax in /etc/systemd/coredump.conf, then run systemctl daemon-reexec.

5. Core dumps inside containers

• kernel.core_pattern is a host-wide setting; the container cannot override it.
• Run the container with --ulimit core=-1 and mount a writable volume matching the pattern path.
• If the pattern pipes to systemd-coredump, the core lands on the host, so debug it there with coredumpctl.
• Debug with the identical binary and libraries: copy the image’s binary out, or run GDB inside the same image.

6. Missing symbols in a release build

If the backtrace is full of ??, install debug symbols or let debuginfod fetch them automatically:

# Debian / Ubuntu
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"
# Fedora / RHEL
sudo dnf debuginfo-install glibc

# separate debug file produced at build time
objcopy --only-keep-debug prog prog.debug
objcopy --strip-debug --add-gnu-debuglink=prog.debug prog

GDB command cheat sheet for segfaults

Command	What it gives you
run / run arg1 arg2	Starts the program, stops automatically on SIGSEGV
bt, bt full, bt 20, bt -20	Backtrace, with locals, limited to the top or bottom frames
frame N, up, down	Move through the call stack to find who passed the bad pointer
list, list 10,30	Show source around the current line
info locals, info args	Values of every local and parameter in the current frame
print ptr, print *ptr, p/x val	Inspect a pointer, its target, or a value in hex
print $_siginfo	Kernel signal details, including si_addr, the exact faulting address
x/16xb ptr, x/s ptr, x/i $pc	Examine raw memory, a string, or the failing instruction
info registers	Register dump, useful on optimized builds
watch ptr, rwatch, awatch	Hardware watchpoint that stops the moment a variable is overwritten
info proc mappings	Shows whether the faulting address belongs to any mapped region at all
thread apply all bt	Backtrace of every thread, essential for multithreaded crashes
gdb -p PID	Attach to a running process instead of restarting it
gdb -batch -ex run -ex bt --args ./prog a b	One-liner backtrace for CI pipelines and scripts

Reading the fault address like a pro

si_addr value	Most likely cause
0x0 or a small offset such as 0x18	NULL pointer dereference, possibly through a struct member
Just below $sp, in the 0x7ff... range	Stack overflow, usually runaway recursion or a huge local array
ASCII-looking value such as 0x6f6c206f	Heap or stack corruption by a string operation
0xffffffffffffffff or an absurd value	Uninitialized pointer, or a pointer used after free()
A valid-looking address, crash on write only	Writing to a read-only page, typically a string literal

Which tool for which symptom

Tool	Best at	Slowdown	Needs recompile
GDB	Where it crashed, call stack, live variable inspection	None until you break	No, only -g recommended
Core dumps	Post-mortem analysis of crashes you cannot reproduce	None	No
AddressSanitizer	Buffer overflows, use-after-free, exact bug line	About 2x, memory about 3x	Yes, -fsanitize=address
UBSan	Undefined behaviour before it turns into a crash	Low	Yes, -fsanitize=undefined
Valgrind (memcheck)	Same class of bugs without recompiling, plus uninitialized reads	10x to 50x	No

Rule of thumb: GDB tells you where the process died. AddressSanitizer tells you where the bug is. Use GDB first because it is instant, and reach for ASan the moment the backtrace lands inside libc. Note that ASan and Valgrind cannot be used on the same run. The team at gnu.org reached a similar conclusion.

Segfault causes ranked by how often they bite

1. Dereferencing a pointer returned by malloc, fopen, strchr or getenv without checking for NULL.
2. Off-by-one loops using <= where < was meant.
3. strcpy, sprintf and gets into a buffer that is too small (use snprintf and strncat with correct sizes).
4. Using memory after free(), or freeing the same block twice.
5. Returning the address of a local variable from a function.
6. Writing to a string literal: char *s = "hi"; s[0] = 'H'; is undefined and segfaults on Linux. Declare char s[] = "hi"; instead.
7. Wrong printf or scanf format specifiers, and forgetting the & in scanf("%d", &x).
8. Unbounded recursion or oversized stack arrays.
9. Data races in threaded code that corrupt a shared pointer.

Make segfaults harder to create in the first place

• Build CI with -fsanitize=address,undefined and run the full test suite there. Sanitizers only find bugs on code paths you actually execute, so test coverage matters.
• Ship release builds with -D_FORTIFY_SOURCE=3 -fstack-protector-strong -Wl,-z,relro,-z,now.
• Turn on -Wall -Wextra -Werror and add -Wanalyzer-null-dereference with GCC’s static analyzer (-fanalyzer).
• Run clang-tidy or cppcheck in your pre-commit hook.
• Keep a .gdbinit in your project with set print pretty on and set pagination off.

FAQ

How do I fix a segmentation fault in C?

You cannot fix it without locating it first. Rebuild with -g -O0, run the program under GDB, type run then bt, and look at the pointer values in the faulting frame. Nine times out of ten you will find a NULL pointer, an out of range index or a freed pointer. If the backtrace stops inside malloc or free, rebuild with -fsanitize=address to get the exact line that corrupted memory.

What exactly is a segmentation fault?

It is the signal SIGSEGV (number 11) sent by the Linux kernel when your process accesses a virtual address that is not mapped, or that it is not allowed to access in that way, for example writing to a read-only page. The MMU raises a page fault, the kernel decides the access is illegal, and the default action terminates the process.

What does “Segmentation fault (core dumped)” mean?

The crash happened and the kernel also wrote a snapshot of the process memory to disk. Find it with coredumpctl list or by checking cat /proc/sys/kernel/core_pattern, then open it with gdb ./prog core and run bt full. If you see the message but no file, your ulimit -c is 0 or the core went to systemd instead of the current directory.

Why does my program crash on Linux but work fine on Windows or macOS?

Undefined behaviour is not portable. Reading one element past an array or using freed memory can silently work on one allocator and crash on another. Different stack sizes, different heap layouts and ASLR change the outcome. The bug exists on both platforms; only one of them is honest about it. Run AddressSanitizer on both, since Clang supports it on macOS and MSVC supports it on Windows.

Can GDB find the bug if the program does not crash every time?

Intermittent segfaults usually mean memory corruption or a data race. Use AddressSanitizer for the first case and ThreadSanitizer (-fsanitize=thread) for the second. You can also set a hardware watchpoint in GDB with watch some_ptr to stop the instant a value changes, or use rr to record a failing run and replay it deterministically.

Do I need root to enable core dumps?

You need root to change kernel.core_pattern or the systemd defaults, but not to raise your own soft limit with ulimit -c unlimited if the hard limit allows it. As a normal user you can always run the program under GDB directly, which requires no privileges at all.

Does AddressSanitizer replace Valgrind?

For overflow and use-after-free detection, ASan is roughly 10 to 20 times faster and gives better reports, so prefer it when you can recompile. Valgrind still wins when you only have a binary, when you need uninitialized-read detection (--track-origins=yes), or when you are debugging a third party library you cannot rebuild.

How do I debug a segfault in a program that has already been deployed?

Keep the unstripped binary or the separate .debug file for every release, enable core dumps on the target, and when a crash happens copy the core plus the matching binary to your workstation. GDB will match them through the build ID. Never debug a core with a different build of the same program; the line numbers will be wrong.

Conclusion

Debugging a segmentation fault in C is a mechanical process, not detective work. Compile with -g -O0, get a backtrace in GDB, read the pointer values and the fault address, capture a core dump when the crash lives on another machine, and switch to AddressSanitizer whenever the stack trace points into libc. That sequence handles the overwhelming majority of segfaults you will meet, including the three we reproduced here, and it takes minutes instead of an afternoon of printf archaeology.

Save the checklist at the top of this page, add the sanitizer flags to your CI configuration, and the next Segmentation fault (core dumped) becomes a line number instead of a bad day.