[COFF] [TUHS] Re: What does `ld` stand for?
Dan Cross via COFF
coff at tuhs.org
Wed Oct 28 03:04:55 AEST 2026
On Mon, Oct 26, 2026 at 6:49 PM Douglas McIlroy
<douglas.mcilroy at dartmouth.edu> wrote:
> Neat hack!
Thanks!
> Am I right in intuiting that this couldn't be done if there
> were two such arrays rather than one?
Not quite. I believe you can do it once per loadable segment, and you
can have a good number of those: the ELF header has a 16 bit field for
the total number program headers, but I suspect most systems will
impose some practical limit well below the full allowable range;
still, it's more than one. Of course, if you're using this trick for
more than one, you'll have to compact the binary more carefully.
> It depends, though, on the data section coming last. That presumably
> explains why gcc doesn't employ the trick.
Along with a couple of other things, yes. GCC (and other compilers)
are likely written in terms of sections, not segments, which don't
have this properly of different size values for memory vs. file. When
GCC writes out the header for a section consisting of only
uninitialized (or explicitly initialized with all zeros) data, it will
set the section type field to `SHT_NOBITS`, and won't write out data
for it in the object file; the linker will almost certainly aggregate
all such sections into one or more BSS segments, but playing the
aforementioned game with the memory size vs the file size on a
loadable segment. For other data-bearing sections, it sets the type to
`SHT_PROGBITS`. When a symbol is partially initialized, it will be put
into a progbits section.
I suppose GCC _could_ try and arrange for an object that is partially
initialized to split across two sections, one containing the
initialization prefix, followed immediately followed by a nobits
section (the section headers have address and offset fields) covering
the uninitialized part, but I don't know that a linker would honor
that; I imagine the linker expects to fill those fields of the section
header in with the values it computes for where the sections are laid
out, and can presumably reorder them any number of ways when the
executable object is generated. In general, I suspect that the once
GCC sees any amount of initialization it punts on being clever about
the layout and just does the naive thing you observed.
- Dan C.
> On Mon, Oct 26, 2026 at 11:53 AM Dan Cross <crossd at gmail.com> wrote:
> >
> > On Sun, Oct 25, 2026 at 10:55 PM Warner Losh <imp at bsdimp.com> wrote:
> > > On Sun, Oct 25, 2026, 8:44 PM Dan Cross via COFF <coff at tuhs.org> wrote:
> > >> > [snip]
> > >> > define N 1<<15
> > >> > static struct { bool h:1; bool v:1; } hv[N][N] = {{1,1}};
> > >> >
> > >> > Apparently there is no way to tell ELF to place a given data section
> > >> > next to an uninitialized section, so gcc had to instantiate the whole
> > >> > 1GB array in the object file, when I had imagined it would take at
> > >> > most 8 bytes plus a few words of metadata.
> > >>
> > >> Curiously, ELF has the expressive power to produce a loadable
> > >> executable object file that could operate in the way you expected at
> > >> load time, but I think you're right that it probably can't split a
> > >> single variable across multiple sections at build time.
> > >>
> > >> For those who are unfamiliar, ELF divides the world into sections and
> > >> segments; to a first order approximation, sections are used at build
> > >> time, consumed by linkers, debuggers, disassemblers, and the like.
> > >> Segments are used at load time by the OS or program loader. Segments
> > >> are far simpler than sections, and are defined by a simple data
> > >> structure called a, "program header." Among the data a program header
> > >> includes are load addresses, file offset, type, flags, permissions,
> > >> and the segment's size. The last is interesting, because there are
> > >> _two_ sizes: the "file size" (p_filesz) is the size of the segment's
> > >> contents in the executable file, while the "memory size" (p_memsz) is
> > >> the amount of memory the segment uses. If the program header has
> > >> `p_memsz > p_filesz` for a loadable ELF segment, then the segment is
> > >> understood to consist of both initialized data and uninitiated data;
> > >> this is how ELF defines a .BSS segment; the delta is filed with zeros
> > >> at load (or CoW's a zero page or whatever).
> > >>
> > >> Thus, one could imagine Doug's large array appearing in a dedicated
> > >> loadable segment, with filesz == 8, and memsz in the giga-range
> > >> (though I suspect that `p_filesz` might be rounded up to a page size,
> > >> regardless). But an ELF linker is allowed to rearrange and combine
> > >> segments; for example, grouping all of the executable text into one
> > >> segment, while moving all of the BSS spans into a dedicated `.bss`
> > >> segment with filesz==0; GCC can't know that this won't happen at
> > >> compile time, so if it sees a single data filled into a large,
> > >> otherwise uninitialized, array, then it hedges its bets and puts it
> > >> all into the same segment in its object file. And of course, there
> > >> could be multiple such values, straddling the boundary between init
> > >> and uninit data; but any given segment can only have, since it only
> > >> has one boundary for it to straddle; without explicit annotations, the
> > >> compiler can't know what to do with any one of them.
> > >>
> > >> However, one _could_ rewrite the ELF binary after linking to juggle
> > >> segments around to accommodate this, but I doubt that's common, and it
> > >> wouldn't help compilation.
> > >
> > > Two comments. Setting the array = {0} makes it initialized so no split between sections.
> >
> > No, initialization to `{0}` specifically is a special case; that
> > remains in the BSS:
> >
> > ```
> > term% cat big.c
> > int big[1<<21] = {0};
> > term% cc -c big.c
> > term% objdump -x big.o
> >
> > big.o: file format elf64-riscv
> > big.o
> > architecture: riscv:rv64, flags 0x00000010:
> > HAS_SYMS
> > start address 0x0000000000000000
> >
> > Sections:
> > Idx Name Size VMA LMA File off Algn
> > 0 .text 00000000 0000000000000000 0000000000000000 00000040 2**1
> > CONTENTS, ALLOC, LOAD, READONLY, CODE
> > 1 .bss 00800000 0000000000000000 0000000000000000 00000040 2**2
> > ALLOC
> > 2 .note.GNU-stack 00000000 0000000000000000 0000000000000000 00000040 2**0
> > CONTENTS, READONLY
> > 3 .riscv.attributes 00000068 0000000000000000 0000000000000000
> > 00000040 2**0
> > CONTENTS, READONLY
> > SYMBOL TABLE:
> > 0000000000000000 l df *ABS* 0000000000000000 big.c
> > 0000000000000000 l .bss 0000000000000000 $d
> > 0000000000000000 l .riscv.attributes 0000000000000000 $d
> > 0000000000000000 g O .bss 0000000000800000 big
> > ```
> >
> > The problem here is that Doug initialized the first value to non-zero
> > which forces it into .data (at least as far as GCC is concerned).
> >
> > ```
> > term% cat big.c
> > struct {_Bool h:1; _Bool v:1;} big[1<<21] = {{1,1}};
> > term% cc -c big.c
> > term% objdump -x big.o
> >
> > big.o: file format elf64-riscv
> > big.o
> > architecture: riscv:rv64, flags 0x00000010:
> > HAS_SYMS
> > start address 0x0000000000000000
> >
> > Sections:
> > Idx Name Size VMA LMA File off Algn
> > 0 .text 00000000 0000000000000000 0000000000000000 00000040 2**1
> > CONTENTS, ALLOC, LOAD, READONLY, CODE
> > 1 .data 00200000 0000000000000000 0000000000000000 00000040 2**0
> > CONTENTS, ALLOC, LOAD, DATA
> > 2 .note.GNU-stack 00000000 0000000000000000 0000000000000000 00200040 2**0
> > CONTENTS, READONLY
> > 3 .riscv.attributes 00000068 0000000000000000 0000000000000000
> > 00200040 2**0
> > CONTENTS, READONLY
> > SYMBOL TABLE:
> > 0000000000000000 l df *ABS* 0000000000000000 big.c
> > 0000000000000000 l .data 0000000000000000 $d
> > 0000000000000000 l .riscv.attributes 0000000000000000 $d
> > 0000000000000000 g O .data 0000000000200000 big
> > ```
> >
> > We can futz with it, though:
> >
> > ```
> > term% cc -fdata-sections -c big.c
> > term% cat big.ld
> > SECTIONS {
> > .data.big ALIGN(4096) : { KEEP(big.o(.data.big)) }
> > }
> > INSERT AFTER .bss;
> > term% cat bigmain.c
> > #include <stdio.h>
> > extern unsigned char big[1<<21]; // On this ABI, Doug's struct is a uchar.
> > int
> > main(void)
> > {
> > printf("big[0] = %d\n", big[0]);
> > printf("big[1] = %d\n", big[1]);
> > printf("big[last] = %d\n", big[(1<<21) - 1]);
> > return 0;
> > }
> > term% cc -v -o big bigmain.o big.o 2>&1 | grep ld
> > "/usr/bin/ld" -e __start --eh-frame-hdr -dynamic-linker
> > /usr/libexec/ld.so -X -o big /usr/lib/crt0.o /usr/lib/crtbegin.o
> > -L/usr/lib bigmain.o big.o -lcompiler_rt -lc -lcompiler_rt
> > /usr/lib/crtend.o
> > term% ld -Tbig.ld -e __start --eh-frame-hdr -dynamic-linker
> > /usr/libexec/ld.so -X -o big /usr/lib/crt0.o /usr/lib/crtbegin.o -L
> > /usr/lib bigmain.o big.o -lcompiler_rt -lc -lcompiler_rt
> > /usr/lib/crtend.o
> > term% ./big
> > big[0] = 3
> > big[1] = 0
> > big[last] = 0
> > term%
> > ```
> > Running `objdump -x` on that executable, I see that the `big` array
> > has indeed been isolated to its own segment:
> > ```
> > PHDR off 0x0000000000000040 vaddr 0x0000000000000040 paddr
> > 0x0000000000000040 align 2**3
> > filesz 0x0000000000000310 memsz 0x0000000000000310 flags r--
> > ...
> > LOAD off 0x0000000000001000 vaddr 0x0000000000004000 paddr
> > 0x0000000000004000 align 2**12
> > filesz 0x0000000000200000 memsz 0x0000000000200000 flags rw-
> > ```
> > Note that `ojbdump` also tells me where the program headers are and
> > how much space they take; they take 784 bytes, starting at 64 bytes
> > into the file. This is a 64-bit binary, so each entry is 56-bytes
> > long and the file size offset is 32 bytes from the start of the
> > struct. With some targeted surgery, I can change that to, say 4KiB:
> >
> > ```
> > LOAD off 0x0000000000001000 vaddr 0x0000000000004000 paddr
> > 0x0000000000004000 align 2**12
> > filesz 0x0000000000001000 memsz 0x0000000000200000 flags rw-
> > ```
> > It is all zeros from there to the end of the file (I'm ignoring the
> > RISC-V attributes), so I can truncate it, too:
> > ```
> > term% ls -l big
> > -rwxr-xr-x 1 cross staff 2104960 Oct 26 15:39 big
> > term% dd if=big of=big.smaller bs=4096 count=2
> > 2+0 records in
> > 2+0 records out
> > 8192 bytes transferred in 0.000 secs (12804761 bytes/sec)
> > term% ls -l big.smaller
> > -rwxr-xr-x 1 cross cross 8192 Oct 26 15:47 big.smaller
> > term% chmod +x big.smaller
> > term% ./big.smaller
> > big[0] = 3
> > big[1] = 0
> > big[last] = 0
> > term%
> > ```
> > Anyway, I thought that was kind of a fun exercise.
> >
> > > And there are also flags that control the alignment of the sections (or you can use a custom linker script).
> >
> > Sure. Though I would expect most loadable sections to be page-aligned
> > in the file so that they can be mapped directly into a process's
> > virtual address space from the page cache. For very small segments,
> > they may just be copied; I didn't look.
> >
> > - Dan C.
More information about the COFF
mailing list