Asuro/doc/planning/x86_64.md

# x86-64 migration

## Status

The x86-64 target boots into Pascal with exception handling, physical/virtual
allocation, the shared heap, PIC/PIT interrupts and tested kernel-context
preemption. Fatal faults report over COM1.
This is a bootstrap kernel, not the complete Asuro desktop or storage stack.
The existing x86 target remains the default.

## Architecture boundaries and reuse

Follow the namespace convention in [namespace.md](namespace.md): architecture
units live under `src/arch/<architecture>/` and use `arch.<architecture>.*` names.
The compiler-required `system.pas` retains its special name.

- `src/arch/x86_64/boot/stub.asm`: 32-bit Multiboot entry and long-mode transition.
- `src/arch/x86_64/arch.x86_64.boot.pas`: bootstrap entry and serial self-checks.
- `src/arch/x86_64/system.pas`: minimal compiler runtime definitions.
- `src/arch/x86_64/arch.x86_64.isr.pas`: IDT/TSS setup, frame validation, and
  allocation-free exception reporting.
- `src/arch/x86_64/isr/entry.asm`: generated entry stubs, register/FX save and
  restore, `iretq`, emergency stacks, and controlled fault probes.
- `src/debug/debug.early.pas`: reusable architecture-independent diagnostic
  formatting, supplied with a character sink by `arch.x86_64.debug`.
- `src/arch/common/system.types.inc`: shared scalar/compiler definitions extracted
  from the existing system unit; native widths follow the selected CPU target.
- `toolchain/target.sh`: target selection shared by the existing build steps.
- `toolchain/x86_64/linker.script`: bootstrap address layout.

Do not copy the i386 kernel, drivers, heap, scheduler policy, or runtime wholesale.
As each subsystem is enabled, retain portable logic in its existing namespace
and put architecture-specific operations behind the appropriate architecture
interface. External `lvgl/` and `wasuro/` sources must remain unchanged.

## Build and boot

Rebuild the builder image when first adding the native x86-64 FPC compiler:

```powershell
docker-compose down --remove-orphans
docker-compose build builder
```

Build x86-64 using the normal build command with an explicit target:

```powershell
$env:ASURO_ARCH = 'x86_64'
docker-compose run builder | Tee-Object -FilePath .\build.log
Remove-Item Env:ASURO_ARCH
Select-String -Path .\build.log -Pattern "Error|Fatal|Success|No errors|Failed" -CaseSensitive
```

Outputs are `Asuro-x86_64.iso`, `bin/x86_64/kernel.bin`, and `lib/x86_64/`.
Without `ASURO_ARCH`, the existing x86 build still produces `Asuro.iso`,
`bin/kernel.bin`, and `lib/`. Cleaning and linking only consider the selected
target's objects. The x86 compiler search path excludes `src/arch/x86_64`.

Use VirtualBox for runtime validation. With the configured Asuro VM stopped,
select `Asuro-x86_64.iso` as its optical medium, then start it through the normal
VM workflow and read `C:\Temp\asuro.log`. Restore the original optical medium
after testing. The bootstrap does not access storage controllers or disks.

Expected serial output:

```text
ASURO x86_64: IDT/TSS ready; DF/NMI/MC emergency stacks installed
ASURO x86_64: entered Pascal
PASS: long mode, Multiboot handoff, 64-bit Pascal types
PASS: INT3 frame, GPRs, RFLAGS, x87/SSE and IRETQ
PASS: page-fault error code, CR2 and IRETQ recovery
EXCEPTION_TESTS_PASSED=0000000000000002
EXCEPTION_TESTS_FAILED=0000000000000000
ASURO x86_64: bootstrap complete; interrupts disabled
```

An early single `C` means CPUID/long-mode support is unavailable; `B` means an
invalid Multiboot magic value. The Pascal checks validate the handoff magic,
a non-null info pointer, native/fixed type widths, wide arithmetic, and EFER.LMA.

### Repeatable VirtualBox fault checks

The default build runs two returning self-tests. `INT3` verifies the saved frame,
all 15 general registers including high halves, CF/DF, x87 and XMM0/XMM15 state,
and `iretq`. The handler intentionally destroys FX state before the return path
restores it. A write to unmapped `0x100000000` verifies hardware error code `2`,
CR2, and recovery by changing RIP only for the exact expected probe.

Use the bounded test harness with the configured VM powered off:

```powershell
.\toolchain\x86_64\test_boot.ps1 -Test none
```

For deliberately fatal tests, select `ud`, `pf`, or `df` when building, then use
the matching harness option. For example:

```powershell
$env:ASURO_ARCH = 'x86_64'
$env:ASURO_BOOT_TEST = 'df'
docker-compose run builder | Tee-Object -FilePath .\build.log
Remove-Item Env:ASURO_BOOT_TEST
Remove-Item Env:ASURO_ARCH
.\toolchain\x86_64\test_boot.ps1 -Test df
```

The harness uses `localenv.json`, temporarily attaches the ISO, starts the VM
through `virtualbox-wrapper.ps1 -command up -Headless -NoMonitor`, validates
serial output, captures registers, stops the VM, and restores the original
optical medium in a `finally` block. The new wrapper switches are optional;
the existing interactive workflow is unchanged. Storage disks are not changed.

Rebuild without `ASURO_BOOT_TEST` after testing to restore the normal image.

## Bootstrap layout and limitations

- GRUB loads the kernel at physical/virtual 1 MiB using explicit Multiboot v1
  load-address fields; the payload is ELF64 with a 32-bit assembly entry.
- Six reserved 4 KiB pages hold the PML4, PDPT, and four page directories.
  They identity-map the first 4 GiB with 2 MiB leaves.
- A reserved 16 KiB stack is aligned for the SysV AMD64 calling convention.
  These bounded early allocations exist before any heap and must remain
  reserved when the physical allocator is introduced.
- Paging, PAE, long mode, x87, and SSE2 support are enabled before calling Pascal.
- A 256-entry IDT uses normalized 64-bit frames. TSS IST1 supplies a 16 KiB
  double-fault stack; IST2 and IST3 supply separate 8 KiB NMI and machine-check
  stacks. The 4 KiB IDT, 104-byte TSS, and emergency stacks are bounded early
  reservations before the allocator exists. Stack guard pages remain future work.
- Entry preserves GPRs and captures CR2 before Pascal code. It clears DF, aligns
  the call stack, and saves/restores x87/SSE state using FXSAVE64/FXRSTOR64.
  AVX/OSXSAVE and lazy-FPU switching are not enabled. Future changes must extend
  this state policy before enabling those features.
- IRQ0 is enabled during timer and context tests, then masked before the final
  halt. PIC acknowledgement and spurious IRQ7/15 handling are implemented.
  General driver hooks, APIC/SMP, Pascal unit-init
  dispatch remain future work.
- The low identity map is temporary. It is not a final process address-space
  design and does not enforce page-level W^X. New heap mappings use 4 KiB pages
  at `0xFFFF800000000000`, spanning up to 16 GiB with NX when supported.
  Their backing frames can reside above 4 GiB.
- Unsupported runtime operations intentionally fail to link rather than silently
  delegating to a hosted runtime or pretending to succeed.

## Remaining migration work

| Area | Existing assumptions / next work |
|---|---|
| Runtime | Keep compiler metadata shared; implement required x86-64 helpers and initialization dispatch as consumers are ported. |
| Common kernel entry | `src/asuro.pas` currently imports x86 units; separate common boot sequencing from architecture setup before sharing full initialization. |
| Exceptions | IDT/TSS, IST stacks, normalized frames, diagnostics, and tested `iretq` paths are implemented. NMI/MC injection, user transitions, guard pages, and general IRQ dispatch remain. |
| ABI | Bootstrap interrupt alignment and x87/SSE preservation are tested. Audit red-zone assumptions for each added compiler/library and extend preservation before enabling AVX or lazy-FPU switching. |
| Memory | PMM, VMM and shared heap implemented. Final process layout, guards, MMIO and DMA APIs remain. |
| Scheduler/loader | Shared scheduling, kernel preemption and process lifecycle tested. Process address spaces and executable loading remain. |
| Drivers/storage | Audit `uint32(pointer)` conversions, MMIO and DMA translations, and hardware address limits. Keep disk/USB/controller structures fixed-width. |
| Graphics | VESA calls into V86, which cannot operate in long mode. Initially use a bootloader framebuffer; rebuild LVGL and verify Pascal/C bindings later. |
| Validation | Add deliberate exception tests, allocator checks, preemption checks, and storage tests as their implementations become available. |

Next milestone: port filesystem/console services and complete driver address/DMA
audits. Kernel process lifecycle now uses the shared manager. Do not enable the
desktop/storage stack before its dependencies are ported and tested.

## Validation performed — 2026-11-07

- x86-64 compile, link, GRUB header validation, and ISO generation passed.
- The existing Asuro VirtualBox VM printed all three expected serial lines.
- First-milestone VirtualBox registers showed `CS=0x18`, `CR0=0x80000013`,
  `CR3=0x103000`, `CR4=0x620`, and `EFER=0x500` (LME and LMA set).
- Disassembly confirmed 64-bit Pascal code, wide arithmetic, and aligned calls
  in the bootstrap. This does not establish interrupt safety for future code.
- The full default x86 build passed, including compilation, linking, ISO, and
  documentation generation, with the x86-64 output directory still present.
- The VM was stopped and its original optical medium restored after testing.
- VirtualBox passed the normal boot tests and deliberate invalid-opcode,
  page-fault, and double-fault builds. The final write-fault test preserved
  error code `2` and CR2 `0x100000000`.
- The exception milestone's normal boot captured `TR=0x20`, confirming the
  installed TSS selector, with long mode still active (`EFER=0x500`).
- Double-fault injection set RSP to unmapped `0x100001000`. Diagnostics reported
  vector `8`, error `0`, the original RSP, and a frame inside the IST1 stack.
- Fatal exception reports include all 15 GPRs, RIP/CS/RFLAGS/RSP/SS, error code,
  CR2, and frame address. CR2 is the last page-fault address and may be stale
  for other exception types.

Runtime evidence is saved locally in the ignored `bin/x86_64/boot.log` and
`bin/x86_64/registers.log` files for the first milestone, and `boot-<test>.log` /
`registers-<test>.log` for the exception milestone. Unsupported-CPU boot checks and NMI/MC injection have not yet been performed.

The interrupt-frame and IST layout follows the
[AMD64 System Programming manual, chapter 8](https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/24593.pdf).

## Memory, timer and shared scheduling progress

The PMM uses 4 KiB frames and dynamically placed bitmaps, with a 64 GiB management
cap. Boot metadata, modules, kernel ranges, low bootstrap memory and allocator
metadata stay reserved. VMM allocates high virtual arenas and rolls back both
page tables and backing frames on failure. Page tables remain below 4 GiB;
backing frames may reside above it.

`memory.heap` is shared between targets, with native-width pointer/size APIs,
16-byte alignment, overflow rejection and large-allocation rollback.
`memory.bitmap` supplies the common bitmap operations. `boot.multiboot` owns
fixed-width protocol records, with aliases retained in the x86 boot unit.
`arch.common.legacyirq` shares PIC programming and PIT constants.

`io.types` separates data contracts from console initialization. `proc.types`
uses native saved-stack addresses. The existing round-robin/quantum policy now
lives in `proc.scheduler`, called by both the i386 process manager and x86-64
preemption tests. Process creation, messaging, return/exit, resource cleanup and reaping now use
`proc.mgr` on x86-64. Descriptor tables are now active; console commands, executable loading and
full filesystem/controller integration remain unavailable in the bootstrap.

VirtualBox validation has passed at 64 MiB, 2 GiB and 6 GiB. The 6 GiB run mapped,
wrote, read and released RAM above 4 GiB. Tests cover PMM accounting and invalid
frees, VMM rollback under injected allocation failures, heap alignment/zeroing/
large allocations, timer delivery, spurious IRQ7/15, IF preservation and CLI.
Two heap-backed kernel contexts preserve distinct 64-bit registers, x87 and XMM
state across timer preemption, then release their stacks after returning to the
bootstrap context. Only the BSP is active; IRQ exclusion is not an SMP lock.

```powershell
.\toolchain\x86_64\test_boot.ps1 -Test none -MemoryMB 64
.\toolchain\x86_64\test_boot.ps1 -Test none -MemoryMB 6144
```

The harness restores original VM RAM and optical media. Logs use
`bin/x86_64/boot-<test>-<MemoryMB>.log` and matching register filenames; zero means
use the configured RAM. All subsystem `*_TESTS_FAILED` fields must be zero.
PIC handling follows the [Intel 8259A datasheet](https://www.cs.umb.edu/cs341/Intel8259/I8259APIC.pdf).

The final i386 regression build passed compilation, linking, ISO and documentation
steps. VirtualBox reached `Boot complete - halting into IDLE`; reported boot test
suites had zero failures. Evidence: `bin/x86_64/i386-regression.log`.
Fatal UD, write-PF and broken-stack DF tests also passed after the context-return
path was extended. Final normal images are built without fault injection.

## Shared process lifecycle

`proc.mgr` now runs on both targets with native process-pointer slots, stable
monotonic IDs, suspended creation, messaging, exit and resource ownership.
Initial stack layout lives in the architecture context units. x86-64's scheduler
adapter saves the current stack and calls the same scheduler/manager as i386.
A shared reaper process handles destruction on both targets outside IRQ context.
Timer selection and kill requests coalesce worker wakeups; cleanup runs after
registry/resource detachment. The reaper sleeps while there is no pending work.

`core.ds.dynamic` contains the common dynamic-list implementation; existing
`core.ds.lists` exports compatible aliases/wrappers. Native addresses, checked
capacity arithmetic, overlap-safe insertion and empty-list regrowth replace
32-bit arithmetic. Deletion retains capacity; explicit `DL_ShrinkToFit` frees
excess storage without allocating inside every scheduler deletion.
`memory.raw` supplies allocation-free scalar fill/move operations.

Process tests cover suspended publication, high-address lookup, killed tasks
never running, normal return, explicit exit, child notification waking an
awaiting parent, callback reentry without double cleanup, command argument/
stream publication, sleep using the PIT divisor, and restored heap accounting.
The full i386 build and VirtualBox boot passed after these shared changes;
evidence is in `bin/x86_64/i386-lifecycle-regression.log`.

The dedicated reaper revision passed all 15 process/container checks at 64 MiB
and 6 GiB, including the corrected check-before-await notification protocol.
The final i386 build and VirtualBox boot also passed with the shared reaper.
VM RAM and original optical media were restored and the VM was powered off.

## Descriptor tables

The shared `driver.storage.fdtable` now uses native memory operations and balanced
IRQ locking. Every x86-64 process, including system tasks, owns its normal FD
table. Storage protocol types depend on shared record definitions rather than
bringing in the linked-list implementation and its console dependencies.

Close reserves an entry until callbacks and owned-string cleanup finish, rejects
active async references and reference-count overflow, and prevents callback
reentry from freeing it twice. Table teardown atomically rejects new users before
process removal. An exited process with active FD references remains alive; final
completion only wakes the shared reaper, which performs cleanup in process
context. Waiting does not repeatedly scan descriptors on every timer tick.

VirtualBox passes 15 descriptor checks plus 16 lifecycle/container checks,
including an exited child held alive by an async FD reference until another task
releases it. No disk I/O is performed by these bootstrap tests. This validates
FD ownership, not full VFS operation or controller DMA support in long mode.

The descriptor-table revision also passes the complete i386 build and VirtualBox
boot, including the existing storage/VFS suites and normal volume discovery.
Evidence: `bin/x86_64/i386-fd-regression.log`. Original VM settings are restored.

## Shared strings and console command infrastructure

`core.util` memory helpers accept native addresses and sizes, reusing scalar
`memory.raw` operations while retaining i386's existing large-copy REP path.
`core.strings` no longer imports the x86 utility or linked-list implementation;
all string copy addresses are native. Its common tests cover signed 64-bit
minimum formatting, overflowing substring/match bounds, and terminator size
rejection. The x86-64 system unit supplies Double-to-int64 truncation using
CVTTSD2SI without changing rounding control state.

`io.stdio` now builds on both targets: growable buffers, bounded command parsing,
case-insensitive allocation-free lookup, registry growth, ANSI output, and
ECHO/CLEAR/HELP share one implementation. Buffer appends reserve space in bulk
and support self-append across relocation. Allocation failures retain the old
buffer; parser failures release all partially built parameters. Completion
detaches the halt owner before invoking callbacks, allowing safe reentry.

VFS working-directory delegation and VERSION/TIME/REBOOT registration remain
gated until their platform dependencies are integrated. This is command
infrastructure, not yet an interactive keyboard/display/serial shell. No VFS or
storage-controller support is implied by compiling the console unit.

The 64-bit bootstrap runs 162 shared string tests and 17 stdio tests. The process
suite also launches the registered ECHO through `runCommand` with real parsed
arguments and a growable output buffer, bringing it to 17 lifecycle/container
checks. The 6 GiB VirtualBox run verifies scheduled ECHO and physical RAM above
4 GiB. The complete i386 build and boot also pass, including 162 string tests,
stdio tests, normal mounts, and the existing storage/VFS suites. Evidence:
`bin/x86_64/i386-console-regression.log` and the normal x86-64 boot logs.

## Build-selected platform adapters

Shared Pascal units no longer select CPUs with `CPU64`/`CPU386` conditionals.
`ASURO_ARCH` selects a single `src/arch/<target>/platform` directory in the
compiler search path. Both targets implement the same `arch.platform.cpu`,
`arch.platform.memory`, `arch.platform.process`, and
`arch.platform.diagnostics` unit names. Their declarations are shared through
`src/arch/common/platform.*.inc`; implementations delegate to the existing
architecture units. Calls are resolved at compile/link time, with no runtime
architecture selection or function-pointer dispatch.

The bitmap heap, allocation headers, process registry, scheduling policy,
strings, console buffers/parser, and descriptor lifecycle remain single shared
implementations. Native scalar widths live in each selected system unit.
`memory.raw` builds its fill pattern from native word width without a CPU branch.

`arch.x86.startup` now registers heap/process/console startup with the boot
manager. `arch.x86.console` owns VERSION/TIME/REBOOT commands and their driver
dependencies. The x86-64 bootstrap retains its explicit startup sequence.
Unused working-directory wrappers moved from `io.stdio` into `io.directory`,
which explicitly depends on VFS. Existing in-repository callers need no changes.

Existing `DEBUG_LMM` allocator guards and architecture-local fault-injection
build flags remain; these are feature/test options, not architecture selection.

Validation: complete i386 build and VirtualBox boot, including 17 stdio checks,
162 string checks, storage/VFS suites and normal mounts. Evidence:
`bin/x86_64/i386-platform-regression.log`. x86-64 boot validation covers heap,
IRQs, contexts, descriptors, strings, console and scheduled command execution.

The final adapter build also passes the 6 GiB VirtualBox bootstrap, including
physical memory above 4 GiB and all 162 string, 17 console, 15 descriptor,
and 17 lifecycle/container checks. VM RAM and optical media were restored;
the VM is powered off.

Source search paths are discovered recursively from shared roots and the selected
architecture tree, with other architectures excluded. Both targets use the same
compiler invocation; `target.sh` loads compiler options and entry points from
`toolchain/targets/<architecture>.sh`.
Directory arguments are NUL-delimited, sorted, and passed as an array so new
subdirectories and spaces in names do not require compile-script edits. Full
i386 and x86-64 builds pass using this discovery path.

All shared build steps now consume the selected profile: pipeline stages,
assembler format and source roots, C/Pascal compiler options, linker format,
entry points, output locations, optional compiler runtime, and image validation.
`target.sh` loads the profile by filename instead of keeping an architecture
switch. Assembly files, object files, and static libraries are discovered;
the configured assembly entry is linked first to preserve the Multiboot header.
Initialization-table, icon, LVGL, version, and checksum generation use target
settings too. Full i386 and x86-64 builds and VirtualBox boot checks validate
this pipeline; the i386 boot includes normal mounts and existing storage tests.