Compare commits

..
Author SHA1 Message Date
Aaron e69b134bff Integrate x86_64 desktop boot and extend VirtualBox validation
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2026-11-08 09:02:34 +00:00
Aaron a6a55153af Enable native WASM execution with process-owned contexts and cleanup 2026-11-08 09:02:34 +00:00
Aaron e462165049 Enable native desktop applications and synchronized resolution changes 2026-11-08 09:02:34 +00:00
Aaron b71e6d67b4 Port E1000 networking and bound packet and DMA handling 2026-11-08 09:02:34 +00:00
Aaron 465cf1deca Enable native USB core, OHCI, mass storage and HID drivers 2026-11-08 09:02:33 +00:00
Aaron f8911ccdec Port storage and filesystems to x86_64 with stable device ownership 2026-11-08 09:02:33 +00:00
Aaron a27d9cb39d Port shared core and platform services to native pointer widths 2026-11-08 09:02:33 +00:00
Aaron 7a6be7f66e Refactor builds to use target profiles and discovery; remove architecture checker 2026-11-07 20:50:04 +00:00
Aaron 6a1eb5e2ac Add x86_64 kernel bring-up with build-selected platform adapters 2026-11-07 20:13:20 +00:00
Aaron e8e4061564 fix file header
continuous-integration/drone Build is failing
2026-11-07 10:19:01 +00:00
Aaron 10691a9e03 fix file headers 2026-11-07 10:18:33 +00:00
Aaron cebad0b583 Fix USB volume discovery, automounting, and file saves 2026-11-01 20:22:07 +00:00
Aaron 53cc758e60 Add standard license headers and Aaron Hance attribution 2026-10-31 20:56:15 +00:00
Aaron 6c7619bc2c Keep development plan local and remove it from version control 2026-10-31 20:50:31 +00:00
Aaron d318eca262 Refactor VFS shared operations and remove unused legacy storage wrappers 2026-10-31 20:46:48 +00:00
Aaron c28b9dd77f Implement USB mass storage and batch OHCI bulk transfers 2026-10-31 20:36:52 +00:00
Aaron c0c56a9c84 Add initial ext2 read/write and formatting support
Register ext2 at boot, integrate mode-aware VFS opens and volume cleanup, and queue asynchronous transfers through a persistent worker.

Validation: Pascal compiler check and git diff --check passed. Full build blocked by Docker engine HTTP 500; runtime and Linux interoperability remain unverified.
2026-10-31 19:49:36 +00:00
Aaron 720ba46630 Improve disk benchmark suites and timing, and fix command launch race 2026-10-31 19:06:46 +00:00
Aaron 8e0ab3da8b Improve storage performance with read pipelining, run lookup, request merging, and counters 2026-10-31 17:32:59 +00:00
Aaron 3d4f846a8f fix(ahci): harden command submission, DMA ownership, and timeout recovery 2026-10-31 16:56:18 +00:00
Aaron d8d51ba5df feat(storage): use volume labels for default mount folders 2026-10-31 16:39:43 +00:00
Aaron d1ecd3fc41 fix(storage): harden disk operations, add volume labels, and stabilize AHCI discovery 2026-10-31 16:30:40 +00:00
Aaron 7e99bc2db1 fat
continuous-integration/drone/push Build is passing
2026-03-24 18:10:04 +00:00
Aaron 096db148d0 fat32: fix pre-alloc cluster leak; add write-handle ref-count
- Remove FAT_WRITE_PREALLOC_BYTES (redundant at typical cluster sizes)
- Simplify fatPreallocClusterCount to always return MAX_CLUSTERS
- FAT32CloseFile: truncate pre-allocated tail clusters back to ByteSize
  on close, freeing FAT chain entries beyond the file's real size
- Add TFATWriteRef + WriteRefs[32] in TFATVolumeCtx and Registered flag
  on TFATOpenFile to track concurrent write handles per dir entry
- fatRegisterWriteHandle / fatUnregisterWriteHandle: CLI-guarded slot scan
  keyed by (DirLoc.SectorLBA, DirLoc.EntryIdx)
- Register on first FAT chain extension in FAT32EnsureCapacityForWrite
- Truncation in FAT32CloseFile skipped unless this is the last writer
  (prevents RunMap corruption in concurrent-open scenarios)
- Add forward decl for fatFreeClusterChain to fix ordering issue
- app.iotest: update default path to /disk/vol2/IOTEST.BIN
2026-03-18 21:05:36 +00:00
Aaron 663bfbb0e4 storage: lifecycle watch system, PFSFormatParams, vol busy/format flags, diskutil improvements
- Add TStorageLifecycleWatch with watch_lifecycle/unwatch_lifecycle API (up to 32 watchers)
- Add notify_lifecycle to broadcast mount/unmount/format events to registered callbacks
- Replace raw puint32 config in format_volume with typed PFSFormatParams
- Add volume_is_busy, volume_is_formatting, volume_set_mounted helpers
- Add lifecycleFlags field to TStorage_Volume for tracking busy/formatting state
- Expand app.diskutil with improved commands and volume state reporting
- Minor improvements to core.util, arch.x86.util, proc.mgr, VFS, and flatfs
- Remove obsolete driver.storage.fs.fat32.old.pas
2026-03-16 23:03:43 +00:00
Aaron aadc59f831 fat32 rewrite and storage stability fixes 2026-03-16 21:18:44 +00:00
Aaron 5d7aacc28f more improvements 2026-03-15 23:58:52 +00:00
Aaron 4810617922 FAT improvements and switch to 2-way associative cache 2026-03-15 19:46:31 +00:00
Aaron 666a011a9f refactor fat32 2026-03-15 15:11:00 +00:00
Aaron a6e81bf5d3 VFS improvements, IO fixes, IO performance improvements. 2026-03-15 14:02:17 +00:00
200 changed files with 21488 additions and 8805 deletions

No files matched your search

-1
View File
@@ -17,7 +17,6 @@ steps:
- name: compile - name: compile
image: t3hn3rd/asuro-build:latest image: t3hn3rd/asuro-build:latest
pull: always
depends_on: depends_on:
- build-image - build-image
commands: commands:
+3 -4
View File
@@ -1,6 +1,8 @@
/lib/*.ppu /lib/*.ppu
/lib/*.a /lib/*.a
/lib/*.o /lib/*.o
/lib/x86_64/
/bin/x86_64/
/release/*.svg /release/*.svg
/release/*.iso /release/*.iso
/bin/*.bin /bin/*.bin
@@ -15,12 +17,9 @@ localenv.json
dockerout.txt dockerout.txt
AGENTS.md AGENTS.md
lessons_learnt.md lessons_learnt.md
/plan.md
*.log *.log
wasuro wasuro
src/core/core.version.pas src/core/core.version.pas
site/* site/*
!site/.gitkeep !site/.gitkeep
/src/boot/dangling_units.inc
/lib/dangling/
.moderne
CLAUDE.md
+4 -4
View File
@@ -1,5 +1,4 @@
ARG RECIPES_IMAGE=t3hn3rd/moderne-asuro-recipes:0.1.1@sha256:eaf54188053511dad9c620a8fb98b0f50c41770cd49a0b3d71a0538386522011 FROM ubuntu:latest
FROM ${RECIPES_IMAGE}
VOLUME ["/code"] VOLUME ["/code"]
@@ -7,8 +6,8 @@ ENV DEBIAN_FRONTEND=noninteractive
RUN dpkg --add-architecture i386 RUN dpkg --add-architecture i386
RUN apt-get update && apt-get install -y \ RUN apt-get update && apt-get install -y \
curl dos2unix wget git make nasm binutils xorriso grub-pc-bin gcc gcc-multilib \ curl dos2unix wget git make nasm binutils xorriso grub-pc-bin gcc gcc-multilib \
python3 python3-pip python3-pil && \ python3 python3-pip python3-pil fp-compiler && \
apt-get clean && rm -rf /var/lib/apt/lists/* apt-get clean my room
RUN pip3 install --no-cache-dir --break-system-packages "mkdocs>=1.6,<2" mkdocs-material RUN pip3 install --no-cache-dir --break-system-packages "mkdocs>=1.6,<2" mkdocs-material
@@ -21,5 +20,6 @@ RUN curl -sL https://sourceforge.net/projects/freepascal/files/Linux/$FPC_VERSIO
ADD https://raw.githubusercontent.com/fsaintjacques/semver-tool/master/src/semver /usr/bin/semver ADD https://raw.githubusercontent.com/fsaintjacques/semver-tool/master/src/semver /usr/bin/semver
RUN chmod +x /usr/bin/semver RUN chmod +x /usr/bin/semver
WORKDIR /code WORKDIR /code
# Sources are mounted at runtime; the CMD below normalizes build scripts.
ENTRYPOINT ["/bin/bash", "-c"] ENTRYPOINT ["/bin/bash", "-c"]
CMD ["find toolchain -name '*.sh' -exec dos2unix {} + 2>/dev/null; find toolchain -name '*.sh' -exec chmod +x {} +; bash toolchain/compile.sh"] CMD ["find toolchain -name '*.sh' -exec dos2unix {} + 2>/dev/null; find toolchain -name '*.sh' -exec chmod +x {} +; bash toolchain/compile.sh"]
-8
View File
@@ -1,8 +0,0 @@
**/.git
**/.gradle
**/build
**/.moderne
wasuro
site
lib
release
+10 -10
View File
@@ -22,12 +22,12 @@ type
THeapPageHeader = memory.heap.THeapPageHeader; THeapPageHeader = memory.heap.THeapPageHeader;
procedure init; procedure init;
function kalloc(size : uint32) : void; function kalloc(size : SizeUInt) : void;
function klalloc(size : uint32) : void; function klalloc(size : SizeUInt) : void;
procedure klfree(address : uint32); procedure klfree(address : PtrUInt);
function kpalloc(address : uint32) : void; function kpalloc(address : PtrUInt) : void;
procedure kfree(area : void); procedure kfree(area : void);
function lmm_total_free : uint32; function lmm_total_free : SizeUInt;
function lmm_page_count : uint32; function lmm_page_count : uint32;
implementation implementation
@@ -37,22 +37,22 @@ begin
memory.heap.init; memory.heap.init;
end; end;
function kalloc(size : uint32) : void; inline; function kalloc(size : SizeUInt) : void; inline;
begin begin
kalloc := memory.heap.kalloc(size); kalloc := memory.heap.kalloc(size);
end; end;
function klalloc(size : uint32) : void; inline; function klalloc(size : SizeUInt) : void; inline;
begin begin
klalloc := memory.heap.klalloc(size); klalloc := memory.heap.klalloc(size);
end; end;
procedure klfree(address : uint32); inline; procedure klfree(address : PtrUInt); inline;
begin begin
memory.heap.klfree(address); memory.heap.klfree(address);
end; end;
function kpalloc(address : uint32) : void; inline; function kpalloc(address : PtrUInt) : void; inline;
begin begin
kpalloc := memory.heap.kpalloc(address); kpalloc := memory.heap.kpalloc(address);
end; end;
@@ -62,7 +62,7 @@ begin
memory.heap.kfree(area); memory.heap.kfree(area);
end; end;
function lmm_total_free : uint32; inline; function lmm_total_free : SizeUInt; inline;
begin begin
lmm_total_free := memory.heap.lmm_total_free; lmm_total_free := memory.heap.lmm_total_free;
end; end;
+2 -2
View File
@@ -205,7 +205,7 @@ Key constants:
- `DATA_OFFSET = $10000` — data area starts at 64 KB into each page - `DATA_OFFSET = $10000` — data area starts at 64 KB into each page
- `LARGE_ALLOC_MAGIC = $FFFFFFFE` — sentinel for klalloc'd multi-page allocations - `LARGE_ALLOC_MAGIC = $FFFFFFFE` — sentinel for klalloc'd multi-page allocations
Interrupt safety: `pushfd; cli; pop saved_flags` (stack-balanced) + `restore_if(saved_flags)` Interrupt safety: `pushfd; cli; pop saved_flags` (stack-balanced) + `arch.platform.cpu.unlock(saved_flags)`
inline procedure that only restores IF via `sti`, avoiding dangerous EFLAGS restoration inline procedure that only restores IF via `sti`, avoiding dangerous EFLAGS restoration
(IOPL, NT, VM, AC bits that caused Bad-TSS on IRET). (IOPL, NT, VM, AC bits that caused Bad-TSS on IRET).
@@ -256,7 +256,7 @@ list, fixes Search_Page, calls `free_page_at_address`). Skips Root_Page.
| Bug | Root Cause | Fix | | Bug | Root Cause | Fix |
|-----|-----------|-----| |-----|-----------|-----|
| Bad TSS exception during network processing | `pushfd`/`popfd` across separate asm blocks: FPC doesn't track ESP changes in asm, causing EFLAGS corruption (NT=1, IOPL wrong) → Bad TSS on IRET | Stack-balanced `pushfd; cli; pop local_var` + `restore_if()` that only does `sti` if IF was set | | Bad TSS exception during network processing | `pushfd`/`popfd` across separate asm blocks: FPC doesn't track ESP changes in asm, causing EFLAGS corruption (NT=1, IOPL wrong) → Bad TSS on IRET | Stack-balanced `pushfd; cli; pop local_var` + `arch.platform.cpu.unlock()` that only does `sti` if IF was set |
| GPF from SSE MOVAPS in doublebuffer flush | `SIZE_PREFIX=4`→8 gave 8-byte alignment; `MOVAPS` needs 16-byte. `klalloc` returned `page+8` (not 16-aligned) | `SIZE_PREFIX=16` (4B size + 12B pad), klalloc header expanded to 16 bytes | | GPF from SSE MOVAPS in doublebuffer flush | `SIZE_PREFIX=4`→8 gave 8-byte alignment; `MOVAPS` needs 16-byte. `klalloc` returned `page+8` (not 16-aligned) | `SIZE_PREFIX=16` (4B size + 12B pad), klalloc header expanded to 16 bytes |
| Pages never freed (historic) | `free_page` extracted wrong block index from PDE Address field | Fixed extraction to `Address SHR 10` — page freeing works for the first time | | Pages never freed (historic) | `free_page` extracted wrong block index from PDE Address field | Fixed extraction to `Address SHR 10` — page freeing works for the first time |
File diff suppressed because it is too large. Load diff
@@ -1,49 +1,85 @@
# driver.storage.ctl.usb <!--
Copyright 2026 Aaron Hance
Author: Aaron Hance <[email protected]>
USB Mass Storage Class driver stub. Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
## Overview http://www.apache.org/licenses/LICENSE-2.0
This unit registers a USB class driver with the driver manager (`driver.mgr`) to claim USB Mass Storage Class interfaces (class `$08`, subclass `$06` SCSI, protocol `$50` Bulk-Only). When the USB stack enumerates a matching interface, the `load` callback is invoked. Currently the callback only logs a detection message; no I/O infrastructure is established. Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
Full Bulk-Only Transport (BOT) support — including Max LUN query, endpoint setup, CBW/CSW command exchange, and `driver.storage.mgr` device registration — is planned but not yet implemented. # USB mass storage
## Boot Registration `driver.storage.ctl.usb` implements SCSI transparent commands over USB Mass
Storage Bulk-Only Transport (interface class/subclass/protocol `08/06/50`).
Registered with `boot.mgr` as `driver.storage.ctl.usb` at the `device` barrier. The barrier order places `device` after `bus`, ensuring `driver.bus.usb.core.init` has run before the storage class driver registers with the driver manager. It probes INQUIRY, TEST UNIT READY and READ CAPACITY(10), registers LUN 0 with
the storage manager, and exposes READ(10), WRITE(10), REQUEST SENSE and
SYNCHRONIZE CACHE(10). Existing partition discovery and filesystem mounting
apply to the resulting USB disk. Writes complete only after cache synchronization.
## Dependencies ## Execution and ownership
- `boot.mgr` An attachment task probes and registers each disk after scheduling starts; boot
initialization never waits on a worker that cannot run yet.
- `driver.mgr` Storage dispatch only publishes a request. One persistent `usb.storage` worker
- `memory.heap` performs I/O in task context, with one command active per device and a reusable,
- `driver.storage.types` page-aligned 4 KiB bounce buffer. Caller buffers are never submitted for DMA.
- `io.syslog` Commands are split at the bounce-buffer boundary; all LBA/count and buffer-end
- `debug.tracer` arithmetic is checked before submission. CBW/CSW wire sizes, tags, status,
- `driver.bus.usb.core` residue, and actual transfer lengths are validated.
- `driver.bus.usb.types`
- `core.util`, `arch.x86.util`
## Constants OHCI batches bulk data into spans of at most 4 KiB per transfer descriptor,
using the endpoint's hardware toggle carry across packets. A 4 KiB full-speed
read uses one data descriptor instead of 64. Short packets still halt the chain
and report the actual byte count; interrupt endpoint packet sizing is unchanged.
| Constant | Value | Description | The worker performs reset recovery (class reset, clear bulk-IN halt, clear
|---|---|---| bulk-OUT halt) after malformed status or phase errors. Failed SCSI commands
| `USB_CLASS_MASS_STORAGE` | `$08` | USB interface class for mass storage | request sense and return storage errors, including write protection.
| `USB_SC_SCSI` | `$06` | SCSI transparent command set subclass |
| `USB_PROTO_BBB` | `$50` | Bulk-Only (BOT) transport protocol |
## Functions and Procedures On disconnect the disk is marked removed immediately. Accepted requests are
completed with errors, and idle volumes are invalidated/unmounted by the worker;
busy volumes are retried. Storage-manager records remain stable tombstones.
Class-driver buffers and context are freed after pending hardware work drains.
### init Host controllers currently have no cancellation API. A timed-out transfer is
therefore quarantined with its private device snapshot and DMA memory. No further
commands are issued to that device. If hardware never completes, those allocations
remain retained until reboot; freeing them would permit DMA into reused memory.
The timeout uses the PIT's actual frequency and a bounded polling fallback.
```pascal ## Supported scope
procedure init;
```
Constructs a `TDeviceIdentifier` matching any USB device with `bInterfaceClass = $08`, `bInterfaceSubClass = $06`, `bInterfaceProtocol = $50`, and registers the `load` callback with `driver.mgr`. - LUN 0 on a single-interface device with one bulk-IN and one bulk-OUT endpoint.
- Direct-access disks with power-of-two sectors from 512 through 4096 bytes.
- Capacity representable by the existing 32-bit storage-sector API; the
READ CAPACITY(16) sentinel is rejected.
- OHCI, EHCI and xHCI bulk transfers. OHCI/EHCI report actual data lengths and
terminate bulk reads on short packets.
## Notes UAS, additional LUNs, composite/alternate-setting interfaces, UHCI storage,
optical devices, and media replacement without USB reconnect are not supported.
The current xHCI driver lacks endpoint-ring reset support: a halted bulk endpoint
fails the device closed rather than attempting an unsafe ring reuse.
- The `load` callback currently returns true unconditionally after logging a detection message. ## Validation
- Once full BOT support is implemented, `load` should query the Max LUN, allocate endpoint structures, and call `driver.storage.mgr.register_device` for each logical unit.
`UnitTest` is called by the boot test suite and is available as `USBMSCTEST`.
Tests use a memory-only mock host controller: wire layout, byte order, capacity
limits, reads, writes, sense errors, short transfers, status stalls, reset ordering,
and request bounds. They perform no disk I/O.
For hardware/VM checks, boot with a disposable USB disk, confirm `USBMSC` discovery
and filesystem mounting, test file read/write/read-back, then unplug while idle
and during I/O. Do not run destructive disk tests against a disk containing data.
Protocol: [USB-IF Bulk-Only Transport 1.0](https://www.usb.org/sites/default/files/usbmassbulk_10.pdf).
+52 -10
View File
@@ -1,3 +1,20 @@
<!--
Copyright 2026 Aaron Hance
Author: Aaron Hance <[email protected]>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# driver.storage.mgr # driver.storage.mgr
Physical storage device registry and I/O dispatch manager. Physical storage device registry and I/O dispatch manager.
@@ -14,7 +31,7 @@ The I/O model operates as follows:
- `storage_read` / `storage_write` are convenience wrappers that build a stack-local `TIORequest` and call `submit_io_wait`. - `storage_read` / `storage_write` are convenience wrappers that build a stack-local `TIORequest` and call `submit_io_wait`.
- `storage_read_async` / `storage_write_async` build and submit a request with a caller-supplied callback, returning immediately. - `storage_read_async` / `storage_write_async` build and submit a request with a caller-supplied callback, returning immediately.
Legacy `storage_read_legacy` / `storage_write_legacy` procedure wrappers are retained for callers not yet migrated to the function-returning API. Blocking callers use `storage_read` / `storage_write` and check the returned `TError`.
## Dependencies ## Dependencies
@@ -205,17 +222,42 @@ function storage_write_async(device : PStorage_Device; addr : uint32; sectors :
Non-blocking write. Analogous to `storage_read_async`. Non-blocking write. Analogous to `storage_read_async`.
### storage_read_legacy / storage_write_legacy
```pascal
procedure storage_read_legacy(device : PStorage_Device; addr : uint32; sectors : uint32; buffer : puint32);
procedure storage_write_legacy(device : PStorage_Device; addr : uint32; sectors : uint32; buffer : puint32);
```
Drop-in procedure replacements for callers using the old void-returning API. Forward to `storage_read` / `storage_write` and discard the error code. Marked for removal once all callers are migrated.
## Notes ## Notes
- `dispatch_next` is an internal procedure called with interrupts disabled. It dequeues the head of the CFIFO, allocates a heap copy via `ioreq_copy` (for ISR safety), and invokes the appropriate `dispatchRead` or `dispatchWrite` on the device. If neither dispatch pointer is set, the request is failed with `eDeviceNotFound`. - `dispatch_next` is an internal procedure called with interrupts disabled. It dequeues the head of the CFIFO, allocates a heap copy via `ioreq_copy` (for ISR safety), and invokes the appropriate `dispatchRead` or `dispatchWrite` on the device. If neither dispatch pointer is set, the request is failed with `eDeviceNotFound`.
- The blocking path relies on pointer-dereferenced spin (`puint32(@wait.Done)^`) to prevent FPC from caching the `Done` flag in a register across the `hlt` instruction. - The blocking path relies on pointer-dereferenced spin (`puint32(@wait.Done)^`) to prevent FPC from caching the `Done` flag in a register across the `hlt` instruction.
- Devices without `dispatchRead`/`dispatchWrite` set (i.e., drivers not yet migrated to Phase 2) will have `submit_io` return `eDeviceNotFound`. Legacy drivers must be migrated before they can use the new I/O path. - Devices without `dispatchRead`/`dispatchWrite` set (i.e., drivers not yet migrated to Phase 2) will have `submit_io` return `eDeviceNotFound`. Legacy drivers must be migrated before they can use the new I/O path.
## Bounded performance path
The manager merges up to 32 consecutive queued requests when operation, device,
LBA range, and virtual buffer range are contiguous. It never searches past the
queue head or delays a lone request to create a merge. Drivers opt in through
`mergeMaxSectors`; SATA AHCI advertises 65,536 sectors, while other drivers default
to zero (disabled). A merged command delivers one completion to each original
request, in FIFO order. A command error fails every member conservatively.
`storage_get_stats` takes an interrupt-protected snapshot of the device counters.
`DISK info <index>` displays accepted/completed/failed logical requests, hardware
commands, merged requests, queue-full rejections, queue and active high-water
marks, successful read/write sectors, request-size buckets, queue/service ticks,
and FAT metadata cache hits/misses. Queue ticks are accumulated per logical
request; service ticks are accumulated per hardware command. Tick values use
`Counters.c32`, not milliseconds. Counters are unsigned 32-bit values, start at
device registration, and wrap; use snapshot differences over bounded benchmark
intervals. Size buckets are <=8, 9..256, and >256 sectors.
FAT32 blocking and async reads share a pooled transfer engine with batches of up
to four outstanding reads, capped by device queue depth. Destination slices are
disjoint. On any error, the engine drains every accepted request before releasing
its context and reports only the contiguous successful byte prefix. Unaligned
edges and odd DMA addresses use the context's private scratch sector. Writes
remain sequential. Run maps store cumulative file-cluster offsets; an open-file
cursor handles sequential locality and binary search handles random seeks.
`STORPERF` runs memory-backed regression checks (also included in boot unit tests)
for reverse completion order, failures, unaligned reads, run lookup, queue
rejection, merging limits, and callback accounting. It does not access real disks
or measure hardware throughput. Compare counters alongside `STORBENCH` and file
benchmarks on contiguous and fragmented files for actual performance validation.
@@ -58,3 +58,111 @@ Registers the `STORTEST` and `DISKTEST` shell commands with `io.stdio`. Must be
- `run_tests` exercises `makeAbsolutePathFrom`, `resolvePathFrom`, `changeDirectoryFrom`, `GetDirectoryListingFrom`, `newVirtualDirectory`, and `PathValid` including `.` and `..` traversal. - `run_tests` exercises `makeAbsolutePathFrom`, `resolvePathFrom`, `changeDirectoryFrom`, `GetDirectoryListingFrom`, `newVirtualDirectory`, and `PathValid` including `.` and `..` traversal.
- `run_disk_tests` is destructive and will irreversibly wipe the partition table of disk 0. It is never called automatically. - `run_disk_tests` is destructive and will irreversibly wipe the partition table of disk 0. It is never called automatically.
- The `Assert` helper used in both procedures increments `passed` or `failed` counters and logs the test name on failure; passing tests produce no output to avoid noise. - The `Assert` helper used in both procedures increments `passed` or `failed` counters and logs the test name on failure; passing tests produce no output to avoid noise.
## Raw read benchmark
`STORBENCH` is implemented and registered by `driver.storage.bench`. It performs
read-only raw-device I/O; it does not invoke the destructive `DISKTEST` path.
```text
STORBENCH [disk] [MiB]
STORBENCH list
STORBENCH run [disk] [MiB] [KiB/request] [depth] [seq|random] [start_lba]
STORBENCH compare [disk] [MiB]
STORBENCH help
```
Manual (`run`) defaults are disk 0, 128 MiB, 1024 KiB/request, sequential reads, and a depth of
`min(4, hardware depth)` (at least one). Examples:
```text
STORBENCH 0 128 64 4
STORBENCH 0 128 4 16 random
STORBENCH 0 256 1024 1 seq 2048
STORBENCH compare 0 128
```
The comparison runs sequential reads at 4, 64, and 1024 KiB/request, each at depths
1, 4, 16, and 32. The MiB argument applies to each of the twelve cases. Queue depth
may exceed hardware depth to expose software queuing and request merging.
Limits are 1..16384 MiB, 1..1024 KiB/request, and depth 1..32. Request size must be
an exact number of device sectors. Buffers are allocated once per case, with a
separate slice for every outstanding request and a maximum total size of 32 MiB.
The transfer range is clamped to the device. Without an explicit LBA, reads start
at sector 2048 when the full requested range fits there; otherwise they start at
zero. Random mode uses a fixed xorshift seed and samples that range with replacement,
so total bytes read are bounded but unique coverage is not guaranteed.
Reports include successful bytes, elapsed microseconds, MiB/s, IOPS, accepted and
completed request counts, peak outstanding requests, average/p95/max completion
latency, and per-run storage-counter deltas. Latencies are measured from submission
to callback. The p95 is an upper bound from logarithmic microsecond buckets, not an exact
microsecond percentile. Runs shorter than 10 ms are excluded from suite speed comparisons.
No console output occurs inside the measured loop. Any failure stops new work;
already accepted DMA is drained before buffers are freed, and failed requests
contribute no bytes or successful IOPS.
Timing prefers the Hyper-V-compatible VM reference counter (100 ns ticks), detected
through its Hv#1 interface and reference-counter privilege. It needs no calibration.
Other platforms use an invariant TSC calibrated against two complete RTC seconds,
with up to three attempts to tolerate transient RTC updates or scheduling delays.
Calibration requires agreement within 1%; failures include the specific reason.
Unlike delivered timer interrupts, this clock advances while interrupts are masked.
Queue submission restores interrupts between requests. Filesystem caches are
bypassed, but drive and virtual-machine/host caches may still affect results.
Virtual or sparse disks can report throughput above physical-media speed.
Device counter deltas may include concurrent I/O from other processes;
benchmark-owned byte, request, and latency counters do not.
`driver.storage.bench.UnitTest` runs memory-only parser, arithmetic, range,
queue-depth, completion-order, and error-accounting tests at boot. It never reads
or writes a real disk. The clock tests also verify that an advertised VM reference
counter advances with interrupts masked; they never calibrate the RTC at boot.
### Automatic suite and table output
`STORBENCH` without arguments, or `STORBENCH auto [disk] [MiB]`, runs six
read-only profiles. Auto defaults to disk 0 and **32 MiB per profile** (up to
192 MiB total before device-size clamping). One or two numeric arguments select the auto suite: `STORBENCH 2 512` tests
disk 2 with 512 MiB per profile. `STORBENCH list` lists disk indices, capacities,
controllers, sector sizes, and hardware queue depths. Three or more numeric
arguments retain the legacy manual syntax; `run` selects manual mode explicitly.
| Profile | Request size | Queue depth |
| --- | --- | --- |
| Seq large | 1024 KiB | 1 |
| Seq queued | 1024 KiB | queued |
| Seq medium | 64 KiB | queued |
| Random small | 4 KiB | 1 |
| Random queued | 4 KiB | queued |
| Random medium | 64 KiB | queued |
Queued depth is the device hardware depth clamped to 4..16, allowing software
queue testing even on serial devices. A request smaller than a device sector is
increased to one sector. Each profile samples the same byte-span size, with the
existing fixed random seed and range clamping rules.
Auto and compare modes print an aligned ASCII table with request KiB, queue
depth, MiB/s, IOPS, average latency, p95 latency upper bound, and status. Rows are
printed outside the measured loop. Successful runs shorter than 10 ms show `-` metrics and
`SHORT`, and are excluded from best-result and queue-gain summaries. Errors show
`FAILED`, stop the suite after accepted DMA has drained, and leave remaining
profiles unrun. The summary includes successful profiles, bytes actually read,
measured time, best sequential throughput, peak random IOPS, and (in auto mode)
queued/QD1 throughput ratios. These ratios describe the measured cases, including
any controller-cache effects; they are not a hardware rating.
```text
STORBENCH
STORBENCH list
STORBENCH 2 512
STORBENCH auto 0 64
STORBENCH compare 0 32
```
Profiles reuse the same raw device range. Host/drive caches are not flushed, so
later profiles may read warm data. These results are observed raw read throughput,
not a claim of uncached physical-media performance.
+2 -2
View File
@@ -19,7 +19,7 @@ Every allocation is preceded by a 16-byte prefix: 4 bytes for the unit count, 12
Allocations larger than a single heap page's capacity are handled by `klalloc`, which maps contiguous virtual pages and stores a magic sentinel (`$FFFFFFFE`) in the header so `kfree` can identify them and call `klfree` instead of the normal bitmap path. Allocations larger than a single heap page's capacity are handled by `klalloc`, which maps contiguous virtual pages and stores a magic sentinel (`$FFFFFFFE`) in the header so `kfree` can identify them and call `klfree` instead of the normal bitmap path.
`kalloc` and `kfree` disable interrupts for the duration of their critical sections using a stack-balanced `pushfd`/`cli`/`pop` sequence and restore the interrupt flag via `restore_if` on all exit paths. `kalloc` and `kfree` disable interrupts for the duration of their critical sections using a stack-balanced `pushfd`/`cli`/`pop` sequence and restore the interrupt flag via `arch.platform.cpu.unlock` on all exit paths.
## Dependencies ## Dependencies
@@ -132,7 +132,7 @@ Returns the number of heap pages currently in the linked list.
## Notes ## Notes
The `restore_if` helper checks bit 9 (IF) of the saved EFLAGS word and issues `STI` only if interrupts were enabled on entry. It uses `STI` rather than `POPFD` to avoid restoring potentially dangerous EFLAGS bits (IOPL, NT, VM, AC) that could trigger a General Protection Fault or Bad-TSS on the next `IRET`. The `arch.platform.cpu.unlock` helper checks bit 9 (IF) of the saved EFLAGS word and issues `STI` only if interrupts were enabled on entry. It uses `STI` rather than `POPFD` to avoid restoring potentially dangerous EFLAGS bits (IOPL, NT, VM, AC) that could trigger a General Protection Fault or Bad-TSS on the next `IRET`.
When `DEBUG_LMM` is defined, `kalloc` allocates one additional unit and writes `GUARD_MAGIC` at the end of the allocation. `kfree` verifies this sentinel before freeing; a mismatch triggers a GPF with a syslog error, catching buffer overruns at free time. When `DEBUG_LMM` is defined, `kalloc` allocates one additional unit and writes `GUARD_MAGIC` at the end of the allocation. `kfree` verifies this sentinel before freeing; a mismatch triggers a GPF with a syslog error, catching buffer overruns at free time.
-45
View File
@@ -1,45 +0,0 @@
# Automatic boot-unit inclusion
Run the usual `docker compose build builder` and `docker compose run builder`
from Asuro. Only this repository is required. Compose and Drone use the same
Dockerfile and repository build context, producing `t3hn3rd/asuro-build:latest`.
The Dockerfile inherits the published `t3hn3rd/moderne-asuro-recipes:0.1.1` image,
pinned by digest. It already includes Java, Moderne CLI, the Pascal LST generator,
and the registered recipe. The Asuro layer installs FPC and the kernel build tools.
To upgrade tooling, publish the upstream images first and update `RECIPES_IMAGE`
in the Dockerfile to the new tag and digest. Local tooling image testing can use
`docker compose build --build-arg RECIPES_IMAGE=<local-tag> builder`.
`ASURO_ARCH` selects the source architecture, defaulting to `x86`. For example:
```sh
export ASURO_ARCH=x86
docker compose run --rm builder
```
Compose forwards this variable to the builder; Drone inherits it from its compile
environment. Only `src/arch/$ASURO_ARCH` is added to FPC's unit search paths.
Other architecture folders are excluded from LST generation and from the recipe's
dependency graph and results. Common units outside `src/arch` remain eligible.
The selected architecture folder must exist. This source selection does not
change the existing i386 compiler target, assembly stub, or linker settings;
an x86_64 port still needs those platform-specific changes.
`compile_dangling.sh` runs after Wasuro synchronization and before FPC. It copies
only `src/` and `compat/` into an isolated temporary analysis repository, generates
LSTs with rewrite-pascal, runs the standalone Moderne recipe, exports its
datatable, and writes `src/boot/dangling_units.inc`. Wasuro is not parsed.
Datatable conversion is provided by the standalone project's Java exporter.
Parse errors, recipe failures, and malformed datatables stop the build. The
generated include is ignored by Git and removed before each generation attempt.
Use `{$DEFINE NOT_DANGLING}` near a unit declaration to exclude an experimental
or legacy unit from automatic inclusion. It can still be referenced normally.
This is a file-level marker; conditional definitions are not evaluated. The
analysis follows both interface and implementation uses, without expanding
includes or evaluating conditional compilation.
Moderne runs through its CLI normally; any license or tenant configuration
required by your CLI installation must be provided to the container separately.
No host credentials are copied into the image.
+1 -1
View File
@@ -1,8 +1,8 @@
services: services:
builder: builder:
image: t3hn3rd/asuro-build:latest
build: . build: .
environment: environment:
ASURO_ARCH: ${ASURO_ARCH:-x86} ASURO_ARCH: ${ASURO_ARCH:-x86}
ASURO_BOOT_TEST: ${ASURO_BOOT_TEST:-none}
volumes: volumes:
- .:/code - .:/code
+1 -1
View File
@@ -283,7 +283,6 @@ nav:
- Version Generation: toolchain/compile_vergen.md - Version Generation: toolchain/compile_vergen.md
- LVGL Compilation: toolchain/compile_lvgl.md - LVGL Compilation: toolchain/compile_lvgl.md
- Wasuro Pull: toolchain/compile_wasuro.md - Wasuro Pull: toolchain/compile_wasuro.md
- Automatic Boot Units: toolchain/dangling_units.md
- Source List: toolchain/compile_sourcelist.md - Source List: toolchain/compile_sourcelist.md
- Source Compilation: toolchain/compile_sources.md - Source Compilation: toolchain/compile_sources.md
- Checksum: toolchain/compile_checksum.md - Checksum: toolchain/compile_checksum.md
@@ -296,6 +295,7 @@ nav:
- Planning: - Planning:
- Architecture: planning/architecture.md - Architecture: planning/architecture.md
- x86-64 Migration: planning/x86_64.md
- USB: planning/USB.md - USB: planning/USB.md
- XHCI: planning/XHCI.md - XHCI: planning/XHCI.md
- FlatFS: planning/flatfs.md - FlatFS: planning/flatfs.md
+1 -2
View File
@@ -28,8 +28,7 @@ We welcome everyone to give building/breaking/fixing/shooting Asuro a go, feel f
4. Install Git for Windows. 4. Install Git for Windows.
5. Install VSCode & the listed plugins. 5. Install VSCode & the listed plugins.
6. Install VirtualBox (v7+). 6. Install VirtualBox (v7+).
7. Clone this repository. The builder inherits the published Moderne/Pascal 7. Clone this repository.
tooling image; see [boot-unit generation](doc/toolchain/dangling_units.md).
8. Run the following command in the root of the repo to build the docker image: 8. Run the following command in the root of the repo to build the docker image:
```powershell ```powershell
docker compose build builder docker compose build builder
+1 -1
View File
@@ -22,7 +22,7 @@ unit app.base64;
interface interface
uses uses
io.stdio, core.util, arch.x86.util, core.strings, debug.tracer, core.enc.base64, memory.heap; io.stdio, core.util, arch.platform.cpu, core.strings, debug.tracer, core.enc.base64, memory.heap;
procedure init(); procedure init();
+1 -1
View File
@@ -22,7 +22,7 @@ unit app.dhclient;
interface interface
uses uses
io.stdio, core.util, arch.x86.util, core.strings, debug.tracer, driver.net.proto.dhcp; io.stdio, core.util, arch.platform.cpu, core.strings, debug.tracer, driver.net.proto.dhcp;
procedure init(); procedure init();
+13 -2
View File
@@ -25,7 +25,7 @@ uses
core.strings, core.strings,
io.stdio, io.stdio,
debug.tracer, debug.tracer,
core.util, arch.x86.util, core.util, arch.platform.cpu,
driver.storage.vol.mgr; driver.storage.vol.mgr;
var var
@@ -150,6 +150,7 @@ var
device : PStorage_Device; device : PStorage_Device;
volCount : uint32; volCount : uint32;
freeSec : uint32; freeSec : uint32;
stats : TStorageStats;
begin begin
if paramCount(params) < 2 then begin if paramCount(params) < 2 then begin
println('Usage: DISK info <disk>'); println('Usage: DISK info <disk>');
@@ -191,6 +192,16 @@ begin
else else
volCount := 0; volCount := 0;
print('Partitions: '); printint(volCount); println(''); print('Partitions: '); printint(volCount); println('');
storage_get_stats(device,stats);
print('I/O submitted/completed/failed: '); printint(stats.Submitted); print(' / '); printint(stats.Completed); print(' / '); printint(stats.Failed); println('');
print('Commands / merged requests: '); printint(stats.Commands); print(' / '); printint(stats.Merged); println('');
print('Queue peak / active peak / queue full: '); printint(stats.QueuePeak); print(' / '); printint(stats.ActivePeak); print(' / '); printint(stats.QueueFull); println('');
print('Read / written sectors: '); printint(stats.ReadSectors); print(' / '); printint(stats.WriteSectors); println('');
print('Requests <=8 / <=256 / >256 sectors: '); printint(stats.SmallRequests); print(' / '); printint(stats.MediumRequests); print(' / '); printint(stats.LargeRequests); println('');
print('Queue ticks total / max: '); printint(stats.QueueTicks); print(' / '); printint(stats.MaxQueueTicks); println('');
print('Service ticks total / max: '); printint(stats.ServiceTicks); print(' / '); printint(stats.MaxServiceTicks); println('');
print('FAT cache hits / misses: '); printint(stats.FatCacheHits); print(' / '); printint(stats.FatCacheMisses); println('');
end; end;
{ ---------- DISK WIPE ---------- } { ---------- DISK WIPE ---------- }
@@ -243,7 +254,7 @@ begin
{ Allocate a zero buffer - 64 sectors at a time } { Allocate a zero buffer - 64 sectors at a time }
bufSects := 64; bufSects := 64;
buf := puint32(kalloc(bufSects * device^.sectorSize)); buf := puint32(kalloc(bufSects * device^.sectorSize));
memset(uint32(buf), 0, bufSects * device^.sectorSize); memset(PtrUInt(buf), 0, bufSects * device^.sectorSize);
print('Wiping device '); print('Wiping device ');
printint(idx); printint(idx);
+654 -176
View File
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
Loaded 20 of 200 files, more files were not shown because too many files have changed in this diff. Show more