Feature/sotrage system overhaul #73

Open
Aaron wants to merge 21 commits from feature/sotrage-system-overhaul into develop
pull from: feature/sotrage-system-overhaul
8 changed files with 1057 additions and 181 deletions
Showing only changes of commit 720ba46630 - Show all commits

No files matched your search

@@ -58,3 +58,111 @@ Registers the `STORTEST` and `DISKTEST` shell commands with `io.stdio`. Must be
- `run_tests` exercises `makeAbsolutePathFrom`, `resolvePathFrom`, `changeDirectoryFrom`, `GetDirectoryListingFrom`, `newVirtualDirectory`, and `PathValid` including `.` and `..` traversal.
- `run_disk_tests` is destructive and will irreversibly wipe the partition table of disk 0. It is never called automatically.
- The `Assert` helper used in both procedures increments `passed` or `failed` counters and logs the test name on failure; passing tests produce no output to avoid noise.
## Raw read benchmark
`STORBENCH` is implemented and registered by `driver.storage.bench`. It performs
read-only raw-device I/O; it does not invoke the destructive `DISKTEST` path.
```text
STORBENCH [disk] [MiB]
STORBENCH list
STORBENCH run [disk] [MiB] [KiB/request] [depth] [seq|random] [start_lba]
STORBENCH compare [disk] [MiB]
STORBENCH help
```
Manual (`run`) defaults are disk 0, 128 MiB, 1024 KiB/request, sequential reads, and a depth of
`min(4, hardware depth)` (at least one). Examples:
```text
STORBENCH 0 128 64 4
STORBENCH 0 128 4 16 random
STORBENCH 0 256 1024 1 seq 2048
STORBENCH compare 0 128
```
The comparison runs sequential reads at 4, 64, and 1024 KiB/request, each at depths
1, 4, 16, and 32. The MiB argument applies to each of the twelve cases. Queue depth
may exceed hardware depth to expose software queuing and request merging.
Limits are 1..16384 MiB, 1..1024 KiB/request, and depth 1..32. Request size must be
an exact number of device sectors. Buffers are allocated once per case, with a
separate slice for every outstanding request and a maximum total size of 32 MiB.
The transfer range is clamped to the device. Without an explicit LBA, reads start
at sector 2048 when the full requested range fits there; otherwise they start at
zero. Random mode uses a fixed xorshift seed and samples that range with replacement,
so total bytes read are bounded but unique coverage is not guaranteed.
Reports include successful bytes, elapsed microseconds, MiB/s, IOPS, accepted and
completed request counts, peak outstanding requests, average/p95/max completion
latency, and per-run storage-counter deltas. Latencies are measured from submission
to callback. The p95 is an upper bound from logarithmic microsecond buckets, not an exact
microsecond percentile. Runs shorter than 10 ms are excluded from suite speed comparisons.
No console output occurs inside the measured loop. Any failure stops new work;
already accepted DMA is drained before buffers are freed, and failed requests
contribute no bytes or successful IOPS.
Timing prefers the Hyper-V-compatible VM reference counter (100 ns ticks), detected
through its Hv#1 interface and reference-counter privilege. It needs no calibration.
Other platforms use an invariant TSC calibrated against two complete RTC seconds,
with up to three attempts to tolerate transient RTC updates or scheduling delays.
Calibration requires agreement within 1%; failures include the specific reason.
Unlike delivered timer interrupts, this clock advances while interrupts are masked.
Queue submission restores interrupts between requests. Filesystem caches are
bypassed, but drive and virtual-machine/host caches may still affect results.
Virtual or sparse disks can report throughput above physical-media speed.
Device counter deltas may include concurrent I/O from other processes;
benchmark-owned byte, request, and latency counters do not.
`driver.storage.bench.UnitTest` runs memory-only parser, arithmetic, range,
queue-depth, completion-order, and error-accounting tests at boot. It never reads
or writes a real disk. The clock tests also verify that an advertised VM reference
counter advances with interrupts masked; they never calibrate the RTC at boot.
### Automatic suite and table output
`STORBENCH` without arguments, or `STORBENCH auto [disk] [MiB]`, runs six
read-only profiles. Auto defaults to disk 0 and **32 MiB per profile** (up to
192 MiB total before device-size clamping). One or two numeric arguments select the auto suite: `STORBENCH 2 512` tests
disk 2 with 512 MiB per profile. `STORBENCH list` lists disk indices, capacities,
controllers, sector sizes, and hardware queue depths. Three or more numeric
arguments retain the legacy manual syntax; `run` selects manual mode explicitly.
| Profile | Request size | Queue depth |
| --- | --- | --- |
| Seq large | 1024 KiB | 1 |
| Seq queued | 1024 KiB | queued |
| Seq medium | 64 KiB | queued |
| Random small | 4 KiB | 1 |
| Random queued | 4 KiB | queued |
| Random medium | 64 KiB | queued |
Queued depth is the device hardware depth clamped to 4..16, allowing software
queue testing even on serial devices. A request smaller than a device sector is
increased to one sector. Each profile samples the same byte-span size, with the
existing fixed random seed and range clamping rules.
Auto and compare modes print an aligned ASCII table with request KiB, queue
depth, MiB/s, IOPS, average latency, p95 latency upper bound, and status. Rows are
printed outside the measured loop. Successful runs shorter than 10 ms show `-` metrics and
`SHORT`, and are excluded from best-result and queue-gain summaries. Errors show
`FAILED`, stop the suite after accepted DMA has drained, and leave remaining
profiles unrun. The summary includes successful profiles, bytes actually read,
measured time, best sequential throughput, peak random IOPS, and (in auto mode)
queued/QD1 throughput ratios. These ratios describe the measured cases, including
any controller-cache effects; they are not a hardware rating.
```text
STORBENCH
STORBENCH list
STORBENCH 2 512
STORBENCH auto 0 64
STORBENCH compare 0 32
```
Profiles reuse the same raw device range. Host/drive caches are not flushed, so
later profiles may read warm data. These results are observed raw read throughput,
not a claim of uncached physical-media performance.
+5 -1
View File
@@ -27,6 +27,10 @@ uses
arch.x86.isr.mgr,
arch.x86.idt;
const
PIT_INPUT_HZ = 1193182;
PIT_DIVISOR = 149;
procedure register();
procedure hook(hook_method : uint32);
procedure unhook(hook_method : uint32);
@@ -54,7 +58,7 @@ procedure register();
begin
if not registered then begin
asm
mov ax, 149
mov ax, PIT_DIVISOR
out $40, al
mov al, ah
out $40, al
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+4
View File
@@ -486,6 +486,10 @@ var
begin
search := params;
for i := 0 to index do begin
if search = nil then begin
getParam := nil;
exit;
end;
if search^.next = nil then begin
getParam := nil;
exit;
+20 -7
View File
@@ -107,10 +107,10 @@ procedure process_trampoline; forward;
{ ---- Process creation ---- }
function create(name : pchar;
function createInternal(name : pchar;
entry : TProcessEntryPoint;
localData : void;
priority : uint8) : PProcessContext;
priority : uint8; runnable : boolean) : PProcessContext;
var
ctx : PProcessContext;
slotPtr : ^uint32;
@@ -119,7 +119,7 @@ var
prio : uint8;
begin
push_trace('proc.mgr.create');
create := nil;
createInternal := nil;
if priority = 0 then prio := 1 else prio := priority;
@@ -140,7 +140,7 @@ begin
ctx^.ParentID := CurrentProcess^.ProcessID;
{ State }
ctx^.State := psCreated;
ctx^.State := psSuspended;
ctx^.ExitCode := 0;
{ Entry point }
1
@@ -269,14 +269,22 @@ begin
slotPtr^ := uint32(ctx);
{ Mark as ready for scheduling }
ctx^.State := psReady;
if runnable then ctx^.State := psReady;
asm popf end;
io.syslog.log('PROCMGR', 'Created process: ');
io.syslog.writestringln(name);
pop_trace;
create := ctx;
createInternal := ctx;
end;
{ Public creation retains its ready-on-return behavior. Commands need their
arguments and streams installed before the scheduler may enter them. }
function create(name : pchar; entry : TProcessEntryPoint;
localData : void; priority : uint8) : PProcessContext;
begin
create := createInternal(name, entry, localData, priority, true);
end;
{ ---- Process trampoline ---- }
@@ -543,12 +551,17 @@ function runCommand(name : pchar; method : TCommandMethod; params : PParamList;
var
ctx : PProcessContext;
begin
ctx := create(name, @legacy_entry, void(method), 1);
ctx := createInternal(name, @legacy_entry, void(method), 1, false);
if ctx <> nil then begin
ctx^.MsgData := void(params);
ctx^.StdIn := stdin_buf;
ctx^.StdOut := stdout_buf;
ctx^.StdErr := stderr_buf;
{ Publish only after all fields read by legacy_entry are initialized.
psCreated is also runnable, so creation must use psSuspended. }
asm pushf; cli end;
ctx^.State := psReady;
asm popf end;
end;
runCommand := ctx;
end;
+2
View File
@@ -33,6 +33,7 @@ uses
core.ds.prio,
driver.storage.vfs,
driver.storage.test,
driver.storage.bench,
driver.storage.metatest,
driver.storage.perftest,
driver.storage.ctl.ahci,
@@ -71,6 +72,7 @@ begin
core.ds.prio.UnitTest;
driver.storage.vfs.UnitTest;
driver.storage.test.UnitTest;
driver.storage.bench.UnitTest;
driver.storage.metatest.UnitTest;
driver.storage.perftest.UnitTest;
driver.storage.ctl.ahci.UnitTest;