Compare commits

..
1 Commits
Author SHA1 Message Date
t3hn3rd 8b4335768b refactor(memory): complete PMM/VMM/LMM rewrite - bitmap allocators, bug fixes, hardening
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
PMM Rewrite:
- Replace ~8KB record array with compact bitmaps
  (PhysPresent/PhysAlloc/PhysScanned, 128B each)
- Add PhysOwner[0..1023] for per-block ownership tracking
- new_block: O(1) amortised via NextFreeHint + dword-level BSF scan,
  returns 0 as OOM sentinel
- free_block: O(1) direct bit clear with double-free detection
- alloc_block: validates PhysPresent and PhysAlloc bits before allocation
- Add pmm_free_blocks/pmm_total_blocks stats functions

VMM Correctness Fixes:
- Fix free_page block extraction (Address SHR 10, was using
  raw Address field)
- Fix invlpg operand (use virtual address, was passing PDE index)
- Clear PDE entry on free_page before returning block to PMM
- Remove map_page double-write (delegate to map_page_ex only)
- Add OOM propagation (new_page returns false if PMM returns 0)
- Fix missing pop_trace in new_page error path

LMM Rewrite:
- Replace per-entry record array with bitmap allocator
  (93% vs 43% page efficiency)
- Layout: Header(24B) + Bitmap(64512B) + Padding + Data at 0x10000
- SIZE_PREFIX=16 for 16-byte aligned returns (SSE MOVAPS requirement)
- Next-fit bitmap scan with roving hint (NextFree) and dword-level
  fast skip
- Large alloc sentinel (LARGE_ALLOC_MAGIC=$FFFFFFFE) so kfree detects
  klalloc'd memory
- Implement klfree (was previously a no-op - pages could never
  be returned)
- Implement try_release_page (return fully-free heap pages to VMM/PMM)
- Fix interrupt safety: stack-balanced pushfd/cli/pop +
  restore_if (sti only)
  Fixes Bad TSS caused by popfd restoring dangerous EFLAGS bits (NT, IOPL)
- Zero memory via rep stosd instead of byte-by-byte loop

Hardening:
- kalloc OOM now triggers BSOD instead of returning nil
  (90% of callers don't check)
- Double-free detection in kfree (verify bitmap bits set before clearing)
- Guard bytes ({$IFDEF DEBUG_LMM}): $DEADBEEF sentinel for buffer overrun detection
- kpalloc: bounds check (block >= 1024 -> GPF) + syslog MMIO mapping audit log

Observability:
- Add MEMINFO command as src/prog/meminfo.pas (moved from inline kernel.pas)
- Displays multiboot memory, PMM block stats, LMM heap stats
- Register via progmanager.pas following standard prog pattern

Files changed:
- src/pmemorymanager.pas
- src/vmemorymanager.pas
- src/lmemorymanager.pas
- src/kernel.pas
- src/progmanager.pas
- src/prog/meminfo.pas
2026-03-04 10:36:09 +00:00
473 changed files with 8616 additions and 130704 deletions
+3 -33
View File
@@ -22,33 +22,9 @@ steps:
commands:
- git fetch --tags
- find . -type f -print0 | xargs -0 dos2unix
- chmod +x /drone/src/toolchain/*.sh
- /drone/src/toolchain/compile.sh
- chmod +x /drone/src/*.sh
- /drone/src/compile.sh
- name: deploy-docs
image: alpine:latest
depends_on:
- compile
when:
branch:
- master
- develop
- feature/mkdocs
volumes:
- name: docs-htdocs
path: /htdocs
commands:
- apk add --no-cache rsync
- |
if [ "${DRONE_BRANCH}" = "master" ]; then
DEST="/htdocs/master"
else
DEST="/htdocs/develop"
fi
- mkdir -p "$DEST"
- rsync -a --delete site/ "$DEST/"
- echo "Documentation deployed to ${DRONE_BRANCH}"
- name: upload-iso-artifact
image: alpine/git
depends_on:
@@ -87,10 +63,4 @@ steps:
from_secret: discord_webhook_id
webhook_token:
from_secret: discord_webhook_secret
message: "**Asuro Build**\n\n{{#success build.status}}✅ Build successful!\n\n{{else}}❌ Build failed!\n\n{{/success}}Repository: `{{repo.namespace}}/{{repo.name}}`\nBranch: `{{commit.branch}}`\nCommit: `{{commit.sha}}`\nAuthor: `{{commit.author}} <{{commit.email}}>`\n\nGitea Diff: [Link](<{{commit.link}}>)\nDrone Build: [Link](<{{build.link}}>)\n\nMessage: {{commit.message}}"
volumes:
- name: docs-htdocs
host:
path: /mnt/user/htdocs/asuro_docs
message: "**Asuro Build**\n\n{{#success build.status}}✅ Build successful!\n\n{{else}}❌ Build failed!\n\n{{/success}}Repository: `{{repo.namespace}}/{{repo.name}}`\nBranch: `{{commit.branch}}`\nCommit: `{{commit.sha}}`\nAuthor: `{{commit.author}} <{{commit.email}}>`\n\nGitea Diff: [Link](<{{commit.link}}>)\nDrone Build: [Link](<{{build.link}}>)\n\nMessage: {{commit.message}}"
+1 -5
View File
@@ -14,9 +14,5 @@ localenv.json
/lvgl/
dockerout.txt
AGENTS.md
lessons_learnt.md
*.log
wasuro
src/core/core.version.pas
site/*
!site/.gitkeep
/doc/*.md
+3 -5
View File
@@ -5,21 +5,19 @@ VOLUME ["/code"]
ENV DEBIAN_FRONTEND=noninteractive
RUN dpkg --add-architecture i386
RUN apt-get update && apt-get install -y \
curl dos2unix wget git make nasm binutils xorriso grub-pc-bin gcc gcc-multilib \
python3 python3-pip python3-pil && \
curl dos2unix wget git make nasm binutils xorriso grub-pc-bin gcc gcc-multilib && \
apt-get clean my room
RUN pip3 install --no-cache-dir --break-system-packages "mkdocs>=1.6,<2" mkdocs-material
SHELL ["/bin/bash", "-c"]
ARG FPC_VERSION=3.2.2
RUN curl -sL https://sourceforge.net/projects/freepascal/files/Linux/$FPC_VERSION/fpc-$FPC_VERSION.i386-linux.tar/download | tar -xf - && \
pushd fpc-$FPC_VERSION.i386-linux && ./install.sh && popd && \
rm -rf fpc-$FPC_VERSION.i386-linux
COPY compile.sh /compile.sh
ADD https://raw.githubusercontent.com/fsaintjacques/semver-tool/master/src/semver /usr/bin/semver
RUN chmod +x /usr/bin/semver
WORKDIR /code
RUN find . -type f -print0 | xargs -0 dos2unix
ENTRYPOINT ["/bin/bash", "-c"]
CMD ["find toolchain -name '*.sh' -exec dos2unix {} + 2>/dev/null; find toolchain -name '*.sh' -exec chmod +x {} +; bash toolchain/compile.sh"]
CMD ["/compile.sh"]
-75
View File
@@ -1,75 +0,0 @@
{ Compatibility shim: wasuro references the old unit name 'lmemorymanager'.
Since wasuro/ cannot be modified, this unit re-exports the public API of
memory.heap so that 'uses lmemorymanager' continues to compile. }
unit lmemorymanager;
interface
uses
memory.heap;
const
ALLOC_UNIT = memory.heap.ALLOC_UNIT;
DATA_OFFSET = memory.heap.DATA_OFFSET;
PAGE_SIZE_LMM = memory.heap.PAGE_SIZE_LMM;
TOTAL_UNITS = memory.heap.TOTAL_UNITS;
BITMAP_DWORDS = memory.heap.BITMAP_DWORDS;
SIZE_PREFIX = memory.heap.SIZE_PREFIX;
LARGE_ALLOC_MAGIC = memory.heap.LARGE_ALLOC_MAGIC;
type
PHeapPageHeader = memory.heap.PHeapPageHeader;
THeapPageHeader = memory.heap.THeapPageHeader;
procedure init;
function kalloc(size : uint32) : void;
function klalloc(size : uint32) : void;
procedure klfree(address : uint32);
function kpalloc(address : uint32) : void;
procedure kfree(area : void);
function lmm_total_free : uint32;
function lmm_page_count : uint32;
implementation
procedure init;
begin
memory.heap.init;
end;
function kalloc(size : uint32) : void; inline;
begin
kalloc := memory.heap.kalloc(size);
end;
function klalloc(size : uint32) : void; inline;
begin
klalloc := memory.heap.klalloc(size);
end;
procedure klfree(address : uint32); inline;
begin
memory.heap.klfree(address);
end;
function kpalloc(address : uint32) : void; inline;
begin
kpalloc := memory.heap.kpalloc(address);
end;
procedure kfree(area : void); inline;
begin
memory.heap.kfree(area);
end;
function lmm_total_free : uint32; inline;
begin
lmm_total_free := memory.heap.lmm_total_free;
end;
function lmm_page_count : uint32; inline;
begin
lmm_page_count := memory.heap.lmm_page_count;
end;
end.
-7
View File
@@ -1,7 +0,0 @@
unit types;
interface
implementation
end.
+3 -9
View File
@@ -20,26 +20,20 @@ runOrFail() {
fi
}
TOOLCHAIN_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
declare -a run_steps=(
"compile_stub.sh" "Failed to compile stub!"
"compile_vergen.sh" "Versions failed to compile"
"compile_lvgl.sh" "Failed to compile LVGL!"
"compile_icons.sh" "Failed to generate icons!"
"compile_wasuro.sh" "Failed to pull Wasuro!"
"compile_sources.sh" "Failed to compile FPC Sources!"
"compile_initcalls.sh" "Failed to generate INITFINAL table!"
"compile_link.sh" "Failed linking!"
"compile_isogen.sh" "Failed to create ISO!"
"compile_docs.sh" "Failed to generate documentation!"
)
for ((i=0; i<${#run_steps[@]}; i+=2))
do
if [ "$ERRCOUNT" -eq "0" ]
then
script="${TOOLCHAIN_DIR}/${run_steps[$i]}"
script=$(pwd)/"${run_steps[$i]}"
message="${run_steps[$i+1]}"
runOrFail "$script" "$message"
fi
@@ -48,9 +42,9 @@ done
#Call generate final artifacts based on failure or success of the above.
if [ "$ERRCOUNT" -ne "0" ]
then
. "${TOOLCHAIN_DIR}/compile_finish.sh" "failed"
. ./compile_finish.sh "failed"
else
. "${TOOLCHAIN_DIR}/compile_finish.sh" "success"
. ./compile_finish.sh "success"
fi
cd ..
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
echo " "
echo "======================="
echo " "
echo "Generating Documentation..."
echo " "
echo "Dowloading Pasdoc..."
wget https://github.com/pasdoc/pasdoc/releases/download/v0.16.0/pasdoc-0.16.0-linux-x86_64.tar.gz -O pasdoc.tar.gz
echo "Extracting Pasdoc..."
tar -xf pasdoc.tar.gz
echo "Creating doc output directory..."
mkdir ./doc
echo "Removing old docs"
rm -rf ./doc/*
echo "Generating Docs..."
./pasdoc/bin/pasdoc -N "Asuro" -T "Asuro OS Documentation" -O "html" -E ./doc/ -S sources.list --use-tipue-search
echo "Docgen finished."
@@ -19,4 +19,4 @@ echo " "
LIBGCC=$(gcc -m32 -print-libgcc-file-name)
echo "libgcc: ${LIBGCC}"
ld -m elf_i386 -s --gc-sections --no-warn-execstack -Ttoolchain/linker.script -o bin/kernel.bin $objstring --start-group lib/liblvgl.a ${LIBGCC} --end-group
ld -m elf_i386 -s --gc-sections -Tlinker.script -o bin/kernel.bin $objstring --start-group lib/liblvgl.a ${LIBGCC} --end-group
+8 -22
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
# compile_lvgl.sh — Download LVGL v9.2 source and compile into lib/liblvgl.a
# Clone & compile in /tmp (fast container-local fs).
# Cache liblvgl.a on host mount (/code/lvgl/) to skip rebuild.
# Copy source + objects to /lvgl (host mount) for debugging.
set -e
LVGL_VERSION="v9.2.2"
@@ -10,7 +10,6 @@ LVGL_DIR="/tmp/lvgl"
OBJ_DIR="/tmp/lvgl_obj"
CONF_DIR="$(pwd)/lvglh"
OUT_DIR="$(pwd)/lib"
CACHE_DIR="/code/lvgl"
CC="gcc"
CFLAGS="-m32 -march=i686 -ffreestanding \
@@ -27,17 +26,6 @@ echo " "
echo "Compiling LVGL..."
echo " "
# If cached liblvgl.a exists on host mount, just copy it and skip everything
if [ -f "${CACHE_DIR}/liblvgl.a" ]; then
echo "Found cached liblvgl.a in ${CACHE_DIR}, copying to ${OUT_DIR}..."
cp "${CACHE_DIR}/liblvgl.a" "${OUT_DIR}/liblvgl.a"
echo "(Delete lvgl/liblvgl.a to force a full rebuild.)"
echo " "
echo "LVGL compilation complete (cached)."
echo " "
exit 0
fi
# Clone LVGL into /tmp (container-local)
rm -rf "$LVGL_DIR"
echo "Downloading LVGL ${LVGL_VERSION}..."
@@ -117,16 +105,14 @@ else
echo "No custom LVGL files in lvglh/."
fi
# Cache liblvgl.a and source to host mount for next build
echo "Caching LVGL to host mount..."
mkdir -p "$CACHE_DIR"
rm -rf ${CACHE_DIR}/* ${CACHE_DIR}/.[!.]* ${CACHE_DIR}/..?* 2>/dev/null || true
cp "${OUT_DIR}/liblvgl.a" "${CACHE_DIR}/liblvgl.a"
cp -a "$LVGL_DIR/src" "${CACHE_DIR}/src"
cp -a "$LVGL_DIR"/*.h "${CACHE_DIR}/" 2>/dev/null || true
cp -a "$OBJ_DIR" "${CACHE_DIR}/obj"
# Copy source + objects to /lvgl host mount for debugging
echo "Copying LVGL source and objects to host mount..."
mkdir /code/lvgl 2>/dev/null || true
rm -rf /code/lvgl/* /code/lvgl/.[!.]* /code/lvgl/..?* 2>/dev/null || true
cp -a "$LVGL_DIR/src" /code/lvgl/src
cp -a "$OBJ_DIR" /code/lvgl/obj
echo "Done."
echo " "
echo "LVGL compilation complete."
echo " "
echo " "
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
echo " "
echo "======================="
echo " "
echo "Compiling FPC Sources..."
echo " "
fpc -Aelf -gw -g -gl -n -v0ew -O3 -OpPENTIUM3 -Si -Sc -Sg -Xd -CX -XXs -CfSSE -CfSSE2 -Rintel -Pi386 -Tlinux -FElib/ -Fusrc/* -Fusrc/driver/* -Fusrc/driver/net/* -Fusrc/driver/bus/* -Fusrc/driver/bus/usb/* -Fusrc/driver/hid/* src/kernel.pas
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
echo " "
echo "======================="
echo " "
echo "Compiling Stub..."
echo " "
nasm -f elf src/stub/stub.asm -o lib/stub.o
@@ -1,15 +1,14 @@
#!/usr/bin/env bash
set -e
TOOLCHAIN_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
echo " "
echo "======================="
echo " "
echo "Generating Versioning Info..."
echo " "
chmod +x "${TOOLCHAIN_DIR}/compile_checksum.sh"
"${TOOLCHAIN_DIR}/compile_checksum.sh"
outfile="src/core/core.version.pas"
file="toolchain/version"
chmod +x ./compile_checksum.sh
./compile_checksum.sh
outfile="src/include/asuro.pas"
file="version"
{
# this script requires semver tool
wget -q https://raw.githubusercontent.com/fsaintjacques/semver-tool/master/src/semver -O bin/semver && chmod +x bin/semver
@@ -23,7 +22,7 @@ sub=$(semver get patch $tagref)
version=$(semver get release $tagref)
release=$(semver get prerel $tagref)
build=$(semver get build $tagref)
linecount=$("${TOOLCHAIN_DIR}/loc.sh" | awk '{print $1}')
linecount=$(./loc.sh | awk '{print $1}')
sourcecount=$(find src -type f | wc -l)
drivercount=$(find src/driver -type f | wc -l)
fpcversion=$(fpc -h | grep -m 1 version | awk '{print $5}')
@@ -35,7 +34,7 @@ checksum=$(md5sum checksums.md5 | awk '{print $1}')
[[ -n "$release" ]] && version="$version-$release"
cat > $outfile <<EOF
unit core.version;
unit asuro;
interface
-94
View File
@@ -1,94 +0,0 @@
# lmemorymanager
Compatibility shim that re-exports the `memory.heap` API under the legacy unit name.
## Overview
The Wasuro WASM VM project references the old unit name `lmemorymanager` for heap memory operations. Because the Wasuro source tree cannot be modified, this shim unit re-exports every public symbol from `memory.heap` so that `uses lmemorymanager` continues to compile without changes. All functions are thin inline wrappers that delegate directly to their `memory.heap` counterparts.
## Dependencies
- `memory.heap` -- the canonical heap allocator implementation in the Asuro kernel.
## Constants
### ALLOC_UNIT
Re-exported from `memory.heap.ALLOC_UNIT`. The base allocation unit size used by the heap allocator.
### DATA_OFFSET
Re-exported from `memory.heap.DATA_OFFSET`. Byte offset from a heap block header to the start of user data.
### PAGE_SIZE_LMM
Re-exported from `memory.heap.PAGE_SIZE_LMM`. Page size used by the lightweight memory manager.
### TOTAL_UNITS
Re-exported from `memory.heap.TOTAL_UNITS`. Total number of allocation units per heap page.
### BITMAP_DWORDS
Re-exported from `memory.heap.BITMAP_DWORDS`. Number of 32-bit words in the per-page allocation bitmap.
### SIZE_PREFIX
Re-exported from `memory.heap.SIZE_PREFIX`. Size of the prefix stored before each allocation to record its length.
### LARGE_ALLOC_MAGIC
Re-exported from `memory.heap.LARGE_ALLOC_MAGIC`. Magic value used to identify large (multi-page) allocations.
## Types
### PHeapPageHeader / THeapPageHeader
Re-exported from `memory.heap`. Pointer and record types describing the header structure at the beginning of each heap page.
## Functions and Procedures
### init
```pascal
procedure init;
```
Initializes the heap memory manager by delegating to `memory.heap.init`.
### kalloc
```pascal
function kalloc(size: uint32): void;
```
Allocates `size` bytes from the kernel heap and returns a pointer to the allocated memory.
### klalloc
```pascal
function klalloc(size: uint32): void;
```
Performs a large kernel allocation of `size` bytes and returns a pointer to the allocated memory.
### klfree
```pascal
procedure klfree(address: uint32);
```
Frees a large allocation previously obtained via `klalloc`.
### kpalloc
```pascal
function kpalloc(address: uint32): void;
```
Allocates a heap page at the specified address and returns a pointer to it.
### kfree
```pascal
procedure kfree(area: void);
```
Frees a standard allocation previously obtained via `kalloc`.
### lmm_total_free
```pascal
function lmm_total_free: uint32;
```
Returns the total number of free bytes available across all heap pages.
### lmm_page_count
```pascal
function lmm_page_count: uint32;
```
Returns the current number of heap pages managed by the allocator.
## Notes
- Every function and procedure in the implementation section is marked `inline`, so the compiler eliminates the wrapper overhead entirely.
- This unit exists solely for backward compatibility with the Wasuro WASM VM build. New kernel code should use `memory.heap` directly.
-16
View File
@@ -1,16 +0,0 @@
# types
Empty compatibility shim that satisfies `uses types` references from the Wasuro WASM VM project.
## Overview
Some units in the Wasuro WASM VM source tree include `uses types` to pull in shared type definitions from the Asuro kernel. In the kernel proper, those types may be declared elsewhere or may no longer be needed in the WASM context. This stub unit provides an empty `types` compilation unit so that `uses types` resolves without error during the Wasuro build.
## Dependencies
None.
## Notes
- The unit declares no constants, types, variables, or routines. Its only purpose is to exist as a valid compilation unit.
- If Wasuro code is ever updated to remove the `uses types` dependency, this shim can be deleted.
-32
View File
@@ -1,32 +0,0 @@
# Asuro OS
Asuro is a 32-bit x86 operating system kernel written in Free Pascal and x86 assembly.
## Documentation Structure
- **Kernel Entry** -- The `asuro.pas` main unit and boot sequence.
- **Architecture (x86)** -- CPU initialization, descriptor tables, interrupt handling, fault handlers, memory management, and process scheduling for the i386 target.
- **Boot** -- Splash screen and early boot visuals.
- **Core** -- Foundational libraries including data structures, encoding algorithms, string handling, graphics primitives, and the kernel panic subsystem.
- **Memory** -- Heap allocator (`kalloc`/`kfree`) and page-level allocation.
- **Processes** -- Process lifecycle, round-robin scheduling, and inter-process messaging.
- **I/O** -- Standard I/O (shell command dispatch) and the system log fan-out.
- **Debug** -- Execution tracer with ring-buffer call stack recording.
- **Drivers** -- Hardware abstraction covering PCI/USB buses, HID devices, networking (E1000, TCP/IP stack), storage (IDE, AHCI, VFS, file systems), video (VESA, LVGL, double-buffering), serial I/O, and timers.
- **Services** -- Background daemons for graphics rendering and USB hotplug.
- **Applications** -- Userland commands and utilities: terminal, text editor, disk tools, network tools, and a WebAssembly runtime.
- **Compatibility** -- Shim layers for legacy code.
- **LVGL Headers** -- Configuration and patches for the LVGL v9.2.2 GUI library.
- **Planning** -- Design documents and architectural notes for subsystems under development.
- **Toolchain** -- Build pipeline documentation covering compilation, linking, and ISO generation.
## Building
The kernel is built inside Docker using a containerized FreePascal 3.2.2 toolchain:
```bash
docker compose build builder
docker compose run builder
```
The documentation site will be available at `http://docs.asuro.xyz`.

Some files were not shown because too many files have changed in this diff Show More