|
|
|
# Tutorial 16 - Virtual Memory Part 4: Higher-Half Kernel
|
|
|
|
|
|
|
|
## tl;dr
|
|
|
|
|
|
|
|
- The time has come: We map and run the kernel from the top of the 64 bit virtual address space! 🥳
|
|
|
|
|
|
|
|
## Table of Contents
|
|
|
|
|
|
|
|
- [Introduction](#introduction)
|
|
|
|
- [Implementation](#implementation)
|
|
|
|
- [Position-Independent Boot Code](#position-independent-boot-code)
|
|
|
|
- [Test it](#test-it)
|
|
|
|
- [Diff to previous](#diff-to-previous)
|
|
|
|
|
|
|
|
## Introduction
|
|
|
|
|
|
|
|
A long time in the making, in this tutorial we finally map the kernel to the most significant area
|
|
|
|
(alternatively: higher-half) of the 64 bit virtual address space. This makes room for future
|
|
|
|
applications to use the whole of the least significant area of the virtual memory space.
|
|
|
|
|
|
|
|
As has been teased since `tutorial 14`, we will make use of the `AArch64`'s `TTBR1`. Since the
|
|
|
|
kernel's virtual address space size is `2 GiB` since the last tutorial, `TTBR1` will cover the range
|
|
|
|
from `0xffff_ffff_ffff_ffff` down to `ffff_ffff_8000_0000` (both inclusive).
|
|
|
|
|
|
|
|
## Implementation
|
|
|
|
|
|
|
|
In `src/memory/mmu.rs`, we extend the `AssociatedTranslationTable` trait with a `TableStartFromTop`
|
|
|
|
associated type:
|
|
|
|
|
|
|
|
```rust
|
|
|
|
/// Intended to be implemented for [`AddressSpace`].
|
|
|
|
pub trait AssociatedTranslationTable {
|
|
|
|
/// A translation table whose address range is:
|
|
|
|
///
|
|
|
|
/// [u64::MAX, (u64::MAX - AS_SIZE) + 1]
|
|
|
|
type TableStartFromTop;
|
|
|
|
|
|
|
|
/// A translation table whose address range is:
|
|
|
|
///
|
|
|
|
/// [AS_SIZE - 1, 0]
|
|
|
|
type TableStartFromBottom;
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
Architecture specific code in `_arch/aarch64/memory/mmu/translation_table.rs` populates both types
|
|
|
|
now by making use of a new generic that is added to `FixedSizeTranslationTable`, which defines
|
|
|
|
whether the covered address space starts from the top or the bottom:
|
|
|
|
|
|
|
|
```rust
|
|
|
|
pub struct FixedSizeTranslationTable<const NUM_TABLES: usize, const START_FROM_TOP: bool> {
|
|
|
|
...
|
|
|
|
```
|
|
|
|
|
|
|
|
```rust
|
|
|
|
impl<const AS_SIZE: usize> memory::mmu::AssociatedTranslationTable
|
|
|
|
for memory::mmu::AddressSpace<AS_SIZE>
|
|
|
|
where
|
|
|
|
[u8; Self::SIZE >> Granule512MiB::SHIFT]: Sized,
|
|
|
|
{
|
|
|
|
type TableStartFromTop =
|
|
|
|
FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, true>;
|
|
|
|
|
|
|
|
type TableStartFromBottom =
|
|
|
|
FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, false>;
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
Thanks to this infrastructure, `BSP` Rust code in `bsp/raspberrypi/memory/mmu.rs` only needs to
|
|
|
|
change to this newly introduced type in order to switch from lower half to higher half translation
|
|
|
|
tables for the kernel:
|
|
|
|
|
|
|
|
```rust
|
|
|
|
type KernelTranslationTable =
|
|
|
|
<KernelVirtAddrSpace as AssociatedTranslationTable>::TableStartFromTop;
|
|
|
|
```
|
|
|
|
|
|
|
|
In the `link.ld` linker script, we define a new symbol `__kernel_virt_start_addr` now, which is the
|
|
|
|
start address of the kernel's virtual address space, calculated as `(u64::MAX -
|
|
|
|
__kernel_virt_addr_space_size) + 1`. In order to make virtual-to-physical address translation easier
|
|
|
|
for the human eye (and mind), we link the kernel itself at `__kernel_virt_start_addr +
|
|
|
|
__rpi_load_addr`.
|
|
|
|
|
|
|
|
Before these tutorials, the first mapped address of the kernel binary was always located at
|
|
|
|
`__rpi_load_addr == 0x8_0000`. Starting with this tutorial, due to the `2 GiB` virtual address space
|
|
|
|
size, the new first mapped address is `ffff_ffff_8008_0000`. So by ignoring the upper bits of the
|
|
|
|
address, you can easily derive the physical address.
|
|
|
|
|
|
|
|
The changes in the `_arch` `MMU` driver are minimal, and mostly concerned with configuring `TCR_EL1`
|
|
|
|
for use with `TTBR1_EL1` now. And of course, setting `TTBR1_EL1` in `fn
|
|
|
|
enable_mmu_and_caching(...)`.
|
|
|
|
|
|
|
|
### Position-Independent Boot Code
|
|
|
|
|
|
|
|
Remember all the fuss that we made about `position-independent code` that will be needed until the
|
|
|
|
`MMU` is enabled. Let's quickly check what it means for us in reality now:
|
|
|
|
|
|
|
|
In `_arch/aarch64/cpu/boot.rs`, we turn on the `MMU` just before returning from `EL2` to `EL1`. So
|
|
|
|
by the time the CPU enters `EL1`, virtual memory will be active, and the CPU must therefore use the
|
|
|
|
new higher-half `virtual addresses` for everything it does.
|
|
|
|
|
|
|
|
Specifically, this means the address from which the CPU should execute upon entering `EL1` (function
|
|
|
|
`kernel_init()`) must be a valid _virtual address_, same as the stack pointer's address. Both of
|
|
|
|
them are programmed in function `fn prepare_el2_to_el1_transition(...)`, so we must ensure now that
|
|
|
|
_link-time_ addresses are used here. For this reason, retrieval of these addresses happens in
|
|
|
|
`assembly` in `boot.s`, where we can explicitly enforce generation of **absolute** addresses:
|
|
|
|
|
|
|
|
```asm
|
|
|
|
// Load the _absolute_ addresses of the following symbols. Since the kernel is linked at
|
|
|
|
// the top of the 64 bit address space, these are effectively virtual addresses.
|
|
|
|
ADR_ABS x1, __boot_core_stack_end_exclusive
|
|
|
|
ADR_ABS x2, kernel_init
|
|
|
|
```
|
|
|
|
|
|
|
|
Both values are forwarded to the Rust entry point function `_start_rust()`, which in turn forwards
|
|
|
|
them to `fn prepare_el2_to_el1_transition(...)`.
|
|
|
|
|
|
|
|
One more thing to consider is that we keep on programming the boot core's stack address for `EL2`
|
|
|
|
using an address that is calculated `PC-relative`, because all the `EL2` code will still run while
|
|
|
|
virtual memory _is disabled_. As such, we need the "physical" address of the stack, so to speak.
|
|
|
|
|
|
|
|
The previous tutorial also explained that it is not easily possible to compile select files using
|
|
|
|
`-fpic` in `Rust`. Still, we are doing some function calls in `Rust` before virtual memory is
|
|
|
|
enabled, so _theoretically_, there is room for failure. However, branches to local code in `AArch64`
|
|
|
|
are usually generated PC-relative. So it is a small risk worth taking. Should it still fail someday,
|
|
|
|
at least our automated CI pipeline would give notice when the tests start to fail.
|
|
|
|
|
|
|
|
## Test it
|
|
|
|
|
|
|
|
That's it! We are ready for a higher-half kernel now. Power up your Raspberrys and marvel at those
|
|
|
|
beautiful (virtual) addresses:
|
|
|
|
|
|
|
|
Raspberry Pi 3:
|
|
|
|
|
|
|
|
```console
|
|
|
|
$ make chainboot
|
|
|
|
[...]
|
|
|
|
|
|
|
|
Precomputing kernel translation tables and patching kernel ELF
|
|
|
|
--------------------------------------------------
|
|
|
|
Section Start Virt Addr Size
|
|
|
|
--------------------------------------------------
|
|
|
|
Generating Code and RO data | 0xffff_ffff_8008_0000 | 64 KiB
|
|
|
|
Generating Data and bss | 0xffff_ffff_8009_0000 | 384 KiB
|
|
|
|
Generating Boot-core stack | 0xffff_ffff_8010_0000 | 512 KiB
|
|
|
|
--------------------------------------------------
|
|
|
|
Patching Kernel table struct at physical 0x9_0000
|
|
|
|
Patching Value of kernel table physical base address (0xd_0000) at physical 0x8_0060
|
|
|
|
Finished in 0.03s
|
|
|
|
|
|
|
|
Minipush 1.0
|
|
|
|
|
|
|
|
[MP] ⏳ Waiting for /dev/ttyUSB0
|
|
|
|
[MP] ✅ Serial connected
|
|
|
|
[MP] 🔌 Please power the target now
|
|
|
|
__ __ _ _ _ _
|
|
|
|
| \/ (_)_ _ (_) | ___ __ _ __| |
|
|
|
|
| |\/| | | ' \| | |__/ _ \/ _` / _` |
|
|
|
|
|_| |_|_|_||_|_|____\___/\__,_\__,_|
|
|
|
|
|
|
|
|
Raspberry Pi 3
|
|
|
|
|
|
|
|
[ML] Requesting binary
|
|
|
|
[MP] ⏩ Pushing 387 KiB =======================================🦀 100% 96 KiB/s Time: 00:00:04
|
|
|
|
[ML] Loaded! Executing the payload now
|
|
|
|
|
|
|
|
[ 4.316420] mingo version 0.16.0
|
|
|
|
[ 4.316627] Booting on: Raspberry Pi 3
|
|
|
|
[ 4.317082] MMU online:
|
|
|
|
[ 4.317375] -------------------------------------------------------------------------------------------------------------------------------------------
|
|
|
|
[ 4.319119] Virtual Physical Size Attr Entity
|
|
|
|
[ 4.320863] -------------------------------------------------------------------------------------------------------------------------------------------
|
|
|
|
[ 4.322610] 0xffff_ffff_8008_0000..0xffff_ffff_8008_ffff --> 0x00_0008_0000..0x00_0008_ffff | 64 KiB | C RO X | Kernel code and RO data
|
|
|
|
[ 4.324223] 0xffff_ffff_8009_0000..0xffff_ffff_800e_ffff --> 0x00_0009_0000..0x00_000e_ffff | 384 KiB | C RW XN | Kernel data and bss
|
|
|
|
[ 4.325793] 0xffff_ffff_8010_0000..0xffff_ffff_8017_ffff --> 0x00_0010_0000..0x00_0017_ffff | 512 KiB | C RW XN | Kernel boot-core stack
|
|
|
|
[ 4.327397] 0xffff_ffff_f000_0000..0xffff_ffff_f000_ffff --> 0x00_3f20_0000..0x00_3f20_ffff | 64 KiB | Dev RW XN | BCM GPIO
|
|
|
|
[ 4.328847] | BCM PL011 UART
|
|
|
|
[ 4.330365] 0xffff_ffff_f001_0000..0xffff_ffff_f001_ffff --> 0x00_3f00_0000..0x00_3f00_ffff | 64 KiB | Dev RW XN | BCM Peripheral Interrupt Controller
|
|
|
|
[ 4.332108] -------------------------------------------------------------------------------------------------------------------------------------------
|
|
|
|
```
|
|
|
|
|
|
|
|
Raspberry Pi 4:
|
|
|
|
|
|
|
|
```console
|
|
|
|
$ BSP=rpi4 make chainboot
|
|
|
|
[...]
|
|
|
|
|
|
|
|
Precomputing kernel translation tables and patching kernel ELF
|
|
|
|
--------------------------------------------------
|
|
|
|
Section Start Virt Addr Size
|
|
|
|
--------------------------------------------------
|
|
|
|
Generating Code and RO data | 0xffff_ffff_8008_0000 | 64 KiB
|
|
|
|
Generating Data and bss | 0xffff_ffff_8009_0000 | 448 KiB
|
|
|
|
Generating Boot-core stack | 0xffff_ffff_8011_0000 | 512 KiB
|
|
|
|
--------------------------------------------------
|
|
|
|
Patching Kernel table struct at physical 0xa_0000
|
|
|
|
Patching Value of kernel table physical base address (0xe_0000) at physical 0x8_0068
|
|
|
|
Finished in 0.03s
|
|
|
|
|
|
|
|
Minipush 1.0
|
|
|
|
|
|
|
|
[MP] ⏳ Waiting for /dev/ttyUSB0
|
|
|
|
[MP] ✅ Serial connected
|
|
|
|
[MP] 🔌 Please power the target now
|
|
|
|
__ __ _ _ _ _
|
|
|
|
| \/ (_)_ _ (_) | ___ __ _ __| |
|
|
|
|
| |\/| | | ' \| | |__/ _ \/ _` / _` |
|
|
|
|
|_| |_|_|_||_|_|____\___/\__,_\__,_|
|
|
|
|
|
|
|
|
Raspberry Pi 4
|
|
|
|
|
|
|
|
[ML] Requesting binary
|
|
|
|
[MP] ⏩ Pushing 449 KiB ======================================🦀 100% 112 KiB/s Time: 00:00:04
|
|
|
|
[ML] Loaded! Executing the payload now
|
|
|
|
|
|
|
|
[ 5.009551] mingo version 0.16.0
|
|
|
|
[ 5.009585] Booting on: Raspberry Pi 4
|
|
|
|
[ 5.010040] MMU online:
|
|
|
|
[ 5.010332] -------------------------------------------------------------------------------------------------------------------------------------------
|
|
|
|
[ 5.012077] Virtual Physical Size Attr Entity
|
|
|
|
[ 5.013821] -------------------------------------------------------------------------------------------------------------------------------------------
|
|
|
|
[ 5.015566] 0xffff_ffff_8008_0000..0xffff_ffff_8008_ffff --> 0x00_0008_0000..0x00_0008_ffff | 64 KiB | C RO X | Kernel code and RO data
|
|
|
|
[ 5.017181] 0xffff_ffff_8009_0000..0xffff_ffff_800f_ffff --> 0x00_0009_0000..0x00_000f_ffff | 448 KiB | C RW XN | Kernel data and bss
|
|
|
|
[ 5.018751] 0xffff_ffff_8011_0000..0xffff_ffff_8018_ffff --> 0x00_0011_0000..0x00_0018_ffff | 512 KiB | C RW XN | Kernel boot-core stack
|
|
|
|
[ 5.020354] 0xffff_ffff_f000_0000..0xffff_ffff_f000_ffff --> 0x00_fe20_0000..0x00_fe20_ffff | 64 KiB | Dev RW XN | BCM GPIO
|
|
|
|
[ 5.021805] | BCM PL011 UART
|
|
|
|
[ 5.023322] 0xffff_ffff_f001_0000..0xffff_ffff_f001_ffff --> 0x00_ff84_0000..0x00_ff84_ffff | 64 KiB | Dev RW XN | GICD
|
|
|
|
[ 5.024730] | GICC
|
|
|
|
[ 5.026138] -------------------------------------------------------------------------------------------------------------------------------------------
|
|
|
|
```
|
|
|
|
|
|
|
|
## Diff to previous
|
|
|
|
```diff
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/Cargo.toml 16_virtual_mem_part4_higher_half_kernel/Cargo.toml
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/Cargo.toml
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/Cargo.toml
|
|
|
|
@@ -1,6 +1,6 @@
|
|
|
|
[package]
|
|
|
|
name = "mingo"
|
|
|
|
-version = "0.15.0"
|
|
|
|
+version = "0.16.0"
|
|
|
|
authors = ["Andre Richter <andre.o.richter@gmail.com>"]
|
|
|
|
edition = "2018"
|
|
|
|
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/cpu/boot.rs 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/cpu/boot.rs
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/cpu/boot.rs
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/cpu/boot.rs
|
|
|
|
@@ -30,7 +30,10 @@
|
|
|
|
/// - The `bss` section is not initialized yet. The code must not use or reference it in any way.
|
|
|
|
/// - The HW state of EL1 must be prepared in a sound way.
|
|
|
|
#[inline(always)]
|
|
|
|
-unsafe fn prepare_el2_to_el1_transition(phys_boot_core_stack_end_exclusive_addr: u64) {
|
|
|
|
+unsafe fn prepare_el2_to_el1_transition(
|
|
|
|
+ virt_boot_core_stack_end_exclusive_addr: u64,
|
|
|
|
+ virt_kernel_init_addr: u64,
|
|
|
|
+) {
|
|
|
|
// Enable timer counter registers for EL1.
|
|
|
|
CNTHCTL_EL2.write(CNTHCTL_EL2::EL1PCEN::SET + CNTHCTL_EL2::EL1PCTEN::SET);
|
|
|
|
|
|
|
|
@@ -53,11 +56,11 @@
|
|
|
|
);
|
|
|
|
|
|
|
|
// Second, let the link register point to kernel_init().
|
|
|
|
- ELR_EL2.set(crate::kernel_init as *const () as u64);
|
|
|
|
+ ELR_EL2.set(virt_kernel_init_addr);
|
|
|
|
|
|
|
|
// Set up SP_EL1 (stack pointer), which will be used by EL1 once we "return" to it. Since there
|
|
|
|
// are no plans to ever return to EL2, just re-use the same stack.
|
|
|
|
- SP_EL1.set(phys_boot_core_stack_end_exclusive_addr);
|
|
|
|
+ SP_EL1.set(virt_boot_core_stack_end_exclusive_addr);
|
|
|
|
}
|
|
|
|
|
|
|
|
//--------------------------------------------------------------------------------------------------
|
|
|
|
@@ -74,9 +77,13 @@
|
|
|
|
#[no_mangle]
|
|
|
|
pub unsafe extern "C" fn _start_rust(
|
|
|
|
phys_kernel_tables_base_addr: u64,
|
|
|
|
- phys_boot_core_stack_end_exclusive_addr: u64,
|
|
|
|
+ virt_boot_core_stack_end_exclusive_addr: u64,
|
|
|
|
+ virt_kernel_init_addr: u64,
|
|
|
|
) -> ! {
|
|
|
|
- prepare_el2_to_el1_transition(phys_boot_core_stack_end_exclusive_addr);
|
|
|
|
+ prepare_el2_to_el1_transition(
|
|
|
|
+ virt_boot_core_stack_end_exclusive_addr,
|
|
|
|
+ virt_kernel_init_addr,
|
|
|
|
+ );
|
|
|
|
|
|
|
|
// Turn on the MMU for EL1.
|
|
|
|
let addr = Address::new(phys_kernel_tables_base_addr as usize);
|
|
|
|
@@ -84,6 +91,7 @@
|
|
|
|
cpu::wait_forever();
|
|
|
|
}
|
|
|
|
|
|
|
|
- // Use `eret` to "return" to EL1. This results in execution of kernel_init() in EL1.
|
|
|
|
+ // Use `eret` to "return" to EL1. Since virtual memory will already be enabled, this results in
|
|
|
|
+ // execution of kernel_init() in EL1 from its _virtual address_.
|
|
|
|
asm::eret()
|
|
|
|
}
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/cpu/boot.s 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/cpu/boot.s
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/cpu/boot.s
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/cpu/boot.s
|
|
|
|
@@ -18,6 +18,18 @@
|
|
|
|
add \register, \register, #:lo12:\symbol
|
|
|
|
.endm
|
|
|
|
|
|
|
|
+// Load the address of a symbol into a register, absolute.
|
|
|
|
+//
|
|
|
|
+// # Resources
|
|
|
|
+//
|
|
|
|
+// - https://sourceware.org/binutils/docs-2.36/as/AArch64_002dRelocations.html
|
|
|
|
+.macro ADR_ABS register, symbol
|
|
|
|
+ movz \register, #:abs_g3:\symbol
|
|
|
|
+ movk \register, #:abs_g2_nc:\symbol
|
|
|
|
+ movk \register, #:abs_g1_nc:\symbol
|
|
|
|
+ movk \register, #:abs_g0_nc:\symbol
|
|
|
|
+.endm
|
|
|
|
+
|
|
|
|
.equ _EL2, 0x8
|
|
|
|
.equ _core_id_mask, 0b11
|
|
|
|
|
|
|
|
@@ -59,11 +71,23 @@
|
|
|
|
// Load the base address of the kernel's translation tables.
|
|
|
|
ldr x0, PHYS_KERNEL_TABLES_BASE_ADDR // provided by bsp/__board_name__/memory/mmu.rs
|
|
|
|
|
|
|
|
- // Set the stack pointer. This ensures that any code in EL2 that needs the stack will work.
|
|
|
|
- ADR_REL x1, __boot_core_stack_end_exclusive
|
|
|
|
- mov sp, x1
|
|
|
|
+ // Load the _absolute_ addresses of the following symbols. Since the kernel is linked at
|
|
|
|
+ // the top of the 64 bit address space, these are effectively virtual addresses.
|
|
|
|
+ ADR_ABS x1, __boot_core_stack_end_exclusive
|
|
|
|
+ ADR_ABS x2, kernel_init
|
|
|
|
+
|
|
|
|
+ // Load the PC-relative address of the stack and set the stack pointer.
|
|
|
|
+ //
|
|
|
|
+ // Since _start() is the first function that runs after the firmware has loaded the kernel
|
|
|
|
+ // into memory, retrieving this symbol PC-relative returns the "physical" address.
|
|
|
|
+ //
|
|
|
|
+ // Setting the stack pointer to this value ensures that anything that still runs in EL2,
|
|
|
|
+ // until the kernel returns to EL1 with the MMU enabled, works as well. After the return to
|
|
|
|
+ // EL1, the virtual address of the stack retrieved above will be used.
|
|
|
|
+ ADR_REL x4, __boot_core_stack_end_exclusive
|
|
|
|
+ mov sp, x4
|
|
|
|
|
|
|
|
- // Jump to Rust code. x0 and x1 hold the function arguments provided to _start_rust().
|
|
|
|
+ // Jump to Rust code. x0, x1 and x2 hold the function arguments provided to _start_rust().
|
|
|
|
b _start_rust
|
|
|
|
|
|
|
|
// Infinitely wait for events (aka "park the core").
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/memory/mmu/translation_table.rs 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/memory/mmu/translation_table.rs
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/memory/mmu/translation_table.rs
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/memory/mmu/translation_table.rs
|
|
|
|
@@ -135,7 +135,7 @@
|
|
|
|
/// aligned, so the lvl3 is put first.
|
|
|
|
#[repr(C)]
|
|
|
|
#[repr(align(65536))]
|
|
|
|
-pub struct FixedSizeTranslationTable<const NUM_TABLES: usize> {
|
|
|
|
+pub struct FixedSizeTranslationTable<const NUM_TABLES: usize, const START_FROM_TOP: bool> {
|
|
|
|
/// Page descriptors, covering 64 KiB windows per entry.
|
|
|
|
lvl3: [[PageDescriptor; 8192]; NUM_TABLES],
|
|
|
|
|
|
|
|
@@ -262,14 +262,23 @@
|
|
|
|
where
|
|
|
|
[u8; Self::SIZE >> Granule512MiB::SHIFT]: Sized,
|
|
|
|
{
|
|
|
|
- type TableStartFromBottom = FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }>;
|
|
|
|
+ type TableStartFromTop =
|
|
|
|
+ FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, true>;
|
|
|
|
+
|
|
|
|
+ type TableStartFromBottom =
|
|
|
|
+ FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, false>;
|
|
|
|
}
|
|
|
|
|
|
|
|
-impl<const NUM_TABLES: usize> FixedSizeTranslationTable<NUM_TABLES> {
|
|
|
|
+impl<const NUM_TABLES: usize, const START_FROM_TOP: bool>
|
|
|
|
+ FixedSizeTranslationTable<NUM_TABLES, START_FROM_TOP>
|
|
|
|
+{
|
|
|
|
// Reserve the last 256 MiB of the address space for MMIO mappings.
|
|
|
|
const L2_MMIO_START_INDEX: usize = NUM_TABLES - 1;
|
|
|
|
const L3_MMIO_START_INDEX: usize = 8192 / 2;
|
|
|
|
|
|
|
|
+ const START_FROM_TOP_OFFSET: Address<Virtual> =
|
|
|
|
+ Address::new((usize::MAX - (Granule512MiB::SIZE * NUM_TABLES)) + 1);
|
|
|
|
+
|
|
|
|
/// Create an instance.
|
|
|
|
#[allow(clippy::assertions_on_constants)]
|
|
|
|
const fn _new(for_precompute: bool) -> Self {
|
|
|
|
@@ -298,20 +307,32 @@
|
|
|
|
/// The start address of the table's MMIO range.
|
|
|
|
#[inline(always)]
|
|
|
|
fn mmio_start_addr(&self) -> Address<Virtual> {
|
|
|
|
- Address::new(
|
|
|
|
+ let mut addr = Address::new(
|
|
|
|
(Self::L2_MMIO_START_INDEX << Granule512MiB::SHIFT)
|
|
|
|
| (Self::L3_MMIO_START_INDEX << Granule64KiB::SHIFT),
|
|
|
|
- )
|
|
|
|
+ );
|
|
|
|
+
|
|
|
|
+ if START_FROM_TOP {
|
|
|
|
+ addr += Self::START_FROM_TOP_OFFSET;
|
|
|
|
+ }
|
|
|
|
+
|
|
|
|
+ addr
|
|
|
|
}
|
|
|
|
|
|
|
|
/// The inclusive end address of the table's MMIO range.
|
|
|
|
#[inline(always)]
|
|
|
|
fn mmio_end_addr_inclusive(&self) -> Address<Virtual> {
|
|
|
|
- Address::new(
|
|
|
|
+ let mut addr = Address::new(
|
|
|
|
(Self::L2_MMIO_START_INDEX << Granule512MiB::SHIFT)
|
|
|
|
| (8191 << Granule64KiB::SHIFT)
|
|
|
|
| (Granule64KiB::SIZE - 1),
|
|
|
|
- )
|
|
|
|
+ );
|
|
|
|
+
|
|
|
|
+ if START_FROM_TOP {
|
|
|
|
+ addr += Self::START_FROM_TOP_OFFSET;
|
|
|
|
+ }
|
|
|
|
+
|
|
|
|
+ addr
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Helper to calculate the lvl2 and lvl3 indices from an address.
|
|
|
|
@@ -320,7 +341,12 @@
|
|
|
|
&self,
|
|
|
|
addr: *const Page<Virtual>,
|
|
|
|
) -> Result<(usize, usize), &'static str> {
|
|
|
|
- let addr = addr as usize;
|
|
|
|
+ let mut addr = addr as usize;
|
|
|
|
+
|
|
|
|
+ if START_FROM_TOP {
|
|
|
|
+ addr -= Self::START_FROM_TOP_OFFSET.into_usize()
|
|
|
|
+ }
|
|
|
|
+
|
|
|
|
let lvl2_index = addr >> Granule512MiB::SHIFT;
|
|
|
|
let lvl3_index = (addr & Granule512MiB::MASK) >> Granule64KiB::SHIFT;
|
|
|
|
|
|
|
|
@@ -347,8 +373,9 @@
|
|
|
|
// OS Interface Code
|
|
|
|
//------------------------------------------------------------------------------
|
|
|
|
|
|
|
|
-impl<const NUM_TABLES: usize> memory::mmu::translation_table::interface::TranslationTable
|
|
|
|
- for FixedSizeTranslationTable<NUM_TABLES>
|
|
|
|
+impl<const NUM_TABLES: usize, const START_FROM_TOP: bool>
|
|
|
|
+ memory::mmu::translation_table::interface::TranslationTable
|
|
|
|
+ for FixedSizeTranslationTable<NUM_TABLES, START_FROM_TOP>
|
|
|
|
{
|
|
|
|
fn init(&mut self) -> Result<(), &'static str> {
|
|
|
|
if self.initialized {
|
|
|
|
@@ -423,12 +450,16 @@
|
|
|
|
return Err("Not enough MMIO space left");
|
|
|
|
}
|
|
|
|
|
|
|
|
- let addr = Address::new(
|
|
|
|
+ let mut addr = Address::new(
|
|
|
|
(Self::L2_MMIO_START_INDEX << Granule512MiB::SHIFT)
|
|
|
|
| (self.cur_l3_mmio_index << Granule64KiB::SHIFT),
|
|
|
|
);
|
|
|
|
self.cur_l3_mmio_index += num_pages;
|
|
|
|
|
|
|
|
+ if START_FROM_TOP {
|
|
|
|
+ addr += Self::START_FROM_TOP_OFFSET;
|
|
|
|
+ }
|
|
|
|
+
|
|
|
|
Ok(PageSliceDescriptor::from_addr(addr, num_pages))
|
|
|
|
}
|
|
|
|
|
|
|
|
@@ -451,7 +482,7 @@
|
|
|
|
//--------------------------------------------------------------------------------------------------
|
|
|
|
|
|
|
|
#[cfg(test)]
|
|
|
|
-pub type MinSizeTranslationTable = FixedSizeTranslationTable<1>;
|
|
|
|
+pub type MinSizeTranslationTable = FixedSizeTranslationTable<1, false>;
|
|
|
|
|
|
|
|
#[cfg(test)]
|
|
|
|
mod tests {
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/memory/mmu.rs
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/_arch/aarch64/memory/mmu.rs
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/_arch/aarch64/memory/mmu.rs
|
|
|
|
@@ -66,6 +66,7 @@
|
|
|
|
|
|
|
|
impl MemoryManagementUnit {
|
|
|
|
/// Setup function for the MAIR_EL1 register.
|
|
|
|
+ #[inline(always)]
|
|
|
|
fn set_up_mair(&self) {
|
|
|
|
// Define the memory types being mapped.
|
|
|
|
MAIR_EL1.write(
|
|
|
|
@@ -79,20 +80,21 @@
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Configure various settings of stage 1 of the EL1 translation regime.
|
|
|
|
+ #[inline(always)]
|
|
|
|
fn configure_translation_control(&self) {
|
|
|
|
- let t0sz = (64 - bsp::memory::mmu::KernelVirtAddrSpace::SIZE_SHIFT) as u64;
|
|
|
|
+ let t1sz = (64 - bsp::memory::mmu::KernelVirtAddrSpace::SIZE_SHIFT) as u64;
|
|
|
|
|
|
|
|
TCR_EL1.write(
|
|
|
|
- TCR_EL1::TBI0::Used
|
|
|
|
+ TCR_EL1::TBI1::Used
|
|
|
|
+ TCR_EL1::IPS::Bits_40
|
|
|
|
- + TCR_EL1::TG0::KiB_64
|
|
|
|
- + TCR_EL1::SH0::Inner
|
|
|
|
- + TCR_EL1::ORGN0::WriteBack_ReadAlloc_WriteAlloc_Cacheable
|
|
|
|
- + TCR_EL1::IRGN0::WriteBack_ReadAlloc_WriteAlloc_Cacheable
|
|
|
|
- + TCR_EL1::EPD0::EnableTTBR0Walks
|
|
|
|
- + TCR_EL1::A1::TTBR0
|
|
|
|
- + TCR_EL1::T0SZ.val(t0sz)
|
|
|
|
- + TCR_EL1::EPD1::DisableTTBR1Walks,
|
|
|
|
+ + TCR_EL1::TG1::KiB_64
|
|
|
|
+ + TCR_EL1::SH1::Inner
|
|
|
|
+ + TCR_EL1::ORGN1::WriteBack_ReadAlloc_WriteAlloc_Cacheable
|
|
|
|
+ + TCR_EL1::IRGN1::WriteBack_ReadAlloc_WriteAlloc_Cacheable
|
|
|
|
+ + TCR_EL1::EPD1::EnableTTBR1Walks
|
|
|
|
+ + TCR_EL1::A1::TTBR1
|
|
|
|
+ + TCR_EL1::T1SZ.val(t1sz)
|
|
|
|
+ + TCR_EL1::EPD0::DisableTTBR0Walks,
|
|
|
|
);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
@@ -131,7 +133,7 @@
|
|
|
|
self.set_up_mair();
|
|
|
|
|
|
|
|
// Set the "Translation Table Base Register".
|
|
|
|
- TTBR0_EL1.set_baddr(phys_tables_base_addr.into_usize() as u64);
|
|
|
|
+ TTBR1_EL1.set_baddr(phys_tables_base_addr.into_usize() as u64);
|
|
|
|
|
|
|
|
self.configure_translation_control();
|
|
|
|
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/bsp/raspberrypi/link.ld 16_virtual_mem_part4_higher_half_kernel/src/bsp/raspberrypi/link.ld
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/bsp/raspberrypi/link.ld
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/bsp/raspberrypi/link.ld
|
|
|
|
@@ -6,6 +6,15 @@
|
|
|
|
/* This file provides __kernel_virt_addr_space_size */
|
|
|
|
INCLUDE src/bsp/raspberrypi/kernel_virt_addr_space_size.ld;
|
|
|
|
|
|
|
|
+/* The kernel's virtual address range will be:
|
|
|
|
+ *
|
|
|
|
+ * [END_ADDRESS_INCLUSIVE, START_ADDRESS]
|
|
|
|
+ * [u64::MAX , (u64::MAX - __kernel_virt_addr_space_size) + 1]
|
|
|
|
+ *
|
|
|
|
+ * Since the start address is needed to set the linker address below, calculate it now.
|
|
|
|
+ */
|
|
|
|
+__kernel_virt_start_addr = ((0xffffffffffffffff - __kernel_virt_addr_space_size) + 1);
|
|
|
|
+
|
|
|
|
/* The address at which the the kernel binary will be loaded by the Raspberry's firmware */
|
|
|
|
__rpi_load_addr = 0x80000;
|
|
|
|
|
|
|
|
@@ -19,13 +28,14 @@
|
|
|
|
|
|
|
|
SECTIONS
|
|
|
|
{
|
|
|
|
- . = __rpi_load_addr;
|
|
|
|
+ /* Add the load address as an offset. Makes virt-to-phys translation easier for the human eye */
|
|
|
|
+ . = __kernel_virt_start_addr + __rpi_load_addr;
|
|
|
|
|
|
|
|
/***********************************************************************************************
|
|
|
|
* Code + RO Data + Global Offset Table
|
|
|
|
***********************************************************************************************/
|
|
|
|
__rx_start = .;
|
|
|
|
- .text :
|
|
|
|
+ .text : AT(__rpi_load_addr)
|
|
|
|
{
|
|
|
|
KEEP(*(.text._start))
|
|
|
|
*(.text._start_arguments) /* Constants (or statics in Rust speak) read by _start(). */
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/bsp/raspberrypi/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/src/bsp/raspberrypi/memory/mmu.rs
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/bsp/raspberrypi/memory/mmu.rs
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/bsp/raspberrypi/memory/mmu.rs
|
|
|
|
@@ -23,7 +23,7 @@
|
|
|
|
//--------------------------------------------------------------------------------------------------
|
|
|
|
|
|
|
|
type KernelTranslationTable =
|
|
|
|
- <KernelVirtAddrSpace as AssociatedTranslationTable>::TableStartFromBottom;
|
|
|
|
+ <KernelVirtAddrSpace as AssociatedTranslationTable>::TableStartFromTop;
|
|
|
|
|
|
|
|
//--------------------------------------------------------------------------------------------------
|
|
|
|
// Public Definitions
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/lib.rs 16_virtual_mem_part4_higher_half_kernel/src/lib.rs
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/lib.rs
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/lib.rs
|
|
|
|
@@ -152,11 +152,6 @@
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
-#[cfg(not(test))]
|
|
|
|
-extern "Rust" {
|
|
|
|
- fn kernel_init() -> !;
|
|
|
|
-}
|
|
|
|
-
|
|
|
|
//--------------------------------------------------------------------------------------------------
|
|
|
|
// Testing
|
|
|
|
//--------------------------------------------------------------------------------------------------
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/src/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/src/memory/mmu.rs
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/src/memory/mmu.rs
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/src/memory/mmu.rs
|
|
|
|
@@ -80,6 +80,11 @@
|
|
|
|
pub trait AssociatedTranslationTable {
|
|
|
|
/// A translation table whose address range is:
|
|
|
|
///
|
|
|
|
+ /// [u64::MAX, (u64::MAX - AS_SIZE) + 1]
|
|
|
|
+ type TableStartFromTop;
|
|
|
|
+
|
|
|
|
+ /// A translation table whose address range is:
|
|
|
|
+ ///
|
|
|
|
/// [AS_SIZE - 1, 0]
|
|
|
|
type TableStartFromBottom;
|
|
|
|
}
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/tests/02_exception_sync_page_fault.rs 16_virtual_mem_part4_higher_half_kernel/tests/02_exception_sync_page_fault.rs
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/tests/02_exception_sync_page_fault.rs
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/tests/02_exception_sync_page_fault.rs
|
|
|
|
@@ -27,8 +27,8 @@
|
|
|
|
// This line will be printed as the test header.
|
|
|
|
println!("Testing synchronous exception handling by causing a page fault");
|
|
|
|
|
|
|
|
- println!("Writing beyond mapped area to address 9 GiB...");
|
|
|
|
- let big_addr: u64 = 9 * 1024 * 1024 * 1024;
|
|
|
|
+ println!("Writing to bottom of address space to address 1 GiB...");
|
|
|
|
+ let big_addr: u64 = 1 * 1024 * 1024 * 1024;
|
|
|
|
core::ptr::read_volatile(big_addr as *mut u64);
|
|
|
|
|
|
|
|
// If execution reaches here, the memory access above did not cause a page fault exception.
|
|
|
|
|
|
|
|
diff -uNr 15_virtual_mem_part3_precomputed_tables/translation_table_tool/bsp.rb 16_virtual_mem_part4_higher_half_kernel/translation_table_tool/bsp.rb
|
|
|
|
--- 15_virtual_mem_part3_precomputed_tables/translation_table_tool/bsp.rb
|
|
|
|
+++ 16_virtual_mem_part4_higher_half_kernel/translation_table_tool/bsp.rb
|
|
|
|
@@ -31,7 +31,7 @@
|
|
|
|
|
|
|
|
symbols = `#{NM_BINARY} --demangle #{kernel_elf}`.split("\n")
|
|
|
|
@kernel_virt_addr_space_size = parse_from_symbols(symbols, /__kernel_virt_addr_space_size/)
|
|
|
|
- @kernel_virt_start_addr = 0
|
|
|
|
+ @kernel_virt_start_addr = parse_from_symbols(symbols, /__kernel_virt_start_addr/)
|
|
|
|
@virt_addresses = parse_from_symbols(symbols, @virt_addresses)
|
|
|
|
@phys_addresses = virt_to_phys(@virt_addresses)
|
|
|
|
|
|
|
|
```
|