# Tutorial 14 - Virtual Memory Part 2: MMIO Remap ## tl;dr - We introduce a first set of changes which is eventually needed for separating `kernel` and `user` address spaces. - The memory mapping strategy gets more sophisticated as we do away with `identity mapping` the whole of the board's address space. - Instead, only ranges that are actually needed are mapped: - The `kernel binary` stays `identity mapped` for now. - Device `MMIO regions` are remapped lazily (to a special reserved virtual address region) during the device driver's `init()`. ## Table of Contents - [Introduction](#introduction) - [Implementation](#implementation) - [A New Mapping API in `src/memory/mmu.rs`](#a-new-mapping-api-in-srcmemorymmutranslationtablers) - [The new APIs in action](#the-new-apis-in-action) - [MMIO Virtual Address Allocation](#mmio-virtual-address-allocation) - [Supporting Changes](#supporting-changes) - [Test it](#test-it) - [Diff to previous](#diff-to-previous) ## Introduction This tutorial is a first step of many needed for enabling `userspace applications` (which we hopefully will have some day in the very distant future). For this, one of the features we want is a clean separation of `kernel` and `user` address spaces. Fortunately, `ARMv8` has convenient architecture support to realize this. The following text and pictue gives some more motivation and technical information. It is quoted from the _[ARM Cortex-A Series Programmer’s Guide for ARMv8-A], Chapter 12.2, Separation of kernel and application Virtual Address spaces_: > Operating systems typically have a number of applications or tasks running concurrently. Each of > these has its own unique set of translation tables and the kernel switches from one to another as > part of the process of switching context between one task and another. However, much of the memory > system is used only by the kernel and has fixed virtual to Physical Address mappings where the > translation table entries rarely change. The ARMv8 architecture provides a number of features to > efficiently handle this requirement. > > The table base addresses are specified in the Translation Table Base Registers `TTBR0_EL1` and > `TTBR1_EL1`. The translation table pointed to by `TTBR0` is selected when the upper bits of the VA > are all 0. `TTBR1` is selected when the upper bits of the VA are all set to 1. [...] > > Figure 12-4 shows how the kernel space can be mapped to the most significant area of memory and > the Virtual Address space associated with each application mapped to the least significant area of > memory. However, both of these are mapped to a much smaller Physical Address space.

This approach is also sometimes called a "[higher half kernel]". To eventually achieve this separation, this tutorial makes a start by changing the following things: 1. Instead of bulk-`identity mapping` the whole of the board's address space, only the particular parts that are needed will be mapped. 1. For now, the `kernel binary` stays identity mapped. This will be changed in the coming tutorials as it is a quite difficult and peculiar exercise to remap the kernel. 1. Device `MMIO regions` are lazily remapped during a device driver's `init()`. 1. A dedicated region of virtual addresses that we reserve using `BSP` code and the `linker script` is used for this. 1. We keep using `TTBR0` for the kernel translation tables for now. This will be changed when we remap the `kernel binary` in the coming tutorials. [ARM Cortex-A Series Programmer’s Guide for ARMv8-A]: https://developer.arm.com/documentation/den0024/latest/ [higher half kernel]: https://wiki.osdev.org/Higher_Half_Kernel ## Implementation Until now, the whole address space of the board was identity mapped at once. The **architecture** (`src/_arch/_/memory/**`) and **bsp** (`src/bsp/_/memory/**`) parts of the kernel worked together directly while setting up the translation tables, without any indirection through **generic kernel code** (`src/memory/**`). The way it worked was that the `architectural MMU code` would query the `bsp code` about the start and end of the physical address space, and any special regions in this space that need a mapping that _is not_ normal chacheable DRAM. It would then go ahead and map the whole address space at once and never touch the translation tables again during runtime. Changing in this tutorial, **architecture** and **bsp** code will no longer autonomously create the virtual memory mappings. Instead, this is now orchestrated by the kernel's **generic MMU subsystem code**. ### A New Mapping API in `src/memory/mmu/translation_table.rs` First, we define an interface for operating on `translation tables`: ```rust /// Translation table operations. pub trait TranslationTable { /// Anything that needs to run before any of the other provided functions can be used. /// /// # Safety /// /// - Implementor must ensure that this function can run only once or is harmless if invoked /// multiple times. fn init(&mut self); /// The translation table's base address to be used for programming the MMU. fn phys_base_address(&self) -> Address; /// Map the given virtual memory region to the given physical memory region. unsafe fn map_at( &mut self, virt_region: &MemoryRegion, phys_region: &MemoryRegion, attr: &AttributeFields, ) -> Result<(), &'static str>; } ``` In order to enable the generic kernel code to manipulate the kernel's translation tables, they must first be made accessible. Until now, they were just a "hidden" struct in the `architectural` MMU driver (`src/arch/.../memory/mmu.rs`). This made sense because the MMU driver code was the only code that needed to be concerned with the table data structure, so having it accessible locally simplified things. Since the tables need to be exposed to the rest of the kernel code now, it makes sense to move them to `BSP` code. Because ultimately, it is the `BSP` that is defining the translation table's properties, such as the size of the virtual address space that the tables need to cover. They are now defined in the global instances region of `src/bsp/.../memory/mmu.rs`. To control access, they are guarded by an `InitStateLock`. ```rust //-------------------------------------------------------------------------------------------------- // Global instances //-------------------------------------------------------------------------------------------------- /// The kernel translation tables. static KERNEL_TABLES: InitStateLock = InitStateLock::new(KernelTranslationTable::new()); ``` The struct `KernelTranslationTable` is a type alias defined in the same file, which in turn gets its definition from an associated type of type `KernelVirtAddrSpace`, which itself is a type alias of `memory::mmu::AddressSpace`. I know this sounds horribly complicated, but in the end this is just some layers of `const generics` whose implementation is scattered between `generic` and `arch` code. This is done to (1) ensure a sane compile-time definition of the translation table struct (by doing various bounds checks), and (2) to separate concerns between generic `MMU` code and specializations that come from the `architectural` part. In the end, these tables can be accessed by calling `bsp::memory::mmu::kernel_translation_tables()`: ```rust /// Return a reference to the kernel's translation tables. pub fn kernel_translation_tables() -> &'static InitStateLock { &KERNEL_TABLES } ``` Finally, the generic kernel code (`src/memory/mmu.rs`) now provides a couple of memory mapping functions that access and manipulate this instance. They are exported for the rest of the kernel to use: ```rust /// Raw mapping of a virtual to physical region in the kernel translation tables. /// /// Prevents mapping into the MMIO range of the tables. pub unsafe fn kernel_map_at( name: &'static str, virt_region: &MemoryRegion, phys_region: &MemoryRegion, attr: &AttributeFields, ) -> Result<(), &'static str>; /// MMIO remapping in the kernel translation tables. /// /// Typically used by device drivers. pub unsafe fn kernel_map_mmio( name: &'static str, mmio_descriptor: &MMIODescriptor, ) -> Result, &'static str>; /// Map the kernel's binary. Returns the translation table's base address. pub unsafe fn kernel_map_binary() -> Result, &'static str>; /// Enable the MMU and data + instruction caching. pub unsafe fn enable_mmu_and_caching( phys_tables_base_addr: Address, ) -> Result<(), MMUEnableError>; ``` ### The new APIs in action `kernel_map_binary()` and `enable_mmu_and_caching()` are used early in `kernel_init()` to set up virtual memory: ```rust let phys_kernel_tables_base_addr = match memory::mmu::kernel_map_binary() { Err(string) => panic!("Error mapping kernel binary: {}", string), Ok(addr) => addr, }; if let Err(e) = memory::mmu::enable_mmu_and_caching(phys_kernel_tables_base_addr) { panic!("Enabling MMU failed: {}", e); } ``` Both functions internally use `bsp` and `arch` specific code to achieve their goals. For example, `memory::mmu::kernel_map_binary()` itself wraps around a `bsp` function of the same name (`bsp::memory::mmu::kernel_map_binary()`): ```rust /// Map the kernel binary. pub unsafe fn kernel_map_binary() -> Result<(), &'static str> { generic_mmu::kernel_map_at( "Kernel boot-core stack", // omitted for brevity. )?; generic_mmu::kernel_map_at( "Kernel code and RO data", &virt_code_region(), &kernel_virt_to_phys_region(virt_code_region()), &AttributeFields { mem_attributes: MemAttributes::CacheableDRAM, acc_perms: AccessPermissions::ReadOnly, execute_never: false, }, )?; generic_mmu::kernel_map_at( "Kernel data and bss", // omitted for brevity. )?; Ok(()) } ``` Another user of the new APIs are device drivers, which now expect an `MMIODescriptor` type instead of a raw address. The following is an example for the `UART`: ```rust impl PL011Uart { /// Create an instance. pub const unsafe fn new( mmio_descriptor: memory::mmu::MMIODescriptor, irq_number: bsp::device_driver::IRQNumber, ) -> Self { Self { // omitted for brevity. } } } ``` When the kernel calls the driver's implementation of `driver::interface::DeviceDriver::init()` during kernel boot, the MMIO Descriptor is used to remap the MMIO region on demand: ```rust unsafe fn init(&self) -> Result<(), &'static str> { let virt_addr = memory::mmu::kernel_map_mmio(self.compatible(), &self.mmio_descriptor)?; self.inner .lock(|inner| inner.init(Some(virt_addr.as_usize())))?; // omitted for brevity. Ok(()) } ``` ### MMIO Virtual Address Allocation Peeking inside `memory::mmu::kernel_map_mmio()`, we can see that a `virtual address region` is obtained from an `allocator` before remapping: ```rust pub unsafe fn kernel_map_mmio( name: &'static str, mmio_descriptor: &MMIODescriptor, ) -> Result, &'static str> { // omitted let virt_region = page_alloc::kernel_mmio_va_allocator().lock(|allocator| allocator.alloc(num_pages))?; kernel_map_at_unchecked( name, &virt_region, &phys_region, &AttributeFields { mem_attributes: MemAttributes::Device, acc_perms: AccessPermissions::ReadWrite, execute_never: true, }, )?; // omitted } ``` This allocator is defined and implemented in the added file `src/memory/mmu/paeg_alloc.rs`. Like other parts of the mapping code, its implementation makes use of the newly introduced `PageAddress` and `MemoryRegion` types (in [`src/memory/mmu/types.rs`](kernel/src/memory/mmu/types.rs)), but apart from that is rather straight forward. Therefore, it won't be covered in details here. The more interesting question is: How does the allocator get to learn which VAs it can use? This is happening in the following function, which gets called as part of `memory::mmu::post_enable_init()`, which in turn gets called in `kernel_init()` after the MMU has been turned on. ```rust /// Query the BSP for the reserved virtual addresses for MMIO remapping and initialize the kernel's /// MMIO VA allocator with it. fn kernel_init_mmio_va_allocator() { let region = bsp::memory::mmu::virt_mmio_remap_region(); page_alloc::kernel_mmio_va_allocator().lock(|allocator| allocator.initialize(region)); } ``` Again, it is the `BSP` that provides the information. The `BSP` itself indirectly gets it from the linker script. In it, we have defined an `8 MiB` region right after the `.data` segment: ```ld.s __data_end_exclusive = .; /*********************************************************************************************** * MMIO Remap Reserved ***********************************************************************************************/ __mmio_remap_start = .; . += 8 * 1024 * 1024; __mmio_remap_end_exclusive = .; ASSERT((. & PAGE_MASK) == 0, "MMIO remap reservation is not page aligned") ``` The two symbols `__mmio_remap_start` and `__mmio_remap_end_exclusive` are used by the `BSP` to learn the VA range. ### Supporting Changes There's a couple of changes not covered in this tutorial text, but the reader should ideally skim through them: - [`src/memory.rs`](kernel/src/memory.rs) and [`src/memory/mmu/types.rs`](kernel/src/memory/mmu/types.rs) introduce a couple of supporting types, like`Address`, `PageAddress` and `MemoryRegion`. It is worth reading their implementations. - [`src/memory/mmu/mapping_record.rs`](kernel/src/memory/mmu/mapping_record.rs) provides the generic kernel code's way of tracking previous memory mappings for use cases such as reusing existing mappings (in case of drivers that have their MMIO ranges in the same `64 KiB` page) or printing mappings statistics. ## Test it When you load the kernel, you can now see that the driver's MMIO virtual addresses start right after the `.data` section: Raspberry Pi 3: ```console $ make chainboot [...] Minipush 1.0 [MP] ⏳ Waiting for /dev/ttyUSB0 [MP] ✅ Serial connected [MP] 🔌 Please power the target now __ __ _ _ _ _ | \/ (_)_ _ (_) | ___ __ _ __| | | |\/| | | ' \| | |__/ _ \/ _` / _` | |_| |_|_|_||_|_|____\___/\__,_\__,_| Raspberry Pi 3 [ML] Requesting binary [MP] ⏩ Pushing 67 KiB =========================================🦀 100% 0 KiB/s Time: 00:00:00 [ML] Loaded! Executing the payload now [ 0.758253] mingo version 0.14.0 [ 0.758460] Booting on: Raspberry Pi 3 [ 0.758915] MMU online: [ 0.759208] ------------------------------------------------------------------------------------------------------------------------------------------- [ 0.760952] Virtual Physical Size Attr Entity [ 0.762696] ------------------------------------------------------------------------------------------------------------------------------------------- [ 0.764441] 0x0000_0000_0000_0000..0x0000_0000_0007_ffff --> 0x00_0000_0000..0x00_0007_ffff | 512 KiB | C RW XN | Kernel boot-core stack [ 0.766044] 0x0000_0000_0008_0000..0x0000_0000_0008_ffff --> 0x00_0008_0000..0x00_0008_ffff | 64 KiB | C RO X | Kernel code and RO data [ 0.767658] 0x0000_0000_0009_0000..0x0000_0000_000d_ffff --> 0x00_0009_0000..0x00_000d_ffff | 320 KiB | C RW XN | Kernel data and bss [ 0.769229] 0x0000_0000_000e_0000..0x0000_0000_000e_ffff --> 0x00_3f20_0000..0x00_3f20_ffff | 64 KiB | Dev RW XN | BCM GPIO [ 0.770680] | BCM PL011 UART [ 0.772197] 0x0000_0000_000f_0000..0x0000_0000_000f_ffff --> 0x00_3f00_0000..0x00_3f00_ffff | 64 KiB | Dev RW XN | BCM Peripheral Interrupt Controller [ 0.773941] ------------------------------------------------------------------------------------------------------------------------------------------- ``` Raspberry Pi 4: ```console $ BSP=rpi4 make chainboot [...] Minipush 1.0 [MP] ⏳ Waiting for /dev/ttyUSB0 [MP] ✅ Serial connected [MP] 🔌 Please power the target now __ __ _ _ _ _ | \/ (_)_ _ (_) | ___ __ _ __| | | |\/| | | ' \| | |__/ _ \/ _` / _` | |_| |_|_|_||_|_|____\___/\__,_\__,_| Raspberry Pi 4 [ML] Requesting binary [MP] ⏩ Pushing 74 KiB =========================================🦀 100% 0 KiB/s Time: 00:00:00 [ML] Loaded! Executing the payload now [ 0.842275] mingo version 0.14.0 [ 0.842308] Booting on: Raspberry Pi 4 [ 0.842763] MMU online: [ 0.843055] ------------------------------------------------------------------------------------------------------------------------------------------- [ 0.844800] Virtual Physical Size Attr Entity [ 0.846544] ------------------------------------------------------------------------------------------------------------------------------------------- [ 0.848288] 0x0000_0000_0000_0000..0x0000_0000_0007_ffff --> 0x00_0000_0000..0x00_0007_ffff | 512 KiB | C RW XN | Kernel boot-core stack [ 0.849892] 0x0000_0000_0008_0000..0x0000_0000_0008_ffff --> 0x00_0008_0000..0x00_0008_ffff | 64 KiB | C RO X | Kernel code and RO data [ 0.851505] 0x0000_0000_0009_0000..0x0000_0000_000d_ffff --> 0x00_0009_0000..0x00_000d_ffff | 320 KiB | C RW XN | Kernel data and bss [ 0.853076] 0x0000_0000_000e_0000..0x0000_0000_000e_ffff --> 0x00_fe20_0000..0x00_fe20_ffff | 64 KiB | Dev RW XN | BCM GPIO [ 0.854528] | BCM PL011 UART [ 0.856045] 0x0000_0000_000f_0000..0x0000_0000_000f_ffff --> 0x00_ff84_0000..0x00_ff84_ffff | 64 KiB | Dev RW XN | GICD [ 0.857453] | GICC [ 0.858862] ------------------------------------------------------------------------------------------------------------------------------------------- ``` ## Diff to previous ```diff diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/Cargo.toml 14_virtual_mem_part2_mmio_remap/kernel/Cargo.toml --- 13_exceptions_part2_peripheral_IRQs/kernel/Cargo.toml +++ 14_virtual_mem_part2_mmio_remap/kernel/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "mingo" -version = "0.13.0" +version = "0.14.0" authors = ["Andre Richter "] edition = "2021" diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs 14_virtual_mem_part2_mmio_remap/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs @@ -14,10 +14,14 @@ //! crate::memory::mmu::translation_table::arch_translation_table use crate::{ - bsp, memory, - memory::mmu::{ - arch_mmu::{Granule512MiB, Granule64KiB}, - AccessPermissions, AttributeFields, MemAttributes, + bsp, + memory::{ + self, + mmu::{ + arch_mmu::{Granule512MiB, Granule64KiB}, + AccessPermissions, AttributeFields, MemAttributes, MemoryRegion, PageAddress, + }, + Address, Physical, Virtual, }, }; use core::convert; @@ -121,12 +125,9 @@ } trait StartAddr { - fn phys_start_addr_u64(&self) -> u64; - fn phys_start_addr_usize(&self) -> usize; + fn phys_start_addr(&self) -> Address; } -const NUM_LVL2_TABLES: usize = bsp::memory::mmu::KernelAddrSpace::SIZE >> Granule512MiB::SHIFT; - //-------------------------------------------------------------------------------------------------- // Public Definitions //-------------------------------------------------------------------------------------------------- @@ -141,10 +142,10 @@ /// Table descriptors, covering 512 MiB windows. lvl2: [TableDescriptor; NUM_TABLES], -} -/// A translation table type for the kernel space. -pub type KernelTranslationTable = FixedSizeTranslationTable; + /// Have the tables been initialized? + initialized: bool, +} //-------------------------------------------------------------------------------------------------- // Private Code @@ -152,12 +153,8 @@ // The binary is still identity mapped, so we don't need to convert here. impl StartAddr for [T; N] { - fn phys_start_addr_u64(&self) -> u64 { - self as *const T as u64 - } - - fn phys_start_addr_usize(&self) -> usize { - self as *const _ as usize + fn phys_start_addr(&self) -> Address { + Address::new(self as *const _ as usize) } } @@ -170,10 +167,10 @@ } /// Create an instance pointing to the supplied address. - pub fn from_next_lvl_table_addr(phys_next_lvl_table_addr: usize) -> Self { + pub fn from_next_lvl_table_addr(phys_next_lvl_table_addr: Address) -> Self { let val = InMemoryRegister::::new(0); - let shifted = phys_next_lvl_table_addr >> Granule64KiB::SHIFT; + let shifted = phys_next_lvl_table_addr.as_usize() >> Granule64KiB::SHIFT; val.write( STAGE1_TABLE_DESCRIPTOR::NEXT_LEVEL_TABLE_ADDR_64KiB.val(shifted as u64) + STAGE1_TABLE_DESCRIPTOR::TYPE::Table @@ -230,12 +227,15 @@ } /// Create an instance. - pub fn from_output_addr(phys_output_addr: usize, attribute_fields: &AttributeFields) -> Self { + pub fn from_output_page_addr( + phys_output_page_addr: PageAddress, + attribute_fields: &AttributeFields, + ) -> Self { let val = InMemoryRegister::::new(0); - let shifted = phys_output_addr as u64 >> Granule64KiB::SHIFT; + let shifted = phys_output_page_addr.into_inner().as_usize() >> Granule64KiB::SHIFT; val.write( - STAGE1_PAGE_DESCRIPTOR::OUTPUT_ADDR_64KiB.val(shifted) + STAGE1_PAGE_DESCRIPTOR::OUTPUT_ADDR_64KiB.val(shifted as u64) + STAGE1_PAGE_DESCRIPTOR::AF::True + STAGE1_PAGE_DESCRIPTOR::TYPE::Page + STAGE1_PAGE_DESCRIPTOR::VALID::True @@ -244,50 +244,133 @@ Self { value: val.get() } } + + /// Returns the valid bit. + fn is_valid(&self) -> bool { + InMemoryRegister::::new(self.value) + .is_set(STAGE1_PAGE_DESCRIPTOR::VALID) + } } //-------------------------------------------------------------------------------------------------- // Public Code //-------------------------------------------------------------------------------------------------- +impl memory::mmu::AssociatedTranslationTable + for memory::mmu::AddressSpace +where + [u8; Self::SIZE >> Granule512MiB::SHIFT]: Sized, +{ + type TableStartFromBottom = FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }>; +} + impl FixedSizeTranslationTable { /// Create an instance. + #[allow(clippy::assertions_on_constants)] pub const fn new() -> Self { + assert!(bsp::memory::mmu::KernelGranule::SIZE == Granule64KiB::SIZE); + // Can't have a zero-sized address space. assert!(NUM_TABLES > 0); Self { lvl3: [[PageDescriptor::new_zeroed(); 8192]; NUM_TABLES], lvl2: [TableDescriptor::new_zeroed(); NUM_TABLES], + initialized: false, } } - /// Iterates over all static translation table entries and fills them at once. - /// - /// # Safety - /// - /// - Modifies a `static mut`. Ensure it only happens from here. - pub unsafe fn populate_tt_entries(&mut self) -> Result<(), &'static str> { - for (l2_nr, l2_entry) in self.lvl2.iter_mut().enumerate() { - *l2_entry = - TableDescriptor::from_next_lvl_table_addr(self.lvl3[l2_nr].phys_start_addr_usize()); + /// Helper to calculate the lvl2 and lvl3 indices from an address. + #[inline(always)] + fn lvl2_lvl3_index_from_page_addr( + &self, + virt_page_addr: PageAddress, + ) -> Result<(usize, usize), &'static str> { + let addr = virt_page_addr.into_inner().as_usize(); + let lvl2_index = addr >> Granule512MiB::SHIFT; + let lvl3_index = (addr & Granule512MiB::MASK) >> Granule64KiB::SHIFT; - for (l3_nr, l3_entry) in self.lvl3[l2_nr].iter_mut().enumerate() { - let virt_addr = (l2_nr << Granule512MiB::SHIFT) + (l3_nr << Granule64KiB::SHIFT); + if lvl2_index > (NUM_TABLES - 1) { + return Err("Virtual page is out of bounds of translation table"); + } - let (phys_output_addr, attribute_fields) = - bsp::memory::mmu::virt_mem_layout().virt_addr_properties(virt_addr)?; + Ok((lvl2_index, lvl3_index)) + } - *l3_entry = PageDescriptor::from_output_addr(phys_output_addr, &attribute_fields); - } + /// Sets the PageDescriptor corresponding to the supplied page address. + /// + /// Doesn't allow overriding an already valid page. + #[inline(always)] + fn set_page_descriptor_from_page_addr( + &mut self, + virt_page_addr: PageAddress, + new_desc: &PageDescriptor, + ) -> Result<(), &'static str> { + let (lvl2_index, lvl3_index) = self.lvl2_lvl3_index_from_page_addr(virt_page_addr)?; + let desc = &mut self.lvl3[lvl2_index][lvl3_index]; + + if desc.is_valid() { + return Err("Virtual page is already mapped"); } + *desc = *new_desc; Ok(()) } +} - /// The translation table's base address to be used for programming the MMU. - pub fn phys_base_address(&self) -> u64 { - self.lvl2.phys_start_addr_u64() +//------------------------------------------------------------------------------ +// OS Interface Code +//------------------------------------------------------------------------------ + +impl memory::mmu::translation_table::interface::TranslationTable + for FixedSizeTranslationTable +{ + fn init(&mut self) { + if self.initialized { + return; + } + + // Populate the l2 entries. + for (lvl2_nr, lvl2_entry) in self.lvl2.iter_mut().enumerate() { + let phys_table_addr = self.lvl3[lvl2_nr].phys_start_addr(); + + let new_desc = TableDescriptor::from_next_lvl_table_addr(phys_table_addr); + *lvl2_entry = new_desc; + } + + self.initialized = true; + } + + fn phys_base_address(&self) -> Address { + self.lvl2.phys_start_addr() + } + + unsafe fn map_at( + &mut self, + virt_region: &MemoryRegion, + phys_region: &MemoryRegion, + attr: &AttributeFields, + ) -> Result<(), &'static str> { + assert!(self.initialized, "Translation tables not initialized"); + + if virt_region.size() != phys_region.size() { + return Err("Tried to map memory regions with unequal sizes"); + } + + if phys_region.end_exclusive_page_addr() > bsp::memory::phys_addr_space_end_exclusive_addr() + { + return Err("Tried to map outside of physical address space"); + } + + let iter = phys_region.into_iter().zip(virt_region.into_iter()); + for (phys_page_addr, virt_page_addr) in iter { + let new_desc = PageDescriptor::from_output_page_addr(phys_page_addr, attr); + let virt_page = virt_page_addr; + + self.set_page_descriptor_from_page_addr(virt_page, &new_desc)?; + } + + Ok(()) } } @@ -296,6 +379,9 @@ //-------------------------------------------------------------------------------------------------- #[cfg(test)] +pub type MinSizeTranslationTable = FixedSizeTranslationTable<1>; + +#[cfg(test)] mod tests { use super::*; use test_macros::kernel_test; diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/_arch/aarch64/memory/mmu.rs 14_virtual_mem_part2_mmio_remap/kernel/src/_arch/aarch64/memory/mmu.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/_arch/aarch64/memory/mmu.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/_arch/aarch64/memory/mmu.rs @@ -15,7 +15,7 @@ use crate::{ bsp, memory, - memory::mmu::{translation_table::KernelTranslationTable, TranslationGranule}, + memory::{mmu::TranslationGranule, Address, Physical}, }; use core::intrinsics::unlikely; use cortex_a::{asm::barrier, registers::*}; @@ -46,13 +46,6 @@ // Global instances //-------------------------------------------------------------------------------------------------- -/// The kernel translation tables. -/// -/// # Safety -/// -/// - Supposed to land in `.bss`. Therefore, ensure that all initial member values boil down to "0". -static mut KERNEL_TABLES: KernelTranslationTable = KernelTranslationTable::new(); - static MMU: MemoryManagementUnit = MemoryManagementUnit; //-------------------------------------------------------------------------------------------------- @@ -87,7 +80,7 @@ /// Configure various settings of stage 1 of the EL1 translation regime. fn configure_translation_control(&self) { - let t0sz = (64 - bsp::memory::mmu::KernelAddrSpace::SIZE_SHIFT) as u64; + let t0sz = (64 - bsp::memory::mmu::KernelVirtAddrSpace::SIZE_SHIFT) as u64; TCR_EL1.write( TCR_EL1::TBI0::Used @@ -119,7 +112,10 @@ use memory::mmu::MMUEnableError; impl memory::mmu::interface::MMU for MemoryManagementUnit { - unsafe fn enable_mmu_and_caching(&self) -> Result<(), MMUEnableError> { + unsafe fn enable_mmu_and_caching( + &self, + phys_tables_base_addr: Address, + ) -> Result<(), MMUEnableError> { if unlikely(self.is_enabled()) { return Err(MMUEnableError::AlreadyEnabled); } @@ -134,13 +130,8 @@ // Prepare the memory attribute indirection register. self.set_up_mair(); - // Populate translation tables. - KERNEL_TABLES - .populate_tt_entries() - .map_err(MMUEnableError::Other)?; - // Set the "Translation Table Base Register". - TTBR0_EL1.set_baddr(KERNEL_TABLES.phys_base_address()); + TTBR0_EL1.set_baddr(phys_tables_base_addr.as_usize() as u64); self.configure_translation_control(); @@ -163,33 +154,3 @@ SCTLR_EL1.matches_all(SCTLR_EL1::M::Enable) } } - -//-------------------------------------------------------------------------------------------------- -// Testing -//-------------------------------------------------------------------------------------------------- - -#[cfg(test)] -mod tests { - use super::*; - use core::{cell::UnsafeCell, ops::Range}; - use test_macros::kernel_test; - - /// Check if KERNEL_TABLES is in .bss. - #[kernel_test] - fn kernel_tables_in_bss() { - extern "Rust" { - static __bss_start: UnsafeCell; - static __bss_end_exclusive: UnsafeCell; - } - - let bss_range = unsafe { - Range { - start: __bss_start.get(), - end: __bss_end_exclusive.get(), - } - }; - let kernel_tables_addr = unsafe { &KERNEL_TABLES as *const _ as usize as *mut u64 }; - - assert!(bss_range.contains(&kernel_tables_addr)); - } -} diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/arm/gicv2/gicc.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/arm/gicv2/gicc.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/arm/gicv2/gicc.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/arm/gicv2/gicc.rs @@ -4,7 +4,9 @@ //! GICC Driver - GIC CPU interface. -use crate::{bsp::device_driver::common::MMIODerefWrapper, exception}; +use crate::{ + bsp::device_driver::common::MMIODerefWrapper, exception, synchronization::InitStateLock, +}; use tock_registers::{ interfaces::{Readable, Writeable}, register_bitfields, register_structs, @@ -60,12 +62,13 @@ /// Representation of the GIC CPU interface. pub struct GICC { - registers: Registers, + registers: InitStateLock, } //-------------------------------------------------------------------------------------------------- // Public Code //-------------------------------------------------------------------------------------------------- +use crate::synchronization::interface::ReadWriteEx; impl GICC { /// Create an instance. @@ -75,10 +78,15 @@ /// - The user must ensure to provide a correct MMIO start address. pub const unsafe fn new(mmio_start_addr: usize) -> Self { Self { - registers: Registers::new(mmio_start_addr), + registers: InitStateLock::new(Registers::new(mmio_start_addr)), } } + pub unsafe fn set_mmio(&self, new_mmio_start_addr: usize) { + self.registers + .write(|regs| *regs = Registers::new(new_mmio_start_addr)); + } + /// Accept interrupts of any priority. /// /// Quoting the GICv2 Architecture Specification: @@ -91,7 +99,9 @@ /// - GICC MMIO registers are banked per CPU core. It is therefore safe to have `&self` instead /// of `&mut self`. pub fn priority_accept_all(&self) { - self.registers.PMR.write(PMR::Priority.val(255)); // Comment in arch spec. + self.registers.read(|regs| { + regs.PMR.write(PMR::Priority.val(255)); // Comment in arch spec. + }); } /// Enable the interface - start accepting IRQs. @@ -101,7 +111,9 @@ /// - GICC MMIO registers are banked per CPU core. It is therefore safe to have `&self` instead /// of `&mut self`. pub fn enable(&self) { - self.registers.CTLR.write(CTLR::Enable::SET); + self.registers.read(|regs| { + regs.CTLR.write(CTLR::Enable::SET); + }); } /// Extract the number of the highest-priority pending IRQ. @@ -117,7 +129,8 @@ &self, _ic: &exception::asynchronous::IRQContext<'irq_context>, ) -> usize { - self.registers.IAR.read(IAR::InterruptID) as usize + self.registers + .read(|regs| regs.IAR.read(IAR::InterruptID) as usize) } /// Complete handling of the currently active IRQ. @@ -136,6 +149,8 @@ irq_number: u32, _ic: &exception::asynchronous::IRQContext<'irq_context>, ) { - self.registers.EOIR.write(EOIR::EOIINTID.val(irq_number)); + self.registers.read(|regs| { + regs.EOIR.write(EOIR::EOIINTID.val(irq_number)); + }); } } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/arm/gicv2/gicd.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/arm/gicv2/gicd.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/arm/gicv2/gicd.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/arm/gicv2/gicd.rs @@ -8,8 +8,9 @@ //! - SPI - Shared Peripheral Interrupt. use crate::{ - bsp::device_driver::common::MMIODerefWrapper, state, synchronization, - synchronization::IRQSafeNullLock, + bsp::device_driver::common::MMIODerefWrapper, + state, synchronization, + synchronization::{IRQSafeNullLock, InitStateLock}, }; use tock_registers::{ interfaces::{Readable, Writeable}, @@ -83,7 +84,7 @@ shared_registers: IRQSafeNullLock, /// Access to banked registers is unguarded. - banked_registers: BankedRegisters, + banked_registers: InitStateLock, } //-------------------------------------------------------------------------------------------------- @@ -120,6 +121,7 @@ //-------------------------------------------------------------------------------------------------- // Public Code //-------------------------------------------------------------------------------------------------- +use crate::synchronization::interface::ReadWriteEx; use synchronization::interface::Mutex; impl GICD { @@ -131,10 +133,17 @@ pub const unsafe fn new(mmio_start_addr: usize) -> Self { Self { shared_registers: IRQSafeNullLock::new(SharedRegisters::new(mmio_start_addr)), - banked_registers: BankedRegisters::new(mmio_start_addr), + banked_registers: InitStateLock::new(BankedRegisters::new(mmio_start_addr)), } } + pub unsafe fn set_mmio(&self, new_mmio_start_addr: usize) { + self.shared_registers + .lock(|regs| *regs = SharedRegisters::new(new_mmio_start_addr)); + self.banked_registers + .write(|regs| *regs = BankedRegisters::new(new_mmio_start_addr)); + } + /// Use a banked ITARGETSR to retrieve the executing core's GIC target mask. /// /// Quoting the GICv2 Architecture Specification: @@ -142,7 +151,8 @@ /// "GICD_ITARGETSR0 to GICD_ITARGETSR7 are read-only, and each field returns a value that /// corresponds only to the processor reading the register." fn local_gic_target_mask(&self) -> u32 { - self.banked_registers.ITARGETSR[0].read(ITARGETSR::Offset0) + self.banked_registers + .read(|regs| regs.ITARGETSR[0].read(ITARGETSR::Offset0)) } /// Route all SPIs to the boot core and enable the distributor. @@ -181,10 +191,10 @@ // Check if we are handling a private or shared IRQ. match irq_num { // Private. - 0..=31 => { - let enable_reg = &self.banked_registers.ISENABLER; + 0..=31 => self.banked_registers.read(|regs| { + let enable_reg = ®s.ISENABLER; enable_reg.set(enable_reg.get() | enable_bit); - } + }), // Shared. _ => { let enable_reg_index_shared = enable_reg_index - 1; diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/arm/gicv2.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/arm/gicv2.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/arm/gicv2.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/arm/gicv2.rs @@ -79,7 +79,8 @@ mod gicc; mod gicd; -use crate::{bsp, cpu, driver, exception, synchronization, synchronization::InitStateLock}; +use crate::{bsp, cpu, driver, exception, memory, synchronization, synchronization::InitStateLock}; +use core::sync::atomic::{AtomicBool, Ordering}; //-------------------------------------------------------------------------------------------------- // Private Definitions @@ -96,12 +97,18 @@ /// Representation of the GIC. pub struct GICv2 { + gicd_mmio_descriptor: memory::mmu::MMIODescriptor, + gicc_mmio_descriptor: memory::mmu::MMIODescriptor, + /// The Distributor. gicd: gicd::GICD, /// The CPU Interface. gicc: gicc::GICC, + /// Have the MMIO regions been remapped yet? + is_mmio_remapped: AtomicBool, + /// Stores registered IRQ handlers. Writable only during kernel init. RO afterwards. handler_table: InitStateLock, } @@ -118,11 +125,17 @@ /// /// # Safety /// - /// - The user must ensure to provide a correct MMIO start address. - pub const unsafe fn new(gicd_mmio_start_addr: usize, gicc_mmio_start_addr: usize) -> Self { + /// - The user must ensure to provide correct MMIO descriptors. + pub const unsafe fn new( + gicd_mmio_descriptor: memory::mmu::MMIODescriptor, + gicc_mmio_descriptor: memory::mmu::MMIODescriptor, + ) -> Self { Self { - gicd: gicd::GICD::new(gicd_mmio_start_addr), - gicc: gicc::GICC::new(gicc_mmio_start_addr), + gicd_mmio_descriptor, + gicc_mmio_descriptor, + gicd: gicd::GICD::new(gicd_mmio_descriptor.start_addr().as_usize()), + gicc: gicc::GICC::new(gicc_mmio_descriptor.start_addr().as_usize()), + is_mmio_remapped: AtomicBool::new(false), handler_table: InitStateLock::new([None; Self::NUM_IRQS]), } } @@ -139,6 +152,20 @@ } unsafe fn init(&self) -> Result<(), &'static str> { + let remapped = self.is_mmio_remapped.load(Ordering::Relaxed); + if !remapped { + // GICD + let mut virt_addr = memory::mmu::kernel_map_mmio("GICD", &self.gicd_mmio_descriptor)?; + self.gicd.set_mmio(virt_addr.as_usize()); + + // GICC + virt_addr = memory::mmu::kernel_map_mmio("GICC", &self.gicc_mmio_descriptor)?; + self.gicc.set_mmio(virt_addr.as_usize()); + + // Conclude remapping. + self.is_mmio_remapped.store(true, Ordering::Relaxed); + } + if bsp::cpu::BOOT_CORE_ID == cpu::smp::core_id() { self.gicd.boot_core_init(); } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_gpio.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_gpio.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_gpio.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_gpio.rs @@ -5,9 +5,10 @@ //! GPIO Driver. use crate::{ - bsp::device_driver::common::MMIODerefWrapper, driver, synchronization, + bsp::device_driver::common::MMIODerefWrapper, driver, memory, synchronization, synchronization::IRQSafeNullLock, }; +use core::sync::atomic::{AtomicUsize, Ordering}; use tock_registers::{ interfaces::{ReadWriteable, Writeable}, register_bitfields, register_structs, @@ -121,6 +122,8 @@ /// Representation of the GPIO HW. pub struct GPIO { + mmio_descriptor: memory::mmu::MMIODescriptor, + virt_mmio_start_addr: AtomicUsize, inner: IRQSafeNullLock, } @@ -140,6 +143,19 @@ } } + /// Init code. + /// + /// # Safety + /// + /// - The user must ensure to provide a correct MMIO start address. + pub unsafe fn init(&mut self, new_mmio_start_addr: Option) -> Result<(), &'static str> { + if let Some(addr) = new_mmio_start_addr { + self.registers = Registers::new(addr); + } + + Ok(()) + } + /// Disable pull-up/down on pins 14 and 15. #[cfg(feature = "bsp_rpi3")] fn disable_pud_14_15_bcm2837(&mut self) { @@ -194,10 +210,12 @@ /// /// # Safety /// - /// - The user must ensure to provide a correct MMIO start address. - pub const unsafe fn new(mmio_start_addr: usize) -> Self { + /// - The user must ensure to provide correct MMIO descriptors. + pub const unsafe fn new(mmio_descriptor: memory::mmu::MMIODescriptor) -> Self { Self { - inner: IRQSafeNullLock::new(GPIOInner::new(mmio_start_addr)), + mmio_descriptor, + virt_mmio_start_addr: AtomicUsize::new(0), + inner: IRQSafeNullLock::new(GPIOInner::new(mmio_descriptor.start_addr().as_usize())), } } @@ -216,4 +234,26 @@ fn compatible(&self) -> &'static str { "BCM GPIO" } + + unsafe fn init(&self) -> Result<(), &'static str> { + let virt_addr = memory::mmu::kernel_map_mmio(self.compatible(), &self.mmio_descriptor)?; + + self.inner + .lock(|inner| inner.init(Some(virt_addr.as_usize())))?; + + self.virt_mmio_start_addr + .store(virt_addr.as_usize(), Ordering::Relaxed); + + Ok(()) + } + + fn virt_mmio_start_addr(&self) -> Option { + let addr = self.virt_mmio_start_addr.load(Ordering::Relaxed); + + if addr == 0 { + return None; + } + + Some(addr) + } } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller/peripheral_ic.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller/peripheral_ic.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller/peripheral_ic.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller/peripheral_ic.rs @@ -7,7 +7,7 @@ use super::{InterruptController, PendingIRQs, PeripheralIRQ}; use crate::{ bsp::device_driver::common::MMIODerefWrapper, - exception, synchronization, + driver, exception, memory, synchronization, synchronization::{IRQSafeNullLock, InitStateLock}, }; use tock_registers::{ @@ -55,11 +55,13 @@ /// Representation of the peripheral interrupt controller. pub struct PeripheralIC { + mmio_descriptor: memory::mmu::MMIODescriptor, + /// Access to write registers is guarded with a lock. wo_registers: IRQSafeNullLock, /// Register read access is unguarded. - ro_registers: ReadOnlyRegisters, + ro_registers: InitStateLock, /// Stores registered IRQ handlers. Writable only during kernel init. RO afterwards. handler_table: InitStateLock, @@ -74,21 +76,26 @@ /// /// # Safety /// - /// - The user must ensure to provide a correct MMIO start address. - pub const unsafe fn new(mmio_start_addr: usize) -> Self { + /// - The user must ensure to provide correct MMIO descriptors. + pub const unsafe fn new(mmio_descriptor: memory::mmu::MMIODescriptor) -> Self { + let addr = mmio_descriptor.start_addr().as_usize(); + Self { - wo_registers: IRQSafeNullLock::new(WriteOnlyRegisters::new(mmio_start_addr)), - ro_registers: ReadOnlyRegisters::new(mmio_start_addr), + mmio_descriptor, + wo_registers: IRQSafeNullLock::new(WriteOnlyRegisters::new(addr)), + ro_registers: InitStateLock::new(ReadOnlyRegisters::new(addr)), handler_table: InitStateLock::new([None; InterruptController::NUM_PERIPHERAL_IRQS]), } } /// Query the list of pending IRQs. fn pending_irqs(&self) -> PendingIRQs { - let pending_mask: u64 = (u64::from(self.ro_registers.PENDING_2.get()) << 32) - | u64::from(self.ro_registers.PENDING_1.get()); + self.ro_registers.read(|regs| { + let pending_mask: u64 = + (u64::from(regs.PENDING_2.get()) << 32) | u64::from(regs.PENDING_1.get()); - PendingIRQs::new(pending_mask) + PendingIRQs::new(pending_mask) + }) } } @@ -97,6 +104,24 @@ //------------------------------------------------------------------------------ use synchronization::interface::{Mutex, ReadWriteEx}; +impl driver::interface::DeviceDriver for PeripheralIC { + fn compatible(&self) -> &'static str { + "BCM Peripheral Interrupt Controller" + } + + unsafe fn init(&self) -> Result<(), &'static str> { + let virt_addr = + memory::mmu::kernel_map_mmio(self.compatible(), &self.mmio_descriptor)?.as_usize(); + + self.wo_registers + .lock(|regs| *regs = WriteOnlyRegisters::new(virt_addr)); + self.ro_registers + .write(|regs| *regs = ReadOnlyRegisters::new(virt_addr)); + + Ok(()) + } +} + impl exception::asynchronous::interface::IRQManager for PeripheralIC { type IRQNumberType = PeripheralIRQ; diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_interrupt_controller.rs @@ -6,7 +6,7 @@ mod peripheral_ic; -use crate::{driver, exception}; +use crate::{driver, exception, memory}; //-------------------------------------------------------------------------------------------------- // Private Definitions @@ -78,10 +78,13 @@ /// /// # Safety /// - /// - The user must ensure to provide a correct MMIO start address. - pub const unsafe fn new(_local_mmio_start_addr: usize, periph_mmio_start_addr: usize) -> Self { + /// - The user must ensure to provide correct MMIO descriptors. + pub const unsafe fn new( + _local_mmio_descriptor: memory::mmu::MMIODescriptor, + periph_mmio_descriptor: memory::mmu::MMIODescriptor, + ) -> Self { Self { - periph: peripheral_ic::PeripheralIC::new(periph_mmio_start_addr), + periph: peripheral_ic::PeripheralIC::new(periph_mmio_descriptor), } } } @@ -94,6 +97,10 @@ fn compatible(&self) -> &'static str { "BCM Interrupt Controller" } + + unsafe fn init(&self) -> Result<(), &'static str> { + self.periph.init() + } } impl exception::asynchronous::interface::IRQManager for InterruptController { diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_pl011_uart.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_pl011_uart.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/device_driver/bcm/bcm2xxx_pl011_uart.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/device_driver/bcm/bcm2xxx_pl011_uart.rs @@ -10,10 +10,13 @@ //! - use crate::{ - bsp, bsp::device_driver::common::MMIODerefWrapper, console, cpu, driver, exception, + bsp, bsp::device_driver::common::MMIODerefWrapper, console, cpu, driver, exception, memory, synchronization, synchronization::IRQSafeNullLock, }; -use core::fmt; +use core::{ + fmt, + sync::atomic::{AtomicUsize, Ordering}, +}; use tock_registers::{ interfaces::{Readable, Writeable}, register_bitfields, register_structs, @@ -231,6 +234,8 @@ /// Representation of the UART. pub struct PL011Uart { + mmio_descriptor: memory::mmu::MMIODescriptor, + virt_mmio_start_addr: AtomicUsize, inner: IRQSafeNullLock, irq_number: bsp::device_driver::IRQNumber, } @@ -270,7 +275,15 @@ /// genrated baud rate of `48_000_000 / (16 * 3.25) = 923_077`. /// /// Error = `((923_077 - 921_600) / 921_600) * 100 = 0.16modulo`. - pub fn init(&mut self) { + /// + /// # Safety + /// + /// - The user must ensure to provide a correct MMIO start address. + pub unsafe fn init(&mut self, new_mmio_start_addr: Option) -> Result<(), &'static str> { + if let Some(addr) = new_mmio_start_addr { + self.registers = Registers::new(addr); + } + // Execution can arrive here while there are still characters queued in the TX FIFO and // actively being sent out by the UART hardware. If the UART is turned off in this case, // those queued characters would be lost. @@ -312,6 +325,8 @@ self.registers .CR .write(CR::UARTEN::Enabled + CR::TXE::Enabled + CR::RXE::Enabled); + + Ok(()) } /// Send a character. @@ -389,13 +404,18 @@ /// /// # Safety /// - /// - The user must ensure to provide a correct MMIO start address. + /// - The user must ensure to provide correct MMIO descriptors. + /// - The user must ensure to provide correct IRQ numbers. pub const unsafe fn new( - mmio_start_addr: usize, + mmio_descriptor: memory::mmu::MMIODescriptor, irq_number: bsp::device_driver::IRQNumber, ) -> Self { Self { - inner: IRQSafeNullLock::new(PL011UartInner::new(mmio_start_addr)), + mmio_descriptor, + virt_mmio_start_addr: AtomicUsize::new(0), + inner: IRQSafeNullLock::new(PL011UartInner::new( + mmio_descriptor.start_addr().as_usize(), + )), irq_number, } } @@ -412,7 +432,13 @@ } unsafe fn init(&self) -> Result<(), &'static str> { - self.inner.lock(|inner| inner.init()); + let virt_addr = memory::mmu::kernel_map_mmio(self.compatible(), &self.mmio_descriptor)?; + + self.inner + .lock(|inner| inner.init(Some(virt_addr.as_usize())))?; + + self.virt_mmio_start_addr + .store(virt_addr.as_usize(), Ordering::Relaxed); Ok(()) } @@ -431,6 +457,16 @@ Ok(()) } + + fn virt_mmio_start_addr(&self) -> Option { + let addr = self.virt_mmio_start_addr.load(Ordering::Relaxed); + + if addr == 0 { + return None; + } + + Some(addr) + } } impl console::interface::Write for PL011Uart { diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/console.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/console.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/console.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/console.rs @@ -5,7 +5,7 @@ //! BSP console facilities. use super::memory; -use crate::{bsp::device_driver, console}; +use crate::{bsp::device_driver, console, cpu, driver}; use core::fmt; //-------------------------------------------------------------------------------------------------- @@ -23,11 +23,25 @@ /// /// - Use only for printing during a panic. pub unsafe fn panic_console_out() -> impl fmt::Write { - let mut panic_gpio = device_driver::PanicGPIO::new(memory::map::mmio::GPIO_START); - let mut panic_uart = device_driver::PanicUart::new(memory::map::mmio::PL011_UART_START); + use driver::interface::DeviceDriver; + let mut panic_gpio = device_driver::PanicGPIO::new(memory::map::mmio::GPIO_START.as_usize()); + let mut panic_uart = + device_driver::PanicUart::new(memory::map::mmio::PL011_UART_START.as_usize()); + + // If remapping of the driver's MMIO already happened, take the remapped start address. + // Otherwise, take a chance with the default physical address. + let maybe_gpio_mmio_start_addr = super::GPIO.virt_mmio_start_addr(); + let maybe_uart_mmio_start_addr = super::PL011_UART.virt_mmio_start_addr(); + + panic_gpio + .init(maybe_gpio_mmio_start_addr) + .unwrap_or_else(|_| cpu::wait_forever()); panic_gpio.map_pl011_uart(); - panic_uart.init(); + panic_uart + .init(maybe_uart_mmio_start_addr) + .unwrap_or_else(|_| cpu::wait_forever()); + panic_uart } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/driver.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/driver.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/driver.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/driver.rs @@ -46,7 +46,15 @@ &self.device_drivers[..] } - fn post_device_driver_init(&self) { + fn early_print_device_drivers(&self) -> &[&'static (dyn DeviceDriver + Sync)] { + &self.device_drivers[0..=1] + } + + fn non_early_print_device_drivers(&self) -> &[&'static (dyn DeviceDriver + Sync)] { + &self.device_drivers[2..] + } + + fn post_early_print_device_driver_init(&self) { // Configure PL011Uart's output pins. super::GPIO.map_pl011_uart(); } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/kernel.ld 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/kernel.ld --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/kernel.ld +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/kernel.ld @@ -38,7 +38,7 @@ ***********************************************************************************************/ .boot_core_stack (NOLOAD) : { - /* ^ */ + __boot_core_stack_start = .; /* ^ */ /* | stack */ . += __rpi_phys_binary_load_addr; /* | growth */ /* | direction */ @@ -68,6 +68,7 @@ /*********************************************************************************************** * Data + BSS ***********************************************************************************************/ + __data_start = .; .data : { *(.data*) } :segment_data /* Section is zeroed in pairs of u64. Align start and end to 16 bytes */ @@ -78,4 +79,16 @@ . = ALIGN(16); __bss_end_exclusive = .; } :segment_data + + . = ALIGN(PAGE_SIZE); + __data_end_exclusive = .; + + /*********************************************************************************************** + * MMIO Remap Reserved + ***********************************************************************************************/ + __mmio_remap_start = .; + . += 8 * 1024 * 1024; + __mmio_remap_end_exclusive = .; + + ASSERT((. & PAGE_MASK) == 0, "MMIO remap reservation is not page aligned") } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/memory/mmu.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/memory/mmu.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/memory/mmu.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/memory/mmu.rs @@ -4,70 +4,163 @@ //! BSP Memory Management Unit. -use super::map as memory_map; -use crate::memory::mmu::*; -use core::ops::RangeInclusive; +use crate::{ + memory::{ + mmu::{ + self as generic_mmu, AccessPermissions, AddressSpace, AssociatedTranslationTable, + AttributeFields, MemAttributes, MemoryRegion, PageAddress, TranslationGranule, + }, + Physical, Virtual, + }, + synchronization::InitStateLock, +}; + +//-------------------------------------------------------------------------------------------------- +// Private Definitions +//-------------------------------------------------------------------------------------------------- + +type KernelTranslationTable = + ::TableStartFromBottom; //-------------------------------------------------------------------------------------------------- // Public Definitions //-------------------------------------------------------------------------------------------------- -/// The kernel's address space defined by this BSP. -pub type KernelAddrSpace = AddressSpace<{ memory_map::END_INCLUSIVE + 1 }>; +/// The translation granule chosen by this BSP. This will be used everywhere else in the kernel to +/// derive respective data structures and their sizes. For example, the `crate::memory::mmu::Page`. +pub type KernelGranule = TranslationGranule<{ 64 * 1024 }>; + +/// The kernel's virtual address space defined by this BSP. +pub type KernelVirtAddrSpace = AddressSpace<{ 1024 * 1024 * 1024 }>; -const NUM_MEM_RANGES: usize = 2; +//-------------------------------------------------------------------------------------------------- +// Global instances +//-------------------------------------------------------------------------------------------------- -/// The virtual memory layout. +/// The kernel translation tables. /// -/// The layout must contain only special ranges, aka anything that is _not_ normal cacheable DRAM. -/// It is agnostic of the paging granularity that the architecture's MMU will use. -pub static LAYOUT: KernelVirtualLayout = KernelVirtualLayout::new( - memory_map::END_INCLUSIVE, - [ - TranslationDescriptor { - name: "Kernel code and RO data", - virtual_range: code_range_inclusive, - physical_range_translation: Translation::Identity, - attribute_fields: AttributeFields { - mem_attributes: MemAttributes::CacheableDRAM, - acc_perms: AccessPermissions::ReadOnly, - execute_never: false, - }, - }, - TranslationDescriptor { - name: "Device MMIO", - virtual_range: mmio_range_inclusive, - physical_range_translation: Translation::Identity, - attribute_fields: AttributeFields { - mem_attributes: MemAttributes::Device, - acc_perms: AccessPermissions::ReadWrite, - execute_never: true, - }, - }, - ], -); +/// It is mandatory that InitStateLock is transparent. +/// +/// That is, `size_of(InitStateLock) == size_of(KernelTranslationTable)`. +/// There is a unit tests that checks this porperty. +static KERNEL_TABLES: InitStateLock = + InitStateLock::new(KernelTranslationTable::new()); //-------------------------------------------------------------------------------------------------- // Private Code //-------------------------------------------------------------------------------------------------- -fn code_range_inclusive() -> RangeInclusive { - // Notice the subtraction to turn the exclusive end into an inclusive end. - #[allow(clippy::range_minus_one)] - RangeInclusive::new(super::code_start(), super::code_end_exclusive() - 1) +/// Helper function for calculating the number of pages the given parameter spans. +const fn size_to_num_pages(size: usize) -> usize { + assert!(size > 0); + assert!(size modulo KernelGranule::SIZE == 0); + + size >> KernelGranule::SHIFT +} + +/// The code pages of the kernel binary. +fn virt_code_region() -> MemoryRegion { + let num_pages = size_to_num_pages(super::code_size()); + + let start_page_addr = super::virt_code_start(); + let end_exclusive_page_addr = start_page_addr.checked_offset(num_pages as isize).unwrap(); + + MemoryRegion::new(start_page_addr, end_exclusive_page_addr) +} + +/// The data pages of the kernel binary. +fn virt_data_region() -> MemoryRegion { + let num_pages = size_to_num_pages(super::data_size()); + + let start_page_addr = super::virt_data_start(); + let end_exclusive_page_addr = start_page_addr.checked_offset(num_pages as isize).unwrap(); + + MemoryRegion::new(start_page_addr, end_exclusive_page_addr) +} + +/// The boot core stack pages. +fn virt_boot_core_stack_region() -> MemoryRegion { + let num_pages = size_to_num_pages(super::boot_core_stack_size()); + + let start_page_addr = super::virt_boot_core_stack_start(); + let end_exclusive_page_addr = start_page_addr.checked_offset(num_pages as isize).unwrap(); + + MemoryRegion::new(start_page_addr, end_exclusive_page_addr) } -fn mmio_range_inclusive() -> RangeInclusive { - RangeInclusive::new(memory_map::mmio::START, memory_map::mmio::END_INCLUSIVE) +// The binary is still identity mapped, so use this trivial conversion function for mapping below. + +fn kernel_virt_to_phys_region(virt_region: MemoryRegion) -> MemoryRegion { + MemoryRegion::new( + PageAddress::from(virt_region.start_page_addr().into_inner().as_usize()), + PageAddress::from( + virt_region + .end_exclusive_page_addr() + .into_inner() + .as_usize(), + ), + ) } //-------------------------------------------------------------------------------------------------- // Public Code //-------------------------------------------------------------------------------------------------- -/// Return a reference to the virtual memory layout. -pub fn virt_mem_layout() -> &'static KernelVirtualLayout { - &LAYOUT +/// Return a reference to the kernel's translation tables. +pub fn kernel_translation_tables() -> &'static InitStateLock { + &KERNEL_TABLES +} + +/// The MMIO remap pages. +pub fn virt_mmio_remap_region() -> MemoryRegion { + let num_pages = size_to_num_pages(super::mmio_remap_size()); + + let start_page_addr = super::virt_mmio_remap_start(); + let end_exclusive_page_addr = start_page_addr.checked_offset(num_pages as isize).unwrap(); + + MemoryRegion::new(start_page_addr, end_exclusive_page_addr) +} + +/// Map the kernel binary. +/// +/// # Safety +/// +/// - Any miscalculation or attribute error will likely be fatal. Needs careful manual checking. +pub unsafe fn kernel_map_binary() -> Result<(), &'static str> { + generic_mmu::kernel_map_at( + "Kernel boot-core stack", + &virt_boot_core_stack_region(), + &kernel_virt_to_phys_region(virt_boot_core_stack_region()), + &AttributeFields { + mem_attributes: MemAttributes::CacheableDRAM, + acc_perms: AccessPermissions::ReadWrite, + execute_never: true, + }, + )?; + + generic_mmu::kernel_map_at( + "Kernel code and RO data", + &virt_code_region(), + &kernel_virt_to_phys_region(virt_code_region()), + &AttributeFields { + mem_attributes: MemAttributes::CacheableDRAM, + acc_perms: AccessPermissions::ReadOnly, + execute_never: false, + }, + )?; + + generic_mmu::kernel_map_at( + "Kernel data and bss", + &virt_data_region(), + &kernel_virt_to_phys_region(virt_data_region()), + &AttributeFields { + mem_attributes: MemAttributes::CacheableDRAM, + acc_perms: AccessPermissions::ReadWrite, + execute_never: true, + }, + )?; + + Ok(()) } //-------------------------------------------------------------------------------------------------- @@ -77,38 +170,60 @@ #[cfg(test)] mod tests { use super::*; + use core::{cell::UnsafeCell, ops::Range}; use test_macros::kernel_test; /// Check alignment of the kernel's virtual memory layout sections. #[kernel_test] fn virt_mem_layout_sections_are_64KiB_aligned() { - const SIXTYFOUR_KIB: usize = 65536; - - for i in LAYOUT.inner().iter() { - let start: usize = *(i.virtual_range)().start(); - let end: usize = *(i.virtual_range)().end() + 1; - - assert_eq!(start modulo SIXTYFOUR_KIB, 0); - assert_eq!(end modulo SIXTYFOUR_KIB, 0); - assert!(end >= start); + for i in [ + virt_boot_core_stack_region, + virt_code_region, + virt_data_region, + ] + .iter() + { + let start = i().start_page_addr().into_inner(); + let end_exclusive = i().end_exclusive_page_addr().into_inner(); + + assert!(start.is_page_aligned()); + assert!(end_exclusive.is_page_aligned()); + assert!(end_exclusive >= start); } } /// Ensure the kernel's virtual memory layout is free of overlaps. #[kernel_test] fn virt_mem_layout_has_no_overlaps() { - let layout = virt_mem_layout().inner(); - - for (i, first) in layout.iter().enumerate() { - for second in layout.iter().skip(i + 1) { - let first_range = first.virtual_range; - let second_range = second.virtual_range; - - assert!(!first_range().contains(second_range().start())); - assert!(!first_range().contains(second_range().end())); - assert!(!second_range().contains(first_range().start())); - assert!(!second_range().contains(first_range().end())); + let layout = [ + virt_boot_core_stack_region(), + virt_code_region(), + virt_data_region(), + ]; + + for (i, first_range) in layout.iter().enumerate() { + for second_range in layout.iter().skip(i + 1) { + assert!(!first_range.overlaps(second_range)) } } } + + /// Check if KERNEL_TABLES is in .bss. + #[kernel_test] + fn kernel_tables_in_bss() { + extern "Rust" { + static __bss_start: UnsafeCell; + static __bss_end_exclusive: UnsafeCell; + } + + let bss_range = unsafe { + Range { + start: __bss_start.get(), + end: __bss_end_exclusive.get(), + } + }; + let kernel_tables_addr = &KERNEL_TABLES as *const _ as usize as *mut u64; + + assert!(bss_range.contains(&kernel_tables_addr)); + } } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/memory.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/memory.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi/memory.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi/memory.rs @@ -10,27 +10,59 @@ //! as the boot core's stack. //! //! +---------------------------------------+ -//! | | 0x0 +//! | | boot_core_stack_start @ 0x0 //! | | ^ //! | Boot-core Stack | | stack //! | | | growth //! | | | direction //! +---------------------------------------+ -//! | | code_start @ 0x8_0000 +//! | | code_start @ 0x8_0000 == boot_core_stack_end_exclusive //! | .text | //! | .rodata | //! | .got | //! | | //! +---------------------------------------+ -//! | | code_end_exclusive +//! | | data_start == code_end_exclusive //! | .data | //! | .bss | //! | | //! +---------------------------------------+ +//! | | data_end_exclusive //! | | +//! +//! +//! +//! +//! +//! The virtual memory layout is as follows: +//! +//! +---------------------------------------+ +//! | | boot_core_stack_start @ 0x0 +//! | | ^ +//! | Boot-core Stack | | stack +//! | | | growth +//! | | | direction +//! +---------------------------------------+ +//! | | code_start @ 0x8_0000 == boot_core_stack_end_exclusive +//! | .text | +//! | .rodata | +//! | .got | +//! | | +//! +---------------------------------------+ +//! | | data_start == code_end_exclusive +//! | .data | +//! | .bss | +//! | | +//! +---------------------------------------+ +//! | | mmio_remap_start == data_end_exclusive +//! | VA region for MMIO remapping | +//! | | +//! +---------------------------------------+ +//! | | mmio_remap_end_exclusive //! | | pub mod mmu; +use crate::memory::{mmu::PageAddress, Address, Physical, Virtual}; use core::cell::UnsafeCell; //-------------------------------------------------------------------------------------------------- @@ -41,6 +73,15 @@ extern "Rust" { static __code_start: UnsafeCell<()>; static __code_end_exclusive: UnsafeCell<()>; + + static __data_start: UnsafeCell<()>; + static __data_end_exclusive: UnsafeCell<()>; + + static __mmio_remap_start: UnsafeCell<()>; + static __mmio_remap_end_exclusive: UnsafeCell<()>; + + static __boot_core_stack_start: UnsafeCell<()>; + static __boot_core_stack_end_exclusive: UnsafeCell<()>; } //-------------------------------------------------------------------------------------------------- @@ -50,35 +91,26 @@ /// The board's physical memory map. #[rustfmt::skip] pub(super) mod map { - /// The inclusive end address of the memory map. - /// - /// End address + 1 must be power of two. - /// - /// # Note - /// - /// RPi3 and RPi4 boards can have different amounts of RAM. To make our code lean for - /// educational purposes, we set the max size of the address space to 4 GiB regardless of board. - /// This way, we can map the entire range that we need (end of MMIO for RPi4) in one take. - /// - /// However, making this trade-off has the downside of making it possible for the CPU to assert a - /// physical address that is not backed by any DRAM (e.g. accessing an address close to 4 GiB on - /// an RPi3 that comes with 1 GiB of RAM). This would result in a crash or other kind of error. - pub const END_INCLUSIVE: usize = 0xFFFF_FFFF; - - pub const GPIO_OFFSET: usize = 0x0020_0000; - pub const UART_OFFSET: usize = 0x0020_1000; + use super::*; /// Physical devices. #[cfg(feature = "bsp_rpi3")] pub mod mmio { use super::*; - pub const START: usize = 0x3F00_0000; - pub const PERIPHERAL_INTERRUPT_CONTROLLER_START: usize = START + 0x0000_B200; - pub const GPIO_START: usize = START + GPIO_OFFSET; - pub const PL011_UART_START: usize = START + UART_OFFSET; - pub const LOCAL_INTERRUPT_CONTROLLER_START: usize = 0x4000_0000; - pub const END_INCLUSIVE: usize = 0x4000_FFFF; + pub const PERIPHERAL_IC_START: Address = Address::new(0x3F00_B200); + pub const PERIPHERAL_IC_SIZE: usize = 0x24; + + pub const GPIO_START: Address = Address::new(0x3F20_0000); + pub const GPIO_SIZE: usize = 0xA0; + + pub const PL011_UART_START: Address = Address::new(0x3F20_1000); + pub const PL011_UART_SIZE: usize = 0x48; + + pub const LOCAL_IC_START: Address = Address::new(0x4000_0000); + pub const LOCAL_IC_SIZE: usize = 0x100; + + pub const END: Address = Address::new(0x4001_0000); } /// Physical devices. @@ -86,13 +118,22 @@ pub mod mmio { use super::*; - pub const START: usize = 0xFE00_0000; - pub const GPIO_START: usize = START + GPIO_OFFSET; - pub const PL011_UART_START: usize = START + UART_OFFSET; - pub const GICD_START: usize = 0xFF84_1000; - pub const GICC_START: usize = 0xFF84_2000; - pub const END_INCLUSIVE: usize = 0xFF84_FFFF; + pub const GPIO_START: Address = Address::new(0xFE20_0000); + pub const GPIO_SIZE: usize = 0xA0; + + pub const PL011_UART_START: Address = Address::new(0xFE20_1000); + pub const PL011_UART_SIZE: usize = 0x48; + + pub const GICD_START: Address = Address::new(0xFF84_1000); + pub const GICD_SIZE: usize = 0x824; + + pub const GICC_START: Address = Address::new(0xFF84_2000); + pub const GICC_SIZE: usize = 0x14; + + pub const END: Address = Address::new(0xFF85_0000); } + + pub const END: Address = mmio::END; } //-------------------------------------------------------------------------------------------------- @@ -105,15 +146,76 @@ /// /// - Value is provided by the linker script and must be trusted as-is. #[inline(always)] -fn code_start() -> usize { - unsafe { __code_start.get() as usize } +fn virt_code_start() -> PageAddress { + PageAddress::from(unsafe { __code_start.get() as usize }) +} + +/// Size of the code segment. +/// +/// # Safety +/// +/// - Value is provided by the linker script and must be trusted as-is. +#[inline(always)] +fn code_size() -> usize { + unsafe { (__code_end_exclusive.get() as usize) - (__code_start.get() as usize) } } -/// Exclusive end page address of the code segment. +/// Start page address of the data segment. +#[inline(always)] +fn virt_data_start() -> PageAddress { + PageAddress::from(unsafe { __data_start.get() as usize }) +} + +/// Size of the data segment. +/// /// # Safety /// /// - Value is provided by the linker script and must be trusted as-is. #[inline(always)] -fn code_end_exclusive() -> usize { - unsafe { __code_end_exclusive.get() as usize } +fn data_size() -> usize { + unsafe { (__data_end_exclusive.get() as usize) - (__data_start.get() as usize) } +} + +/// Start page address of the MMIO remap reservation. +/// +/// # Safety +/// +/// - Value is provided by the linker script and must be trusted as-is. +#[inline(always)] +fn virt_mmio_remap_start() -> PageAddress { + PageAddress::from(unsafe { __mmio_remap_start.get() as usize }) +} + +/// Size of the MMIO remap reservation. +/// +/// # Safety +/// +/// - Value is provided by the linker script and must be trusted as-is. +#[inline(always)] +fn mmio_remap_size() -> usize { + unsafe { (__mmio_remap_end_exclusive.get() as usize) - (__mmio_remap_start.get() as usize) } +} + +/// Start page address of the boot core's stack. +#[inline(always)] +fn virt_boot_core_stack_start() -> PageAddress { + PageAddress::from(unsafe { __boot_core_stack_start.get() as usize }) +} + +/// Size of the boot core's stack. +#[inline(always)] +fn boot_core_stack_size() -> usize { + unsafe { + (__boot_core_stack_end_exclusive.get() as usize) - (__boot_core_stack_start.get() as usize) + } +} + +//-------------------------------------------------------------------------------------------------- +// Public Code +//-------------------------------------------------------------------------------------------------- + +/// Exclusive end address of the physical address space. +#[inline(always)] +pub fn phys_addr_space_end_exclusive_addr() -> PageAddress { + PageAddress::from(map::END) } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi.rs 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/bsp/raspberrypi.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/bsp/raspberrypi.rs @@ -10,17 +10,20 @@ pub mod exception; pub mod memory; +use super::device_driver; +use crate::memory::mmu::MMIODescriptor; +use memory::map::mmio; + //-------------------------------------------------------------------------------------------------- // Global instances //-------------------------------------------------------------------------------------------------- -use super::device_driver; static GPIO: device_driver::GPIO = - unsafe { device_driver::GPIO::new(memory::map::mmio::GPIO_START) }; + unsafe { device_driver::GPIO::new(MMIODescriptor::new(mmio::GPIO_START, mmio::GPIO_SIZE)) }; static PL011_UART: device_driver::PL011Uart = unsafe { device_driver::PL011Uart::new( - memory::map::mmio::PL011_UART_START, + MMIODescriptor::new(mmio::PL011_UART_START, mmio::PL011_UART_SIZE), exception::asynchronous::irq_map::PL011_UART, ) }; @@ -28,14 +31,17 @@ #[cfg(feature = "bsp_rpi3")] static INTERRUPT_CONTROLLER: device_driver::InterruptController = unsafe { device_driver::InterruptController::new( - memory::map::mmio::LOCAL_INTERRUPT_CONTROLLER_START, - memory::map::mmio::PERIPHERAL_INTERRUPT_CONTROLLER_START, + MMIODescriptor::new(mmio::LOCAL_IC_START, mmio::LOCAL_IC_SIZE), + MMIODescriptor::new(mmio::PERIPHERAL_IC_START, mmio::PERIPHERAL_IC_SIZE), ) }; #[cfg(feature = "bsp_rpi4")] static INTERRUPT_CONTROLLER: device_driver::GICv2 = unsafe { - device_driver::GICv2::new(memory::map::mmio::GICD_START, memory::map::mmio::GICC_START) + device_driver::GICv2::new( + MMIODescriptor::new(mmio::GICD_START, mmio::GICD_SIZE), + MMIODescriptor::new(mmio::GICC_START, mmio::GICC_SIZE), + ) }; //-------------------------------------------------------------------------------------------------- diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/common.rs 14_virtual_mem_part2_mmio_remap/kernel/src/common.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/common.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/common.rs @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 +// +// Copyright (c) 2020-2022 Andre Richter + +//! General purpose code. + +/// Check if a value is aligned to a given size. +#[inline(always)] +pub const fn is_aligned(value: usize, alignment: usize) -> bool { + assert!(alignment.is_power_of_two()); + + (value & (alignment - 1)) == 0 +} + +/// Align down. +#[inline(always)] +pub const fn align_down(value: usize, alignment: usize) -> usize { + assert!(alignment.is_power_of_two()); + + value & !(alignment - 1) +} + +/// Align up. +#[inline(always)] +pub const fn align_up(value: usize, alignment: usize) -> usize { + assert!(alignment.is_power_of_two()); + + (value + alignment - 1) & !(alignment - 1) +} diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/driver.rs 14_virtual_mem_part2_mmio_remap/kernel/src/driver.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/driver.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/driver.rs @@ -31,6 +31,14 @@ fn register_and_enable_irq_handler(&'static self) -> Result<(), &'static str> { Ok(()) } + + /// After MMIO remapping, returns the new virtual start address. + /// + /// This API assumes a driver has only a single, contiguous MMIO aperture, which will not be + /// the case for more complex devices. This API will likely change in future tutorials. + fn virt_mmio_start_addr(&self) -> Option { + None + } } /// Device driver management functions. @@ -38,15 +46,17 @@ /// The `BSP` is supposed to supply one global instance. pub trait DriverManager { /// Return a slice of references to all `BSP`-instantiated drivers. - /// - /// # Safety - /// - /// - The order of devices is the order in which `DeviceDriver::init()` is called. fn all_device_drivers(&self) -> &[&'static (dyn DeviceDriver + Sync)]; - /// Initialization code that runs after driver init. + /// Return only those drivers needed for the BSP's early printing functionality. /// - /// For example, device driver code that depends on other drivers already being online. - fn post_device_driver_init(&self); + /// For example, the default UART. + fn early_print_device_drivers(&self) -> &[&'static (dyn DeviceDriver + Sync)]; + + /// Return all drivers minus early-print drivers. + fn non_early_print_device_drivers(&self) -> &[&'static (dyn DeviceDriver + Sync)]; + + /// Initialization code that runs after the early print driver init. + fn post_early_print_device_driver_init(&self); } } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/lib.rs 14_virtual_mem_part2_mmio_remap/kernel/src/lib.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/lib.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/lib.rs @@ -113,8 +113,10 @@ #![feature(asm_const)] #![feature(core_intrinsics)] #![feature(format_args_nl)] +#![feature(generic_const_exprs)] #![feature(linkage)] #![feature(panic_info_message)] +#![feature(step_trait)] #![feature(trait_alias)] #![no_std] // Testing @@ -127,6 +129,7 @@ mod synchronization; pub mod bsp; +pub mod common; pub mod console; pub mod cpu; pub mod driver; @@ -179,6 +182,7 @@ #[no_mangle] unsafe fn kernel_init() -> ! { exception::handling_init(); + memory::mmu::post_enable_init(); bsp::console::qemu_bring_up_console(); test_main(); diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/main.rs 14_virtual_mem_part2_mmio_remap/kernel/src/main.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/main.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/main.rs @@ -27,21 +27,41 @@ #[no_mangle] unsafe fn kernel_init() -> ! { use driver::interface::DriverManager; - use memory::mmu::interface::MMU; exception::handling_init(); - if let Err(string) = memory::mmu::mmu().enable_mmu_and_caching() { - panic!("MMU: {}", string); + let phys_kernel_tables_base_addr = match memory::mmu::kernel_map_binary() { + Err(string) => panic!("Error mapping kernel binary: {}", string), + Ok(addr) => addr, + }; + + if let Err(e) = memory::mmu::enable_mmu_and_caching(phys_kernel_tables_base_addr) { + panic!("Enabling MMU failed: {}", e); + } + // Printing will silently fail from here on, because the driver's MMIO is not remapped yet. + + memory::mmu::post_enable_init(); + + // Bring up the drivers needed for printing first. + for i in bsp::driver::driver_manager() + .early_print_device_drivers() + .iter() + { + // Any encountered errors cannot be printed yet, obviously, so just safely park the CPU. + i.init().unwrap_or_else(|_| cpu::wait_forever()); } + bsp::driver::driver_manager().post_early_print_device_driver_init(); + // Printing available again from here on. - for i in bsp::driver::driver_manager().all_device_drivers().iter() { + // Now bring up the remaining drivers. + for i in bsp::driver::driver_manager() + .non_early_print_device_drivers() + .iter() + { if let Err(x) = i.init() { panic!("Error loading driver: {}: {}", i.compatible(), x); } } - bsp::driver::driver_manager().post_device_driver_init(); - // println! is usable from here on. // Let device drivers register and enable their handlers with the interrupt controller. for i in bsp::driver::driver_manager().all_device_drivers() { @@ -68,8 +88,8 @@ info!("{}", libkernel::version()); info!("Booting on: {}", bsp::board_name()); - info!("MMU online. Special regions:"); - bsp::memory::mmu::virt_mem_layout().print_layout(); + info!("MMU online:"); + memory::mmu::kernel_print_mappings(); let (_, privilege_level) = exception::current_privilege_level(); info!("Current privilege level: {}", privilege_level); diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/mapping_record.rs 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/mapping_record.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/mapping_record.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/mapping_record.rs @@ -0,0 +1,233 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 +// +// Copyright (c) 2020-2022 Andre Richter + +//! A record of mapped pages. + +use super::{ + AccessPermissions, Address, AttributeFields, MMIODescriptor, MemAttributes, MemoryRegion, + Physical, Virtual, +}; +use crate::{bsp, info, synchronization, synchronization::InitStateLock, warn}; + +//-------------------------------------------------------------------------------------------------- +// Private Definitions +//-------------------------------------------------------------------------------------------------- + +/// Type describing a virtual memory mapping. +#[allow(missing_docs)] +#[derive(Copy, Clone)] +struct MappingRecordEntry { + pub users: [Option<&'static str>; 5], + pub phys_start_addr: Address, + pub virt_start_addr: Address, + pub num_pages: usize, + pub attribute_fields: AttributeFields, +} + +struct MappingRecord { + inner: [Option; 12], +} + +//-------------------------------------------------------------------------------------------------- +// Global instances +//-------------------------------------------------------------------------------------------------- + +static KERNEL_MAPPING_RECORD: InitStateLock = + InitStateLock::new(MappingRecord::new()); + +//-------------------------------------------------------------------------------------------------- +// Private Code +//-------------------------------------------------------------------------------------------------- + +impl MappingRecordEntry { + pub fn new( + name: &'static str, + virt_region: &MemoryRegion, + phys_region: &MemoryRegion, + attr: &AttributeFields, + ) -> Self { + Self { + users: [Some(name), None, None, None, None], + phys_start_addr: phys_region.start_addr(), + virt_start_addr: virt_region.start_addr(), + num_pages: phys_region.num_pages(), + attribute_fields: *attr, + } + } + + fn find_next_free_user(&mut self) -> Result<&mut Option<&'static str>, &'static str> { + if let Some(x) = self.users.iter_mut().find(|x| x.is_none()) { + return Ok(x); + }; + + Err("Storage for user info exhausted") + } + + pub fn add_user(&mut self, user: &'static str) -> Result<(), &'static str> { + let x = self.find_next_free_user()?; + *x = Some(user); + Ok(()) + } +} + +impl MappingRecord { + pub const fn new() -> Self { + Self { inner: [None; 12] } + } + + fn find_next_free(&mut self) -> Result<&mut Option, &'static str> { + if let Some(x) = self.inner.iter_mut().find(|x| x.is_none()) { + return Ok(x); + } + + Err("Storage for mapping info exhausted") + } + + fn find_duplicate( + &mut self, + phys_region: &MemoryRegion, + ) -> Option<&mut MappingRecordEntry> { + self.inner + .iter_mut() + .filter(|x| x.is_some()) + .map(|x| x.as_mut().unwrap()) + .filter(|x| x.attribute_fields.mem_attributes == MemAttributes::Device) + .find(|x| { + if x.phys_start_addr != phys_region.start_addr() { + return false; + } + + if x.num_pages != phys_region.num_pages() { + return false; + } + + true + }) + } + + pub fn add( + &mut self, + name: &'static str, + virt_region: &MemoryRegion, + phys_region: &MemoryRegion, + attr: &AttributeFields, + ) -> Result<(), &'static str> { + let x = self.find_next_free()?; + + *x = Some(MappingRecordEntry::new( + name, + virt_region, + phys_region, + attr, + )); + Ok(()) + } + + pub fn print(&self) { + const KIB_RSHIFT: u32 = 10; // log2(1024). + const MIB_RSHIFT: u32 = 20; // log2(1024 * 1024). + + info!(" -------------------------------------------------------------------------------------------------------------------------------------------"); + info!( + " {:^44} {:^30} {:^7} {:^9} {:^35}", + "Virtual", "Physical", "Size", "Attr", "Entity" + ); + info!(" -------------------------------------------------------------------------------------------------------------------------------------------"); + + for i in self.inner.iter().flatten() { + let size = i.num_pages * bsp::memory::mmu::KernelGranule::SIZE; + let virt_start = i.virt_start_addr; + let virt_end_inclusive = virt_start + (size - 1); + let phys_start = i.phys_start_addr; + let phys_end_inclusive = phys_start + (size - 1); + + let (size, unit) = if (size >> MIB_RSHIFT) > 0 { + (size >> MIB_RSHIFT, "MiB") + } else if (size >> KIB_RSHIFT) > 0 { + (size >> KIB_RSHIFT, "KiB") + } else { + (size, "Byte") + }; + + let attr = match i.attribute_fields.mem_attributes { + MemAttributes::CacheableDRAM => "C", + MemAttributes::Device => "Dev", + }; + + let acc_p = match i.attribute_fields.acc_perms { + AccessPermissions::ReadOnly => "RO", + AccessPermissions::ReadWrite => "RW", + }; + + let xn = if i.attribute_fields.execute_never { + "XN" + } else { + "X" + }; + + info!( + " {}..{} --> {}..{} | \ + {: >3} {} | {: <3} {} {: <2} | {}", + virt_start, + virt_end_inclusive, + phys_start, + phys_end_inclusive, + size, + unit, + attr, + acc_p, + xn, + i.users[0].unwrap() + ); + + for k in i.users[1..].iter() { + if let Some(additional_user) = *k { + info!( + " | {}", + additional_user + ); + } + } + } + + info!(" -------------------------------------------------------------------------------------------------------------------------------------------"); + } +} + +//-------------------------------------------------------------------------------------------------- +// Public Code +//-------------------------------------------------------------------------------------------------- +use synchronization::interface::ReadWriteEx; + +/// Add an entry to the mapping info record. +pub fn kernel_add( + name: &'static str, + virt_region: &MemoryRegion, + phys_region: &MemoryRegion, + attr: &AttributeFields, +) -> Result<(), &'static str> { + KERNEL_MAPPING_RECORD.write(|mr| mr.add(name, virt_region, phys_region, attr)) +} + +pub fn kernel_find_and_insert_mmio_duplicate( + mmio_descriptor: &MMIODescriptor, + new_user: &'static str, +) -> Option> { + let phys_region: MemoryRegion = (*mmio_descriptor).into(); + + KERNEL_MAPPING_RECORD.write(|mr| { + let dup = mr.find_duplicate(&phys_region)?; + + if let Err(x) = dup.add_user(new_user) { + warn!("{}", x); + } + + Some(dup.virt_start_addr) + }) +} + +/// Human-readable print of all recorded kernel mappings. +pub fn kernel_print() { + KERNEL_MAPPING_RECORD.read(|mr| mr.print()); +} diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/page_alloc.rs 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/page_alloc.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/page_alloc.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/page_alloc.rs @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 +// +// Copyright (c) 2021-2022 Andre Richter + +//! Page allocation. + +use super::MemoryRegion; +use crate::{ + memory::{AddressType, Virtual}, + synchronization::IRQSafeNullLock, + warn, +}; +use core::num::NonZeroUsize; + +//-------------------------------------------------------------------------------------------------- +// Public Definitions +//-------------------------------------------------------------------------------------------------- + +/// A page allocator that can be lazyily initialized. +pub struct PageAllocator { + pool: Option>, +} + +//-------------------------------------------------------------------------------------------------- +// Global instances +//-------------------------------------------------------------------------------------------------- + +static KERNEL_MMIO_VA_ALLOCATOR: IRQSafeNullLock> = + IRQSafeNullLock::new(PageAllocator::new()); + +//-------------------------------------------------------------------------------------------------- +// Public Code +//-------------------------------------------------------------------------------------------------- + +/// Return a reference to the kernel's MMIO virtual address allocator. +pub fn kernel_mmio_va_allocator() -> &'static IRQSafeNullLock> { + &KERNEL_MMIO_VA_ALLOCATOR +} + +impl PageAllocator { + /// Create an instance. + pub const fn new() -> Self { + Self { pool: None } + } + + /// Initialize the allocator. + pub fn initialize(&mut self, pool: MemoryRegion) { + if self.pool.is_some() { + warn!("Already initialized"); + return; + } + + self.pool = Some(pool); + } + + /// Allocate a number of pages. + pub fn alloc( + &mut self, + num_requested_pages: NonZeroUsize, + ) -> Result, &'static str> { + if self.pool.is_none() { + return Err("Allocator not initialized"); + } + + self.pool + .as_mut() + .unwrap() + .take_first_n_pages(num_requested_pages) + } +} diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/translation_table.rs 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/translation_table.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/translation_table.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/translation_table.rs @@ -8,7 +8,91 @@ #[path = "../../_arch/aarch64/memory/mmu/translation_table.rs"] mod arch_translation_table; +use super::{AttributeFields, MemoryRegion}; +use crate::memory::{Address, Physical, Virtual}; + //-------------------------------------------------------------------------------------------------- // Architectural Public Reexports //-------------------------------------------------------------------------------------------------- -pub use arch_translation_table::KernelTranslationTable; +#[cfg(target_arch = "aarch64")] +pub use arch_translation_table::FixedSizeTranslationTable; + +//-------------------------------------------------------------------------------------------------- +// Public Definitions +//-------------------------------------------------------------------------------------------------- + +/// Translation table interfaces. +pub mod interface { + use super::*; + + /// Translation table operations. + pub trait TranslationTable { + /// Anything that needs to run before any of the other provided functions can be used. + /// + /// # Safety + /// + /// - Implementor must ensure that this function can run only once or is harmless if invoked + /// multiple times. + fn init(&mut self); + + /// The translation table's base address to be used for programming the MMU. + fn phys_base_address(&self) -> Address; + + /// Map the given virtual memory region to the given physical memory region. + /// + /// # Safety + /// + /// - Using wrong attributes can cause multiple issues of different nature in the system. + /// - It is not required that the architectural implementation prevents aliasing. That is, + /// mapping to the same physical memory using multiple virtual addresses, which would + /// break Rust's ownership assumptions. This should be protected against in the kernel's + /// generic MMU code. + unsafe fn map_at( + &mut self, + virt_region: &MemoryRegion, + phys_region: &MemoryRegion, + attr: &AttributeFields, + ) -> Result<(), &'static str>; + } +} + +//-------------------------------------------------------------------------------------------------- +// Testing +//-------------------------------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + use crate::memory::mmu::{AccessPermissions, MemAttributes, PageAddress}; + use arch_translation_table::MinSizeTranslationTable; + use interface::TranslationTable; + use test_macros::kernel_test; + + /// Sanity checks for the TranslationTable implementation. + #[kernel_test] + fn translationtable_implementation_sanity() { + // This will occupy a lot of space on the stack. + let mut tables = MinSizeTranslationTable::new(); + + tables.init(); + + let virt_start_page_addr: PageAddress = PageAddress::from(0); + let virt_end_exclusive_page_addr: PageAddress = + virt_start_page_addr.checked_offset(5).unwrap(); + + let phys_start_page_addr: PageAddress = PageAddress::from(0); + let phys_end_exclusive_page_addr: PageAddress = + phys_start_page_addr.checked_offset(5).unwrap(); + + let virt_region = MemoryRegion::new(virt_start_page_addr, virt_end_exclusive_page_addr); + let phys_region = MemoryRegion::new(phys_start_page_addr, phys_end_exclusive_page_addr); + + let attr = AttributeFields { + mem_attributes: MemAttributes::CacheableDRAM, + acc_perms: AccessPermissions::ReadWrite, + execute_never: true, + }; + + unsafe { assert_eq!(tables.map_at(&virt_region, &phys_region, &attr), Ok(())) }; + } +} diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/types.rs 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/types.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu/types.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu/types.rs @@ -0,0 +1,373 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 +// +// Copyright (c) 2020-2022 Andre Richter + +//! Memory Management Unit types. + +use crate::{ + bsp, common, + memory::{Address, AddressType, Physical}, +}; +use core::{convert::From, iter::Step, num::NonZeroUsize, ops::Range}; + +//-------------------------------------------------------------------------------------------------- +// Public Definitions +//-------------------------------------------------------------------------------------------------- + +/// A wrapper type around [Address] that ensures page alignment. +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub struct PageAddress { + inner: Address, +} + +/// A type that describes a region of memory in quantities of pages. +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub struct MemoryRegion { + start: PageAddress, + end_exclusive: PageAddress, +} + +/// Architecture agnostic memory attributes. +#[allow(missing_docs)] +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub enum MemAttributes { + CacheableDRAM, + Device, +} + +/// Architecture agnostic access permissions. +#[allow(missing_docs)] +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub enum AccessPermissions { + ReadOnly, + ReadWrite, +} + +/// Collection of memory attributes. +#[allow(missing_docs)] +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub struct AttributeFields { + pub mem_attributes: MemAttributes, + pub acc_perms: AccessPermissions, + pub execute_never: bool, +} + +/// An MMIO descriptor for use in device drivers. +#[derive(Copy, Clone)] +pub struct MMIODescriptor { + start_addr: Address, + end_addr_exclusive: Address, +} + +//-------------------------------------------------------------------------------------------------- +// Public Code +//-------------------------------------------------------------------------------------------------- + +//------------------------------------------------------------------------------ +// PageAddress +//------------------------------------------------------------------------------ +impl PageAddress { + /// Unwraps the value. + pub fn into_inner(self) -> Address { + self.inner + } + + /// Calculates the offset from the page address. + /// + /// `count` is in units of [PageAddress]. For example, a count of 2 means `result = self + 2 * + /// page_size`. + pub fn checked_offset(self, count: isize) -> Option { + if count == 0 { + return Some(self); + } + + let delta = count + .unsigned_abs() + .checked_mul(bsp::memory::mmu::KernelGranule::SIZE)?; + let result = if count.is_positive() { + self.inner.as_usize().checked_add(delta)? + } else { + self.inner.as_usize().checked_sub(delta)? + }; + + Some(Self { + inner: Address::new(result), + }) + } +} + +impl From for PageAddress { + fn from(addr: usize) -> Self { + assert!( + common::is_aligned(addr, bsp::memory::mmu::KernelGranule::SIZE), + "Input usize not page aligned" + ); + + Self { + inner: Address::new(addr), + } + } +} + +impl From> for PageAddress { + fn from(addr: Address) -> Self { + assert!(addr.is_page_aligned(), "Input Address not page aligned"); + + Self { inner: addr } + } +} + +impl Step for PageAddress { + fn steps_between(start: &Self, end: &Self) -> Option { + if start > end { + return None; + } + + // Since start <= end, do unchecked arithmetic. + Some( + (end.inner.as_usize() - start.inner.as_usize()) + >> bsp::memory::mmu::KernelGranule::SHIFT, + ) + } + + fn forward_checked(start: Self, count: usize) -> Option { + start.checked_offset(count as isize) + } + + fn backward_checked(start: Self, count: usize) -> Option { + start.checked_offset(-(count as isize)) + } +} + +//------------------------------------------------------------------------------ +// MemoryRegion +//------------------------------------------------------------------------------ +impl MemoryRegion { + /// Create an instance. + pub fn new(start: PageAddress, end_exclusive: PageAddress) -> Self { + assert!(start <= end_exclusive); + + Self { + start, + end_exclusive, + } + } + + fn as_range(&self) -> Range> { + self.into_iter() + } + + /// Returns the start page address. + pub fn start_page_addr(&self) -> PageAddress { + self.start + } + + /// Returns the start address. + pub fn start_addr(&self) -> Address { + self.start.into_inner() + } + + /// Returns the exclusive end page address. + pub fn end_exclusive_page_addr(&self) -> PageAddress { + self.end_exclusive + } + + /// Returns the exclusive end page address. + pub fn end_inclusive_page_addr(&self) -> PageAddress { + self.end_exclusive.checked_offset(-1).unwrap() + } + + /// Checks if self contains an address. + pub fn contains(&self, addr: Address) -> bool { + let page_addr = PageAddress::from(addr.align_down_page()); + self.as_range().contains(&page_addr) + } + + /// Checks if there is an overlap with another memory region. + pub fn overlaps(&self, other_region: &Self) -> bool { + let self_range = self.as_range(); + + self_range.contains(&other_region.start_page_addr()) + || self_range.contains(&other_region.end_inclusive_page_addr()) + } + + /// Returns the number of pages contained in this region. + pub fn num_pages(&self) -> usize { + PageAddress::steps_between(&self.start, &self.end_exclusive).unwrap() + } + + /// Returns the size in bytes of this region. + pub fn size(&self) -> usize { + // Invariant: start <= end_exclusive, so do unchecked arithmetic. + let end_exclusive = self.end_exclusive.into_inner().as_usize(); + let start = self.start.into_inner().as_usize(); + + end_exclusive - start + } + + /// Splits the MemoryRegion like: + /// + /// -------------------------------------------------------------------------------- + /// | | | | | | | | | | | | | | | | | | | + /// -------------------------------------------------------------------------------- + /// ^ ^ ^ + /// | | | + /// left_start left_end_exclusive | + /// | + /// ^ | + /// | | + /// right_start right_end_exclusive + /// + /// Left region is returned to the caller. Right region is the new region for this struct. + pub fn take_first_n_pages(&mut self, num_pages: NonZeroUsize) -> Result { + let count: usize = num_pages.into(); + + let left_end_exclusive = self.start.checked_offset(count as isize); + let left_end_exclusive = match left_end_exclusive { + None => return Err("Overflow while calculating left_end_exclusive"), + Some(x) => x, + }; + + if left_end_exclusive > self.end_exclusive { + return Err("Not enough free pages"); + } + + let allocation = Self { + start: self.start, + end_exclusive: left_end_exclusive, + }; + self.start = left_end_exclusive; + + Ok(allocation) + } +} + +impl IntoIterator for MemoryRegion { + type Item = PageAddress; + type IntoIter = Range; + + fn into_iter(self) -> Self::IntoIter { + Range { + start: self.start, + end: self.end_exclusive, + } + } +} + +impl From for MemoryRegion { + fn from(desc: MMIODescriptor) -> Self { + let start = PageAddress::from(desc.start_addr.align_down_page()); + let end_exclusive = PageAddress::from(desc.end_addr_exclusive().align_up_page()); + + Self { + start, + end_exclusive, + } + } +} + +//------------------------------------------------------------------------------ +// MMIODescriptor +//------------------------------------------------------------------------------ + +impl MMIODescriptor { + /// Create an instance. + pub const fn new(start_addr: Address, size: usize) -> Self { + assert!(size > 0); + let end_addr_exclusive = Address::new(start_addr.as_usize() + size); + + Self { + start_addr, + end_addr_exclusive, + } + } + + /// Return the start address. + pub const fn start_addr(&self) -> Address { + self.start_addr + } + + /// Return the exclusive end address. + pub fn end_addr_exclusive(&self) -> Address { + self.end_addr_exclusive + } +} + +//-------------------------------------------------------------------------------------------------- +// Testing +//-------------------------------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + use crate::memory::Virtual; + use test_macros::kernel_test; + + /// Sanity of [PageAddress] methods. + #[kernel_test] + fn pageaddress_type_method_sanity() { + let page_addr: PageAddress = + PageAddress::from(bsp::memory::mmu::KernelGranule::SIZE * 2); + + assert_eq!( + page_addr.checked_offset(-2), + Some(PageAddress::::from(0)) + ); + + assert_eq!( + page_addr.checked_offset(2), + Some(PageAddress::::from( + bsp::memory::mmu::KernelGranule::SIZE * 4 + )) + ); + + assert_eq!( + PageAddress::::from(0).checked_offset(0), + Some(PageAddress::::from(0)) + ); + assert_eq!(PageAddress::::from(0).checked_offset(-1), None); + + let max_page_addr = Address::::new(usize::MAX).align_down_page(); + assert_eq!( + PageAddress::::from(max_page_addr).checked_offset(1), + None + ); + + let zero = PageAddress::::from(0); + let three = PageAddress::::from(bsp::memory::mmu::KernelGranule::SIZE * 3); + assert_eq!(PageAddress::steps_between(&zero, &three), Some(3)); + } + + /// Sanity of [MemoryRegion] methods. + #[kernel_test] + fn memoryregion_type_method_sanity() { + let zero = PageAddress::::from(0); + let zero_region = MemoryRegion::new(zero, zero); + assert_eq!(zero_region.num_pages(), 0); + assert_eq!(zero_region.size(), 0); + + let one = PageAddress::::from(bsp::memory::mmu::KernelGranule::SIZE); + let one_region = MemoryRegion::new(zero, one); + assert_eq!(one_region.num_pages(), 1); + assert_eq!(one_region.size(), bsp::memory::mmu::KernelGranule::SIZE); + + let three = PageAddress::::from(bsp::memory::mmu::KernelGranule::SIZE * 3); + let mut three_region = MemoryRegion::new(zero, three); + assert!(three_region.contains(zero.into_inner())); + assert!(!three_region.contains(three.into_inner())); + assert!(three_region.overlaps(&one_region)); + + let allocation = three_region + .take_first_n_pages(NonZeroUsize::new(2).unwrap()) + .unwrap(); + assert_eq!(allocation.num_pages(), 2); + assert_eq!(three_region.num_pages(), 1); + + for (i, alloc) in allocation.into_iter().enumerate() { + assert_eq!( + alloc.into_inner().as_usize(), + i * bsp::memory::mmu::KernelGranule::SIZE + ); + } + } +} diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu.rs 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/memory/mmu.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/memory/mmu.rs @@ -3,29 +3,24 @@ // Copyright (c) 2020-2022 Andre Richter //! Memory Management Unit. -//! -//! In order to decouple `BSP` and `arch` parts of the MMU code (to keep them pluggable), this file -//! provides types for composing an architecture-agnostic description of the kernel's virtual memory -//! layout. -//! -//! The `BSP` provides such a description through the `bsp::memory::mmu::virt_mem_layout()` -//! function. -//! -//! The `MMU` driver of the `arch` code uses `bsp::memory::mmu::virt_mem_layout()` to compile and -//! install respective translation tables. #[cfg(target_arch = "aarch64")] #[path = "../_arch/aarch64/memory/mmu.rs"] mod arch_mmu; +mod mapping_record; +mod page_alloc; mod translation_table; +mod types; -use core::{fmt, ops::RangeInclusive}; +use crate::{ + bsp, + memory::{Address, Physical, Virtual}, + synchronization, warn, +}; +use core::{fmt, num::NonZeroUsize}; -//-------------------------------------------------------------------------------------------------- -// Architectural Public Reexports -//-------------------------------------------------------------------------------------------------- -pub use arch_mmu::mmu; +pub use types::*; //-------------------------------------------------------------------------------------------------- // Public Definitions @@ -45,13 +40,15 @@ /// MMU functions. pub trait MMU { - /// Called by the kernel during early init. Supposed to take the translation tables from the - /// `BSP`-supplied `virt_mem_layout()` and install/activate them for the respective MMU. + /// Turns on the MMU for the first time and enables data and instruction caching. /// /// # Safety /// /// - Changes the HW's global state. - unsafe fn enable_mmu_and_caching(&self) -> Result<(), MMUEnableError>; + unsafe fn enable_mmu_and_caching( + &self, + phys_tables_base_addr: Address, + ) -> Result<(), MMUEnableError>; /// Returns true if the MMU is enabled, false otherwise. fn is_enabled(&self) -> bool; @@ -64,55 +61,51 @@ /// Describes properties of an address space. pub struct AddressSpace; -/// Architecture agnostic translation types. -#[allow(missing_docs)] -#[derive(Copy, Clone)] -pub enum Translation { - Identity, - Offset(usize), -} - -/// Architecture agnostic memory attributes. -#[allow(missing_docs)] -#[derive(Copy, Clone)] -pub enum MemAttributes { - CacheableDRAM, - Device, -} - -/// Architecture agnostic access permissions. -#[allow(missing_docs)] -#[derive(Copy, Clone)] -pub enum AccessPermissions { - ReadOnly, - ReadWrite, -} - -/// Collection of memory attributes. -#[allow(missing_docs)] -#[derive(Copy, Clone)] -pub struct AttributeFields { - pub mem_attributes: MemAttributes, - pub acc_perms: AccessPermissions, - pub execute_never: bool, +/// Intended to be implemented for [`AddressSpace`]. +pub trait AssociatedTranslationTable { + /// A translation table whose address range is: + /// + /// [AS_SIZE - 1, 0] + type TableStartFromBottom; } -/// Architecture agnostic descriptor for a memory range. -#[allow(missing_docs)] -pub struct TranslationDescriptor { - pub name: &'static str, - pub virtual_range: fn() -> RangeInclusive, - pub physical_range_translation: Translation, - pub attribute_fields: AttributeFields, -} +//-------------------------------------------------------------------------------------------------- +// Private Code +//-------------------------------------------------------------------------------------------------- +use interface::MMU; +use synchronization::interface::*; +use translation_table::interface::TranslationTable; + +/// Query the BSP for the reserved virtual addresses for MMIO remapping and initialize the kernel's +/// MMIO VA allocator with it. +fn kernel_init_mmio_va_allocator() { + let region = bsp::memory::mmu::virt_mmio_remap_region(); + + page_alloc::kernel_mmio_va_allocator().lock(|allocator| allocator.initialize(region)); +} + +/// Map a region in the kernel's translation tables. +/// +/// No input checks done, input is passed through to the architectural implementation. +/// +/// # Safety +/// +/// - See `map_at()`. +/// - Does not prevent aliasing. +unsafe fn kernel_map_at_unchecked( + name: &'static str, + virt_region: &MemoryRegion, + phys_region: &MemoryRegion, + attr: &AttributeFields, +) -> Result<(), &'static str> { + bsp::memory::mmu::kernel_translation_tables() + .write(|tables| tables.map_at(virt_region, phys_region, attr))?; -/// Type for expressing the kernel's virtual memory layout. -pub struct KernelVirtualLayout { - /// The last (inclusive) address of the address space. - max_virt_addr_inclusive: usize, + if let Err(x) = mapping_record::kernel_add(name, virt_region, phys_region, attr) { + warn!("{}", x); + } - /// Array of descriptors for non-standard (normal cacheable DRAM) memory regions. - inner: [TranslationDescriptor; NUM_SPECIAL_RANGES], + Ok(()) } //-------------------------------------------------------------------------------------------------- @@ -132,6 +125,9 @@ /// The granule's size. pub const SIZE: usize = Self::size_checked(); + /// The granule's mask. + pub const MASK: usize = Self::SIZE - 1; + /// The granule's shift, aka log2(size). pub const SHIFT: usize = Self::SIZE.trailing_zeros() as usize; @@ -159,110 +155,147 @@ } } -impl Default for AttributeFields { - fn default() -> AttributeFields { - AttributeFields { - mem_attributes: MemAttributes::CacheableDRAM, - acc_perms: AccessPermissions::ReadWrite, - execute_never: true, - } +/// Raw mapping of a virtual to physical region in the kernel translation tables. +/// +/// Prevents mapping into the MMIO range of the tables. +/// +/// # Safety +/// +/// - See `kernel_map_at_unchecked()`. +/// - Does not prevent aliasing. Currently, the callers must be trusted. +pub unsafe fn kernel_map_at( + name: &'static str, + virt_region: &MemoryRegion, + phys_region: &MemoryRegion, + attr: &AttributeFields, +) -> Result<(), &'static str> { + if bsp::memory::mmu::virt_mmio_remap_region().overlaps(virt_region) { + return Err("Attempt to manually map into MMIO region"); } -} -/// Human-readable output of a TranslationDescriptor. -impl fmt::Display for TranslationDescriptor { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - // Call the function to which self.range points, and dereference the result, which causes - // Rust to copy the value. - let start = *(self.virtual_range)().start(); - let end = *(self.virtual_range)().end(); - let size = end - start + 1; - - // log2(1024). - const KIB_RSHIFT: u32 = 10; - - // log2(1024 * 1024). - const MIB_RSHIFT: u32 = 20; - - let (size, unit) = if (size >> MIB_RSHIFT) > 0 { - (size >> MIB_RSHIFT, "MiB") - } else if (size >> KIB_RSHIFT) > 0 { - (size >> KIB_RSHIFT, "KiB") - } else { - (size, "Byte") - }; + kernel_map_at_unchecked(name, virt_region, phys_region, attr)?; - let attr = match self.attribute_fields.mem_attributes { - MemAttributes::CacheableDRAM => "C", - MemAttributes::Device => "Dev", - }; + Ok(()) +} - let acc_p = match self.attribute_fields.acc_perms { - AccessPermissions::ReadOnly => "RO", - AccessPermissions::ReadWrite => "RW", +/// MMIO remapping in the kernel translation tables. +/// +/// Typically used by device drivers. +/// +/// # Safety +/// +/// - Same as `kernel_map_at_unchecked()`, minus the aliasing part. +pub unsafe fn kernel_map_mmio( + name: &'static str, + mmio_descriptor: &MMIODescriptor, +) -> Result, &'static str> { + let phys_region = MemoryRegion::from(*mmio_descriptor); + let offset_into_start_page = mmio_descriptor.start_addr().offset_into_page(); + + // Check if an identical region has been mapped for another driver. If so, reuse it. + let virt_addr = if let Some(addr) = + mapping_record::kernel_find_and_insert_mmio_duplicate(mmio_descriptor, name) + { + addr + // Otherwise, allocate a new region and map it. + } else { + let num_pages = match NonZeroUsize::new(phys_region.num_pages()) { + None => return Err("Requested 0 pages"), + Some(x) => x, }; - let xn = if self.attribute_fields.execute_never { - "PXN" - } else { - "PX" - }; + let virt_region = + page_alloc::kernel_mmio_va_allocator().lock(|allocator| allocator.alloc(num_pages))?; - write!( - f, - " {:#010x} - {:#010x} | {: >3} {} | {: <3} {} {: <3} | {}", - start, end, size, unit, attr, acc_p, xn, self.name - ) - } + kernel_map_at_unchecked( + name, + &virt_region, + &phys_region, + &AttributeFields { + mem_attributes: MemAttributes::Device, + acc_perms: AccessPermissions::ReadWrite, + execute_never: true, + }, + )?; + + virt_region.start_addr() + }; + + Ok(virt_addr + offset_into_start_page) +} + +/// Map the kernel's binary. Returns the translation table's base address. +/// +/// # Safety +/// +/// - See [`bsp::memory::mmu::kernel_map_binary()`]. +pub unsafe fn kernel_map_binary() -> Result, &'static str> { + let phys_kernel_tables_base_addr = + bsp::memory::mmu::kernel_translation_tables().write(|tables| { + tables.init(); + tables.phys_base_address() + }); + + bsp::memory::mmu::kernel_map_binary()?; + + Ok(phys_kernel_tables_base_addr) +} + +/// Enable the MMU and data + instruction caching. +/// +/// # Safety +/// +/// - Crucial function during kernel init. Changes the the complete memory view of the processor. +pub unsafe fn enable_mmu_and_caching( + phys_tables_base_addr: Address, +) -> Result<(), MMUEnableError> { + arch_mmu::mmu().enable_mmu_and_caching(phys_tables_base_addr) +} + +/// Finish initialization of the MMU subsystem. +pub fn post_enable_init() { + kernel_init_mmio_va_allocator(); +} + +/// Human-readable print of all recorded kernel mappings. +pub fn kernel_print_mappings() { + mapping_record::kernel_print() } -impl KernelVirtualLayout<{ NUM_SPECIAL_RANGES }> { - /// Create a new instance. - pub const fn new(max: usize, layout: [TranslationDescriptor; NUM_SPECIAL_RANGES]) -> Self { - Self { - max_virt_addr_inclusive: max, - inner: layout, - } - } - - /// For a virtual address, find and return the physical output address and corresponding - /// attributes. - /// - /// If the address is not found in `inner`, return an identity mapped default with normal - /// cacheable DRAM attributes. - pub fn virt_addr_properties( - &self, - virt_addr: usize, - ) -> Result<(usize, AttributeFields), &'static str> { - if virt_addr > self.max_virt_addr_inclusive { - return Err("Address out of range"); - } - - for i in self.inner.iter() { - if (i.virtual_range)().contains(&virt_addr) { - let output_addr = match i.physical_range_translation { - Translation::Identity => virt_addr, - Translation::Offset(a) => a + (virt_addr - (i.virtual_range)().start()), - }; - - return Ok((output_addr, i.attribute_fields)); - } - } +//-------------------------------------------------------------------------------------------------- +// Testing +//-------------------------------------------------------------------------------------------------- - Ok((virt_addr, AttributeFields::default())) - } +#[cfg(test)] +mod tests { + use super::*; + use crate::memory::mmu::{AccessPermissions, MemAttributes, PageAddress}; + use test_macros::kernel_test; - /// Print the memory layout. - pub fn print_layout(&self) { - use crate::info; + /// Check that you cannot map into the MMIO VA range from kernel_map_at(). + #[kernel_test] + fn no_manual_mmio_map() { + let phys_start_page_addr: PageAddress = PageAddress::from(0); + let phys_end_exclusive_page_addr: PageAddress = + phys_start_page_addr.checked_offset(5).unwrap(); + let phys_region = MemoryRegion::new(phys_start_page_addr, phys_end_exclusive_page_addr); + + let num_pages = NonZeroUsize::new(phys_region.num_pages()).unwrap(); + let virt_region = page_alloc::kernel_mmio_va_allocator() + .lock(|allocator| allocator.alloc(num_pages)) + .unwrap(); - for i in self.inner.iter() { - info!("{}", i); - } - } + let attr = AttributeFields { + mem_attributes: MemAttributes::CacheableDRAM, + acc_perms: AccessPermissions::ReadWrite, + execute_never: true, + }; - #[cfg(test)] - pub fn inner(&self) -> &[TranslationDescriptor; NUM_SPECIAL_RANGES] { - &self.inner + unsafe { + assert_eq!( + kernel_map_at("test", &virt_region, &phys_region, &attr), + Err("Attempt to manually map into MMIO region") + ) + }; } } diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/src/memory.rs 14_virtual_mem_part2_mmio_remap/kernel/src/memory.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/src/memory.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/src/memory.rs @@ -5,3 +5,163 @@ //! Memory Management. pub mod mmu; + +use crate::{bsp, common}; +use core::{ + fmt, + marker::PhantomData, + ops::{Add, Sub}, +}; + +//-------------------------------------------------------------------------------------------------- +// Public Definitions +//-------------------------------------------------------------------------------------------------- + +/// Metadata trait for marking the type of an address. +pub trait AddressType: Copy + Clone + PartialOrd + PartialEq {} + +/// Zero-sized type to mark a physical address. +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub enum Physical {} + +/// Zero-sized type to mark a virtual address. +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub enum Virtual {} + +/// Generic address type. +#[derive(Copy, Clone, Debug, PartialOrd, PartialEq)] +pub struct Address { + value: usize, + _address_type: PhantomData ATYPE>, +} + +//-------------------------------------------------------------------------------------------------- +// Public Code +//-------------------------------------------------------------------------------------------------- + +impl AddressType for Physical {} +impl AddressType for Virtual {} + +impl Address { + /// Create an instance. + pub const fn new(value: usize) -> Self { + Self { + value, + _address_type: PhantomData, + } + } + + /// Convert to usize. + pub const fn as_usize(self) -> usize { + self.value + } + + /// Align down to page size. + #[must_use] + pub const fn align_down_page(self) -> Self { + let aligned = common::align_down(self.value, bsp::memory::mmu::KernelGranule::SIZE); + + Self::new(aligned) + } + + /// Align up to page size. + #[must_use] + pub const fn align_up_page(self) -> Self { + let aligned = common::align_up(self.value, bsp::memory::mmu::KernelGranule::SIZE); + + Self::new(aligned) + } + + /// Checks if the address is page aligned. + pub const fn is_page_aligned(&self) -> bool { + common::is_aligned(self.value, bsp::memory::mmu::KernelGranule::SIZE) + } + + /// Return the address' offset into the corresponding page. + pub const fn offset_into_page(&self) -> usize { + self.value & bsp::memory::mmu::KernelGranule::MASK + } +} + +impl Add for Address { + type Output = Self; + + #[inline(always)] + fn add(self, rhs: usize) -> Self::Output { + match self.value.checked_add(rhs) { + None => panic!("Overflow on Address::add"), + Some(x) => Self::new(x), + } + } +} + +impl Sub> for Address { + type Output = Self; + + #[inline(always)] + fn sub(self, rhs: Address) -> Self::Output { + match self.value.checked_sub(rhs.value) { + None => panic!("Overflow on Address::sub"), + Some(x) => Self::new(x), + } + } +} + +impl fmt::Display for Address { + // Don't expect to see physical addresses greater than 40 bit. + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let q3: u8 = ((self.value >> 32) & 0xff) as u8; + let q2: u16 = ((self.value >> 16) & 0xffff) as u16; + let q1: u16 = (self.value & 0xffff) as u16; + + write!(f, "0x")?; + write!(f, "{:02x}_", q3)?; + write!(f, "{:04x}_", q2)?; + write!(f, "{:04x}", q1) + } +} + +impl fmt::Display for Address { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let q4: u16 = ((self.value >> 48) & 0xffff) as u16; + let q3: u16 = ((self.value >> 32) & 0xffff) as u16; + let q2: u16 = ((self.value >> 16) & 0xffff) as u16; + let q1: u16 = (self.value & 0xffff) as u16; + + write!(f, "0x")?; + write!(f, "{:04x}_", q4)?; + write!(f, "{:04x}_", q3)?; + write!(f, "{:04x}_", q2)?; + write!(f, "{:04x}", q1) + } +} + +//-------------------------------------------------------------------------------------------------- +// Testing +//-------------------------------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + use test_macros::kernel_test; + + /// Sanity of [Address] methods. + #[kernel_test] + fn address_type_method_sanity() { + let addr = Address::::new(bsp::memory::mmu::KernelGranule::SIZE + 100); + + assert_eq!( + addr.align_down_page().as_usize(), + bsp::memory::mmu::KernelGranule::SIZE + ); + + assert_eq!( + addr.align_up_page().as_usize(), + bsp::memory::mmu::KernelGranule::SIZE * 2 + ); + + assert!(!addr.is_page_aligned()); + + assert_eq!(addr.offset_into_page(), 100); + } +} diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/tests/02_exception_sync_page_fault.rs 14_virtual_mem_part2_mmio_remap/kernel/tests/02_exception_sync_page_fault.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/tests/02_exception_sync_page_fault.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/tests/02_exception_sync_page_fault.rs @@ -21,18 +21,40 @@ #[no_mangle] unsafe fn kernel_init() -> ! { - use memory::mmu::interface::MMU; + use libkernel::driver::interface::DriverManager; exception::handling_init(); - bsp::console::qemu_bring_up_console(); // This line will be printed as the test header. println!("Testing synchronous exception handling by causing a page fault"); - if let Err(string) = memory::mmu::mmu().enable_mmu_and_caching() { - info!("MMU: {}", string); + let phys_kernel_tables_base_addr = match memory::mmu::kernel_map_binary() { + Err(string) => { + info!("Error mapping kernel binary: {}", string); + cpu::qemu_exit_failure() + } + Ok(addr) => addr, + }; + + if let Err(e) = memory::mmu::enable_mmu_and_caching(phys_kernel_tables_base_addr) { + info!("Enabling MMU failed: {}", e); cpu::qemu_exit_failure() } + // Printing will silently fail from here on, because the driver's MMIO is not remapped yet. + + memory::mmu::post_enable_init(); + bsp::console::qemu_bring_up_console(); + + // Bring up the drivers needed for printing first. + for i in bsp::driver::driver_manager() + .early_print_device_drivers() + .iter() + { + // Any encountered errors cannot be printed yet, obviously, so just safely park the CPU. + i.init().unwrap_or_else(|_| cpu::qemu_exit_failure()); + } + bsp::driver::driver_manager().post_early_print_device_driver_init(); + // Printing available again from here on. info!("Writing beyond mapped area to address 9 GiB..."); let big_addr: u64 = 9 * 1024 * 1024 * 1024; diff -uNr 13_exceptions_part2_peripheral_IRQs/kernel/tests/03_exception_restore_sanity.rs 14_virtual_mem_part2_mmio_remap/kernel/tests/03_exception_restore_sanity.rs --- 13_exceptions_part2_peripheral_IRQs/kernel/tests/03_exception_restore_sanity.rs +++ 14_virtual_mem_part2_mmio_remap/kernel/tests/03_exception_restore_sanity.rs @@ -30,18 +30,40 @@ #[no_mangle] unsafe fn kernel_init() -> ! { - use memory::mmu::interface::MMU; + use libkernel::driver::interface::DriverManager; exception::handling_init(); - bsp::console::qemu_bring_up_console(); // This line will be printed as the test header. println!("Testing exception restore"); - if let Err(string) = memory::mmu::mmu().enable_mmu_and_caching() { - info!("MMU: {}", string); + let phys_kernel_tables_base_addr = match memory::mmu::kernel_map_binary() { + Err(string) => { + info!("Error mapping kernel binary: {}", string); + cpu::qemu_exit_failure() + } + Ok(addr) => addr, + }; + + if let Err(e) = memory::mmu::enable_mmu_and_caching(phys_kernel_tables_base_addr) { + info!("Enabling MMU failed: {}", e); cpu::qemu_exit_failure() } + // Printing will silently fail from here on, because the driver's MMIO is not remapped yet. + + memory::mmu::post_enable_init(); + bsp::console::qemu_bring_up_console(); + + // Bring up the drivers needed for printing first. + for i in bsp::driver::driver_manager() + .early_print_device_drivers() + .iter() + { + // Any encountered errors cannot be printed yet, obviously, so just safely park the CPU. + i.init().unwrap_or_else(|_| cpu::qemu_exit_failure()); + } + bsp::driver::driver_manager().post_early_print_device_driver_init(); + // Printing available again from here on. info!("Making a dummy system call"); ```