Memory Usage in ivshmem

ivshmem 處理 shared memory 的原理, 簡單來說就是.

  • ivshmem 建立了一個 RAM type 的 MemoryRegion 來封裝 host OS 上的 shared memory.

  • ivshmem 將該 MemoryRegion 註冊成自己 PCI device 的 BAR2. QEMU PCI framework 會將所有 PCI device 的 BAR 加入到 system_memory 當中, 進行 guest OS memory 的管理.

  • 當 MemoryRegion 被加入到 system_memory 時, QEMU 就會透過 KVM API 對 guest OS 的記憶體進行對應的操作. 舉例來說操作可能是在 page table 建立 memory mapping 或是為了 trap-and-emulate 而把 page table 的 mapping 刪掉, 這根據 MemoryRegion 的 type 決定. ivshmem 這邊的操作則是將 shared memory mapping 到 guest OS 的記憶體.

接下來就個別講解細節.

ivshmem CLI option

ivshmem CLI option 包含 shared memory 的路徑跟大小, 這兩個資料會被傳入 IVShmemState 的 class member 當中.

  • CLI option: -device ivshmem,shm=hello,size=1 => IVShmemState

    IVShmemState->shmobj = "hello" // shm
IVShmemState->sizearg = 1      // size

  • 細節請參考: device parameters

desugar_shm(): create MemoryRegion

Create TYPE_MEMORY_BACKEND_FILE and MemoryRegion by IVShmemState->shmobj. Assign the MemoryRegion to IVShmemState->hostmem.

  • hostmem = object_new(TYPE_MEMORY_BACKEND_FILE)

  • hostmem->properties["mem-path"] = "/dev/shm/" + IVShmemState->shmobj

  • user_creatable_complete(hostmem)

    1. create MemoryRegion

    2. mmap() file to QEMU process memory

register MemoryRegion as PCI BAR2

ivshmem 將該 MemoryRegion 註冊成自己 PCI device 的 BAR2.

  • IVShmemState->ivshmem_bar2 = host_memory_backend_get_memory(IVShmemState->hostmem)

  • pci_register_bar(PCI_DEVICE(s), 2, attr, IVShmemState->ivshmem_bar2);

Then, we should dig into QEMU PCI framework.

PCI framework

QEMU PCI framework 會將所有 PCI device 的 BAR 加入到 system_memory 當中, 進行 guest OS memory 的管理.

  • pci_register_bar(): 把 MemoryRegion 的資料放到 pci_dev->io_region[bar_num]

  • pci_update_mapping():

    • 更新 pci_dev->io_region[bar_num], 將該 MemoryRegion 加入 system_memory 當中 (memory_region_add_subregion())

    • memory_region_add_subregion_overlap(r->address_space, r->addr, r->memory, 1); // PCIIORegion* r;

    • PCI device 使用的 address_space 為何?

      • PCIIORegion->address_space is from PCIBus->address_space_{mem,io}

      • PCIBus->address_space_mem is init at PCI Host Bridge start

    • More: MemoryListener and MemoryRegion transaction, KVMMemoryListener

Details

  1. TYPE_MEMORY_BACKEND_FILE:

    // user_creatable_complete() create MemoryRegion and do mmap()
user_creatable_complete() => host_memory_backend_memory_complete() => file_backend_memory_alloc() => memory_region_init_ram_from_file()

// inheritance: TYPE_MEMORY_BACKEND_FILE => TYPE_MEMORY_BACKEND => TYPE_OBJECT
[TYPE_MEMORY_BACKEND] UserCreatableClass* ucc->complete = host_memory_backend_memory_complete
[TYPE_MEMORY_BACKEND_FILE] HostMemoryBackendClass* bc->alloc = file_backend_memory_alloc

host_memory_backend_memory_complete()
=> bc->alloc() = file_backend_memory_alloc()
=> memory_region_init_ram_from_file()

  2. pci_{set,get}_{byte,word,…}: pointer dereference with endianness.

struct

  • PCIDevice

    • PCIIORegion io_regions[]; // all PCI BAR, each element of array is single PCI BAR

    • PCIBus bus;

    • uint8_t* config; // PCI config space. each PCI BAR has some space.

    • uint8_t* wmask, cmask;

  • PCIIORegion

    • pcibus_t addr, size; // MMIO region, addr may equal to PCI_BAR_UNMAPPED;

    • MemoryRegion memory; // one PCI BAR maps to one QEMU MemoryRegion

    • MemoryRegion address_space; // one of pci_dev->bus->address_space_{mem,io}

  • PCIBus

    • MemoryRegion address_space_io, address_space_mem;

    • init at PCI Host Bridge (e.g. i440fx_init())

misc: gdb print MemoryRegion linked list:

(gdb) memory_region_p mr->subregions->tqh_first
MemoryRegion 'kvm-apic-msi': size=0/100000, is_ram 0, is_ro 0, hwaddr fee00000, callback addr 0x555555eeae00
(gdb) memory_region_p mr->subregions->tqh_first->subregions_link.tqe_next
MemoryRegion 'ram-above-4g': size=0/40000000, is_ram 0, is_ro 0, hwaddr 100000000, callback addr 0x555555ee6780
(gdb) memory_region_p mr->subregions->tqh_first->subregions_link.tqe_next->subregions_link.tqe_next
MemoryRegion 'ram-below-4g': size=0/c0000000, is_ram 0, is_ro 0, hwaddr 0, callback addr 0x555555ee6780
(gdb) memory_region_p mr->subregions->tqh_first->subregions_link.tqe_next->subregions_link.tqe_next->subregions_link.tqe_next
MemoryRegion 'pci': size=1/0, is_ram 0, is_ro 0, hwaddr 0, callback addr 0x555555ee6780