v1 -> v2: - provide CXL exclusive MMIO32 & MMIO64 space - hard coded two cxl root ports
RFC because - Many contents are ported from Jonathan' patch on qemu virt design
- Less experience and not particularly confident in sbsa-ref address space design so this might be stupidly broken in a way I've not considered.
Currently the base CXL support for arm platforms is only on Jonathan's patches[1] which have not yet merged into upstream. SBSA-REF can be more like a real machine, thus the support of cxl could be meaningful.
This series leverages Jonathan's patches[1] to design [SBSA_CXL_HOST] and [SBSA_CXL_FIXED_WINDOW] spaces for sbsa-ref layout.
For [SBSA_CXL_HOST], since this creates a default pxb-cxl (bus_nr=0xc0) bridge with two cxl root ports on sbsa-ref, the new memory layout places 64K space for one hard coded cxl host bridge register regions in the sbsa-ref memmap.
According to above design, for now only two cxl type3 devices could be added on the cxl host.
With the 'create_pxb_cxl', users don't need to input '-device pxb-cxl' and '-device cxl-rp' parameters.
In addition, this support indepentent mmio32(32M) & mmio64(1M) space which are enough for cxl components, because in this way the previous pcie mmio32/64 space would not be divided and affected.
However, with the pxb-cxl-host, any cxl root ports and cxl endpoint devices would occupy the BDF number of the original pcie domain. Hence, the max available pcie devices on sbsa-ref would decrease, which seems to bring a series of trouble. I'm looking for some comments on the problems and suggestions on if there are better ways to do it.
For [SBSA_CXL_FIXED_WINDOW], in order to provide CFMWs on sbsa-ref, this extends 1TB space from the hole above RAM Memory [SBSA_MEM] for CXL Fixed Memory Window. 0xA0000000000 is chosen as the base address of this space because of 3 reasons:
1) It is more suitable to choose a static address instead of that implementation in virt, since a dynamic address space layout of sbsa-ref is not appropriate for its original purpose as a reference platform.
2) The Hotplug Memory address range should in the range of maximum addressable range of sbsa-ref platform(0x10000000000-0x80ffffffffff). It is satisfied the requirements of memory hotplug in linux kernel.
3) The start pfn of CFMW should exceed the reserved_pfn_range for onlined numa node.
Based on 'cxl_fmws_link_targets', this adds a new function 'sbsa_cxl_fmws_link_targets' for binding cfmws.target with the default pxb-cxl-bus on sbsa-ref.
In addition, this also adds 'create_cxl_fixed_window_region' to support creating a static cfmw region on sbsa-ref, so users don't need to input '-M cxl-fmw' parameter.
Thus, to run sbsa-ref with a cxl device could use: qemu-system-aarch64 \ -object memory-backend-file,id=mem2,mem-path=/tmp/mem2,size=256M,share=true \ -device cxl-type3,bus=cxl.0,volatile-memdev=mem2,id=cxl-mem1 \
By the way, since the matched firmware correspond to this patch would allocate pcie bus 0xc0 ~ 0xff to pxb-cxl-host, we should add "bus=pcie.0" when we want to plug some devices on the original pcie bus, for example: -device qemu-xhci,bus=pcie.0 \ or -device nvme,serial=deadbeef,bus=pcie.0,drive=hdd \ -drive file=../disk/hdd.qcow2,format=qcow2,id=hdd,if=none \
This series patches are here to hopefully some comments to guide me!
Link: [1]: https://lore.kernel.org/linux-cxl/20220616141950.23374-1-Jonathan.Cameron@hu... [2]: https://edk2.groups.io/g/devel/topic/rfc_edk2_patch_v3_0_1/109403423# [3]: https://edk2.groups.io/g/devel/topic/rfc_patch_edk2_platforms_v2/109403456
Yuquan Wang (1): hw/arm/sbsa-ref: Support CXL Host Bridge & CFMW
docs/system/arm/sbsa.rst | 4 ++ hw/arm/sbsa-ref.c | 122 +++++++++++++++++++++++++++++++++++++- hw/cxl/cxl-host-stubs.c | 2 + hw/cxl/cxl-host.c | 2 +- include/hw/cxl/cxl_host.h | 2 + 5 files changed, 130 insertions(+), 2 deletions(-)
This creates a default pxb-cxl (bus_nr=0xc0) bridge with two cxl root ports on sbsa-ref. And the memory layout places 64K space for the cxl host bridge register regions(CHBCR) in the sbsa-ref memmap.
In addition, this support indepentent mmio32(32M) & mmio64(1M) space for cxl components.
To provide CFMWs on sbsa-ref, this extends 1TB space from the hole above RAM Memory [SBSA_MEM] for CXL Fixed Memory Window.
Signed-off-by: Yuquan Wang wangyuquan1236@phytium.com.cn --- docs/system/arm/sbsa.rst | 4 ++ hw/arm/sbsa-ref.c | 122 +++++++++++++++++++++++++++++++++++++- hw/cxl/cxl-host-stubs.c | 2 + hw/cxl/cxl-host.c | 2 +- include/hw/cxl/cxl_host.h | 2 + 5 files changed, 130 insertions(+), 2 deletions(-)
diff --git a/docs/system/arm/sbsa.rst b/docs/system/arm/sbsa.rst index 2bf3fc8d59..21b88e88e7 100644 --- a/docs/system/arm/sbsa.rst +++ b/docs/system/arm/sbsa.rst @@ -28,6 +28,7 @@ The ``sbsa-ref`` board supports: - E1000E ethernet card on PCIe bus - Bochs display adapter on PCIe bus - A generic SBSA watchdog device + - CXL host bridge and CXL fixed memory window
Board to firmware interface @@ -92,3 +93,6 @@ Platform version changes:
0.4 CPU topology information is present in devicetree. + +0.5 + CXL host bridge and CXL fixed memory window are supported. diff --git a/hw/arm/sbsa-ref.c b/hw/arm/sbsa-ref.c index e3195d5449..655dc82863 100644 --- a/hw/arm/sbsa-ref.c +++ b/hw/arm/sbsa-ref.c @@ -36,12 +36,18 @@ #include "hw/arm/smmuv3.h" #include "hw/block/flash.h" #include "hw/boards.h" +#include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_host.h" #include "hw/ide/ide-bus.h" #include "hw/ide/ahci-sysbus.h" #include "hw/intc/arm_gicv3_common.h" #include "hw/intc/arm_gicv3_its_common.h" #include "hw/loader.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pci_bus.h" +#include "hw/pci/pcie_port.h" #include "hw/pci-host/gpex.h" +#include "hw/pci-bridge/pci_expander_bridge.h" #include "hw/qdev-properties.h" #include "hw/usb.h" #include "hw/usb/xhci.h" @@ -94,6 +100,10 @@ enum { SBSA_SECURE_MEM, SBSA_AHCI, SBSA_XHCI, + SBSA_CXL_HOST, + SBSA_CXL_MMIO, + SBSA_CXL_MMIO_HIGH, + SBSA_CXL_FIXED_WINDOW, };
struct SBSAMachineState { @@ -105,6 +115,8 @@ struct SBSAMachineState { int psci_conduit; DeviceState *gic; PFlashCFI01 *flash[2]; + CXLState cxl_devices_state; + PCIBus *cxlbus; };
#define TYPE_SBSA_MACHINE MACHINE_TYPE_NAME("sbsa-ref") @@ -132,6 +144,10 @@ static const MemMapEntry sbsa_ref_memmap[] = { /* Space here reserved for more SMMUs */ [SBSA_AHCI] = { 0x60100000, 0x00010000 }, [SBSA_XHCI] = { 0x60110000, 0x00010000 }, + /* 64KiB CXL Host Bridge Registers space */ + [SBSA_CXL_HOST] = { 0x60120000, 0x00010000 }, + /* 32M CXL 32-bit MMIO space */ + [SBSA_CXL_MMIO] = { 0x60130000, 0x02000000 }, /* Space here reserved for other devices */ [SBSA_PCIE_PIO] = { 0x7fff0000, 0x00010000 }, /* 32-bit address PCIE MMIO space */ @@ -141,6 +157,10 @@ static const MemMapEntry sbsa_ref_memmap[] = { /* ~1TB PCIE MMIO space (4GB to 1024GB boundary) */ [SBSA_PCIE_MMIO_HIGH] = { 0x100000000ULL, 0xFF00000000ULL }, [SBSA_MEM] = { 0x10000000000ULL, RAMLIMIT_BYTES }, + /* 1M CXL 64-bit MMIO space */ + [SBSA_CXL_MMIO_HIGH] = { 0x90000000000ULL, 0x00100000 }, + /* 1TB CXL FIXED WINDOW space */ + [SBSA_CXL_FIXED_WINDOW] = { 0xA0000000000ULL, 0x10000000000ULL }, };
static const int sbsa_ref_irqmap[] = { @@ -216,7 +236,7 @@ static void create_fdt(SBSAMachineState *sms) * fw compatibility. */ qemu_fdt_setprop_cell(fdt, "/", "machine-version-major", 0); - qemu_fdt_setprop_cell(fdt, "/", "machine-version-minor", 4); + qemu_fdt_setprop_cell(fdt, "/", "machine-version-minor", 5);
if (ms->numa_state->have_numa_distance) { int size = nb_numa_nodes * nb_numa_nodes * 3 * sizeof(uint32_t); @@ -629,6 +649,35 @@ static void create_smmu(const SBSAMachineState *sms, PCIBus *bus) } }
+static void create_pxb_cxl(SBSAMachineState *sms, PCIBus *bus) +{ + DeviceState *qdev = qdev_new(TYPE_PXB_CXL_DEV); + PCIDevice *dev = PCI_DEVICE(qdev); + CXLHost *host; + PCIHostState *cxl; + PCIDevice *cxlrp; + PCIEPort *p; + PCIESlot *s; + int i; + + sms->cxl_devices_state.is_enabled = true; + qdev_prop_set_uint32(qdev, "bus_nr", 0xc0); + pci_realize_and_unref(dev, bus, &error_fatal); + + host = PXB_CXL_DEV(dev)->cxl_host_bridge; + cxl = PCI_HOST_BRIDGE(host); + sms->cxlbus = cxl->bus; + + for (i = 0; i < 2; i++) { + cxlrp = pci_new(-1, "cxl-rp"); + p = PCIE_PORT(cxlrp); + s = PCIE_SLOT(cxlrp); + p->port = i; + s->slot = i; + pci_realize_and_unref(cxlrp, sms->cxlbus, &error_fatal); + } +} + static void create_pcie(SBSAMachineState *sms) { hwaddr base_ecam = sbsa_ref_memmap[SBSA_PCIE_ECAM].base; @@ -638,6 +687,10 @@ static void create_pcie(SBSAMachineState *sms) hwaddr base_mmio_high = sbsa_ref_memmap[SBSA_PCIE_MMIO_HIGH].base; hwaddr size_mmio_high = sbsa_ref_memmap[SBSA_PCIE_MMIO_HIGH].size; hwaddr base_pio = sbsa_ref_memmap[SBSA_PCIE_PIO].base; + hwaddr cxl_base_mmio = sbsa_ref_memmap[SBSA_CXL_MMIO].base; + hwaddr cxl_size_mmio = sbsa_ref_memmap[SBSA_CXL_MMIO].size; + hwaddr cxl_base_mmio_high = sbsa_ref_memmap[SBSA_CXL_MMIO_HIGH].base; + hwaddr cxl_size_mmio_high = sbsa_ref_memmap[SBSA_CXL_MMIO_HIGH].size; int irq = sbsa_ref_irqmap[SBSA_PCIE]; MachineClass *mc = MACHINE_GET_CLASS(sms); MemoryRegion *mmio_alias, *mmio_alias_high, *mmio_reg; @@ -686,6 +739,67 @@ static void create_pcie(SBSAMachineState *sms) pci_create_simple(pci->bus, -1, "bochs-display");
create_smmu(sms, pci->bus); + + /* Map CXL MMIO space */ + mmio_alias = g_new0(MemoryRegion, 1); + mmio_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 1); + memory_region_init_alias(mmio_alias, OBJECT(dev), "cxl-mmio", + mmio_reg, cxl_base_mmio, cxl_size_mmio); + memory_region_add_subregion(get_system_memory(), cxl_base_mmio, mmio_alias); + + /* Map CXL MMIO_HIGH space */ + mmio_alias_high = g_new0(MemoryRegion, 1); + memory_region_init_alias(mmio_alias_high, OBJECT(dev), "cxl-mmio-high", + mmio_reg, cxl_base_mmio_high, cxl_size_mmio_high); + memory_region_add_subregion(get_system_memory(), cxl_base_mmio_high, + mmio_alias_high); + + create_pxb_cxl(sms, pci->bus); +} + +static void create_cxl_host_reg_region(SBSAMachineState *sms) +{ + MemoryRegion *sysmem = get_system_memory(); + MemoryRegion *mr = &sms->cxl_devices_state.host_mr; + + memory_region_init(mr, OBJECT(sms), "cxl_host_reg", + sbsa_ref_memmap[SBSA_CXL_HOST].size); + memory_region_add_subregion(sysmem, sbsa_ref_memmap[SBSA_CXL_HOST].base, mr); +} + +static void create_cxl_fixed_window_region(SBSAMachineState *sms, + MemoryRegion *mem) +{ + char *cxl_host = object_get_canonical_path(OBJECT(sms->cxlbus)); + hwaddr base = sbsa_ref_memmap[SBSA_CXL_FIXED_WINDOW].base; + GList *it; + strList host_target = { NULL, cxl_host }; + CXLFixedMemoryWindowOptions sbsa_ref_cfmwoptions = { + .size = 1 * TiB, + .has_interleave_granularity = false, + .targets = &host_target, + }; + CXLFixedWindow *fw; + + cxl_fixed_memory_window_config(&sms->cxl_devices_state, + &sbsa_ref_cfmwoptions, &error_fatal); + + it = sms->cxl_devices_state.fixed_windows; + fw = it->data; + fw->base = base; + memory_region_init_io(&fw->mr, OBJECT(sms), &cfmws_ops, fw, + "cxl-fixed-memory-region", fw->size); + + memory_region_add_subregion(mem, fw->base, &fw->mr); +} + +static void sbsa_cxl_fmws_link_targets(SBSAMachineState *sms, + Error **errp) +{ + GList *it = sms->cxl_devices_state.fixed_windows; + CXLFixedWindow *fw = it->data; + + fw->target_hbs[0] = PXB_CXL_DEV(pci_bridge_get_device(sms->cxlbus)); }
static void *sbsa_ref_dtb(const struct arm_boot_info *binfo, int *fdt_size) @@ -821,6 +935,12 @@ static void sbsa_ref_init(MachineState *machine)
create_pcie(sms);
+ create_cxl_host_reg_region(sms); + create_cxl_fixed_window_region(sms, sysmem); + pxb_cxl_hook_up_registers(&sms->cxl_devices_state, sms->cxlbus, + &error_fatal); + sbsa_cxl_fmws_link_targets(sms, &error_fatal); + create_secure_ec(secure_sysmem);
sms->bootinfo.ram_size = machine->ram_size; diff --git a/hw/cxl/cxl-host-stubs.c b/hw/cxl/cxl-host-stubs.c index cae4afcdde..aea94933ba 100644 --- a/hw/cxl/cxl-host-stubs.c +++ b/hw/cxl/cxl-host-stubs.c @@ -11,5 +11,7 @@ void cxl_fmws_link_targets(CXLState *stat, Error **errp) {}; void cxl_machine_init(Object *obj, CXLState *state) {}; void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp) {}; +void cxl_fixed_memory_window_config(CXLState *cxl_state, + CXLFixedMemoryWindowOptions *object, Error **errp) {};
const MemoryRegionOps cfmws_ops; diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c index e9f2543c43..d408c7db15 100644 --- a/hw/cxl/cxl-host.c +++ b/hw/cxl/cxl-host.c @@ -22,7 +22,7 @@ #include "hw/pci/pcie_port.h" #include "hw/pci-bridge/pci_expander_bridge.h"
-static void cxl_fixed_memory_window_config(CXLState *cxl_state, +void cxl_fixed_memory_window_config(CXLState *cxl_state, CXLFixedMemoryWindowOptions *object, Error **errp) { diff --git a/include/hw/cxl/cxl_host.h b/include/hw/cxl/cxl_host.h index c9bc9c7c50..f3184733aa 100644 --- a/include/hw/cxl/cxl_host.h +++ b/include/hw/cxl/cxl_host.h @@ -16,6 +16,8 @@ void cxl_machine_init(Object *obj, CXLState *state); void cxl_fmws_link_targets(CXLState *stat, Error **errp); void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp); +void cxl_fixed_memory_window_config(CXLState *cxl_state, + CXLFixedMemoryWindowOptions *object, Error **errp);
extern const MemoryRegionOps cfmws_ops;
On Tue, 5 Nov 2024 18:43:46 +0800 "Yuquan Wang" wangyuquan1236@phytium.com.cn wrote:
This creates a default pxb-cxl (bus_nr=0xc0) bridge with two cxl root ports on sbsa-ref. And the memory layout places 64K space for the cxl host bridge register regions(CHBCR) in the sbsa-ref memmap.
In addition, this support indepentent mmio32(32M) & mmio64(1M) space for cxl components.
Those are too small. Might work today but not sustainable.
I'm a bit surprised it was this simple to move the MMIO Space away from what is normally done for PXBs. I think it might work because the GPEX memory windows are effectively unlimited in size but I'd like some more eyes on this from people familiar with how all that works and whether there might be some corner cases that you haven't seen yet.
Otherwise this looks ok to me.
To provide CFMWs on sbsa-ref, this extends 1TB space from the hole above RAM Memory [SBSA_MEM] for CXL Fixed Memory Window.
Signed-off-by: Yuquan Wang wangyuquan1236@phytium.com.cn
docs/system/arm/sbsa.rst | 4 ++ hw/arm/sbsa-ref.c | 122 +++++++++++++++++++++++++++++++++++++- hw/cxl/cxl-host-stubs.c | 2 + hw/cxl/cxl-host.c | 2 +- include/hw/cxl/cxl_host.h | 2 + 5 files changed, 130 insertions(+), 2 deletions(-)
diff --git a/docs/system/arm/sbsa.rst b/docs/system/arm/sbsa.rst index 2bf3fc8d59..21b88e88e7 100644 --- a/docs/system/arm/sbsa.rst +++ b/docs/system/arm/sbsa.rst @@ -28,6 +28,7 @@ The ``sbsa-ref`` board supports:
- E1000E ethernet card on PCIe bus
- Bochs display adapter on PCIe bus
- A generic SBSA watchdog device
- CXL host bridge and CXL fixed memory window
Board to firmware interface @@ -92,3 +93,6 @@ Platform version changes: 0.4 CPU topology information is present in devicetree.
+0.5
- CXL host bridge and CXL fixed memory window are supported.
diff --git a/hw/arm/sbsa-ref.c b/hw/arm/sbsa-ref.c index e3195d5449..655dc82863 100644 --- a/hw/arm/sbsa-ref.c +++ b/hw/arm/sbsa-ref.c @@ -36,12 +36,18 @@ #include "hw/arm/smmuv3.h" #include "hw/block/flash.h" #include "hw/boards.h" +#include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_host.h" #include "hw/ide/ide-bus.h" #include "hw/ide/ahci-sysbus.h" #include "hw/intc/arm_gicv3_common.h" #include "hw/intc/arm_gicv3_its_common.h" #include "hw/loader.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pci_bus.h" +#include "hw/pci/pcie_port.h" #include "hw/pci-host/gpex.h" +#include "hw/pci-bridge/pci_expander_bridge.h" #include "hw/qdev-properties.h" #include "hw/usb.h" #include "hw/usb/xhci.h" @@ -94,6 +100,10 @@ enum { SBSA_SECURE_MEM, SBSA_AHCI, SBSA_XHCI,
- SBSA_CXL_HOST,
- SBSA_CXL_MMIO,
- SBSA_CXL_MMIO_HIGH,
- SBSA_CXL_FIXED_WINDOW,
}; struct SBSAMachineState { @@ -105,6 +115,8 @@ struct SBSAMachineState { int psci_conduit; DeviceState *gic; PFlashCFI01 *flash[2];
- CXLState cxl_devices_state;
- PCIBus *cxlbus;
}; #define TYPE_SBSA_MACHINE MACHINE_TYPE_NAME("sbsa-ref") @@ -132,6 +144,10 @@ static const MemMapEntry sbsa_ref_memmap[] = { /* Space here reserved for more SMMUs */ [SBSA_AHCI] = { 0x60100000, 0x00010000 }, [SBSA_XHCI] = { 0x60110000, 0x00010000 },
- /* 64KiB CXL Host Bridge Registers space */
- [SBSA_CXL_HOST] = { 0x60120000, 0x00010000 },
- /* 32M CXL 32-bit MMIO space */
- [SBSA_CXL_MMIO] = { 0x60130000, 0x02000000 }, /* Space here reserved for other devices */ [SBSA_PCIE_PIO] = { 0x7fff0000, 0x00010000 }, /* 32-bit address PCIE MMIO space */
@@ -141,6 +157,10 @@ static const MemMapEntry sbsa_ref_memmap[] = { /* ~1TB PCIE MMIO space (4GB to 1024GB boundary) */ [SBSA_PCIE_MMIO_HIGH] = { 0x100000000ULL, 0xFF00000000ULL }, [SBSA_MEM] = { 0x10000000000ULL, RAMLIMIT_BYTES },
- /* 1M CXL 64-bit MMIO space */
- [SBSA_CXL_MMIO_HIGH] = { 0x90000000000ULL, 0x00100000 },
As above, make this bigger. CXL devices can have substantial BARs.
- /* 1TB CXL FIXED WINDOW space */
- [SBSA_CXL_FIXED_WINDOW] = { 0xA0000000000ULL, 0x10000000000ULL },
}; static const int sbsa_ref_irqmap[] = { @@ -216,7 +236,7 @@ static void create_fdt(SBSAMachineState *sms) * fw compatibility. */ qemu_fdt_setprop_cell(fdt, "/", "machine-version-major", 0);
- qemu_fdt_setprop_cell(fdt, "/", "machine-version-minor", 4);
- qemu_fdt_setprop_cell(fdt, "/", "machine-version-minor", 5);
if (ms->numa_state->have_numa_distance) { int size = nb_numa_nodes * nb_numa_nodes * 3 * sizeof(uint32_t); @@ -629,6 +649,35 @@ static void create_smmu(const SBSAMachineState *sms, PCIBus *bus) } } +static void create_pxb_cxl(SBSAMachineState *sms, PCIBus *bus) +{
- DeviceState *qdev = qdev_new(TYPE_PXB_CXL_DEV);
- PCIDevice *dev = PCI_DEVICE(qdev);
- CXLHost *host;
- PCIHostState *cxl;
- PCIDevice *cxlrp;
- PCIEPort *p;
- PCIESlot *s;
- int i;
- sms->cxl_devices_state.is_enabled = true;
- qdev_prop_set_uint32(qdev, "bus_nr", 0xc0);
- pci_realize_and_unref(dev, bus, &error_fatal);
- host = PXB_CXL_DEV(dev)->cxl_host_bridge;
- cxl = PCI_HOST_BRIDGE(host);
- sms->cxlbus = cxl->bus;
- for (i = 0; i < 2; i++) {
cxlrp = pci_new(-1, "cxl-rp");p = PCIE_PORT(cxlrp);s = PCIE_SLOT(cxlrp);p->port = i;s->slot = i;pci_realize_and_unref(cxlrp, sms->cxlbus, &error_fatal);- }
+}
static void create_pcie(SBSAMachineState *sms) { hwaddr base_ecam = sbsa_ref_memmap[SBSA_PCIE_ECAM].base; @@ -638,6 +687,10 @@ static void create_pcie(SBSAMachineState *sms) hwaddr base_mmio_high = sbsa_ref_memmap[SBSA_PCIE_MMIO_HIGH].base; hwaddr size_mmio_high = sbsa_ref_memmap[SBSA_PCIE_MMIO_HIGH].size; hwaddr base_pio = sbsa_ref_memmap[SBSA_PCIE_PIO].base;
- hwaddr cxl_base_mmio = sbsa_ref_memmap[SBSA_CXL_MMIO].base;
- hwaddr cxl_size_mmio = sbsa_ref_memmap[SBSA_CXL_MMIO].size;
- hwaddr cxl_base_mmio_high = sbsa_ref_memmap[SBSA_CXL_MMIO_HIGH].base;
- hwaddr cxl_size_mmio_high = sbsa_ref_memmap[SBSA_CXL_MMIO_HIGH].size; int irq = sbsa_ref_irqmap[SBSA_PCIE]; MachineClass *mc = MACHINE_GET_CLASS(sms); MemoryRegion *mmio_alias, *mmio_alias_high, *mmio_reg;
@@ -686,6 +739,67 @@ static void create_pcie(SBSAMachineState *sms) pci_create_simple(pci->bus, -1, "bochs-display"); create_smmu(sms, pci->bus);
- /* Map CXL MMIO space */
- mmio_alias = g_new0(MemoryRegion, 1);
- mmio_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 1);
- memory_region_init_alias(mmio_alias, OBJECT(dev), "cxl-mmio",
mmio_reg, cxl_base_mmio, cxl_size_mmio);- memory_region_add_subregion(get_system_memory(), cxl_base_mmio, mmio_alias);
- /* Map CXL MMIO_HIGH space */
- mmio_alias_high = g_new0(MemoryRegion, 1);
- memory_region_init_alias(mmio_alias_high, OBJECT(dev), "cxl-mmio-high",
mmio_reg, cxl_base_mmio_high, cxl_size_mmio_high);- memory_region_add_subregion(get_system_memory(), cxl_base_mmio_high,
mmio_alias_high);- create_pxb_cxl(sms, pci->bus);
+}
static void *sbsa_ref_dtb(const struct arm_boot_info *binfo, int *fdt_size) @@ -821,6 +935,12 @@ static void sbsa_ref_init(MachineState *machine) create_pcie(sms);
- create_cxl_host_reg_region(sms);
- create_cxl_fixed_window_region(sms, sysmem);
- pxb_cxl_hook_up_registers(&sms->cxl_devices_state, sms->cxlbus,
&error_fatal);- sbsa_cxl_fmws_link_targets(sms, &error_fatal);
- create_secure_ec(secure_sysmem);
sms->bootinfo.ram_size = machine->ram_size; diff --git a/hw/cxl/cxl-host-stubs.c b/hw/cxl/cxl-host-stubs.c index cae4afcdde..aea94933ba 100644 --- a/hw/cxl/cxl-host-stubs.c +++ b/hw/cxl/cxl-host-stubs.c @@ -11,5 +11,7 @@ void cxl_fmws_link_targets(CXLState *stat, Error **errp) {}; void cxl_machine_init(Object *obj, CXLState *state) {}; void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp) {}; +void cxl_fixed_memory_window_config(CXLState *cxl_state,
CXLFixedMemoryWindowOptions *object, Error **errp) {};const MemoryRegionOps cfmws_ops; diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c index e9f2543c43..d408c7db15 100644 --- a/hw/cxl/cxl-host.c +++ b/hw/cxl/cxl-host.c @@ -22,7 +22,7 @@ #include "hw/pci/pcie_port.h" #include "hw/pci-bridge/pci_expander_bridge.h" -static void cxl_fixed_memory_window_config(CXLState *cxl_state, +void cxl_fixed_memory_window_config(CXLState *cxl_state, CXLFixedMemoryWindowOptions *object, Error **errp) { diff --git a/include/hw/cxl/cxl_host.h b/include/hw/cxl/cxl_host.h index c9bc9c7c50..f3184733aa 100644 --- a/include/hw/cxl/cxl_host.h +++ b/include/hw/cxl/cxl_host.h @@ -16,6 +16,8 @@ void cxl_machine_init(Object *obj, CXLState *state); void cxl_fmws_link_targets(CXLState *stat, Error **errp); void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp); +void cxl_fixed_memory_window_config(CXLState *cxl_state,
CXLFixedMemoryWindowOptions *object, Error **errp);extern const MemoryRegionOps cfmws_ops;
W dniu 7.11.2024 o 13:04, Jonathan Cameron pisze:
On Tue, 5 Nov 2024 18:43:46 +0800 "Yuquan Wang"wangyuquan1236@phytium.com.cn wrote:
This creates a default pxb-cxl (bus_nr=0xc0) bridge with two cxl root ports on sbsa-ref. And the memory layout places 64K space for the cxl host bridge register regions(CHBCR) in the sbsa-ref memmap.
In addition, this support indepentent mmio32(32M) & mmio64(1M) space for cxl components.
Those are too small. Might work today but not sustainable.
I'm a bit surprised it was this simple to move the MMIO Space away from what is normally done for PXBs. I think it might work because the GPEX memory windows are effectively unlimited in size but I'd like some more eyes on this from people familiar with how all that works and whether there might be some corner cases that you haven't seen yet.
I see the same problem as with multiple PCIe buses (for NUMA systems):
pci 0000:c0:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0000:c0:00.0: bridge window [io size 0x1000]: failed to assign pci 0000:c0:01.0: bridge window [io size 0x1000]: can't assign; no space pci 0000:c0:01.0: bridge window [io size 0x1000]: failed to assign
I do not know how it looks on real hardware (all my systems have one PCIe bus) but shouldn't each host bridge have own separate resource windows for config space, buses, mmio etc.?
Now we squeeze all pcie buses as pcie-pxb devices and this patch adds cxl to the combo.
On Tue, 12 Nov 2024 18:10:56 +0100 Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
W dniu 7.11.2024 o 13:04, Jonathan Cameron pisze:
On Tue, 5 Nov 2024 18:43:46 +0800 "Yuquan Wang"wangyuquan1236@phytium.com.cn wrote:
This creates a default pxb-cxl (bus_nr=0xc0) bridge with two cxl root ports on sbsa-ref. And the memory layout places 64K space for the cxl host bridge register regions(CHBCR) in the sbsa-ref memmap.
In addition, this support indepentent mmio32(32M) & mmio64(1M) space for cxl components.
Those are too small. Might work today but not sustainable.
I'm a bit surprised it was this simple to move the MMIO Space away from what is normally done for PXBs. I think it might work because the GPEX memory windows are effectively unlimited in size but I'd like some more eyes on this from people familiar with how all that works and whether there might be some corner cases that you haven't seen yet.
I see the same problem as with multiple PCIe buses (for NUMA systems):
pci 0000:c0:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0000:c0:00.0: bridge window [io size 0x1000]: failed to assign pci 0000:c0:01.0: bridge window [io size 0x1000]: can't assign; no space pci 0000:c0:01.0: bridge window [io size 0x1000]: failed to assign
I do not know how it looks on real hardware (all my systems have one PCIe bus) but shouldn't each host bridge have own separate resource windows for config space, buses, mmio etc.?
Now we squeeze all pcie buses as pcie-pxb devices and this patch adds cxl to the combo.
In theory fine to break them up because each can have a smaller window they just happen to be next to each other in this configuration.
CXL PXB (maybe the pcie one was well) doesn't IIRC support IO regions in general. So that above is kind of normal and shouldn't matter unless you emulate an ancient PCI device.
Jonathan
-
Jonathan Cameron -
Jonathan Cameron
-
Marcin Juszkiewicz -
Yuquan Wang