[Linaro-open-discussions] ACPI Generic Initiator support status and potential additions

Jonathan Cameron Jonathan.Cameron at Huawei.com
Wed Nov 4 13:23:31 UTC 2020


Hi All,

This one made it onto the list of topics to discuss (now marked as no need
to discuss). I've been meaning to give a status update by email including
what is outstanding here. Please let me know if this fails to cover
some aspect of interest.

Background:

https://github.com/hisilicon/acpi-numa-whitepaper/releases/tag/v0.93 chapter 3.

Generic initiators are a concept in ACPI 6.3 (sec 5.2.16.6) to plug a hole
in the definition of proximity domains.

Proximity domains in ACPI (NUMA nodes in kernel) are defined by entries in SRAT
table.  There are a whole range of different types of SRAT entry but before
ACPI 6.3 this more or less in practice meant that a proximity domain only
existed if it contained either (or both) memory and CPUs.   Other initiators
of memory transactions such as network cards can be assigned to an existing
proximity domain via _PXM in ACPI DSDT. This restricted them to sharing a domain
with either memory or processors.

That doesn't always reflect system architecture, particularly with the addition
of richer descriptions of access characteristics (latency / bandwidth) brought
in by HMAT.  Hence Generic Initiator domains to allow you to specify a
proximity domain with some other type of device in it (such as a network card)
and get all of the descriptive capability available for CPU / memory nodes.

Note that this was brought in prior to CXL becoming public but 1.1 CXL spec
states that initiators on CXL should be described using Generic Initiator nodes.
This should accelerate the number of users of this feature considerably.
It is also useful in some existing systems.

What support was needed in kernel:

1) Parsing of the SRAT Generic Initiator Affinity Structure
2) Instantiating the NUMA nodes that map to the GI PXM nodes to ensure stuff
   like fallback lists for memory allocation work as normal.
3) Richer use of HMAT access characteristics to differentiate nearest CPU
   to memory from nearest initiator to memory.
4) PXM assignment from the SRAT record rather than _PXM (not yet done).
5) PCI PXM assignment (not yet done)

Status:

The kernel patches sat on the list (with rebases) for well over a year
failing to get the architecture review needed (as there was significant
risk of breakage in both ARM64 and x86). It was to break this blockage
that we were interested in an open discussion on this. However, they did
recently get x86 review this and Rafael queued them for 5.10 (now merged)

The PCI PXM issue has been long standing due to some buggy firmware
on certain X86 boards and the need for a clarification in the ACPI spec
(added in 6.3).  To make this safe, needed to ensure that NUMA nodes on
ACPI systems can only be instantiated during the main parse of SRAT.

https://lore.kernel.org/linux-mm/20200818142430.1156547-1-Jonathan.Cameron@huawei.com/

That fix is now in place, and we'll resend the PCI fix shortly.
Note it may be "interesting" to support nodes from CXL CDAT tables at runtime
but that is another topic.
( https://uefi.org/node/4093 https://lore.kernel.org/linux-cxl/20201102183428.00005f4f@Huawei.com/T/#m52fb027e30e7c476c4993ba72aad618620f178c6 )

For a Generic Initiator Nodes, there are two ways a device an be assigned
to the proximity domain. Conventional _PXM in DSDT can be used and
that is now supported. The SRAT entry itself also contains an address
(PCI seg + BDF or Platform UID / HID based).  There is no obligation to
provide both. The SRAT based method will require some level of alternative
infrastructure to that used for _PXM. We may look at this at some stage.

So a few outstanding things but probably not worth discussing on a call
at this stage unless anyone is seeing problems with the stuff already merged.

Thanks,

Jonathan


More information about the Linaro-open-discussions mailing list