When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range.
Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu().
Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange.
Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
--- v2 - clamp p2m_ipa_bits = PADDR_BIT instead --- xen/arch/arm/arm64/head.S | 6 ++++++ xen/arch/arm/p2m.c | 10 +++++----- 2 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index aa1f88c764..057dd5d925 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -473,6 +473,12 @@ cpu_init: ldr x0, =(TCR_RES1|TCR_SH0_IS|TCR_ORGN0_WBWA|TCR_IRGN0_WBWA|TCR_T0SZ(64-48)) /* ID_AA64MMFR0_EL1[3:0] (PARange) corresponds to TCR_EL2[18:16] (PS) */ mrs x1, ID_AA64MMFR0_EL1 + /* Limit to 48 bits, 256TB PA range (#5) */ + ubfm x1, x1, #0, #3 + mov x2, #5 + cmp x1, x2 + csel x1, x1, x2, lt + bfi x0, x1, #16, #3
msr tcr_el2, x0 diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c index fb71fa4c1c..3349b464a3 100644 --- a/xen/arch/arm/p2m.c +++ b/xen/arch/arm/p2m.c @@ -32,10 +32,10 @@ static unsigned int __read_mostly max_vmid = MAX_VMID_8_BIT; #define P2M_ROOT_PAGES (1<<P2M_ROOT_ORDER)
/* - * Set larger than any possible value, so the number of IPA bits can be + * Set to the maximum configured support for IPA bits, so the number of IPA bits can be * restricted by external entity (e.g. IOMMU). */ -unsigned int __read_mostly p2m_ipa_bits = 64; +unsigned int __read_mostly p2m_ipa_bits = PADDR_BITS;
/* Helpers to lookup the properties of each level */ static const paddr_t level_masks[] = @@ -2030,7 +2030,7 @@ void __init setup_virt_paging(void) unsigned int root_order; /* Page order of the root of the p2m */ unsigned int sl0; /* Desired SL0, maximum in comment */ } pa_range_info[] = { - /* T0SZ minimum and SL0 maximum from ARM DDI 0487A.b Table D4-5 */ + /* T0SZ minimum and SL0 maximum from ARM DDI 0487H.a Table D5-6 */ /* PA size, t0sz(min), root-order, sl0(max) */ [0] = { 32, 32/*32*/, 0, 1 }, [1] = { 36, 28/*28*/, 0, 1 }, @@ -2038,7 +2038,7 @@ void __init setup_virt_paging(void) [3] = { 42, 22/*22*/, 3, 1 }, [4] = { 44, 20/*20*/, 0, 2 }, [5] = { 48, 16/*16*/, 0, 2 }, - [6] = { 0 }, /* Invalid */ + [6] = { 52, 12/*12*/, 3, 3 }, [7] = { 0 } /* Invalid */ };
@@ -2069,7 +2069,7 @@ void __init setup_virt_paging(void) } }
- /* pa_range is 4 bits, but the defined encodings are only 3 bits */ + /* pa_range is 4 bits but we don't support all modes */ if ( pa_range >= ARRAY_SIZE(pa_range_info) || !pa_range_info[pa_range].pabits ) panic("Unknown encoding of ID_AA64MMFR0_EL1.PARange %x\n", pa_range);
On 28 Apr 2022, at 11:34, Alex Bennée alex.bennee@linaro.org wrote:
When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range.
Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu().
Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange.
Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
v2
- clamp p2m_ipa_bits = PADDR_BIT instead
Hi Alex,
I’ve tested the patch on fvp and Xen+Dom0 runs fine.
Tested-by: Luca Fancellu luca.fancellu@arm.com
Cheers, Luca
Hi Alex,
On 28/04/2022 11:34, Alex Bennée wrote:
When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range.
Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu().
Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange.
Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
Acked-by: Julien Grall jgrall@amazon.com
Cheers,
Julien Grall julien@xen.org writes:
Hi Alex,
On 28/04/2022 11:34, Alex Bennée wrote:
When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range. Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu(). Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange. Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
Acked-by: Julien Grall jgrall@amazon.com
Will you pick this up via your tree or do I need to do something else to get it upstreamed? I guess it needs to go on master and last stable?
Hi Alex,
On 10/05/2022 15:03, Alex Bennée wrote:
Julien Grall julien@xen.org writes:
Hi Alex,
On 28/04/2022 11:34, Alex Bennée wrote:
When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range. Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu(). Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange. Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
Acked-by: Julien Grall jgrall@amazon.com
Will you pick this up via your tree or do I need to do something else to get it upstreamed? I guess it needs to go on master and last stable?
We only have one tree in Xen where committers (such as Stefano and I) will commit patches regularly to staging. Osstest will then push to master once the testing passed.
I have done that now. Interestingly, git am wasn't able to apply this patch. I had to do with:
42sh> git am --show-current-patch=diff | patch -p1 patching file xen/arch/arm/arm64/head.S Hunk #1 succeeded at 474 (offset 1 line). patching file xen/arch/arm/p2m.c Hunk #1 succeeded at 32 with fuzz 2. Hunk #2 succeeded at 2023 (offset -7 lines). Hunk #3 succeeded at 2031 (offset -7 lines). Hunk #4 succeeded at 2062 (offset -7 lines).
Which branch did you use for sending the patch?
Regarding stable, I will add the patch in my backport candidate list and send a list to Stefano when we prepare the backports.
Cheers,
Julien Grall julien@xen.org writes:
Hi Alex,
On 10/05/2022 15:03, Alex Bennée wrote:
Julien Grall julien@xen.org writes:
Hi Alex,
On 28/04/2022 11:34, Alex Bennée wrote:
When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range. Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu(). Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange. Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
Acked-by: Julien Grall jgrall@amazon.com
Will you pick this up via your tree or do I need to do something else to get it upstreamed? I guess it needs to go on master and last stable?
We only have one tree in Xen where committers (such as Stefano and I) will commit patches regularly to staging. Osstest will then push to master once the testing passed.
I have done that now. Interestingly, git am wasn't able to apply this patch. I had to do with:
42sh> git am --show-current-patch=diff | patch -p1 patching file xen/arch/arm/arm64/head.S Hunk #1 succeeded at 474 (offset 1 line). patching file xen/arch/arm/p2m.c Hunk #1 succeeded at 32 with fuzz 2. Hunk #2 succeeded at 2023 (offset -7 lines). Hunk #3 succeeded at 2031 (offset -7 lines). Hunk #4 succeeded at 2062 (offset -7 lines).
Which branch did you use for sending the patch?
0941d6cb23 from RELEASE-4.16.0
Regarding stable, I will add the patch in my backport candidate list and send a list to Stefano when we prepare the backports.
Cheers,
Hi Alex,
On 10/05/2022 15:47, Alex Bennée wrote:
Julien Grall julien@xen.org writes:
On 10/05/2022 15:03, Alex Bennée wrote:
Julien Grall julien@xen.org writes:
Hi Alex,
On 28/04/2022 11:34, Alex Bennée wrote:
When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range. Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu(). Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange. Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
Acked-by: Julien Grall jgrall@amazon.com
Will you pick this up via your tree or do I need to do something else to get it upstreamed? I guess it needs to go on master and last stable?
We only have one tree in Xen where committers (such as Stefano and I) will commit patches regularly to staging. Osstest will then push to master once the testing passed.
I have done that now. Interestingly, git am wasn't able to apply this patch. I had to do with:
42sh> git am --show-current-patch=diff | patch -p1 patching file xen/arch/arm/arm64/head.S Hunk #1 succeeded at 474 (offset 1 line). patching file xen/arch/arm/p2m.c Hunk #1 succeeded at 32 with fuzz 2. Hunk #2 succeeded at 2023 (offset -7 lines). Hunk #3 succeeded at 2031 (offset -7 lines). Hunk #4 succeeded at 2062 (offset -7 lines).
Which branch did you use for sending the patch?
0941d6cb23 from RELEASE-4.16.0
This would explain why. Patch sent to xen-devel should be based on staging (or master).
Cheers,
Julien Grall julien@xen.org writes:
Hi Alex,
On 10/05/2022 15:47, Alex Bennée wrote:
Julien Grall julien@xen.org writes:
On 10/05/2022 15:03, Alex Bennée wrote:
Julien Grall julien@xen.org writes:
Hi Alex,
On 28/04/2022 11:34, Alex Bennée wrote:
When we introduced FEAT_LPA to QEMU's -cpu max we discovered older kernels had a bug where the physical address was copied directly from ID_AA64MMFR0_EL1.PARange field. The early cpu_init code of Xen commits the same error by blindly copying across the max supported range. Unsurprisingly when the page tables aren't set up for these greater ranges hilarity ensues and the hypervisor crashes fairly early on in the boot-up sequence. This happens when we write to the control register in enable_mmu(). Attempt to fix this the same way as the Linux kernel does by gating PARange to the maximum the hypervisor can handle. I also had to fix up code in p2m which panics when it sees an "invalid" entry in PARange. Signed-off-by: Alex Bennée alex.bennee@linaro.org Cc: Richard Henderson richard.henderson@linaro.org Cc: Stefano Stabellini sstabellini@kernel.org Cc: Julien Grall julien@xen.org Cc: Volodymyr Babchuk Volodymyr_Babchuk@epam.com Cc: Bertrand Marquis bertrand.marquis@arm.com
Acked-by: Julien Grall jgrall@amazon.com
Will you pick this up via your tree or do I need to do something else to get it upstreamed? I guess it needs to go on master and last stable?
We only have one tree in Xen where committers (such as Stefano and I) will commit patches regularly to staging. Osstest will then push to master once the testing passed.
I have done that now. Interestingly, git am wasn't able to apply this patch. I had to do with:
42sh> git am --show-current-patch=diff | patch -p1 patching file xen/arch/arm/arm64/head.S Hunk #1 succeeded at 474 (offset 1 line). patching file xen/arch/arm/p2m.c Hunk #1 succeeded at 32 with fuzz 2. Hunk #2 succeeded at 2023 (offset -7 lines). Hunk #3 succeeded at 2031 (offset -7 lines). Hunk #4 succeeded at 2062 (offset -7 lines).
Which branch did you use for sending the patch?
0941d6cb23 from RELEASE-4.16.0
This would explain why. Patch sent to xen-devel should be based on staging (or master).
Ahh at the time I wasn't sure if there was another regression in master so I was basing of stable. I'll re-base of master next time ;-)
stratos-dev@op-lists.linaro.org
-
Alex Bennée -
Julien Grall -
Luca Fancellu