On Mon, 15 Mar 2021 16:42:34 +0000
Lorenzo Pieralisi via Linaro-open-discussions <linaro-open-discussions(a)op-lists.linaro.org> wrote:
> On Mon, Mar 15, 2021 at 01:00:19AM +0000, Jammy Zhou via Linaro-open-discussions wrote:
> > Hi all,
> >
> > We're going to have the LOD meeting in one week, currently we don't
> > have any topic proposed yet. Please let me know if you have something
> > to discuss.
>
> I would be glad to have a follow-up with Jonathan on CXL expansion memory
> on arm64 if possible, hopefully I can read up in the meanwhile to get up
> to speed on the matter.
I'm always happy to talk on this topic :)
There is a related discussion around the (code-first) proposal to generalize
GI usage to enable some CXL use cases that we might also be able to have
a useful discussion about:
https://lore.kernel.org/linux-acpi/CAPcyv4gmd_cygXK0PpGkXmJLC3_ctEpRvpi5P-Q…
As there are some US folks who are interested in this topic (but super busy),
can we do a straw poll of whether any of them can make a call on Monday?
If not go for a more China/Europe friendly time?
We had a few more topics brewing, but I'm not sure they will be in a state
to discuss next week (or to give anyone else time to think about them
in advance).
Obviously good to touch on any updates to older topics as well if anyone
has any!
Jonathan
>
> Lorenzo
Hello,
Please find the requested slides containing *rough* QEMU boot time figures attached.
I will further refine/update the slides with Firmware/Linux Guest related figures
and will share later.
Many thanks
Salil
> From: Linaro-open-discussions
> [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of
> Jammy Zhou via Linaro-open-discussions
> Sent: Monday, March 22, 2021 12:38 PM
> To: Vincent Guittot <vincent.guittot(a)linaro.org>
> Cc: Lorenzo Pieralisi via Linaro-open-discussions
> <linaro-open-discussions(a)op-lists.linaro.org>; Jon Masters
> <jcm(a)jonmasters.org>
> Subject: Re: [Linaro-open-discussions] LOD meeting agenda for March 22nd
>
> Hi All,
>
> The meeting notes are published in the collaborate page below (with
> Jonathan's CXL slides and the recording).
>
> https://collaborate.linaro.org/display/LOD/2021-3-22+Meeting+Meeting+notes
>
> Regards,
> Jammy
>
> On Thu, 18 Mar 2021 at 22:48, Vincent Guittot <vincent.guittot(a)linaro.org>
> wrote:
>
> > On Thu, 18 Mar 2021 at 15:39, Jammy Zhou via Linaro-open-discussions
> > <linaro-open-discussions(a)op-lists.linaro.org> wrote:
> > >
> > > Hmm, it looks like I forgot to add one event in the calendar. It has been
> > > added by Vincent just a moment ago. Thanks!
> >
> > Yeah, I noticed that the meeting didn't appear in the agenda so I
> > figured out to add it
> >
> > >
> > > On Thu, 18 Mar 2021 at 22:11, Jonathan Cameron <
> > jonathan.cameron(a)huawei.com>
> > > wrote:
> > >
> > > > Ah, I'd missed that as not in the calendar at the bottom of the page
> > yet.
> > > >
> > > > That works fine for me.
> > > >
> > > > Jonathan
> > > >
> > > > -----Original Message-----
> > > > From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> > > > Sent: 18 March 2021 12:28
> > > > To: Jonathan Cameron <jonathan.cameron(a)huawei.com>
> > > > Cc: Lorenzo Pieralisi via Linaro-open-discussions <
> > > > linaro-open-discussions(a)op-lists.linaro.org>; Jammy Zhou <
> > > > jammy.zhou(a)linaro.org>; Jon Masters <jcm(a)jonmasters.org>
> > > > Subject: Re: [Linaro-open-discussions] LOD meeting agenda for March
> > 22nd
> > > >
> > > > On Wed, Mar 17, 2021 at 05:17:05PM +0000, Jonathan Cameron wrote:
> > > >
> > > > [...]
> > > >
> > > > > > I would also take this opportunity to get your (and Salil's)
> > > > > > feedback on the virtual CPU hotplug implementation with PSCI if you
> > > > > > had time to think about what we discussed last time.
> > > > >
> > > > > That should be fine to talk about / keep up momentum on.
> > > > >
> > > > > Have we actually picked a time slot?
> > > > >
> > > > > I have a strangely empty day on Monday, so feel free to suggest one.
> > > > > As Jon hasn't replied, lets go for a time that isn't too unpleasant
> > > > > for those based in China.
> > > >
> > > > I am happy to keep schedule as advertised in:
> > > >
> > > >
> > https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home
> > > >
> > > > namely 10AM GMT. Is it fine for everyone ?
> > > >
> > > > Thanks,
> > > > Lorenzo
> > > >
> > > --
> > > Linaro-open-discussions mailing list
> > > https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home
> > > https://op-lists.linaro.org/mailman/listinfo/linaro-open-discussions
> >
> --
> Linaro-open-discussions mailing list
> https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home
> https://op-lists.linaro.org/mailman/listinfo/linaro-open-discussions
Hi Lorenzo,
> > -----Original Message-----
> > From: Linaro-open-discussions
> > [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of
> > Lorenzo Pieralisi via Linaro-open-discussions
> > Sent: 10 December 2020 09:13
> > To: Jonathan Cameron <jonathan.cameron(a)huawei.com>
> > Cc: linaro-open-discussions(a)op-lists.linaro.org
> > Subject: Re: [Linaro-open-discussions] Suggested Agenda / timings for
> Monday
> > 7 Dec call
> >
> > On Thu, Dec 03, 2020 at 01:40:55PM +0000, Jonathan Cameron wrote:
> >
> > [...]
> >
> > > RMR related topics - follow up on last call: Lorenzo Pieralisi (ARM) - 5 mins
Just a quick one on updated RMR spec publication. I see a revision E.a here,
https://developer.arm.com/documentation/den0049/latest/
This seems to have addressed all the things we discussed. Is this a final one officially?
If yes, I couldn't understand why the RMR node flags are at an offset 8, compared to
other nodes where the node specific data normally starts at offset 16.
Could you please check and let me know.
Thanks,
Shameer
Hi Zhangfei,
On Fri, Mar 26, 2021 at 12:35:49PM +0000, Zhangfei Gao via Linaro-open-discussions wrote:
> Hi,
>
> I am looking for some suggestions about dvm [1].
>
> we are testing sva with openssl, and dvm is enabled by default to broadcast
> TLB maintenance.
> And thp is enabled in system by default, a daemon khugepaged
> (mm/khugepaged.c) is running
> to collapse memory to huge page in a period of time.
>
> And we found thp: khugepaged may cause issue to sva test case.
> https://github.com/Linaro/uadk/issues/215
>
> Two cases:
> 1. Heavy weight test case, async mode, 36+ jobs.
> openssl speed -elapsed -engine uadk -async_jobs 36 rsa2048
>
> Once collapse_huge_page happens.
> hardware may hung in io page fault, there maybe huge numbers of page fault
> keeps happening,
> while usually only several io page fault reported.
>
> 2. With high thp scan frequence, low weight test case, sync mode, 1 job.
> data may not correct.
> sudo openssl speed -engine uadk -seconds 1 rsa2048
> Doing 2048 bits public rsa's for 1s: RSA verify failure
>
> Two workarounds:
> 1. disable thp
> echo never > /sys/kernel/mm/transparent_hugepage/enabled
>
> 2. enable thp but add tlbi and ignore dvm.
> Adding arm_smmu_tlb_inv_range in arm-smmu-v3-sva.c:
> arm_smmu_mm_invalidate_range.
> It is called by khugepaged: collapse_huge_page->
> mmu_notifier_invalidate_range_end
>
>
> Looks dvm is not taking effect in some corner cases.
>
> Questions
> 1. if khugepaged collapse the memory used by device, and then change tlb,
> can dvm sync this tlb change to smmu.
>
> 2. Any possible dma is just using the memory, and collapsed by khugepaged,
> can dvm handle this case?
> Or khugepaged should not touch memory using by device, looks khugepaged
> can not distinguish.
I think hugepage is juste one symptom, and may not be the only way to
trigger the issue. On khugepaged collapse, a large invalidation is issued
which arch/arm64/include/asm/tlbflush.h transforms into a TLBI-by-ASID:
if ((!system_supports_tlb_range() &&
(end - start) >= (MAX_TLBI_OPS * stride)) ||
pages >= MAX_TLBI_RANGE_PAGES) {
flush_tlb_mm(vma->vm_mm);
return;
}
With MAX_TLBI_OPS = 512 and stride = 0x1000 we hit the size limit of 2M,
which is the size of a hugepage. My guess is that on khugepage collapse we
issue a TLBI ASIDE1IS (rather than a TLBI VALE1IS), somehow that isn't
taken into account by the SMMU, and we end up with stale TLB entries
leading to memory corruption. If that's the case, I'd suggest keeping DVM
disabled on this platform (workaround 2), to force all SVA invalidation to
go through the command queue.
Thanks,
Jean
Hi,
I am looking for some suggestions about dvm [1].
we are testing sva with openssl, and dvm is enabled by default to
broadcast TLB maintenance.
And thp is enabled in system by default, a daemon khugepaged
(mm/khugepaged.c) is running
to collapse memory to huge page in a period of time.
And we found thp: khugepaged may cause issue to sva test case.
https://github.com/Linaro/uadk/issues/215
Two cases:
1. Heavy weight test case, async mode, 36+ jobs.
openssl speed -elapsed -engine uadk -async_jobs 36 rsa2048
Once collapse_huge_page happens.
hardware may hung in io page fault, there maybe huge numbers of page
fault keeps happening,
while usually only several io page fault reported.
2. With high thp scan frequence, low weight test case, sync mode, 1 job.
data may not correct.
sudo openssl speed -engine uadk -seconds 1 rsa2048
Doing 2048 bits public rsa's for 1s: RSA verify failure
Two workarounds:
1. disable thp
echo never > /sys/kernel/mm/transparent_hugepage/enabled
2. enable thp but add tlbi and ignore dvm.
Adding arm_smmu_tlb_inv_range in arm-smmu-v3-sva.c:
arm_smmu_mm_invalidate_range.
It is called by khugepaged: collapse_huge_page->
mmu_notifier_invalidate_range_end
Looks dvm is not taking effect in some corner cases.
Questions
1. if khugepaged collapse the memory used by device, and then change
tlb, can dvm sync this tlb change to smmu.
2. Any possible dma is just using the memory, and collapsed by
khugepaged, can dvm handle this case?
Or khugepaged should not touch memory using by device, looks
khugepaged can not distinguish.
[1] DVM
Distributed Virtual Memory, a protocol for interconnect messages to provide
broadcast TLB maintenance operations (among other things).
Thanks
On Thu, 18 Mar 2021 at 15:39, Jammy Zhou via Linaro-open-discussions
<linaro-open-discussions(a)op-lists.linaro.org> wrote:
>
> Hmm, it looks like I forgot to add one event in the calendar. It has been
> added by Vincent just a moment ago. Thanks!
Yeah, I noticed that the meeting didn't appear in the agenda so I
figured out to add it
>
> On Thu, 18 Mar 2021 at 22:11, Jonathan Cameron <jonathan.cameron(a)huawei.com>
> wrote:
>
> > Ah, I'd missed that as not in the calendar at the bottom of the page yet.
> >
> > That works fine for me.
> >
> > Jonathan
> >
> > -----Original Message-----
> > From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> > Sent: 18 March 2021 12:28
> > To: Jonathan Cameron <jonathan.cameron(a)huawei.com>
> > Cc: Lorenzo Pieralisi via Linaro-open-discussions <
> > linaro-open-discussions(a)op-lists.linaro.org>; Jammy Zhou <
> > jammy.zhou(a)linaro.org>; Jon Masters <jcm(a)jonmasters.org>
> > Subject: Re: [Linaro-open-discussions] LOD meeting agenda for March 22nd
> >
> > On Wed, Mar 17, 2021 at 05:17:05PM +0000, Jonathan Cameron wrote:
> >
> > [...]
> >
> > > > I would also take this opportunity to get your (and Salil's)
> > > > feedback on the virtual CPU hotplug implementation with PSCI if you
> > > > had time to think about what we discussed last time.
> > >
> > > That should be fine to talk about / keep up momentum on.
> > >
> > > Have we actually picked a time slot?
> > >
> > > I have a strangely empty day on Monday, so feel free to suggest one.
> > > As Jon hasn't replied, lets go for a time that isn't too unpleasant
> > > for those based in China.
> >
> > I am happy to keep schedule as advertised in:
> >
> > https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home
> >
> > namely 10AM GMT. Is it fine for everyone ?
> >
> > Thanks,
> > Lorenzo
> >
> --
> Linaro-open-discussions mailing list
> https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home
> https://op-lists.linaro.org/mailman/listinfo/linaro-open-discussions
On Mon, Mar 15, 2021 at 01:00:19AM +0000, Jammy Zhou via Linaro-open-discussions wrote:
> Hi all,
>
> We're going to have the LOD meeting in one week, currently we don't
> have any topic proposed yet. Please let me know if you have something
> to discuss.
I would be glad to have a follow-up with Jonathan on CXL expansion memory
on arm64 if possible, hopefully I can read up in the meanwhile to get up
to speed on the matter.
Lorenzo
Hi all,
We're going to have the LOD meeting in one week, currently we don't have
any topic proposed yet. Please let me know if you have something to discuss.
Regards,
Jammy