- Linaro-open-discussions - op-lists.linaro.org

Re: [Linaro-open-discussions] [PATCH 1/4] sched: Add infrastructure to describe if cluster scheduler is really running

by Song Bao Hua (Barry Song)

> -----Original Message----- > From: Tim Chen [mailto:tim.c.chen@linux.intel.com] > Sent: Friday, June 25, 2021 11:03 AM > To: Song Bao Hua (Barry Song) <song.bao.hua(a)hisilicon.com>; yangyicong > <yangyicong(a)huawei.com> > Cc: guodong.xu(a)linaro.org; tangchengchang <tangchengchang(a)huawei.com>; > Zengtao (B) <prime.zeng(a)hisilicon.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; chenxiang (M) <chenxiang66(a)hisilicon.com>; > Linuxarm <linuxarm(a)huawei.com> > Subject: Re: [PATCH 1/4] sched: Add infrastructure to describe if cluster > scheduler is really running > > > > On 6/24/21 2:15 AM, Barry Song wrote: > > use sched_cluster_present static key to describe if cluster scheduler > > is really working. > > and add sched_cluster_active() API to reflect the status of cluster > > scheduler as well. > > Only if cluster is bigger than smt, and less than coregroup, we are > > quite sure cluster scheduler is used. > > > > Signed-off-by: Barry Song <song.bao.hua(a)hisilicon.com> > > --- > > include/linux/sched/cluster.h | 19 +++++++++++++++++++ > > kernel/sched/core.c | 20 ++++++++++++++++++++ > > kernel/sched/fair.c | 4 ++++ > > kernel/sched/sched.h | 1 + > > 4 files changed, 44 insertions(+) > > create mode 100644 include/linux/sched/cluster.h > > > > diff --git a/include/linux/sched/cluster.h b/include/linux/sched/cluster.h > > new file mode 100644 > > index 0000000..ea6c475 > > --- /dev/null > > +++ b/include/linux/sched/cluster.h > > @@ -0,0 +1,19 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +#ifndef _LINUX_SCHED_CLUSTER_H > > +#define _LINUX_SCHED_CLUSTER_H > > + > > +#include <linux/static_key.h> > > + > > +#ifdef CONFIG_SCHED_CLUSTER > > +extern struct static_key_false sched_cluster_present; > > + > > +static __always_inline bool sched_cluster_active(void) > > +{ > > + return static_branch_likely(&sched_cluster_present); > > Barry, we may encounter load where having the cluster sched > domain is not productive. Wonder if we should add a run time > sysfs control to turn on/off the sched cluster feature (for packing) > and cluster sched domain (for spreading). > > This is a bit orthogonal to this patch series but could be > a 3rd patchset. We may try though I am not sure if Peter is a fan of this kind of sysctl. This kind of thing is usually done by sched_feat(): There is a sysfs_feature ctl in /sys/kernel/debug/sched/features: kernel/sched/debug.c static int sched_feat_set(char *cmp) { int i; int neg = 0; if (strncmp(cmp, "NO_", 3) == 0) { neg = 1; cmp += 3; } i = match_string(sched_feat_names, __SCHED_FEAT_NR, cmp); if (i < 0) return i; if (neg) { sysctl_sched_features &= ~(1UL << i); sched_feat_disable(i); } else { sysctl_sched_features |= (1UL << i); sched_feat_enable(i); } return 0; } > > Tim > > > +} > > +#else > > +static inline bool sched_cluster_active(void) { return false; } > > + > > +#endif > > + > > +#endif > > diff --git a/kernel/sched/core.c b/kernel/sched/core.c > > index 9e9a5be..dd5984d 100644 > > --- a/kernel/sched/core.c > > +++ b/kernel/sched/core.c > > @@ -8655,6 +8655,17 @@ int sched_cpu_activate(unsigned int cpu) > > if (cpumask_weight(cpu_smt_mask(cpu)) == 2) > > static_branch_inc_cpuslocked(&sched_smt_present); > > #endif > > + > > +#ifdef CONFIG_SCHED_CLUSTER > > + /* > > + * When going up, increment the number of cluster cpus with > > + * cluster present. > > + */ > > + if (cpumask_weight(cpu_cluster_mask(cpu)) > > cpumask_weight(cpu_smt_mask(cpu)) && > > + cpumask_weight(cpu_cluster_mask(cpu)) < > cpumask_weight(cpu_coregroup_mask(cpu))) > > + static_branch_inc_cpuslocked(&sched_cluster_present); > > +#endif > > + > > set_cpu_active(cpu, true); > > > > if (sched_smp_initialized) { > > @@ -8731,6 +8742,15 @@ int sched_cpu_deactivate(unsigned int cpu) > > static_branch_dec_cpuslocked(&sched_smt_present); > > #endif > > > > +#ifdef CONFIG_SCHED_CLUSTER > > + /* > > + * When going down, decrement the number of cpus with cluster present. > > + */ > > + if (cpumask_weight(cpu_cluster_mask(cpu)) > > cpumask_weight(cpu_smt_mask(cpu)) && > > + cpumask_weight(cpu_cluster_mask(cpu)) < > cpumask_weight(cpu_coregroup_mask(cpu))) > > + static_branch_dec_cpuslocked(&sched_cluster_present); > > +#endif > > + > > if (!sched_smp_initialized) > > return 0; > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > > index ce625bf..8578cb1 100644 > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -5997,6 +5997,10 @@ static inline int __select_idle_cpu(int cpu, struct > task_struct *p) > > return -1; > > } > > > > +#ifdef CONFIG_SCHED_CLUSTER > > +DEFINE_STATIC_KEY_FALSE(sched_cluster_present); > > +#endif > > + > > #ifdef CONFIG_SCHED_SMT > > DEFINE_STATIC_KEY_FALSE(sched_smt_present); > > EXPORT_SYMBOL_GPL(sched_smt_present); > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > > index 8f0194c..0f8f610 100644 > > --- a/kernel/sched/sched.h > > +++ b/kernel/sched/sched.h > > @@ -6,6 +6,7 @@ > > > > #include <linux/sched/autogroup.h> > > #include <linux/sched/clock.h> > > +#include <linux/sched/cluster.h> > > #include <linux/sched/coredump.h> > > #include <linux/sched/cpufreq.h> > > #include <linux/sched/cputime.h> > > Thanks Barry

2 years, 9 months

1
0
0 0

[PATCH 0/3] cluster-scheduler upstream plan

by Barry Song

Hi Tim, I'd like to introduce my colleague Yicong to you. Yicong is also working on scheduler with me. Hi Yicong and Tim, I am planning to send the normal patchset not RFC after 5.14-rc1 if this patchset from TianTao can be merged before that: https://lore.kernel.org/lkml/1622712162-7028-1-git-send-email-tiantao6@hisi… But firstly, I will only send the patchset supporting spreading as packing path is quite tricky. If we put them together, hardly maintainers can review it. Since SCHED_CLUSTER's default status is disabled in Kconfig, lacking the consideration of packing path won't hurt those workloads who like packing. Three patches in this email thread are for the first patchset with spreading path only. Note patch 1/3 will be rebased againest: https://lore.kernel.org/lkml/1622712162-7028-1-git-send-email-tiantao6@hisi… https://lore.kernel.org/lkml/20210611052249.25776-1-song.bao.hua@hisilicon.… In the commit log, I put some TODOs which might need some benchmark data from Tim and Yicong. Hi Tim, we don't have Jacobsville machine, I will appreciate a lot if you can provide some benchmark data for x86. Would you please work on this? Thanks Barry Barry Song (1): scheduler: add scheduler level for clusters Jonathan Cameron (1): topology: Represent clusters of CPUs within a die Tim Chen (1): scheduler: Add cluster scheduler level for x86 Documentation/admin-guide/cputopology.rst | 26 +++++++-- arch/arm64/Kconfig | 7 +++ arch/arm64/kernel/topology.c | 2 + arch/x86/Kconfig | 8 +++ arch/x86/include/asm/smp.h | 7 +++ arch/x86/include/asm/topology.h | 3 + arch/x86/kernel/cpu/cacheinfo.c | 1 + arch/x86/kernel/cpu/common.c | 3 + arch/x86/kernel/smpboot.c | 44 ++++++++++++++- drivers/acpi/pptt.c | 67 +++++++++++++++++++++++ drivers/base/arch_topology.c | 15 +++++ drivers/base/topology.c | 10 ++++ include/linux/acpi.h | 5 ++ include/linux/arch_topology.h | 5 ++ include/linux/sched/topology.h | 7 +++ include/linux/topology.h | 13 +++++ kernel/sched/topology.c | 5 ++ 17 files changed, 223 insertions(+), 5 deletions(-) -- 2.25.1

2 years, 10 months

4
19
0 0

Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28

by Salil Mehta

> From: Linaro-open-discussions > [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of > Lorenzo Pieralisi via Linaro-open-discussions > Sent: Tuesday, June 22, 2021 10:39 AM > > On Mon, Jun 21, 2021 at 09:33:42PM +0000, Song Bao Hua (Barry Song) wrote: > > > > > > > -----Original Message----- > > > From: Linaro-open-discussions > > > [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of > > > Lorenzo Pieralisi via Linaro-open-discussions > > > Sent: Monday, June 21, 2021 9:32 PM > > > To: Jammy Zhou <jammy.zhou(a)linaro.org> > > > Cc: Lorenzo Pieralisi via Linaro-open-discussions > > > <linaro-open-discussions(a)op-lists.linaro.org> > > > Subject: Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28 > > > > > > On Mon, Jun 21, 2021 at 12:37:03AM +0000, Jammy Zhou via Linaro-open-discussions > > > wrote: > > > > Hi all, > > > > > > > > We are only one week away from the next LOD meeting on June 28. If you > > > > have any topic to discuss, please let me know. > > > > > > We could discuss the virt CPU hotplug status/forward plan. > > > > > > I don't know if Barry's scheduler discussion can benefit from an LOD > > > session, please chime in if that's the case. > > > > Hi Lorenzo, > > Thanks for asking. I have no important updates at this time. That > > discussion is mainly for co-working with Tim Chen on the plan of > > upstreaming and benchmarking, also figuring out whether the patches > > are leading to the same result on arm and x86 platforms. > > > > > > > > AOB ? > > Ok. So, anything else to discuss other than the PSCI KVM patches > that are currently on ML ? Jonathan, Salil, anyone please do let > me know. Hi Lorenzo, Do you think it is useful & possible for you guys to present the kernel changes in the upcoming Linux Plumbers Conference 2021 in context to vcpu Hotplug? There are issues like below which might need involvement of larger audience. 1. Kernel sizing over possible vcpus (some patches by Bharath Rao floating around) 2. Any user cases which might be impacted because of this change? 3. Issues related to QEMU.(legal?) etc. Date of conference is in September so there is plenty of time to push the patches in between. Proposal submission cutoff date is 25th June, 2021 Thanks Salil > > Thanks, > Lorenzo > -- > Linaro-open-discussions mailing list > https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home > https://op-lists.linaro.org/mailman/listinfo/linaro-open-discussions

2 years, 10 months

2
2
0 0

[PATCH 0/4] cluster-scheduler upstream plan - packing path

by Barry Song

Hi Yicong and Tim, This is the 2nd patchset following the 1st one: https://op-lists.linaro.org/pipermail/linaro-open-discussions/2021-June/000… While the 1st patchset was only focusing on spreading path, this patchset is mainly for packing path. I have only tested tbench4 on one numa. Without this patchset, I am seeing up to 5% performance decrease on tbench4 by spreading path only; but with it, I can see up to 28% performance increase, compared to the case of w/o cluster scheduler. I am running the benchmark by Mel's mmtests with this config file: configs/config-scheduler-schbench-1numa # MM Test Parameters export MMTESTS="tbench4" # List of monitors export RUN_MONITOR=yes export MONITORS_ALWAYS= export MONITORS_GZIP="proc-vmstat mpstat" export MONITORS_WITH_LATENCY="vmstat" export MONITOR_UPDATE_FREQUENCY=10 # TBench export TBENCH_DURATION=60 export TBENCH_MIN_CLIENTS=1 export TBENCH_MAX_CLIENTS=96 with commands like: numactl -N 0 -m 0 ./run-mmtests.sh --no-monitor -c configs/config-scheduler-schbench-1numa testtag my machine has 4 numa, each numa has 24 cores(6 clusters). Hopefully, we are going to have more benchmark cases like pgbench, hackbench etc on both one numa and four numa. Hi Yicong, Note we might need to test the case jumplabel is disabled. Thanks Barry Barry Song (4): sched: Add infrastructure to describe if cluster scheduler is really running sched: Add per_cpu cluster domain info and cpus_share_cluster API sched/fair: Scan cluster before scanning llc in wake-up path sched/fair: Use cpus_share_cluster to further pull wakee include/linux/sched/cluster.h | 19 ++++++++++++++ include/linux/sched/sd_flags.h | 9 +++++++ include/linux/sched/topology.h | 8 +++++- kernel/sched/core.c | 28 ++++++++++++++++++++ kernel/sched/fair.c | 58 +++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 3 +++ kernel/sched/topology.c | 11 ++++++++ 7 files changed, 131 insertions(+), 5 deletions(-) create mode 100644 include/linux/sched/cluster.h -- 1.8.3.1

2 years, 10 months

1
4
0 0

Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28

by Jonathan Cameron

On Tue, 22 Jun 2021 12:17:33 +0000 Jonathan Cameron via Linaro-open-discussions <linaro-open-discussions(a)op-lists.linaro.org> wrote: > On Tue, 22 Jun 2021 09:39:15 +0000 > Lorenzo Pieralisi via Linaro-open-discussions <linaro-open-discussions(a)op-lists.linaro.org> wrote: > > > On Mon, Jun 21, 2021 at 09:33:42PM +0000, Song Bao Hua (Barry Song) wrote: > > > > > > > > > > -----Original Message----- > > > > From: Linaro-open-discussions > > > > [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of > > > > Lorenzo Pieralisi via Linaro-open-discussions > > > > Sent: Monday, June 21, 2021 9:32 PM > > > > To: Jammy Zhou <jammy.zhou(a)linaro.org> > > > > Cc: Lorenzo Pieralisi via Linaro-open-discussions > > > > <linaro-open-discussions(a)op-lists.linaro.org> > > > > Subject: Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28 > > > > > > > > On Mon, Jun 21, 2021 at 12:37:03AM +0000, Jammy Zhou via Linaro-open-discussions > > > > wrote: > > > > > Hi all, > > > > > > > > > > We are only one week away from the next LOD meeting on June 28. If you > > > > > have any topic to discuss, please let me know. > > > > > > > > We could discuss the virt CPU hotplug status/forward plan. > > > > > > > > I don't know if Barry's scheduler discussion can benefit from an LOD > > > > session, please chime in if that's the case. > > > > > > Hi Lorenzo, > > > Thanks for asking. I have no important updates at this time. That > > > discussion is mainly for co-working with Tim Chen on the plan of > > > upstreaming and benchmarking, also figuring out whether the patches > > > are leading to the same result on arm and x86 platforms. > > > > > > > > > > > AOB ? > > > > Ok. So, anything else to discuss other than the PSCI KVM patches > > that are currently on ML ? Jonathan, Salil, anyone please do let > > me know. > > > > On the near term horizon is the confidential compute stuff that there > is an event about tomorrow, but next week feels a little early to try > and have a detailed discussion about that (I assume)? > > A few other things will surface by next month, but not next week ;) > > Otherwise, nothing immediate jumps out to me, beyond any useful > discussion that can be had around the PSCI KVM patches you mention. > > Thanks, > > Jonathan > Leaving that one aside, it seems the main open question from our side is around planning / progress on steps for the vCPU hotplug. Perhaps that's something best formulated over email... My current understanding is that next item would be the Guest Kernel changes? Perhaps a good target for sending out an RFC on that would be just after the merge window? Thanks, Jonathan > > > > > Thanks, > > Lorenzo >

2 years, 10 months

1
0
0 0

[RFC linux 0/5] KVM: arm64: Let userspace handle PSCI

by Jean-Philippe Brucker

As planned during the vCPU hot-add discussions from previous LOD meetings, this prototype lets userspace handle PSCI calls from a guest. The vCPU hot-add model preferred by Arm presents all possible resources through ACPI at boot time, only marking unavailable vCPUs as hidden. The VMM prevents bringing up those vCPUs by rejecting PSCI CPU_ON calls. This allows to keep things simple for vCPU scaling enablement, while leaving the door open for hardware CPU hot-add. This series focuses on moving PSCI support into userspace. Patches 1-3 allow userspace to request WFI to be executed by KVM. That way the VMM can easily implement the CPU_SUSPEND function, which is mandatory from PSCI v0.2 onwards (even if it doesn't have a more useful implementation than WFI, natively available to the guest). An alternative would be to poll the vGIC implemented in KVM for interrupts, but I haven't explored that solution. Patches 4 and 5 let the VMM request PSCI calls. The guest needs additional support to deal with hidden CPUs and to gracefully handle the "NOT_PRESENT" return value from PSCI CPU_ON. The full prototype can be found here: https://jpbrucker.net/git/linux/log/?h=cpuhp/devel https://jpbrucker.net/git/qemu/log/?h=cpuhp/devel Jean-Philippe Brucker (5): KVM: arm64: Replace power_off with mp_state in struct kvm_vcpu_arch KVM: arm64: Move WFI execution to check_vcpu_requests() KVM: arm64: Allow userspace to request WFI KVM: arm64: Pass hypercalls to userspace KVM: arm64: Pass PSCI calls to userspace Documentation/virt/kvm/api.rst | 46 +++++++++++++++---- Documentation/virt/kvm/arm/psci.rst | 1 + arch/arm64/include/asm/kvm_host.h | 10 ++++- include/kvm/arm_hypercalls.h | 1 + include/kvm/arm_psci.h | 4 ++ include/uapi/linux/kvm.h | 3 ++ arch/arm64/kvm/arm.c | 66 +++++++++++++++++++-------- arch/arm64/kvm/handle_exit.c | 3 +- arch/arm64/kvm/hypercalls.c | 28 +++++++++++- arch/arm64/kvm/psci.c | 69 ++++++++++++++--------------- 10 files changed, 165 insertions(+), 66 deletions(-) -- 2.31.1

2 years, 10 months

4
17
0 0

Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28

by Jonathan Cameron

On Tue, 22 Jun 2021 09:39:15 +0000 Lorenzo Pieralisi via Linaro-open-discussions <linaro-open-discussions(a)op-lists.linaro.org> wrote: > On Mon, Jun 21, 2021 at 09:33:42PM +0000, Song Bao Hua (Barry Song) wrote: > > > > > > > -----Original Message----- > > > From: Linaro-open-discussions > > > [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of > > > Lorenzo Pieralisi via Linaro-open-discussions > > > Sent: Monday, June 21, 2021 9:32 PM > > > To: Jammy Zhou <jammy.zhou(a)linaro.org> > > > Cc: Lorenzo Pieralisi via Linaro-open-discussions > > > <linaro-open-discussions(a)op-lists.linaro.org> > > > Subject: Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28 > > > > > > On Mon, Jun 21, 2021 at 12:37:03AM +0000, Jammy Zhou via Linaro-open-discussions > > > wrote: > > > > Hi all, > > > > > > > > We are only one week away from the next LOD meeting on June 28. If you > > > > have any topic to discuss, please let me know. > > > > > > We could discuss the virt CPU hotplug status/forward plan. > > > > > > I don't know if Barry's scheduler discussion can benefit from an LOD > > > session, please chime in if that's the case. > > > > Hi Lorenzo, > > Thanks for asking. I have no important updates at this time. That > > discussion is mainly for co-working with Tim Chen on the plan of > > upstreaming and benchmarking, also figuring out whether the patches > > are leading to the same result on arm and x86 platforms. > > > > > > > > AOB ? > > Ok. So, anything else to discuss other than the PSCI KVM patches > that are currently on ML ? Jonathan, Salil, anyone please do let > me know. > On the near term horizon is the confidential compute stuff that there is an event about tomorrow, but next week feels a little early to try and have a detailed discussion about that (I assume)? A few other things will surface by next month, but not next week ;) Otherwise, nothing immediate jumps out to me, beyond any useful discussion that can be had around the PSCI KVM patches you mention. Thanks, Jonathan > Thanks, > Lorenzo

2 years, 10 months

1
0
0 0

Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28

by Song Bao Hua (Barry Song)

> -----Original Message----- > From: Linaro-open-discussions > [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of > Lorenzo Pieralisi via Linaro-open-discussions > Sent: Monday, June 21, 2021 9:32 PM > To: Jammy Zhou <jammy.zhou(a)linaro.org> > Cc: Lorenzo Pieralisi via Linaro-open-discussions > <linaro-open-discussions(a)op-lists.linaro.org> > Subject: Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28 > > On Mon, Jun 21, 2021 at 12:37:03AM +0000, Jammy Zhou via Linaro-open-discussions > wrote: > > Hi all, > > > > We are only one week away from the next LOD meeting on June 28. If you > > have any topic to discuss, please let me know. > > We could discuss the virt CPU hotplug status/forward plan. > > I don't know if Barry's scheduler discussion can benefit from an LOD > session, please chime in if that's the case. Hi Lorenzo, Thanks for asking. I have no important updates at this time. That discussion is mainly for co-working with Tim Chen on the plan of upstreaming and benchmarking, also figuring out whether the patches are leading to the same result on arm and x86 platforms. > > AOB ? > > Lorenzo > -- > Linaro-open-discussions mailing list > https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home > https://op-lists.linaro.org/mailman/listinfo/linaro-open-discussions Thanks Barry

2 years, 10 months

2
1
0 0

Re: [Linaro-open-discussions] LOD Meeting Agenda for June 28

by Lorenzo Pieralisi

On Mon, Jun 21, 2021 at 12:37:03AM +0000, Jammy Zhou via Linaro-open-discussions wrote: > Hi all, > > We are only one week away from the next LOD meeting on June 28. If you > have any topic to discuss, please let me know. We could discuss the virt CPU hotplug status/forward plan. I don't know if Barry's scheduler discussion can benefit from an LOD session, please chime in if that's the case. AOB ? Lorenzo

2 years, 10 months

1
0
0 0

LOD Meeting Agenda for June 28

by Jammy Zhou

Hi all, We are only one week away from the next LOD meeting on June 28. If you have any topic to discuss, please let me know. Thanks, Jammy

2 years, 10 months

1
0
0 0

2024

2023

2022

2021

2020

Linaro-open-discussions