On Fri, Oct 20, 2023 at 09:51:34AM +0000, Salil Mehta wrote:

...

It has been sent, it has been reviewed, comments have been provided, and suggestions made. James has been silent since posting it.

AFAICS there was only a RFC V2 version which was sent on Wed, 13 Sep 2023 and not a non-RFC version. Sorry, if I am missing something

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-james.morse@a...

Yes, you're right, and I've just noticed I've been rebasing James' old patch set, not his new one (since he used the same tag/branch name in two different trees.) Sorting that out at the moment...

...

Consequently, I've pressed ahead with one of the changes I proposed, giving James a chance to object, but heard nothing. It is now merged into mainline kernels.

https://lore.kernel.org/linux-arm-kernel/E1qkoRr-0088Q8-Da@rmk-PC.armlinux.o...

This one - right?

No, an earlier version was merged:

https://lore.kernel.org/linux-arm-kernel/E1qgnh2-007ZRZ-WD@rmk-PC.armlinux.o...

because I was given misleading feedback on that, which meant that the non-RFC version you linked to was broken. Since the non-RFC version and the RFC v2 version were identical except for the ia64 bits, Thomas merged the RFC v2 version to avoid breaking ia64.

...

I have updated James' patch set and I'm intending to post a new patch set later today, updated with the change which has now been merged.

In the mean time, it's here:

http://git.armlinux.org.uk/cgit/linux-arm.git/log/?h=aarch64/hotplug-vcpu/v6...

Will you be posting James complete series as a 'non-RFC' version later today?

It is now way too late in the kernel cycle for this to be merged, so it doesn't matter whether it's posted as RFC or not, it won't be merged for at least another approx. three weeks.

On Fri, Oct 20, 2023 at 11:13:51AM +0100, Russell King (Oracle) wrote:

On Fri, Oct 20, 2023 at 11:10:06AM +0100, Russell King (Oracle) wrote:

...

On Fri, Oct 20, 2023 at 09:51:34AM +0000, Salil Mehta wrote:

...

...

It has been sent, it has been reviewed, comments have been provided, and suggestions made. James has been silent since posting it.

AFAICS there was only a RFC V2 version which was sent on Wed, 13 Sep 2023 and not a non-RFC version. Sorry, if I am missing something

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-james.morse@a...

Yes, you're right, and I've just noticed I've been rebasing James' old patch set, not his new one (since he used the same tag/branch name in two different trees.) Sorting that out at the moment...

...

...

Consequently, I've pressed ahead with one of the changes I proposed, giving James a chance to object, but heard nothing. It is now merged into mainline kernels.

https://lore.kernel.org/linux-arm-kernel/E1qkoRr-0088Q8-Da@rmk-PC.armlinux.o...

This one - right?

No, an earlier version was merged:

https://lore.kernel.org/linux-arm-kernel/E1qgnh2-007ZRZ-WD@rmk-PC.armlinux.o...

because I was given misleading feedback on that, which meant that the non-RFC version you linked to was broken. Since the non-RFC version and the RFC v2 version were identical except for the ia64 bits, Thomas merged the RFC v2 version to avoid breaking ia64.

... and it looks like James has also suffered the same misinformation problem, ending up dropping all the ia64 changes from his posted series that were in the earlier patch set in his other tree. So I need to do some rework of his series to put everything back together...

So I'm retracting getting another series out today... I will if I can, but I'm not guaranteeing it!

I'm afraid it isn't going to happen - going through the review comments there's some ambiguities there (at least to me) that I can't solve without input from the reviewers. I haven't even managed to get half way through the patches yet.

I've pushed out an updated version to the CGIT URL I gave earlier. Essentially, the commits that have my sign-off have been updated, those which don't have my sign-off are outstanding.

I've re-ordered some of the patches as well (due to some of the review comments) and added one of my own (for parisc) updating James' commit description in one of his later patches. I've updated the commentry below the "---" marker that James had where I've made changes.

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote:

...

I'm afraid it isn't going to happen - going through the review comments there's some ambiguities there (at least to me) that I can't solve without input from the reviewers. I haven't even managed to get half way through the patches yet.

Possible to share the link of the specific review comments and the patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

...

I've pushed out an updated version to the CGIT URL I gave earlier. Essentially, the commits that have my sign-off have been updated, those which don't have my sign-off are outstanding.

Ok.

Also, I am preparing RFC V3 for Qemu changes. Is it safe to use that repository for testing? Or should I keep on using RFC V2 patch-set for testing which James floated in September?

It would be good to test the patches even though I haven't been through all the review comments. They have been updated against v6.6-rc6 which incorporates my change, and testing against it will help to gain confidence in the updates made so far.

Thanks.

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote:

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 20, 2023 5:48 PM To: Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote:

...

...

I'm afraid it isn't going to happen - going through the review comments there's some ambiguities there (at least to me) that I can't solve without input from the reviewers. I haven't even managed to get half way through the patches yet.

Possible to share the link of the specific review comments and the patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-james.morse@a...

IICRC, you mentioned that this patch-set was ready for submission?

I did, then I realised, as I explained, that I'd used James' old patch series from his other git tree, hadn't taken his latest patch set and fixed up the review comments. So it's anything but.

BTW, which comments in specific in above link are you referring to?

All of them from the RFC v2 posting. Since James has been completely silent, as far as I'm aware, no one has done any work to address the review comments.

So I'm having to go through all the comments and update James' patches, otherwise we're going to end up with reviewers deeming the series not worth the effort of reviewing because they've been ignored.

I have no idea what is going on with James - and I haven't been asked to take on this work, but in order to move this thing forward, someone has to step up to the plate and take action... otherwise I don't see vcpu hotplug ever making it into the kernel.

One of the things that I'm trying to do as well with this patch set is to get pieces of it merged where it makes sense to do so, thereby making *some* progress towards reducing the excessive size of this series - and the size of the series is another detractor to getting review comments and eventually getting it merged.

At least we have one patch merged, and three more that are potential for merging. Admittedly I've added two patches... but it's still some forward progress rather than the series continuing to increase in size.

Also, I've had to add back some patches that James dropped between his trees - he dropped the IA64 changes, but that architecture is not going to be removed from the mainline kernel, so its necessary to update that as well. So that's added at least one extra patch to this 35 patch series.

Given that I never got any response to my suggestion concerning the first patch (which moves the arch_.*register_cpu() prototypes to linux/cpu.h rather than the piecemeal changes through James' patch) which I would have expected James to at least reply to, I've come to the conclusion that James' efforts must be directed elsewhere now and he has no time at all for VCPU hotplug. So, I believe it's now down to those who want the feature in the kernel to take up the batton and progress these patches - and that's exactly what I'm now doing.

Hi Joyce,

From: Joyce Qi joyce.qi@linaro.org Sent: Saturday, October 21, 2023 3:43 AM To: Russell King (Oracle) linux@armlinux.org.uk Cc: Salil Mehta salil.mehta@huawei.com; James Morse james.morse@arm.com; linaro-open-discussions@op-lists.linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi all，

Do we need to discuss this non-RFC patch-set of Virtual CPU Hotplug kernel support topic on next Tuesday？

Thanks for your proactive suggestion. Appreciate it.

I think, it is better to discuss this as part of the mail-chain as it is not a kind of topic which needs an audio conference. It will waste other people's time.

Plus, we need to be considerate and not drag someone, who is not able to reply to the mail-chain, into an audio conference discussion of a topic (as indicated by Russell in his reply). We don't know the reason of his pause. Maybe it is because of the time, priority, or the personal reasons i.e. annual leaves etc.

Thanks Salil.

Any other topics ？

Thanks Joyce

...

在 2023年10月21日，01:35，Russell King (Oracle) linux@armlinux.org.uk 写

道：

...

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 20, 2023 5:48 PM To: Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU

Hotplug

...

...

...

kernel support

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote:

...

...

I'm afraid it isn't going to happen - going through the review

comments

...

...

...

...

...

there's some ambiguities there (at least to me) that I can't solve without input from the reviewers. I haven't even managed to get half way through the patches yet.

Possible to share the link of the specific review comments and the patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-

james.morse@arm.com/

...

...

IICRC, you mentioned that this patch-set was ready for submission?

I did, then I realised, as I explained, that I'd used James' old patch series from his other git tree, hadn't taken his latest patch set and fixed up the review comments. So it's anything but.

...

BTW, which comments in specific in above link are you referring to?

All of them from the RFC v2 posting. Since James has been completely silent, as far as I'm aware, no one has done any work to address the review comments.

So I'm having to go through all the comments and update James' patches, otherwise we're going to end up with reviewers deeming the series not worth the effort of reviewing because they've been ignored.

I have no idea what is going on with James - and I haven't been asked to take on this work, but in order to move this thing forward, someone has to step up to the plate and take action... otherwise I don't see vcpu hotplug ever making it into the kernel.

One of the things that I'm trying to do as well with this patch set is to get pieces of it merged where it makes sense to do so, thereby making *some* progress towards reducing the excessive size of this series - and the size of the series is another detractor to getting review comments and eventually getting it merged.

At least we have one patch merged, and three more that are potential for merging. Admittedly I've added two patches... but it's still some forward progress rather than the series continuing to increase in size.

Also, I've had to add back some patches that James dropped between his trees - he dropped the IA64 changes, but that architecture is not going to be removed from the mainline kernel, so its necessary to update that as well. So that's added at least one extra patch to this 35 patch series.

Given that I never got any response to my suggestion concerning the first patch (which moves the arch_.*register_cpu() prototypes to linux/cpu.h rather than the piecemeal changes through James' patch) which I would have expected James to at least reply to, I've come to the conclusion that James' efforts must be directed elsewhere now and he has no time at all for VCPU hotplug. So, I believe it's now down to those who want the feature in the kernel to take up the batton and progress these patches - and that's exactly what I'm now doing.

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

All,

Will cancel today’s meeting since to special topic .

在 2023年10月23日，20:19，Joyce Qi joyce.qi@linaro.org 写道：

Hi Salil，

That’s ok to discuss via email.

Jonathan,James,

Do you have any topic tomorrow?

Thanks:) Joyce

...

在 2023年10月23日，17:38，Salil Mehta salil.mehta@huawei.com 写道：

Hi Joyce,

...

From: Joyce Qi joyce.qi@linaro.org Sent: Saturday, October 21, 2023 3:43 AM To: Russell King (Oracle) linux@armlinux.org.uk Cc: Salil Mehta salil.mehta@huawei.com; James Morse james.morse@arm.com; linaro-open-discussions@op-lists.linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi all，

Do we need to discuss this non-RFC patch-set of Virtual CPU Hotplug kernel support topic on next Tuesday？

Thanks for your proactive suggestion. Appreciate it.

I think, it is better to discuss this as part of the mail-chain as it is not a kind of topic which needs an audio conference. It will waste other people's time.

Plus, we need to be considerate and not drag someone, who is not able to reply to the mail-chain, into an audio conference discussion of a topic (as indicated by Russell in his reply). We don't know the reason of his pause. Maybe it is because of the time, priority, or the personal reasons i.e. annual leaves etc.

Thanks Salil.

...

Any other topics ？

Thanks Joyce

...

在 2023年10月21日，01:35，Russell King (Oracle) linux@armlinux.org.uk 写

道：

...

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 20, 2023 5:48 PM To: Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU

Hotplug

...

...

...

kernel support

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote: >> I'm afraid it isn't going to happen - going through the review

comments

...

...

...

>> there's some ambiguities there (at least to me) that I can't solve >> without input from the reviewers. I haven't even managed to get half >> way through the patches yet. > > Possible to share the link of the specific review comments and the > patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-

james.morse@arm.com/

...

...

IICRC, you mentioned that this patch-set was ready for submission?

I did, then I realised, as I explained, that I'd used James' old patch series from his other git tree, hadn't taken his latest patch set and fixed up the review comments. So it's anything but.

...

BTW, which comments in specific in above link are you referring to?

All of them from the RFC v2 posting. Since James has been completely silent, as far as I'm aware, no one has done any work to address the review comments.

So I'm having to go through all the comments and update James' patches, otherwise we're going to end up with reviewers deeming the series not worth the effort of reviewing because they've been ignored.

I have no idea what is going on with James - and I haven't been asked to take on this work, but in order to move this thing forward, someone has to step up to the plate and take action... otherwise I don't see vcpu hotplug ever making it into the kernel.

One of the things that I'm trying to do as well with this patch set is to get pieces of it merged where it makes sense to do so, thereby making *some* progress towards reducing the excessive size of this series - and the size of the series is another detractor to getting review comments and eventually getting it merged.

At least we have one patch merged, and three more that are potential for merging. Admittedly I've added two patches... but it's still some forward progress rather than the series continuing to increase in size.

Also, I've had to add back some patches that James dropped between his trees - he dropped the IA64 changes, but that architecture is not going to be removed from the mainline kernel, so its necessary to update that as well. So that's added at least one extra patch to this 35 patch series.

Given that I never got any response to my suggestion concerning the first patch (which moves the arch_.*register_cpu() prototypes to linux/cpu.h rather than the piecemeal changes through James' patch) which I would have expected James to at least reply to, I've come to the conclusion that James' efforts must be directed elsewhere now and he has no time at all for VCPU hotplug. So, I believe it's now down to those who want the feature in the kernel to take up the batton and progress these patches - and that's exactly what I'm now doing.

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Thanks Joyce

On Thu, Nov 02, 2023 at 02:30:28PM +0000, James Morse wrote:

Hi Russell,

On 20/10/2023 18:35, Russell King (Oracle) wrote:

...

So I'm having to go through all the comments and update James' patches, otherwise we're going to end up with reviewers deeming the series not worth the effort of reviewing because they've been ignored.

...

I have no idea what is going on with James - and I haven't been asked to take on this work, but in order to move this thing forward, someone has to step up to the plate and take action... otherwise I don't see vcpu hotplug ever making it into the kernel.

I have too much work going on with MPAM. This was posted as an RFC to try and get the Kubernetes and Kata-containers people to test it, and tell us that it works for them.

Without that, there is the risk we've solved the wrong problem.

I think what you're essentially saying is that on the kernel side, we are waiting for confirmation from the Kubernetes and Kata-containers people that this patch is acceptable to them.

What do we do if they don't respond?

...

Also, I've had to add back some patches that James dropped between his trees - he dropped the IA64 changes, but that architecture is not going to be removed from the mainline kernel, so its necessary to update that as well. So that's added at least one extra patch to this 35 patch series.

Fair enough - it didn't even build from rc6 to rc1 of the last cycle, and Ard's series said there was prior agreement with the few (one?) person who still cared about that thing.

It appears that ia64 has now been dropped, so that's one less architecture we have to worry about for the vCPU hotplug changes. I've already rebased my v6.6 patch set against current Linus' tip and dropped the ia64 bits from all patches. Essentially:

ACPI: Move ACPI_HOTPLUG_CPU to be disabled on arm64 * Dropped ia64 changes ia64/topology: Switch over to GENERIC_CPU_DEVICES * Dropped ACPI: Rename ACPI_HOTPLUG_CPU to include 'present' * Dropped ia64 changes ACPI: Add _OSC bits to advertise OS support for toggling CPU present/enabled * Dropped ia64 changes * Updated James' comment to remove reference to ia64

The v6.6 patches are at:

git://git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/v6.6

Other branches can be found under aarch64/hotplug-vcpu/*. Note that the kernel build bot will be building these.

I will be pushing out aarch64/hotplug-vcpu/head - which will be the latest set of patches - thus it isn't expected that pulling this will always fast-forward (it probably never will.) Also may not be tested.

Please do. If you change more than 50% of a patch, feel free to change the author too.

Thanks. I would like your input on the patches I haven't updated from your RFC v2 changes. If you'd like a list, I can send one.

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote:

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 20, 2023 5:48 PM To: Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote:

...

...

I'm afraid it isn't going to happen - going through the review comments there's some ambiguities there (at least to me) that I can't solve without input from the reviewers. I haven't even managed to get half way through the patches yet.

Possible to share the link of the specific review comments and the patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-james.morse@a...

IICRC, you mentioned that this patch-set was ready for submission?

In case you missed it, Rafael Wysocki isn't happy with various patches in this series - we have his comments on one of the patches which sound like the approach with the "enabled" bit won't work - at the very least in an arch-independent context. He also says he has other concerns, but hasn't set out what those are, and says he will do so after the merge window.

He summarises it as "and it doesn't look good" which rings alarm bells. I am hoping this doesn't mean it's back to the drawing board for this feature.

I'm also wondering why it's taken Rafael this long to state this concern - it is not like this has changed between RFC v2 and the series I posted, and Rafael did comment on the RFC v2 series. So I don't really understand why it's taken so long to bring up this concern.

Hopefully we will find out more after the merge window is over in about two weeks time.

Salil - I suspect this means that the qemu patches need to be held as well, because if the kernel side approach has to change it may impact the qemu side as well.

On Fri, Oct 27, 2023 at 03:02:25PM +0000, Salil Mehta wrote:

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 27, 2023 2:47 PM To: Salil Mehta salil.mehta@huawei.com

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 20, 2023 5:48 PM To: Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote:

...

...

I'm afraid it isn't going to happen - going through the review comments there's some ambiguities there (at least to me) that I can't solve without input from the reviewers. I haven't even managed to get half way through the patches yet.

Possible to share the link of the specific review comments and the patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-james.morse@a...

IICRC, you mentioned that this patch-set was ready for submission?

In case you missed it, Rafael Wysocki isn't happy with various patches in this series - we have his comments on one of the patches which sound like the approach with the "enabled" bit won't work - at the very least in an arch-independent context. He also says he has other concerns, but hasn't set out what those are, and says he will do so after the merge window.

Can you please share the link of the specific comments of Rafael you are worried about?

...

He summarises it as "and it doesn't look good" which rings alarm bells. I am hoping this doesn't mean it's back to the drawing board for this feature.

If there is anything other than 'enabled' and 'online-capable' bit discussion then I would suggest to discuss this as part of the mailing-list so that more eyes can be privy to it.

AFAICS, there are only 2 comments from Rafael on this patch-set. Maybe I am failing to see his point. It would be helpful if you can elaborate more.

You were copied on Rafael's comments on the series that I posted, and I was hoping that you have read them, but it seems you haven't. I won't bother answering each of your questions given that everything you're asking is covered in Rafael's comments (which I'll provide at the bottom.)

...

I'm also wondering why it's taken Rafael this long to state this concern - it is not like this has changed between RFC v2 and the series I posted, and Rafael did comment on the RFC v2 series. So I don't really understand why it's taken so long to bring up this concern.

I guess you are referring to some discussion which I am not privy of. What is that concern and where and when it has been discussed?

...

Hopefully we will find out more after the merge window is over in about two weeks time.

BTW, what it has in relation to merge window?

...

Salil - I suspect this means that the qemu patches need to be held as well, because if the kernel side approach has to change it may impact the qemu side as well.

I really do not understand all of this. Stance is vacillating and in fact has been contradictory to what you commented in the mailing-list.

Err, what (to all of the above)?

Anyway, I'm not going to argue with you.

This is Rafael's reply that covers everything that you've questioned me about above. It was also copied not only to you, but almost all the appropriate kernel mailing lists. Everything is out in the open. There is nothing being hidden. I've no idea where that paranoia is coming from.

For the overall series, Rafael said:

"I've gone through the series and there is at least one thing in it that concerns me a lot and some others that at least appear to be really questionable.

"I need more time to send comments which I'm not going to do before the 6.7 merge window (sorry), but from what I can say right now, this is not looking good."

As you can see, Rafael says, "I'm not going to do this before the 6.7 merge window" where "this" is providing further feedback on the concerns he has with the series. This answers your question above "BTW, what it has in relation to merge window?"

As you can also see, Rafael says, "this is not looking good" which at least to me as a native english speaker means that there are serious issues that make this unmergable.

The patch that makes use of the enabled bit received this reply from Rafael:

"TBH, I am expecting problems to be there.

"If something has been interpreted in a specific way for several years, then changing that interpretation is just incompatible with the entire installed base, at least potentially.

"It is not even possible to estimate the potential adverse impact of this change, as it causes a firmware-provided bit that has never been taken into account so far to become meaningful and it does so for every device in the system.

"It will be very hard to convince me that this change is a good idea."

That is quite damning. Rafael said, "very hard to convince me that this change is a good idea"... so using the enabled bit to control whether a CPU is enumerated doesn't sound like a patch that will be accepted into mainline even if all the others are.

Do these responses from Rafael sound to you like this is a patch series that is going to get imminently merged, or does it sound like the patch series needs some major rework and potentially needing a different approach?

Hi Salil,

On 30/10/2023 10:45, Salil Mehta wrote:

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 27, 2023 8:26 PM To: Salil Mehta salil.mehta@huawei.com

On Fri, Oct 27, 2023 at 03:02:25PM +0000, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 27, 2023 2:47 PM To: Salil Mehta salil.mehta@huawei.com

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 20, 2023 5:48 PM To: Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote: >> I'm afraid it isn't going to happen - going through the review comments >> there's some ambiguities there (at least to me) that I can't solve >> without input from the reviewers. I haven't even managed to get half >> way through the patches yet. > > Possible to share the link of the specific review comments and the > patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-james.morse@a...

IICRC, you mentioned that this patch-set was ready for submission?

In case you missed it, Rafael Wysocki isn't happy with various patches in this series - we have his comments on one of the patches which sound like the approach with the "enabled" bit won't work - at the very least in an arch-independent context. He also says he has other concerns, but hasn't set out what those are, and says he will do so after the merge window.

Can you please share the link of the specific comments of Rafael you are worried about?

...

He summarises it as "and it doesn't look good" which rings alarm bells. I am hoping this doesn't mean it's back to the drawing board for this feature.

If there is anything other than 'enabled' and 'online-capable' bit discussion then I would suggest to discuss this as part of the mailing-list so that more eyes can be privy to it.

AFAICS, there are only 2 comments from Rafael on this patch-set. Maybe I am failing to see his point. It would be helpful if you can elaborate more.

You were copied on Rafael's comments on the series that I posted, and I was hoping that you have read them, but it seems you haven't. I won't bother answering each of your questions given that everything you're asking is covered in Rafael's comments (which I'll provide at the bottom.)

Aah, got it. I think you are referring to RFC V3 which you have recently posted on 24th October 2023 and my earlier reply was in context to RFC V2 where Rafael commented as well but there his comments were in no way blocker to the progress of the patch-set.

https://lore.kernel.org/all/ZTffkAdOqL2pI2la@shell.armlinux.org.uk/

Apologies, I did not see that. I have been off earlier last week due to medical condition and was not able to sit much in front of laptop either. Surprisingly, patches did not land in my outlook this time (trying to figure out why...but looks like they were held by server).

...

...

...

I'm also wondering why it's taken Rafael this long to state this concern - it is not like this has changed between RFC v2 and the series I posted, and Rafael did comment on the RFC v2 series. So I don't really understand why it's taken so long to bring up this concern.

I guess you are referring to some discussion which I am not privy of. What is that concern and where and when it has been discussed?

...

Hopefully we will find out more after the merge window is over in about two weeks time.

BTW, what it has in relation to merge window?

...

Salil - I suspect this means that the qemu patches need to be held as well, because if the kernel side approach has to change it may impact the qemu side as well.

I really do not understand all of this. Stance is vacillating and in fact has been contradictory to what you commented in the mailing-list.

Err, what (to all of the above)?

Anyway, I'm not going to argue with you.

This is Rafael's reply that covers everything that you've questioned me about above. It was also copied not only to you, but almost all the appropriate kernel mailing lists. Everything is out in the open. There is nothing being hidden. I've no idea where that paranoia is coming from.

Well, I assumed you were referring to RFC V2 comments because you did not mention the clear context (and Rafael did commented in RFC V2 as well) things got mixed up in my reply.

...

For the overall series, Rafael said:

"I've gone through the series and there is at least one thing in it that concerns me a lot and some others that at least appear to be really questionable.

"I need more time to send comments which I'm not going to do before the 6.7 merge window (sorry), but from what I can say right now, this is not looking good."

As you can see, Rafael says, "I'm not going to do this before the 6.7 merge window" where "this" is providing further feedback on the concerns he has with the series. This answers your question above "BTW, what it has in relation to merge window?"

As you can also see, Rafael says, "this is not looking good" which at least to me as a native english speaker means that there are serious issues that make this unmergable. The patch that makes use of the enabled bit received this reply from Rafael:

"TBH, I am expecting problems to be there.

"If something has been interpreted in a specific way for several years, then changing that interpretation is just incompatible with the entire installed base, at least potentially.

"It is not even possible to estimate the potential adverse impact of this change, as it causes a firmware-provided bit that has never been taken into account so far to become meaningful and it does so for every device in the system.

"It will be very hard to convince me that this change is a good idea."

Thanks for pointing. Yes I can see them now.

https://lore.kernel.org/all/CAJZ5v0hhEeyDEMHnVQEiXzaKK07TSnE6GJhuTW97-XEb9Co...

https://lore.kernel.org/all/CAJZ5v0j-73_+9U3ngDAf9w1ADDhBTKctJdWboqUk-okH2TQ...

AFAICS, what Rafael is pointing is nothing new. We roughly knew these issues from last year when we introduced 'enabled' concept and were discussed as well. Last year, I had also proposed a different kernel solution to mitigate these problems (by faking not-present in the cpumask) which solved some of these but had other problems and James was uncomfortable (because it changed architecture code).

James received a hard 'never' on any interaction with the present bit and the GIC unless there is physical hardware that does this.

If we know there is _never_ going to be physical hardware that does this, we can abuse the present bit. But we don't know that.

Faking it will have consequences for the kernel and user-space once hardware exists, and we will have painted ourselves into an ABI corner.

...

That is quite damning. Rafael said, "very hard to convince me that this change is a good idea"... so using the enabled bit to control whether a CPU is enumerated doesn't sound like a patch that will be accepted into mainline even if all the others are.

Do these responses from Rafael sound to you like this is a patch series that is going to get imminently merged, or does it sound like the patch series needs some major rework and potentially needing a different approach?

Sure, they don't. Your cover letter said something similar :|

Excerpt from RFC V3 cover-letter: " I'm posting this as RFC v3 because there's still some unaddressed comments and it's clearly not ready for merging. Even if it was ready to be merged, it is too late in this development cycle to be taking this change in, so there would be little point posting it non-RFC. Also James stated that he's waiting for confirmation from the Kubernetes/Kata folk - I have no idea what the status is there."

@James, can you please update about what you are waiting for?

(the @ is for twitter right?)

Some confirmation from the people that work on those projects that this works for them. Especially around the way CPUs are visible as present (because they are), and this is visible if you stick your nose in the kernel's sysfs business.

I've not had anything in private. I've not had time to catch up with this RFC until the merge window. (and next week is some internal conference where anything could happen - and the week after is plumbers)

Thanks,

James

Hi Jianyong Wu,

You can find my version of the kernel changes which is against v6.6 at:

git://git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/v6.6

This takes account of the patch that was merged during the v6.6-rc phase. Once v6.7-rc1 is out, shortly thereafter I will be publishing a version against that taking account of further patches that were merged during this window and dropping the ia64 changes since ia64 support has now been removed from the mainline kernel.

Thanks.

On Fri, Nov 03, 2023 at 02:08:43AM +0000, Jianyong Wu wrote:

Hello James, Salil,

I'm the guy you find that work on kata containers. Sorry to have not tested for your patch set for the past long time (I'm just waiting for the patch set a little stable, maybe I miss it). I will start the test it soon. Can you point me the latest code (including Kernel and Qemu) in case that I test based on the old one.

Thanks Jianyong Wu

...

-----Original Message----- From: James Morse via Linaro-open-discussions linaro-open-discussions@op-lists.linaro.org Sent: 2023年11月2日 22:31 To: Salil Mehta salil.mehta@huawei.com; Russell King linux@armlinux.org.uk Cc: linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi Salil,

On 30/10/2023 10:45, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 27, 2023 8:26 PM To: Salil Mehta salil.mehta@huawei.com

On Fri, Oct 27, 2023 at 03:02:25PM +0000, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 27, 2023 2:47 PM To: Salil Mehta salil.mehta@huawei.com

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote: >> From: Russell King linux@armlinux.org.uk >> Sent: Friday, October 20, 2023 5:48 PM >> To: Salil Mehta salil.mehta@huawei.com >> Cc: James Morse james.morse@arm.com; >> linaro-open-discussions@op- lists.linaro.org; Joyce Qi >> joyce.qi@linaro.org; Lorenzo Pieralisi >> lorenzo.pieralisi@linaro.org; Jonathan Cameron >> jonathan.cameron@huawei.com; Shameerali Kolothum Thodi >> shameerali.kolothum.thodi@huawei.com;

karl.heubaum@oracle.com;

...

...

...

...

>> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; >> miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm >> linuxarm@huawei.com; salil.mehta@opnsrc.net >> Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU >> Hotplug kernel support >> >> On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote: >>>> I'm afraid it isn't going to happen - going through the review >>>> comments there's some ambiguities there (at least to me) that I >>>> can't solve without input from the reviewers. I haven't even >>>> managed to get half way through the patches yet. >>> >>> Possible to share the link of the specific review comments and >>> the patches you are referring to here? >> >> I'm preferring to the comments on the RFC v2 posting on the 13th >> September by James. I think you already provided a link. > > > https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-jam > es.morse@arm.com/ > > IICRC, you mentioned that this patch-set was ready for submission?

In case you missed it, Rafael Wysocki isn't happy with various patches in this series - we have his comments on one of the patches which sound like the approach with the "enabled" bit won't work - at the very least in an arch-independent context. He also says he has other concerns, but hasn't set out what those are, and says he will do so after the merge window.

Can you please share the link of the specific comments of Rafael you are worried about?

...

He summarises it as "and it doesn't look good" which rings alarm bells. I am hoping this doesn't mean it's back to the drawing board for this feature.

If there is anything other than 'enabled' and 'online-capable' bit discussion then I would suggest to discuss this as part of the mailing-list so that more eyes can be privy to it.

AFAICS, there are only 2 comments from Rafael on this patch-set. Maybe I am failing to see his point. It would be helpful if you can elaborate more.

You were copied on Rafael's comments on the series that I posted, and I was hoping that you have read them, but it seems you haven't. I won't bother answering each of your questions given that everything you're asking is covered in Rafael's comments (which I'll provide at the bottom.)

Aah, got it. I think you are referring to RFC V3 which you have recently posted on 24th October 2023 and my earlier reply was in context to RFC V2 where Rafael commented as well but there his comments were in no way blocker to the progress of the patch-set.

https://lore.kernel.org/all/ZTffkAdOqL2pI2la@shell.armlinux.org.uk/

Apologies, I did not see that. I have been off earlier last week due to medical condition and was not able to sit much in front of laptop either. Surprisingly, patches did not land in my outlook this time (trying to figure out why...but looks like they were held by server).

...

...

...

I'm also wondering why it's taken Rafael this long to state this concern - it is not like this has changed between RFC v2 and the series I posted, and Rafael did comment on the RFC v2 series. So I don't really understand why it's taken so long to bring up this concern.

I guess you are referring to some discussion which I am not privy of. What is that concern and where and when it has been discussed?

...

Hopefully we will find out more after the merge window is over in about two weeks time.

BTW, what it has in relation to merge window?

...

Salil - I suspect this means that the qemu patches need to be held as well, because if the kernel side approach has to change it may impact the qemu side as well.

I really do not understand all of this. Stance is vacillating and in fact has been contradictory to what you commented in the mailing-list.

Err, what (to all of the above)?

Anyway, I'm not going to argue with you.

This is Rafael's reply that covers everything that you've questioned me about above. It was also copied not only to you, but almost all the appropriate kernel mailing lists. Everything is out in the open. There is nothing being hidden. I've no idea where that paranoia is coming from.

Well, I assumed you were referring to RFC V2 comments because you did not mention the clear context (and Rafael did commented in RFC V2 as well) things got mixed up in my reply.

...

For the overall series, Rafael said:

"I've gone through the series and there is at least one thing in it that concerns me a lot and some others that at least appear to be really questionable.

"I need more time to send comments which I'm not going to do before the 6.7 merge window (sorry), but from what I can say right now, this is not looking good."

As you can see, Rafael says, "I'm not going to do this before the 6.7 merge window" where "this" is providing further feedback on the concerns he has with the series. This answers your question above "BTW, what it has in relation to merge window?"

As you can also see, Rafael says, "this is not looking good" which at least to me as a native english speaker means that there are serious issues that make this unmergable. The patch that makes use of the enabled bit received this reply from Rafael:

"TBH, I am expecting problems to be there.

"If something has been interpreted in a specific way for several years, then changing that interpretation is just incompatible with the entire installed base, at least potentially.

"It is not even possible to estimate the potential adverse impact of this change, as it causes a firmware-provided bit that has never been taken into account so far to become meaningful and it does so for every device in the system.

"It will be very hard to convince me that this change is a good idea."

Thanks for pointing. Yes I can see them now.

https://lore.kernel.org/all/CAJZ5v0hhEeyDEMHnVQEiXzaKK07TSnE6GJhuTW97-

...

XEb9CoSHQ@mail.gmail.com/

https://lore.kernel.org/all/CAJZ5v0j-73_+9U3ngDAf9w1ADDhBTKctJdWboqUk-

...

okH2TQGyg@mail.gmail.com/

AFAICS, what Rafael is pointing is nothing new. We roughly knew these issues from last year when we introduced 'enabled' concept and were discussed as well. Last year, I had also proposed a different kernel solution to mitigate these problems (by faking not-present in the cpumask) which solved some of these but had other problems and James was uncomfortable (because it changed architecture code).

James received a hard 'never' on any interaction with the present bit and the GIC unless there is physical hardware that does this.

If we know there is _never_ going to be physical hardware that does this, we can abuse the present bit. But we don't know that.

Faking it will have consequences for the kernel and user-space once hardware exists, and we will have painted ourselves into an ABI corner.

...

...

That is quite damning. Rafael said, "very hard to convince me that this change is a good idea"... so using the enabled bit to control whether a CPU is enumerated doesn't sound like a patch that will be accepted into mainline even if all the others are.

Do these responses from Rafael sound to you like this is a patch series that is going to get imminently merged, or does it sound like the patch series needs some major rework and potentially needing a different approach?

Sure, they don't. Your cover letter said something similar :|

Excerpt from RFC V3 cover-letter: " I'm posting this as RFC v3 because there's still some unaddressed comments and it's clearly not ready for merging. Even if it was ready to be merged, it is too late in this development cycle to be taking this change in, so there would be little point posting it non-RFC. Also James stated that he's waiting for confirmation from the Kubernetes/Kata folk - I have no idea what the status is there."

...

@James, can you please update about what you are waiting for?

(the @ is for twitter right?)

Some confirmation from the people that work on those projects that this works for them. Especially around the way CPUs are visible as present (because they are), and this is visible if you stick your nose in the kernel's sysfs business.

I've not had anything in private. I've not had time to catch up with this RFC until the merge window. (and next week is some internal conference where anything could happen - and the week after is plumbers)

Thanks,

James

Linaro-open-discussions mailing list -- linaro-open-discussions@op-lists.linaro.org https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+Home

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Jianyong Wu,

They are James' patches rebased on v6.7. Individual changelogs are in the patch descriptions where changes have been necessary in the same style that James was using. See my posting of RFC v3.

https://patchwork.kernel.org/project/linux-pm/cover/ZTffkAdOqL2pI2la@shell.a... https://lore.kernel.org/all/ZTffkAdOqL2pI2la@shell.armlinux.org.uk/

You were Cc'd on that posting, so the patches are in your mailbox...

Thanks.

On Fri, Nov 03, 2023 at 09:41:36AM +0000, Jianyong Wu wrote:

Hello Russell,

Thanks for your link. One question: what's the difference between your git tree with James'. I used to get James' code from git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git in branch of virtual_cpu_hotplug

I have just done an initial test integrating James' and Salil's patch with kata containers on Arm and it functions well. Also, I have a plan to add an integration test CI in kata community for Vcpu Hotplug on Arm. There we can do more tests integrating with k8s. To do the test, I need a stable git link of Kernel and Qemu including the latest change, thus, I can clone the code from a fixed source in each test. It's very helpful if you guys could aid.

Thanks Jianyong Wu

...

-----Original Message----- From: Russell King linux@armlinux.org.uk Sent: 2023年11月3日 16:46 To: Jianyong Wu Jianyong.Wu@arm.com Cc: James Morse James.Morse@arm.com; Salil Mehta salil.mehta@huawei.com; linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi Jianyong Wu,

You can find my version of the kernel changes which is against v6.6 at:

git://git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/v6.6

This takes account of the patch that was merged during the v6.6-rc phase. Once v6.7-rc1 is out, shortly thereafter I will be publishing a version against that taking account of further patches that were merged during this window and dropping the ia64 changes since ia64 support has now been removed from the mainline kernel.

Thanks.

On Fri, Nov 03, 2023 at 02:08:43AM +0000, Jianyong Wu wrote:

...

Hello James, Salil,

I'm the guy you find that work on kata containers. Sorry to have not tested for

your patch set for the past long time (I'm just waiting for the patch set a little stable, maybe I miss it). I will start the test it soon. Can you point me the latest code (including Kernel and Qemu) in case that I test based on the old one.

...

Thanks Jianyong Wu

...

-----Original Message----- From: James Morse via Linaro-open-discussions linaro-open-discussions@op-lists.linaro.org Sent: 2023年11月2日 22:31 To: Salil Mehta salil.mehta@huawei.com; Russell King linux@armlinux.org.uk Cc: linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi Salil,

On 30/10/2023 10:45, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 27, 2023 8:26 PM To: Salil Mehta salil.mehta@huawei.com

On Fri, Oct 27, 2023 at 03:02:25PM +0000, Salil Mehta wrote: >> From: Russell King linux@armlinux.org.uk >> Sent: Friday, October 27, 2023 2:47 PM >> To: Salil Mehta salil.mehta@huawei.com >> >> On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote: >>>> From: Russell King linux@armlinux.org.uk >>>> Sent: Friday, October 20, 2023 5:48 PM >>>> To: Salil Mehta salil.mehta@huawei.com >>>> Cc: James Morse james.morse@arm.com; >>>> linaro-open-discussions@op- lists.linaro.org; Joyce Qi >>>> joyce.qi@linaro.org; Lorenzo Pieralisi >>>> lorenzo.pieralisi@linaro.org; Jonathan Cameron >>>> jonathan.cameron@huawei.com; Shameerali Kolothum Thodi >>>> shameerali.kolothum.thodi@huawei.com;

karl.heubaum@oracle.com;

...

...

>>>> darren@os.amperecomputing.com;

ilkka@os.amperecomputing.com;

...

...

...

...

>>>> miguel.luis@oracle.com; vishnu@os.amperecomputing.com; >>>> Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net >>>> Subject: Re: [Request] Regarding non-RFC patch-set of Virtual >>>> CPU Hotplug kernel support >>>> >>>> On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote: >>>>>> I'm afraid it isn't going to happen - going through the >>>>>> review comments there's some ambiguities there (at least to >>>>>> me) that I can't solve without input from the reviewers. I >>>>>> haven't even managed to get half way through the patches yet. >>>>> >>>>> Possible to share the link of the specific review comments >>>>> and the patches you are referring to here? >>>> >>>> I'm preferring to the comments on the RFC v2 posting on the >>>> 13th September by James. I think you already provided a link. >>> >>> >>> https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1 >>> -jam >>> es.morse@arm.com/ >>> >>> IICRC, you mentioned that this patch-set was ready for submission? >> >> In case you missed it, Rafael Wysocki isn't happy with various >> patches in this series - we have his comments on one of the >> patches which sound like the approach with the "enabled" bit >> won't work - at the very least in an arch-independent context. >> He also says he has other concerns, but hasn't set out what >> those are, and says he will do so after the merge window. > > > Can you please share the link of the specific comments of Rafael > you are worried about? > > > >> He summarises it as "and it doesn't look good" which rings alarm bells. >> I am hoping this doesn't mean it's back to the drawing board >> for this feature. > > If there is anything other than 'enabled' and 'online-capable' > bit discussion then I would suggest to discuss this as part of > the mailing-list so that more eyes can be privy to it. > > AFAICS, there are only 2 comments from Rafael on this patch-set. > Maybe I am failing to see his point. It would be helpful if you > can elaborate more.

You were copied on Rafael's comments on the series that I posted, and I was hoping that you have read them, but it seems you haven't. I won't bother answering each of your questions given that everything you're asking is covered in Rafael's comments (which I'll provide at the bottom.)

Aah, got it. I think you are referring to RFC V3 which you have recently posted on 24th October 2023 and my earlier reply was in context to RFC V2 where Rafael commented as well but there his comments were in no way blocker to the progress of the patch-set.

https://lore.kernel.org/all/ZTffkAdOqL2pI2la@shell.armlinux.org.uk /

Apologies, I did not see that. I have been off earlier last week due to medical condition and was not able to sit much in front of laptop either. Surprisingly, patches did not land in my outlook this time (trying to figure out why...but looks like they were held by server).

...

>> I'm also wondering why it's taken Rafael this long to state >> this concern - it is not like this has changed between RFC v2 >> and the series I posted, and Rafael did comment on the RFC v2 >> series. So I don't really understand why it's taken so long to bring up

this concern.

...

...

...

...

> > I guess you are referring to some discussion which I am not privy of. > What is that concern and where and when it has been discussed? > > >> Hopefully we will find out more after the merge window is over >> in about two weeks time. > > BTW, what it has in relation to merge window? > > >> Salil - I suspect this means that the qemu patches need to be >> held as well, because if the kernel side approach has to change >> it may impact the qemu side as well. > > I really do not understand all of this. Stance is vacillating > and in fact has been contradictory to what you commented in the

mailing-list.

...

...

...

...

Err, what (to all of the above)?

Anyway, I'm not going to argue with you.

This is Rafael's reply that covers everything that you've questioned me about above. It was also copied not only to you, but almost all the appropriate kernel mailing lists. Everything is out in the

open.

...

...

...

...

There is nothing being hidden. I've no idea where that paranoia is coming from.

Well, I assumed you were referring to RFC V2 comments because you did not mention the clear context (and Rafael did commented in RFC V2 as well) things got mixed up in my reply.

...

For the overall series, Rafael said:

"I've gone through the series and there is at least one thing in it that concerns me a lot and some others that at least appear to be really questionable.

"I need more time to send comments which I'm not going to do before the 6.7 merge window (sorry), but from what I can say right now, this is not looking good."

As you can see, Rafael says, "I'm not going to do this before the 6.7 merge window" where "this" is providing further feedback on the concerns he has with the series. This answers your question above "BTW, what it has in relation to merge window?"

As you can also see, Rafael says, "this is not looking good" which at least to me as a native english speaker means that there are serious issues that make this unmergable. The patch that makes use of the enabled bit received this reply from Rafael:

"TBH, I am expecting problems to be there.

"If something has been interpreted in a specific way for several years, then changing that interpretation is just incompatible with the entire installed base, at least potentially.

"It is not even possible to estimate the potential adverse impact of this change, as it causes a firmware-provided bit that has never been taken into account so far to become meaningful and it does so for every device in the system.

"It will be very hard to convince me that this change is a good idea."

Thanks for pointing. Yes I can see them now.

https://lore.kernel.org/all/CAJZ5v0hhEeyDEMHnVQEiXzaKK07TSnE6GJhuTW9

...

...

7-

...

XEb9CoSHQ@mail.gmail.com/

https://lore.kernel.org/all/CAJZ5v0j-73_+9U3ngDAf9w1ADDhBTKctJdWboqU k-

...

okH2TQGyg@mail.gmail.com/

AFAICS, what Rafael is pointing is nothing new. We roughly knew these issues from last year when we introduced 'enabled' concept and were discussed as well. Last year, I had also proposed a different kernel solution to mitigate these problems (by faking not-present in the cpumask) which solved some of these but had other problems and James was uncomfortable (because it changed architecture code).

James received a hard 'never' on any interaction with the present bit and the GIC unless there is physical hardware that does this.

If we know there is _never_ going to be physical hardware that does this, we can abuse the present bit. But we don't know that.

Faking it will have consequences for the kernel and user-space once hardware exists, and we will have painted ourselves into an ABI corner.

...

...

That is quite damning. Rafael said, "very hard to convince me that this change is a good idea"... so using the enabled bit to control whether a CPU is enumerated doesn't sound like a patch that will be accepted into mainline even if all the others are.

Do these responses from Rafael sound to you like this is a patch series that is going to get imminently merged, or does it sound like the patch series needs some major rework and potentially needing a different approach?

Sure, they don't. Your cover letter said something similar :|

Excerpt from RFC V3 cover-letter: " I'm posting this as RFC v3 because there's still some unaddressed comments and it's clearly not ready for merging. Even if it was ready to be merged, it is too late in this development cycle to be taking this change in, so there would be little point posting it

non-RFC.

...

...

...

Also James stated that he's waiting for confirmation from the Kubernetes/Kata folk - I have no idea what the status is there."

...

@James, can you please update about what you are waiting for?

(the @ is for twitter right?)

Some confirmation from the people that work on those projects that this works for them. Especially around the way CPUs are visible as present (because they are), and this is visible if you stick your nose in the kernel's sysfs business.

I've not had anything in private. I've not had time to catch up with this RFC until the merge window. (and next week is some internal conference where anything could happen - and the week after is plumbers)

Thanks,

James

Linaro-open-discussions mailing list -- linaro-open-discussions@op-lists.linaro.org https://collaborate.linaro.org/display/LOD/Linaro+Open+Discussions+H ome

IMPORTANT NOTICE: The contents of this email and any attachments are

confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

On Mon, Nov 06, 2023 at 01:38:26PM +0000, Salil Mehta wrote:

Hi Jianyong,

...

From: Jianyong Wu Jianyong.Wu@arm.com Sent: Friday, November 3, 2023 9:42 AM To: Russell King linux@armlinux.org.uk; James Morse James.Morse@arm.com; Salil Mehta salil.mehta@huawei.com Cc: linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: RE: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hello Russell,

Thanks for your link. One question: what's the difference between your git tree with James'. I used to get James' code from git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git in branch of virtual_cpu_hotplug

I have just done an initial test integrating James' and Salil's patch with kata containers on Arm and it functions well. Also, I have a plan to add an integration test CI in kata community for Vcpu Hotplug on Arm. There we can do more tests integrating with k8s. To do the test, I need a stable git link of Kernel and Qemu including the latest change, thus, I can clone the code from a fixed source in each test. It's very helpful if you guys could aid.

Please use Qemu repositories mentioned in the below link https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v2

Hi Salil,

What is the earliest _host_ kernel that this version of qemu is expected to inter-operate with?

Oracle's UEK7 kernels (current) are based off 5.15-stable, and it appears that this version of qemu requires KVM_ARM_SMCCC_FILTER support which was introduced in v6.4-rc1 - which is a particularly recent kernel.

I'm guessing that vcpu hotplug requires that so it can filter the PSCI ops, but please note the dependency requirement.

Thanks.

Hi Salil,

From the discussion at https://lore.kernel.org/lkml/8634x522z2.wl-maz@kernel.org/T/, I think you can check SMCCC_FILTER in kvm host using KVM_HAS_DEVICE_ATTR and I find that you have done that in RFCv2 but don't abort once error out. WDYT?

Thanks Jianyong

-----Original Message----- From: Salil Mehta salil.mehta@huawei.com Sent: 2023年11月11日 0:40 To: Russell King linux@armlinux.org.uk Cc: Jianyong Wu Jianyong.Wu@arm.com; James Morse James.Morse@arm.com; linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: RE: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi Russell,

...

From: Russell King linux@armlinux.org.uk Sent: Friday, November 10, 2023 4:21 PM To: Salil Mehta salil.mehta@huawei.com

On Mon, Nov 06, 2023 at 01:38:26PM +0000, Salil Mehta wrote:

...

Hi Jianyong,

...

From: Jianyong Wu Jianyong.Wu@arm.com Sent: Friday, November 3, 2023 9:42 AM To: Russell King linux@armlinux.org.uk; James Morse James.Morse@arm.com; Salil Mehta salil.mehta@huawei.com Cc: linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: RE: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hello Russell,

Thanks for your link. One question: what's the difference between your git tree with James'. I used to get James' code from git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git in branch of virtual_cpu_hotplug

I have just done an initial test integrating James' and Salil's patch with kata containers on Arm and it functions well. Also, I have a plan to add an integration test CI in kata community for Vcpu Hotplug on Arm. There we can do more tests integrating with k8s. To do the test, I need a stable git link of Kernel and Qemu including the latest change, thus, I can clone the code from a fixed source in each test. It's very helpful if you guys could aid.

Please use Qemu repositories mentioned in the below link https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v2

Hi Salil,

What is the earliest _host_ kernel that this version of qemu is expected to inter-operate with?

Whatever is the kernel corresponding to the below repository of James,

[...]/morse/linux.git virtual_cpu_hotplug/rfc/v2

Earlier kernel would require changes. Before RFC V2, I have been using my own ported version of 2 of the James private branches part of repository,

[...]linux-arm/Linux-jm.git

...

Oracle's UEK7 kernels (current) are based off 5.15-stable, and it appears that this version of qemu requires KVM_ARM_SMCCC_FILTER support which was introduced in v6.4-rc1 - which is a particularly recent kernel.

Yes, all these changes. You can try using below branch I ported for my own testing (but with a caution - I mean no strings attached :|),

https://github.com/salil-mehta/linux/tree/vcpuhp-rfc-v1-jm-20230224-port-6.4... rc2-20230516

...

I'm guessing that vcpu hotplug requires that so it can filter the PSCI ops, but please note the dependency requirement.

That's correct. I ported above branch at start of May so it already Had those Oliver Upton's SMCC Filter changes part of the kernel.

Thanks Salil.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Russell,

I give a small comment in your RFCv3 patch [34/39], see https://patchwork.kernel.org/project/linux-parisc/patch/E1qvJBQ-00AqS8-8B@rm...

Thanks Jianyong

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

-----Original Message----- From: Russell King linux@armlinux.org.uk Sent: 2023年11月21日 17:45 To: Jianyong Wu Jianyong.Wu@arm.com Cc: James Morse James.Morse@arm.com; salil.mehta@huawei.com; linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Nov 03, 2023 at 09:41:36AM +0000, Jianyong Wu wrote:

...

Also, I have a plan to add an integration test CI in kata community for Vcpu

Hotplug on Arm. There we can do more tests integrating with k8s. To do the test, I need a stable git link of Kernel and Qemu including the latest change, thus, I can clone the code from a fixed source in each test. It's very helpful if you guys could aid.

I don't think I properly replied to this. I think what you want is not a "stable" git tree, but a branch that you can pull that always contains the latest code.

For my tree, that will be:

git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/head

which will be regularly kept updated and rebased against the mainline kernel. It may sometimes be between releases rather than based on -rc or final kernels.

Great! That's what I want.

Does that satisfy your need?

Yes. Thanks Russell. It saves me lots of work.

Thanks Jianyong

Thanks.

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Joyce, I have do not have any.

Thanks Salil From: Joyce Qi joyce.qi@linaro.org Sent: Sunday, November 26, 2023 11:51 AM To: Jianyong Wu Jianyong.Wu@arm.com Cc: Russell King linux@armlinux.org.uk; James Morse James.Morse@arm.com; Salil Mehta salil.mehta@huawei.com; linaro-open-discussions@op-lists.linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net; Jonathan Cameron jonathan.cameron@huawei.com Subject: Re: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi all，

Do we have topic to sync next Tuesday？

Thanks Joyce

在 2023年11月22日，09:22，Jianyong Wu <Jianyong.Wu@arm.commailto:Jianyong.Wu@arm.com> 写道：

-----Original Message----- From: Russell King <linux@armlinux.org.ukmailto:linux@armlinux.org.uk> Sent: 2023年11月21日 17:45 To: Jianyong Wu <Jianyong.Wu@arm.commailto:Jianyong.Wu@arm.com> Cc: James Morse <James.Morse@arm.commailto:James.Morse@arm.com>; salil.mehta@huawei.commailto:salil.mehta@huawei.com; linaro-open-discussions@op-lists.linaro.orgmailto:linaro-open-discussions@op-lists.linaro.org; Joyce Qi <joyce.qi@linaro.orgmailto:joyce.qi@linaro.org>; Lorenzo Pieralisi <lorenzo.pieralisi@linaro.orgmailto:lorenzo.pieralisi@linaro.org>; karl.heubaum@oracle.commailto:karl.heubaum@oracle.com; darren@os.amperecomputing.commailto:darren@os.amperecomputing.com; ilkka@os.amperecomputing.commailto:ilkka@os.amperecomputing.com; miguel.luis@oracle.commailto:miguel.luis@oracle.com; vishnu@os.amperecomputing.commailto:vishnu@os.amperecomputing.com; Linuxarm <linuxarm@huawei.commailto:linuxarm@huawei.com>; salil.mehta@opnsrc.netmailto:salil.mehta@opnsrc.net Subject: Re: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Nov 03, 2023 at 09:41:36AM +0000, Jianyong Wu wrote:

Also, I have a plan to add an integration test CI in kata community for Vcpu Hotplug on Arm. There we can do more tests integrating with k8s. To do the test, I need a stable git link of Kernel and Qemu including the latest change, thus, I can clone the code from a fixed source in each test. It's very helpful if you guys could aid.

I don't think I properly replied to this. I think what you want is not a "stable" git tree, but a branch that you can pull that always contains the latest code.

For my tree, that will be:

git.armlinux.org.uk/~rmk/linux-arm.githttp://git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/head

which will be regularly kept updated and rebased against the mainline kernel. It may sometimes be between releases rather than based on -rc or final kernels.

Great! That's what I want.

Does that satisfy your need? Yes. Thanks Russell. It saves me lots of work.

Thanks Jianyong

Thanks.

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last! IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Jianyong, A request, is it possible to share the results of the testing through mailing list for the wider audience?

Many thanks Salil.

From: Jianyong Wu Jianyong.Wu@arm.com Sent: Wednesday, November 22, 2023 1:22 AM To: Russell King linux@armlinux.org.uk Cc: James Morse James.Morse@arm.com; Salil Mehta salil.mehta@huawei.com; linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: RE: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

...

From: Russell King linux@armlinux.org.uk Sent: 2023年11月21日 17:45 To: Jianyong Wu Jianyong.Wu@arm.com Cc: James Morse James.Morse@arm.com; salil.mehta@huawei.com; linaro-open-discussions@op-lists.linaro.org; Joyce Qi

joyce.qi@linaro.org;

...

Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Nov 03, 2023 at 09:41:36AM +0000, Jianyong Wu wrote:

...

Also, I have a plan to add an integration test CI in kata community for Vcpu

Hotplug on Arm. There we can do more tests integrating with k8s. To do the test, I need a stable git link of Kernel and Qemu including the latest change, thus, I can clone the code from a fixed source in each test. It's very helpful if you guys could aid.

I don't think I properly replied to this. I think what you want is not a "stable" git tree, but a branch that you can pull that always contains the latest code.

For my tree, that will be:

git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/head

which will be regularly kept updated and rebased against the mainline kernel. It may sometimes be between releases rather than based on -rc or final kernels.

Great! That's what I want.

...

Does that satisfy your need?

Yes. Thanks Russell. It saves me lots of work.

Thanks Jianyong

...

Thanks.

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Salil,

Overall, I have done the k8s test. The full test log locates http://jenkins.katacontainers.io/job/kata-containers-tests-experimental-ARM-... The log is so large. You need scroll down to the bottom of it to find the log like:

[Log begin:] 17:18:13 1..1 17:18:18 ok 1 Check CPU constraints 17:18:18 1..1 17:20:21 1..1 17:20:36 ok 1 Check number of cpus 17:20:36 1..1 [log end]

These 2 are the k8s test result related with vCPU Hotplug. You can see that both are PASSED. You can refer to https://github.com/kata-containers/tests/blob/main/integration/kubernetes/k8... and https://github.com/kata-containers/tests/blob/main/integration/kubernetes/k8... to find the detail of the test.

kernel branch in this test: git://git.armlinux.org.uk/~rmk/linux-arm.git branch: origin/aarch64/hotplug-vcpu/v6.6-rc7 and VMM use Salil's v2 patch.

Thanks Jianyong

-----Original Message----- From: Russell King linux@armlinux.org.uk Sent: 2023年11月27日 18:21 To: Salil Mehta salil.mehta@huawei.com Cc: Jianyong Wu Jianyong.Wu@arm.com; James Morse James.Morse@arm.com; linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Mon, Nov 27, 2023 at 10:17:06AM +0000, Salil Mehta wrote:

...

Hi Jianyong, A request, is it possible to share the results of the testing through mailing list for the wider audience?

... or any audience? (I haven't had any results yet.)

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Salil,

I've shared the test results of k8s + kata containers in LKML of Russell's v3 patch set. Hope it helps.

Thanks Jianyong

-----Original Message----- From: Salil Mehta salil.mehta@huawei.com Sent: 2023年12月5日 18:34 To: Jianyong Wu Jianyong.Wu@arm.com; Russell King linux@armlinux.org.uk Cc: James Morse James.Morse@arm.com; linaro-open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: RE: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi Jianyong, Many thanks for taking pains and sharing the results of your testing in this forum and which look quite promising. I would encourage you to share the same to the wider audiences in the mailing-list in reply to some Linux kernel patches.

As per the latest update Greg-Kroah-Hartman has agreed to pull some of the Russell's patches related to VCPU cleanups which is great.

https://lore.kernel.org/lkml/2023120131-leotard-deprecate-4e27@gregkh/

These results will increase confidence on the remaining patches although the key issue of the ACPI still remains unresolved and Russell has already asked that question again from Rafael yesterday

https://lore.kernel.org/lkml/ZW4ZBkj2oCmxv55T@shell.armlinux.org.uk/

Thanks Salil

...

From: Jianyong Wu Jianyong.Wu@arm.com Sent: Tuesday, December 5, 2023 3:48 AM To: Russell King linux@armlinux.org.uk; Salil Mehta salil.mehta@huawei.com Cc: James Morse James.Morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: RE: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

Hi Salil,

Overall, I have done the k8s test. The full test log locates http://jenkins.katacontainers.io/job/kata-containers-tests-experimenta l- ARM-ubuntu/18/console The log is so large. You need scroll down to the bottom of it to find the log like:

[Log begin:] 17:18:13 1..1 17:18:18 ok 1 Check CPU constraints 17:18:18 1..1 17:20:21 1..1 17:20:36 ok 1 Check number of cpus 17:20:36 1..1 [log end]

These 2 are the k8s test result related with vCPU Hotplug. You can see that both are PASSED. You can refer to https://github.com/kata- containers/tests/blob/main/integration/kubernetes/k8s-cpu-ns.bats and https://github.com/kata- containers/tests/blob/main/integration/kubernetes/k8s-number-cpus.bats to find the detail of the test.

kernel branch in this test: git://git.armlinux.org.uk/~rmk/linux-arm.git branch: origin/aarch64/hotplug-vcpu/v6.6-rc7 and VMM use Salil's v2 patch.

Thanks Jianyong

...

-----Original Message----- From: Russell King linux@armlinux.org.uk Sent: 2023年11月27日 18:21 To: Salil Mehta salil.mehta@huawei.com Cc: Jianyong Wu Jianyong.Wu@arm.com; James Morse James.Morse@arm.com; linaro-open-discussions@op-lists.linaro.org; Joyce

Qi

...

joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Linaro-open-discussions] Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Mon, Nov 27, 2023 at 10:17:06AM +0000, Salil Mehta wrote:

...

Hi Jianyong, A request, is it possible to share the results of the testing through mailing list for the wider audience?

... or any audience? (I haven't had any results yet.)

-- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Jianyong,

From: Jianyong Wu Jianyong.Wu@arm.com Sent: Friday, November 3, 2023 2:09 AM To: James Morse James.Morse@arm.com; Salil Mehta salil.mehta@huawei.com

Hello James, Salil,

I'm the guy you find that work on kata containers. Sorry to have not tested for your patch set for the past long time (I'm just waiting for the patch set a little stable, maybe I miss it). I will start the test it soon. Can you point me the latest code (including Kernel and Qemu) in case that I test based on the old one.

Please use the repositories mentioned in the RFC V2 along with the RFC V2/V3 of the kernel part.

Thanks Salil.

Hi Russell,

On 27/10/2023 14:46, Russell King (Oracle) wrote:

On Fri, Oct 20, 2023 at 04:56:20PM +0000, Salil Mehta wrote:

...

...

From: Russell King linux@armlinux.org.uk Sent: Friday, October 20, 2023 5:48 PM To: Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; linaro-open-discussions@op- lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Shameerali Kolothum Thodi shameerali.kolothum.thodi@huawei.com; karl.heubaum@oracle.com; darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; miguel.luis@oracle.com; vishnu@os.amperecomputing.com; Linuxarm linuxarm@huawei.com; salil.mehta@opnsrc.net Subject: Re: [Request] Regarding non-RFC patch-set of Virtual CPU Hotplug kernel support

On Fri, Oct 20, 2023 at 04:40:59PM +0000, Salil Mehta wrote:

...

...

I'm afraid it isn't going to happen - going through the review comments there's some ambiguities there (at least to me) that I can't solve without input from the reviewers. I haven't even managed to get half way through the patches yet.

Possible to share the link of the specific review comments and the patches you are referring to here?

I'm preferring to the comments on the RFC v2 posting on the 13th September by James. I think you already provided a link.

https://lore.kernel.org/linux-arm-kernel/20230913163823.7880-1-james.morse@a...

IICRC, you mentioned that this patch-set was ready for submission?

In case you missed it, Rafael Wysocki isn't happy with various patches in this series - we have his comments on one of the patches which sound like the approach with the "enabled" bit won't work - at the very least in an arch-independent context. He also says he has other concerns, but hasn't set out what those are, and says he will do so after the merge window.

He summarises it as "and it doesn't look good" which rings alarm bells. I am hoping this doesn't mean it's back to the drawing board for this feature.

The angle of attack on this is the enabled bit has always existed, we didn't invent anything new here. Linux has historically ignored it. Its fine to say "its always junk on x86" if we don't trust vendors to get this right.

I'd much prefer we quirk the devices where this is discovered to be junk, instead of declaring outright that we ignore this bit of the spec.

I'm also wondering why it's taken Rafael this long to state this concern - it is not like this has changed between RFC v2 and the series I posted, and Rafael did comment on the RFC v2 series. So I don't really understand why it's taken so long to bring up this concern.

Hopefully we will find out more after the merge window is over in about two weeks time.

Salil - I suspect this means that the qemu patches need to be held as well, because if the kernel side approach has to change it may impact the qemu side as well.

Hi James,

From: James Morse james.morse@arm.com Sent: Thursday, November 2, 2023 2:30 PM To: Salil Mehta salil.mehta@huawei.com; linaro-open-discussions@op-lists.linaro.org

Hi Salil,

On 19/10/2023 12:49, Salil Mehta wrote:

...

Do we have any plan when we want to push the non-RFC Virtual CPU Hotplug

patch-set

...

for the kernel side to the community. I guess people will be more

interested in reviewing

...

if it is floated with a non-RFC tag.

Have we heard from the kubernetes or kata-containers folk that this works for them? ... and that the bitmask visibility problem isn't an issue?

I am not sure. Who are those people and can we include them in the list?

...

If time is a problem at your side then do you need my help in carrying it

forward for you?

I don't think we should consider merging this at all until we know from the intended users that it works for them.

Agreed. But to be able to chase them we should know whom are we waiting for. If you can list down the concerned people then perhaps I can help.

Thanks Salil.