Hello, We have been trying to verify the "system suspend/Restore" with vCPU Hotplug patches recently and found this functionality does not work on ARM64 with VMs even without our patches i.e. using latest kernel and qemu repository.
estuary:/$ cat /sys/power/mem_sleep
[s2idle]
estuary:/$
estuary:/$ cat /sys/power/state
freeze mem disk
estuary:/$
estuary:/$
estuary:/$ echo mem > /sys/power/state
[ 60.458445] PM: suspend entry (s2idle)
[ 60.458840] Filesystems sync: 0.000 seconds
[ 60.459649] Freezing user space processes
[ 60.461149] Freezing user space processes completed (elapsed 0.001 seconds)
[ 60.461830] OOM killer disabled.
[ 60.462144] Freezing remaining freezable tasks
[ 60.463188] Freezing remaining freezable tasks completed (elapsed 0.000 seconds)
[ 60.463920] printk: Suspending console(s) (use no_console_suspend to debug)
(qemu)
(qemu) sys
system_powerdown system_reset system_wakeup
(qemu) system_wakeup
Error: wake-up from suspend is not supported by this guest
(qemu)
Or using # systemctl suspend
What is the expected behavior or are we missing something?
Thanks Salil
On Thu, Jul 27, 2023 at 10:06:58AM +0000, Salil Mehta wrote:
Hello, We have been trying to verify the "system suspend/Restore" with vCPU Hotplug patches recently and found this functionality does not work on ARM64 with VMs even without our patches i.e. using latest kernel and qemu repository.
estuary:/$ cat /sys/power/mem_sleep [s2idle] estuary:/$ estuary:/$ cat /sys/power/state freeze mem disk estuary:/$ estuary:/$ estuary:/$ echo mem > /sys/power/state [ 60.458445] PM: suspend entry (s2idle) [ 60.458840] Filesystems sync: 0.000 seconds [ 60.459649] Freezing user space processes [ 60.461149] Freezing user space processes completed (elapsed 0.001 seconds) [ 60.461830] OOM killer disabled. [ 60.462144] Freezing remaining freezable tasks [ 60.463188] Freezing remaining freezable tasks completed (elapsed 0.000 seconds) [ 60.463920] printk: Suspending console(s) (use no_console_suspend to debug) (qemu) (qemu) sys system_powerdown system_reset system_wakeup (qemu) system_wakeup Error: wake-up from suspend is not supported by this guest (qemu)
Or using # systemctl suspend
What is the expected behavior or are we missing something?
What is the wakeup source in this experiment ? Is it enabled ? You must have done that before executing "echo mem > /sys/power/state". It is tested to generate the interrupt ?
Typically I use RTC on systems as I need not physical access to the platform to test that. $echo +20 > /sys/class/rtc/rtc0/wakealarm $ echo mem > /sys/power/state
Not sure if RTC is available on your system/VM. You just need to find a working wakeup source for your experiment.
Sorry if you have already tried that and seeing issues. You may have just missed to list it here. Thought I will check. The timer are also disabled during any system level sleeps and hence can't wake up without a valid working wakeup source unlike the normal CPU sleep/idle states.
Also make sure all the drivers have working suspend resume routines, sometimes the driver resume fails and if the console is not yet resumed at that point, you get no logs.
-- Regards, Sudeep
Hi Sudeep, Thanks for looking into it.
From: Sudeep Holla sudeep.holla@arm.com Sent: Thursday, July 27, 2023 11:34 AM
On Thu, Jul 27, 2023 at 10:06:58AM +0000, Salil Mehta wrote:
Hello, We have been trying to verify the "system suspend/Restore" with vCPU Hotplug patches recently and found this functionality does not work on ARM64 with VMs even without our patches i.e. using latest kernel and qemu repository.
estuary:/$ cat /sys/power/mem_sleep [s2idle] estuary:/$ estuary:/$ cat /sys/power/state freeze mem disk estuary:/$ estuary:/$ estuary:/$ echo mem > /sys/power/state [ 60.458445] PM: suspend entry (s2idle) [ 60.458840] Filesystems sync: 0.000 seconds [ 60.459649] Freezing user space processes [ 60.461149] Freezing user space processes completed (elapsed 0.001 seconds) [ 60.461830] OOM killer disabled. [ 60.462144] Freezing remaining freezable tasks [ 60.463188] Freezing remaining freezable tasks completed (elapsed 0.000 seconds) [ 60.463920] printk: Suspending console(s) (use no_console_suspend to debug) (qemu) (qemu) sys system_powerdown system_reset system_wakeup (qemu) system_wakeup Error: wake-up from suspend is not supported by this guest (qemu)
Or using # systemctl suspend
What is the expected behavior or are we missing something?
What is the wakeup source in this experiment ? Is it enabled ? You must have done that before executing "echo mem > /sys/power/state". It is tested to generate the interrupt ?
Hmm. I would have thought the keyboard etc. should contribute to those interrupts by default (isn't it?) or do we need to explicitly configure the devices for their interrupts to be use in wakeup from s2idle?
Typically I use RTC on systems as I need not physical access to the platform to test that. $echo +20 > /sys/class/rtc/rtc0/wakealarm $ echo mem > /sys/power/state
This might not work as after putting the VM into s2idle state VM kind of hangs and becomes unresponsive. I would assume that keyboard interrupts are not being considered for waking-up the VM here either.
I can see RTC is configured by Qemu/VMM during init.
Not sure if RTC is available on your system/VM. You just need to find a working wakeup source for your experiment.
Sorry if you have already tried that and seeing issues. You may have just missed to list it here. Thought I will check. The timer are also disabled during any system level sleeps and hence can't wake up without a valid working wakeup source unlike the normal CPU sleep/idle states.
So something like "system_wakeup" from qemu prompt seems to be the way but It looks this is not enabled in Qemu due to missing ACPI handling. Need to dig this further.
Also make sure all the drivers have working suspend resume routines, sometimes the driver resume fails and if the console is not yet resumed at that point, you get no logs.
Good point. This could be a possibility.
Thanks Salil.
On Thu, Jul 27, 2023 at 04:15:46PM +0100, Salil Mehta wrote:
Typically I use RTC on systems as I need not physical access to the platform to test that. $echo +20 > /sys/class/rtc/rtc0/wakealarm $ echo mem > /sys/power/state
This might not work as after putting the VM into s2idle state VM kind of hangs and becomes unresponsive.
That's strange, I tried Sudeep's command on QEMU and Linux master and it does wake up the system after 20 seconds (with /sys/power/mem_sleep as "[s2idle]"). I was curious because last time I needed to test suspend, I had to update the DT to set the console interrupt as a wakeup source. This RTC method is a lot better.
Thanks, Jean
On Thu, Jul 27, 2023 at 03:15:46PM +0000, Salil Mehta wrote:
Hmm. I would have thought the keyboard etc. should contribute to those interrupts by default (isn't it?) or do we need to explicitly configure the devices for their interrupts to be use in wakeup from s2idle?
Hmm, when you say keyboard here, do you mean the serial console or does QEMU emulate a keyboard device ? If latter, then yes but better to check if the device is configured as wakeup source, the keyboard/pad DT node must have the "wakeup-source" property. If former, try if no_console_suspend in the cmdline(or something similar) helps. Again need to check if it is enabled as wakeup source.(HINT: check /sys/class/wakeup/wakeup*/device/power/wakeup)
Typically I use RTC on systems as I need not physical access to the platform to test that. $echo +20 > /sys/class/rtc/rtc0/wakealarm $ echo mem > /sys/power/state
This might not work as after putting the VM into s2idle state VM kind of hangs and becomes unresponsive. I would assume that keyboard interrupts are not being considered for waking-up the VM here either.
What are the keyboards you are referring here ? Just serial port input or emulated keyboards ?
I can see RTC is configured by Qemu/VMM during init.
Good, Jean-Philippe confirms atleast RTC is working in QEMU as wakeup source.
Not sure if RTC is available on your system/VM. You just need to find a working wakeup source for your experiment.
Sorry if you have already tried that and seeing issues. You may have just missed to list it here. Thought I will check. The timer are also disabled during any system level sleeps and hence can't wake up without a valid working wakeup source unlike the normal CPU sleep/idle states.
So something like "system_wakeup" from qemu prompt seems to be the way but It looks this is not enabled in Qemu due to missing ACPI handling. Need to dig this further.
Yes ACPI is a different story. Not sure if you have RTC driver support for ACPI, need to check.
Also make sure all the drivers have working suspend resume routines, sometimes the driver resume fails and if the console is not yet resumed at that point, you get no logs.
Good point. This could be a possibility.
Let me know if you need any help.
Hi Sudeep,
I had to pick on this thread since it is the subject I would like to address.
On 28/07/2023 12:48, Sudeep Holla wrote:
On Thu, Jul 27, 2023 at 03:15:46PM +0000, Salil Mehta wrote:
Hmm. I would have thought the keyboard etc. should contribute to those interrupts by default (isn't it?) or do we need to explicitly configure the devices for their interrupts to be use in wakeup from s2idle?
Hmm, when you say keyboard here, do you mean the serial console or does QEMU emulate a keyboard device ? If latter, then yes but better to check if the device is configured as wakeup source, the keyboard/pad DT node must have the "wakeup-source" property. If former, try if no_console_suspend in the cmdline(or something similar) helps. Again need to check if it is enabled as wakeup source.(HINT: check /sys/class/wakeup/wakeup*/device/power/wakeup)
Typically I use RTC on systems as I need not physical access to the platform to test that. $echo +20 > /sys/class/rtc/rtc0/wakealarm $ echo mem > /sys/power/state
This might not work as after putting the VM into s2idle state VM kind of hangs and becomes unresponsive. I would assume that keyboard interrupts are not being considered for waking-up the VM here either.
What are the keyboards you are referring here ? Just serial port input or emulated keyboards ?
I can see RTC is configured by Qemu/VMM during init.
Good, Jean-Philippe confirms atleast RTC is working in QEMU as wakeup source.
Not sure if RTC is available on your system/VM. You just need to find a working wakeup source for your experiment.
Sorry if you have already tried that and seeing issues. You may have just missed to list it here. Thought I will check. The timer are also disabled during any system level sleeps and hence can't wake up without a valid working wakeup source unlike the normal CPU sleep/idle states.
So something like "system_wakeup" from qemu prompt seems to be the way but It looks this is not enabled in Qemu due to missing ACPI handling. Need to dig this further.
Yes ACPI is a different story. Not sure if you have RTC driver support for ACPI, need to check.
Also make sure all the drivers have working suspend resume routines, sometimes the driver resume fails and if the console is not yet resumed at that point, you get no logs.
Good point. This could be a possibility.
Let me know if you need any help.
I have the following wakeup sources available on the guest:
PNP0C0E:00 PNP0C0C:00 rtc-efi.0 alarmtimer.0.auto
Being able to successfully put the guest to sleep by the interaction of PNP0C0E although the guest couldn't wakeup through the mean of rtc.
That seems related to the timestamp echoed into /sys/class/rtc/rtc0/wakealarm. If it is more than the curret "date '+%s'" a message of write status of 3 and write error is thrown.
I was thinking the wakealarm value would be the timestamp in the future for wakeup. Did I took a wrong assumption?
Any help/direction would be much appreciated.
Thanks in advance.
Regards, Miguel
From: Miguel Luis miguel.luis@oracle.com Sent: Tuesday, November 7, 2023 12:24 PM To: Sudeep Holla sudeep.holla@arm.com; Salil Mehta salil.mehta@huawei.com Cc: James Morse james.morse@arm.com; peter.maydell@linaro.org; linaro- open-discussions@op-lists.linaro.org; Joyce Qi joyce.qi@linaro.org; Jonathan Cameron jonathan.cameron@huawei.com; Lorenzo Pieralisi lorenzo.pieralisi@linaro.org; Jean-Philippe Brucker <jean- philippe@linaro.org>; Jean-Philippe Brucker <jean- philippe.brucker@arm.com>; Vishnu Pajjuri OS vishnu@os.amperecomputing.com; Salil Mehta salil.mehta@opnsrc.net Subject: Re: [Question] System suspend/resume functionality on ARM64 VM does seem to be working?
Hi Sudeep,
I had to pick on this thread since it is the subject I would like to address.
On 28/07/2023 12:48, Sudeep Holla wrote:
On Thu, Jul 27, 2023 at 03:15:46PM +0000, Salil Mehta wrote:
Hmm. I would have thought the keyboard etc. should contribute to those interrupts by default (isn't it?) or do we need to explicitly configure the devices for their interrupts to be use in wakeup from s2idle?
Hmm, when you say keyboard here, do you mean the serial console or does QEMU emulate a keyboard device ? If latter, then yes but better to check if the device is configured as wakeup source, the keyboard/pad DT node must have the "wakeup-source" property. If former, try if no_console_suspend in the cmdline(or something similar) helps. Again need to check if it is enabled as wakeup source.(HINT: check /sys/class/wakeup/wakeup*/device/power/wakeup)
Typically I use RTC on systems as I need not physical access to the platform to test that. $echo +20 > /sys/class/rtc/rtc0/wakealarm $ echo mem > /sys/power/state
This might not work as after putting the VM into s2idle state VM kind of hangs and becomes unresponsive. I would assume that keyboard interrupts are not being considered for waking-up the VM here either.
What are the keyboards you are referring here ? Just serial port input or emulated keyboards ?
I can see RTC is configured by Qemu/VMM during init.
Good, Jean-Philippe confirms atleast RTC is working in QEMU as wakeup source.
Not sure if RTC is available on your system/VM. You just need to find a working wakeup source for your experiment.
Sorry if you have already tried that and seeing issues. You may have just missed to list it here. Thought I will check. The timer are also disabled during any system level sleeps and hence can't wake up without a valid working wakeup source unlike the normal CPU sleep/idle states.
So something like "system_wakeup" from qemu prompt seems to be the way but It looks this is not enabled in Qemu due to missing ACPI handling. Need to dig this further.
Yes ACPI is a different story. Not sure if you have RTC driver support for ACPI, need to check.
Also make sure all the drivers have working suspend resume routines, sometimes the driver resume fails and if the console is not yet resumed at that point, you get no logs.
Good point. This could be a possibility.
Let me know if you need any help.
I have the following wakeup sources available on the guest:
PNP0C0E:00 PNP0C0C:00 rtc-efi.0
This is what is used. But UEFI/QemuVirt Package does not have its corresponding ends implemented for the wakeupalarm. We need a change there. I did verify this earlier.
alarmtimer.0.auto
Being able to successfully put the guest to sleep by the interaction of PNP0C0E although the guest couldn't wakeup through the mean of rtc.
That seems related to the timestamp echoed into /sys/class/rtc/rtc0/wakealarm. If it is more than the curret "date '+%s'" a message of write status of 3 and write error is thrown.
I was thinking the wakealarm value would be the timestamp in the future for wakeup. Did I took a wrong assumption?
Any help/direction would be much appreciated.
Thanks in advance.
Regards, Miguel
On Tue, Nov 07, 2023 at 12:29:30PM +0000, Salil Mehta wrote:
From: Miguel Luis miguel.luis@oracle.com
I have the following wakeup sources available on the guest:
PNP0C0E:00 PNP0C0C:00 rtc-efi.0
This is what is used. But UEFI/QemuVirt Package does not have its corresponding ends implemented for the wakeupalarm. We need a change there. I did verify this earlier.
Correct, AFAIK the EFI RTC doesn't support wakeup alarm feature.
-- Regards, Sudeep
linaro-open-discussions@op-lists.linaro.org
-
Jean-Philippe Brucker -
Miguel Luis -
Salil Mehta -
Sudeep Holla