Hello Everyone, I am seeking assistance with a performance regression observed on an i.MX6SOLO rev 1.4 (996 MHz) board in our LAVA farm. The Issue: A test suite containing 90 test cases, which previously took 10 minutes to complete, is now consistently taking 20 minutes. This issue is unique to this one specific board; other boards of the same type are still completing the run in 10 minutes using the same build/firmware. Observations:
- I'm not using any test_character_delay externally in my YAML. - Identical Logs: The "Fast" and "Slow" jobs produce identical serial output. - Recent Hangs: The test run has recently started to get stuck/hang mid-run, even though CPU usage is not at 100%.
Debugging Performed:
- Verified that CPU thermal throttling is not active. - Network latency and DNS resolution times appear normal. - No hardware errors are reported in dmesg. - Swapped power supplies and network cables with no improvement.
Questions:
1. What is the best way to debug "dead time" between lava-test-case signals? 2. Could the LAVA worker be experiencing serial buffer delays on this specific node that don't appear in the logs?
Thank you for your time.
Best Regards Pavan Kumar
Hi Pawan,
On Fri, Jan 16, 2026 at 9:45 AM pawanKumar pawan4749@gmail.com wrote:
Hello Everyone, I am seeking assistance with a performance regression observed on an i.MX6SOLO rev 1.4 (996 MHz) board in our LAVA farm. The Issue: A test suite containing 90 test cases, which previously took 10 minutes to complete, is now consistently taking 20 minutes. This issue is unique to this one specific board; other boards of the same type are still completing the run in 10 minutes using the same build/firmware. Observations:
I'm not using any test_character_delay externally in my YAML. Identical Logs: The "Fast" and "Slow" jobs produce identical serial output. Recent Hangs: The test run has recently started to get stuck/hang mid-run, even though CPU usage is not at 100%.
Debugging Performed:
Verified that CPU thermal throttling is not active. Network latency and DNS resolution times appear normal. No hardware errors are reported in dmesg. Swapped power supplies and network cables with no improvement.
Questions:
What is the best way to debug "dead time" between lava-test-case signals?
I'm not sure what you mean by that, but you could try dowloading raw log. It contains timestamps for each line in the log.
Could the LAVA worker be experiencing serial buffer delays on this specific node that don't appear in the logs?
Very unlikely. Did you try to run the same test outside of LAVA (manually)?
Best Regards, Milosz
lava-users@lists.lavasoftware.org