- lava-users - op-lists.linaro.org

Issue: Log processor bottleneck and deploy-flasher timeouts on high-frequency output (LAVA 2026.02 )

by pawanKumar

Dear LAVA Community, I am reaching out to report a regression (or significant change in behavior) regarding how the LAVA dispatcher handles high-frequency terminal output and escape sequences during flashing operations. Background: After upgrading from LAVA 2022.06 to a recent version LAVA 2026.02 , our flashing jobs—which utilize mfgtoolcli (NXP) to write images to i.MX6/8 devices—have begun failing consistently with a deploy-flasher timeout. The Issue: The flashing tool outputs a high-frequency progress bar using carriage returns (\r) and ANSI escape codes (e.g., \e[1F, \e[2K). 1. Log Bloat: In the older version, these were handled gracefully. In the current version, every escape sequence is captured as a new log entry, resulting in logs exceeding 5MB for a single flash. 2. Dispatcher Lag: The dispatcher appears to bottleneck while processing this flood of data. This "processing lag" causes the internal action timer to hit the default 500s limit, even if the physical flashing process succeeds locally on the worker. 3. Timeout Overrides: We’ve observed that deploy-flasher often ignores the timeout values specified in the Job YAML, defaulting to 500 seconds unless explicitly overridden in the Device Dictionary. Error Examples: finish programming rootfs-a>> [1F [2K 1 100% [============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================= ... continues >] Donedeploy-flasher timed out after 500 seconds>> uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.243-0-ged48c51>> >> Wait for Known USB Device Appear... [?25l [1F [1F [1F [1F [1F [1F1:241312 1/ 0 [ ] >> [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [ Steps we have attempted without success: - Setting TERM=dumb and piping to cat (the tool continues to output sequences). - Using tr -d '\r' (reduces lines but the data volume still triggers the timeout). - Increasing timeouts in the Job YAML (often ignored by the sub-action). Question: Has there been a change in the dispatcher's log-collection priority or buffering logic that would cause this bottleneck? Are there recommended "best practices" for handling tools that force interactive progress bars in the newer LAVA architecture? Best Regards Pavan Kumar

21 hours, 47 minutes

2
1
0 0

LAVA 2026.02

by Remi Duraffort

Hi folks, The 2026.02 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Security fixes Some security issues has been fixed in this release and we advice to upgrade as soon as possible. The details will be available later on. # Device-types ## New device-types * asus-CX3402CVA-brya * lenovo-chrome-2in1-14iru10-brox * mt8196-rauru-navi-sku1 * r9a07g043u11-smarc * r9a07g044c2-smarc * sc7180-trogdor-wormdingler-rev1-boe # Documentation migration Documentation v3 continues to be updated. All `deploy`, `boot` and `test` methods are now documented and published at https://lava.readthedocs.io/en/latest under **Technical references**. # Debian support Debian 11 (Bullseye) is being phased out. 2026.02 is the last LAVA release to support it. Starting from 2026.02 the oldest supported Debian release will be 12 (Bookworm) # Docker * All LAVA base images are now based on Debian 13 (Trixie) `slim` * Debian 11 (Bullseye) was dropped from all CI jobs * `wget` tool was added to lava-dispatcher-base image # Job logs ## Secret filtering LAVA is now filtering the job logs to remove sensitive data. This is still a work in progress that will be improved in the following releases. With this new feature, values of the variables defined in `secrets` dictionary will be replaced in the job output log with string `[MASKED]`. # Dispatcher changes ## Downloading artifacts Set max_retries to 1 by default when downloading an artifacts. The max_retries value can be set in the job definition using failure_retry key. Example: ```yaml - deploy: failure_retry: 2 images: image: url: https://example.com/foo ``` ## Deploy character delay Add support for `deploy_character_delay` to add delay between each character when interacting with the serial during deployment. To use it add to the device dictionary (in milliseconds): ```jinja2 {% set deploy_character_delay = 30 %} ``` Defaults to 30ms delay for vexpress devices. ## fastboot Improve fastboot device detection by making parsing more generic. This should now work with multiple fastboot versions. ## FVP Use `find` to locate armlm binary instead of hardcoded path to provide compatibility with more FVP models. ## multinode Fix `lava-role list` to properly match the documentation. Also fix multinode collate function used to replace placeholdeers like `$ipaddr` in synchronization messages. ## pyocd Fix flashing with pyocd with latest pycocd versions. ## Transfer overlay Allow to transfer the overlay using zmodem when network is not available. In the job definition, add: ```yaml - boot: transfer_overlay: transfer_method: zmodem unpack_command: tar -C / -xzf ``` The worker and target device **must** have respectively the `sz` and `rz` utilities installed. This is typically provided by: `lrzsz` package on Debian-based or Fedora-based systems. ## uuu Allow to use `uniquify` parameter for uuu deployment action. This is useful when using `downloads://` urls. # Server changes ## Restricting callback URLs By default, notification callbacks can target any URL. Administrators can restrict which hosts are permitted by setting `CALLBACK_ALLOWED_HOSTS` in `/etc/lava-server/settings.yaml` or a file in `/etc/lava-server/settings.d/`: ```yaml CALLBACK_ALLOWED_HOSTS: - "example.com" - "*.example.com" - "*.ci.example.com" ``` Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

3 weeks, 1 day

1
0
0 0

EXT4 officially supported, but are you using anything else?

by Ben Copeland

Hello all, I am preparing a pull request in LAVA and would like to know which filesystems people are using. The PR aims to drop libguestfs and eventually move to e2fsprogs-based tooling. LAVA officially supports ext4, but unofficially, are people using anything else? Could you let me know if any users are using XFS, BTRFS (or other variants)? Any feedback would be appreciated! Regards, Ben

1 month

1
0
0 0

DRF-filters deprecation

by Milosz Wasilewski

Hi, When LAVA introduced REST framework, it allowed for cross model filtering using djangorestframework-filters package. v0.2 LAVA API still allows that. Unfortunately the package has been unmaintained for the last 5 years: https://pypi.org/project/djangorestframework-filters/#history Without updates it will soon become incompatible with DRF and Django itself. In fact Debian has to carry patches to make it still working. On top of that there are performance issues with queries made by the DFR-filters. For these reasons it was decided that DRF-filters will be removed as LAVA dependency starting with version 0.3 of the API. v0.2 will remain supported for now. There is no end of life date agreed yet. In practice this means cross-model queries won't be supported any more in v0.3. As a tradeoff performance of single model queries should improve. If there are any concerns regarding this change, please reply to this thread. Best Regards, Milosz

2 months

1
0
0 0

Auto-registration disabled by default

by Milosz Wasilewski

Hi, Starting from 2026.02 worker autoregistration will be disabled by default. Details can be found in this merge request: https://gitlab.com/lava/lava/-/merge_requests/2081 This may break some setups, so please be aware when upgrading. Local setting can still overwrite the default if needed. Best Regards, Milosz

2 months, 1 week

1
0
0 0

Download action default retry

by Milosz Wasilewski

Hi, As of now the default retry for download action is set to 3 (three). This is an outlier as all other actions have retry set to 1. Some time ago I complained about it in the list. There is also a MR posted to address the problem: https://gitlab.com/lava/lava/-/merge_requests/3012 I just rebased it on top of 2026.01 and I'm planning to merge it in 2026.02 release. It will still be possible to set retry to 3 in download action with addition of failure_retry in a test job: - deploy: failure_retry: 2 images: image: url: 'https://example.com/foo.tgz' timeout: minutes: 20 to: downloads Please reply here if this change is going to fundamentally break your setup. IMHO the worst that can happen is the download timeouts increase 3x which in my experience should not cause problems. I don't know all potential setups, so please reply if this is a breaking change for you. Best Regards, Milosz

2 months, 1 week

1
0
0 0

2026.01 release is out

by Milosz Wasilewski

Hi, 2026.01 release was tagged yesterday. Here are the release notes: https://gitlab.com/lava/lava/-/wikis/releases/2026.01 Please report any issues you find during the deployment or when using. Best Regards, Milosz

2 months, 1 week

1
0
0 0

Sudden 100% Increase in Test Execution Time on Device i.MX6SOLO rev1.4 996 MHz

by pawanKumar

Hello Everyone, I am seeking assistance with a performance regression observed on an i.MX6SOLO rev 1.4 (996 MHz) board in our LAVA farm. The Issue: A test suite containing 90 test cases, which previously took 10 minutes to complete, is now consistently taking 20 minutes. This issue is unique to this one specific board; other boards of the same type are still completing the run in 10 minutes using the same build/firmware. Observations: - I'm not using any test_character_delay externally in my YAML. - Identical Logs: The "Fast" and "Slow" jobs produce identical serial output. - Recent Hangs: The test run has recently started to get stuck/hang mid-run, even though CPU usage is not at 100%. Debugging Performed: - Verified that CPU thermal throttling is not active. - Network latency and DNS resolution times appear normal. - No hardware errors are reported in dmesg. - Swapped power supplies and network cables with no improvement. Questions: 1. What is the best way to debug "dead time" between lava-test-case signals? 2. Could the LAVA worker be experiencing serial buffer delays on this specific node that don't appear in the logs? Thank you for your time. Best Regards Pavan Kumar

2 months, 3 weeks

2
4
0 0

Dropping support for debian 11

by Milosz Wasilewski

Hello LAVA users! Debian 11 bullseye is nearing close to EOL (August 2026). We're planning to retire LAVA support for it in the 2026.03 release. If that breaks your setup and there is a good reason to keep the support for longer, please respond to this message. Dropping debian 11 will allow to remove a few workarounds in LAVA that are currently included in the code. Best Regards, Milosz

3 months

1
0
0 0

Passing runtime parameters to user_commands in device configuration

by מור נגלי

Hello LAVA community, I'm working with LAVA for automated testing of embedded systems with BMC (Baseboard Management Controller) and I've encountered a limitation that I'd like to ask about. My Use Case I'm testing devices that support Redfish API for power management. I need to execute various Redfish commands (forceoff, poweron, forcerestart, etc.) from the dispatcher during test jobs. Currently, I define these as user_commands in my device dictionary: jinja2 {% set user_commands = { 'redfish_forceoff': { 'do': '/opt/lava-lab/shared/lab-scripts/dispatcher_command.sh 192.168.17.111 forceoff' }, 'redfish_poweron': { 'do': '/opt/lava-lab/shared/lab-scripts/dispatcher_command.sh 192.168.17.111 poweron' }, # ... more commands} %} Then in my job file: yaml - command: name: redfish_forceoff The Problem Every time I need to add a new Redfish action, I must modify the device dictionary (Jinja file) to add a new user_command entry. I'd like users to be able to specify the action dynamically from the job file without requiring device configuration changes. *What I want to achieve:* yaml # In job file - specify action dynamicallycontext: redfish_action: forceoff actions: - command: name: redfish_command # Generic command that uses context variable What I've Investigated 1. *Deploy actions* support variable substitution (e.g., {IMAGE}), but this is hardcoded for deploy actions only 2. *Job context variables* can override device dictionary values, but only at device configuration rendering time (job parsing), not at action execution time 3. *user_commands* are static strings with no runtime parameter substitution mechanism 4. *Test definitions* support parameters, but they run on the DUT, not the dispatcher I understand from the documentation that dispatcher commands are intentionally restricted for security reasons, which makes sense. My Current Workaround I'm using Jinja2 loops to auto-generate commands from a list: jinja2 {% set redfish_actions = ['auth_test', 'forceoff', 'poweron', 'forcerestart'] %}{% set user_commands = {} %}{% for action in redfish_actions %} {% set _ = user_commands.update({'rf_' + action: {'do': rf_script + ' ' + action}}) %}{% endfor %} This reduces the pain (just add one word to a list), but still requires device configuration edits. My Questions 1. *Is there any mechanism I've missed* that allows passing runtime parameters from job files to user_commands? 2. *Is this a known limitation* that others have encountered? 3. *Would it be feasible* to add support for parameter substitution in user_commands (similar to how deploy actions work with {IMAGE}), while maintaining security constraints? 4. *Are there alternative approaches* within LAVA's architecture that I should consider? Any guidance or suggestions would be greatly appreciated. If this is a feature that doesn't exist but would be useful, I'd be happy to discuss potential implementation approaches or contribute to a feature request. Thank you for your time and for maintaining LAVA! Best regards, Mor

5 months

2
4
0 0

Configure SSL certifcate

by Gregory Machin

Hi, can you point me to a document for the configuration of SSL certificate on the Lava Docker image ? Thank you. Greg

6 months, 1 week

1
0
0 0

Call for LAVA maintainers

by Remi Duraffort

Hello LAVA users and developers, LAVA has been around and useful for a lot of people for a long time. For some years now, the list of LAVA maintainers as been shrinking down to 2 or even 1 developer (payed by Linaro). In order to review all the patches that the LAVA developers are pushing, we are looking for developers that want to become maintainer. The idea would be to delegate some part of the maintenance of LAVA sub-systems to some developers. If you are interested, please reach out to me in the lava-users ML, privately or in the new LAVA discord server at https://discord.gg/4dVatqyw Thanks for your attention. -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

6 months, 1 week

2
2
0 0

Connect a device through serial without PDU

by Ali Tariq

Hello, I am trying to connect a RISC-V VisionFive 2 device with the LAVA web server. I do not have a PDU at the moment and I do not know which one will be appropriate for this device setup. The LAVA master and worker are running on the same machine, so I have localhost set up. I added the following device dictionary for VisionFive 2. {% extends 'jh7100-visionfive.jinja2' %} {% set console_backend = 'serial' %} {% set serial_port = '/dev/ttyUSB0' %} {% set serial_baud_rate = 115200 %} The dictionary template, which is extended for VisionFive 2, is from the VisionFive 1 device type. As I won't be needing initramfs and other uboot env variables, I am using visionfive 1's dictionary template just for getting the setup to work. Following is the VisionFive 1 dictionary template. {% extends 'base-uboot.jinja2' %} {% set uboot_mkimage_arch = 'riscv' %} {% set console_device = console_device|default('ttyS0') %} {% set baud_rate = baud_rate|default(115200) %} {% set booti_kernel_addr = '0x84000000' %} {% set booti_dtb_addr = '0x88000000' %} {% set booti_ramdisk_addr = '0x88300000' %} {% set uboot_initrd_high = '0xffffffffffffffff' %} {% set uboot_fdt_high = '0xffffffffffffffff' %} {% set bootloader_prompt = 'VisionFive#' %} {% set uboot_tftp_commands = [ "tftpboot {KERNEL_ADDR} {KERNEL}", "tftpboot {RAMDISK_ADDR} {RAMDISK}", "tftpboot {DTB_ADDR} {DTB}"] -%} I was able to run a job on QEMU. But the VisionFive 2 is not detected by the LAVA web server. The physical connection is complete because I can use picocom on my computer to attach to the VisionFive 2 using the command sudo picocom -b 115200 /dev/ttyUSB0. I created several job definitions just for testing the setup but they all are just loading. The device health check is bad. Following is the simplest job that I am trying to run and see if it completes. job_name: simple-uboot-test device_type: visionfive2 priority: medium visibility: public timeouts: job: minutes: 5 actions: - deploy: timeout: minutes: 1 to: u-boot - boot: method: bootloader bootloader: u-boot connection: serial commands: - printenv But all the jobs are on infinite loading. This is my first time connecting a physical device with the LAVA web server, so I would appreciate it if someone could tell me what the issue is here. Regards, Ali

6 months, 1 week

2
1
0 0

Maintainership help request

by Igor Ponomarev

Hello, I recently saw the "Call for LAVA maintainers" lava-announcement email and I am willing to help. I am pretty active in LAVA's gitlab. By the way, your discord link is expired. Nobody will be able to join until a new link is created. (you can make it without expiration timer) -- Igor Ponomarev Consultant Software Engineer Collabora Ltd. Unit 106 St Johns Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK Registered in England & Wales, no. 5513718

6 months, 1 week

2
1
0 0

email notifications not using correct url. issue with absolute_url?

by Michael Peddie

As the subject says, email notifications in lava aren't using the correct url when linking to a job. They use the example.com url, and even after changing the value under Sites in the admin settings, it still does. Looking at the notification template for emails, it uses the job.absolute_url, but I can't quite figure out how it constructs example.com, there shouldn't be anything anywhere that it pulls that from. Any help on this would be appreciated, even if it's just insight and not a solution.

6 months, 2 weeks

2
1
0 0

Is there any job using the ssh.jinja2 template as a learning example?

by 93831920＠qq.com

Is there any job using the ssh.jinja2 template as a learning example? Thank you

7 months, 3 weeks

2
1
0 0

Is there such a job test?

by 93831920＠qq.com

The device already has a Windows system and provides an SSH server, so there is no need for deployment. After LAVA establishes an SSH connection with the device, it can directly send test instructions, and exit the SSH connection after the test is completed?

8 months, 2 weeks

1
0
0 0

Not able to update default to qemu.jinja2 with updated configuration

by sweta.ghosh＠nagarro.com

Hello, I would like to replace -net legacy values with -netdev as netdevice in /etc/lava-server/dispatcher-config/device-types/qemu.jinja2 file but even after creating new jinja file my device is not able to identify it. I have tried restarting all the LAVA services but still not able to . Could you please share the how can I change or update existing jinja file. Thanks, Sweta

8 months, 3 weeks

2
1
0 0

how to discrib a ssh device?

by 93831920＠qq.com

Device is RAM, directly connected to the FT4232H through JTAG, FT4232T connected to the windows host through USB-MicroUsb, the windows host provides the telnet test command interface to the workers of the lava automated test environment through the telnet server, in this scenario, how to create the device type, device and job in lava, Can you use lava to automate testing of RAM?

8 months, 3 weeks

1
1
0 0

Running Lava in docker on Ubuntu.

by Gregory Machin

Hi, I have been asked by our hardware dev team to stand up a server that will run Lava. I was advised that Debian was the only supported OS. But as they will be running the official docker images is there any reason I couldn't deploy use Ubuntu 24.04 LTS ? Thank you Greg

9 months, 2 weeks

2
1
0 0

Using LAVA to transfer file via Ymodem from LAVA dispatcher(HOST machine) to DUT

by hina.hafeez＠10xengineers.ai

Hi LAVA users, I have set-up a LAVA server, dispatcher and connected my device (banana-pi) board serially via UART to host machine on which dispatcher is running. I have defined a job which uses deploy "tmpfs" to download .itb file from local file server. After that, under "boot" action, i am using u-boot method to boot the device. Output: 1. deploy images section runs successfully 2. uboot action starts 3. After connecting to device using telnet, device is reset(pdu-reboot). 4. Device enters in Y modem state waiting for image to get transferred. 5. After some time, "CCC CCCCCCC[ 99.943] spl: ymodem err - Timed out" Issue/Question: 6. At this stage, i want LAVA dispatcher(HOST machine) to send .itb file over UART using Ymodem to device. 7. But LAVA is unable to execute this command "sz -Y --ymodem /uboot-opensbi.itb < /dev/ttyUSB0 > /dev/ttyUSB0" to transfer opensbi.itb file to device. 8. I need to know what am i missing in writing job definition for this case. 9. Which device template should i use and how it should be extended. 10. is there any job definition syntax to fulfill my requirement? Regards,

9 months, 3 weeks

3
5
0 0

Open ID Connect (OIDC) authentication providers with LAVA

by sweta.ghosh＠nagarro.com

Hello Team, I am trying to use Azure AD for user authentication in LAVA. I followed the document https://validation.linaro.org/static/docs/v2/authentication.html#using-open… and created a yaml file with all details under /etc/lava-server/settings.d and restarted lava-server-gunicorn.service. Additionally , added ssl_header configuration to trust X-Forwarded-Proto header for detecting HTTPS. cat secure_proxy_ssl_header.yaml SECURE_PROXY_SSL_HEADER: - HTTP_X_FORWARDED_PROTO - https During sign in I am able get Microsoft login page but after successfully login I get "Bad Request (400)" . Could you please suggest if Lava can work with mozilla-django-oidc or if i am missing any configuration.

10 months, 2 weeks

1
1
0 0

PXE boot qemu vm in LAVA

by sweta.ghosh＠nagarro.com

Hello Team, I have a custom scripts which spins up a pxe server in qemu and on the same network when we try to spin up VM it boot's from network and takes OS image from pxe to boot up. I want to do the same in LAVA and execute all my automated OS test cases over newly booted VM . I have a pipeline which setup's the bridge n/w (pxe network for server and host ) and i want to trigger LAVA job to spin up VM using same network. But to that to work I require image_arg in deploy action and which require mandatory download of image . I even tried to remove the image from img_arg but it wont boot from network . Also I am not able to find option to pass qemu arguments in case I use tftp or nfs . Could you please suggest how pxe boot qemu vm in LAVA

10 months, 2 weeks

1
0
0 0

timeouts are broken

by Milosz Wasilewski

Hi, After the 2025.02 release download timeouts are terribly broken. This should have never happened. LAVA is silently dividing the action timeout by number of repetitions: https://gitlab.com/lava/lava/-/wikis/releases/2025.02#timeout-and-retries https://gitlab.com/lava/lava/-/commit/07b13b5e4a2bb335dcfff155974ade81ef17b… This causes perfectly good jobs to fail. Example: https://lava.infra.foundries.io/scheduler/job/73929 If such change is required it should have been done with explicit parameter set in the job definition. In the current form I consider it a bug. The only way to work around this bug is to set the timeout to some ridiculous number (3 times as long as it is normally required). IMHO this should be reverted before the next release. Best Regards, Milosz

10 months, 4 weeks

2
8
0 0

Environment variables in interactive tests

by Michael Peddie

Hi, I am trying to make use of environment variables defined on my lava-server inside interactive tests. I have had some success with this, but I am not sure why one method works over the other. For example, I do something like the following: echo 'wget --user=some-name --password=${SECRET_PASSWORD}' > somefile.sh And that works just fine. But when I already have a script (say it exists on the DUT already) that just takes it as a parameter, it doesn't work. Example: some_script.sh -p ${SECRET_PASSWORD} -u some-name I have also tried accessing it directly in the script, similar to what somefile.sh would look like on the inside, but had no success with that. I have tried exporting the environment variable as another one to see if that would work, but to no avail. Is there something I'm missing? What I can I do to achieve this without hardcoding it anywhere in my scripts? Regards, Michael

11 months

2
12
0 0

deploy to flasher with download URLs

by Milosz Wasilewski

Hi, I upgraded my LAVA instance to 2025.04 yesterday I skipped one release and the upgrade was from 2024.09. Most of the jobs I run in use deploy-to-flasher with custom script handling the board flashing. Up until 2024.09 (I don't know about 2025.02) "downloads" URLs resulted in all files stored in a flat directory. My flashing scripts made use of this "feature". In 2025.04 this has changed. This commit: https://gitlab.com/lava/lava/-/commit/4d9f0ebdae9ca53baf6633f4a35e716183bd2… makes the files stored in separate directories. It feels like a step in a right direction but it broke my flashing scripts. To make a quick fix I did this in the beginning of the script: DOWNLOAD_DIRS=$(find . -maxdepth 1 -mindepth 1 -type d -printf '%f ') for d in $DOWNLOAD_DIRS; do FILE_TO_COPY=$(find ./$d -maxdepth 1 -mindepth 1 -type f -printf '%f') ln "$PWD/$d/$FILE_TO_COPY" "$PWD/$FILE_TO_COPY" done This isn't great but it solves the issue for now. After a short discussion with Chase we came to conclusion that solution that looks "more correct" is to implement support for "uniquify" for "download" URLs. Chase sent a patch here: https://gitlab.com/lava/lava/-/merge_requests/2795 This should be available in the next release. I hope this message helps someone with similar issues :) Best Regards, Milosz

11 months, 1 week

1
0
0 0

Download to worker mid-test

by Michael Peddie

Hi, I am trying to write a test to conditionally extract files from a zip file for use in some scripts, e.g. unzip the file, copy/move the files I need out, remove the zip file and directory created from unzipping it. However, the zip file size exceeds the disk space of the DUT. I have considered doing something like a MultiNode job, where the DUT is booted as well as a QEMU instance, and the file is downloaded and extracted on the QEMU and moved onto the DUT, but I would prefer not to do that if possible. Is there any way to download the zip file mid-test to the worker instead, extract the files I need, and move them onto the DUT? Ideally no modifications to the code of LAVA, but if that is necessary then I can figure out an interim solution and come back to it later. Regards, Michael

1 year

2
1
0 0

LAVA LDAP mirror groups

by Michael Peddie

Hi, I am trying to get LDAP working in my LAVA instance, and have managed to get logging into it working. The issue comes when I try to use Django's MIRROR_GROUPS setting. I am already aware that it is not built into LAVA to be configurable, and I have already taken steps to add the relevant lines in lava_server/settings/common.py to make it work (initialize to None, values.get() in the update method, then run eval on the value in the LDAP if section), but it still won't work. All other required settings are clearly working just fine, and I can even set USER_FLAGS_BY_GROUP just fine, but I would prefer to mirror certain groups that users are members of and assign permissions to the groups. Do I need to pre-create the groups before logging in to LAVA or am I missing something else/doing something wrong? Original common.py source: https://gitlab.com/lava/lava/-/blob/master/lava_server/settings/common.py

1 year, 1 month

2
8
0 0

Can't do Linux auto login from LAVA job

by Ramasamy, Rajeshkumar

Hi All, I am trying to setup LAVA and execute test cases. I am using Intel x86-64 machine. LAVA able to boot the Linux without any issues, but it is stuck just before login prompt. I am able to boot the machine with manual procedure. Pls find the log below. [[0;32m OK [0m] Started [0;1;39mBerkeley Internet Name Domain (DNS)[0m. [[0;32m OK [0m] Started [0;1;39mPostfix Mail Transport Agent[0m. [[0;32m OK [0m] Reached target [0;1;39mMulti-User System[0m. [[0;32m OK [0m] Reached target [0;1;39mHost and Network Name Lookups[0m. Starting [0;1;39mRecord Runlevel Change in UTMP[0m... [[0;32m OK [0m] Finished [0;1;39mRecord Runlevel Change in UTMP[0m. Linux Update 6 intel-x86-64 ttyUSB0 <Login prompt print expected here ("intel-x86-64 login:"), but not displayed on the log> bootloader-interrupt timed out after 280 seconds end: 2.3.2 bootloader-interrupt (duration 00:04:40) [common] case: bootloader-interrupt case_id: 185 definition: lava duration: 280.00 extra: ... level: 2.3.2 namespace: common result: fail health check yaml file snippet: - boot: timeout: minutes: 5 method: ipxe commands: nfs failure_retry: 3 prompts: - '[a-zA-Z0-9\-\_]+@[a-zA-Z0-9\-\_]+:.*?#' auto_login: login_prompt: "intel-x86-64 login:" username: root password_prompt: "password:" password: root login_commands: - cat /proc/cmdline - cd / - ls - test: failure_retry: 3 timeout: minutes: 5 definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: smoke-tests-basic description: "Basic system test command for Linaro Ubuntu images" run: steps: - printenv Thanks Rajesh

1 year, 2 months

1
0
0 0

Can't do Linux login from LAVA job

by Ramasamy, Rajeshkumar

Hi All, I am trying to setup LAVA and execute test cases. I am using Intel x86-64 machine. LAVA able to boot the Linux without any issues, but it is stuck just before login prompt. I am able to boot the machine with manual procedure. Pls find the log below. [[0;32m OK [0m] Started [0;1;39mBerkeley Internet Name Domain (DNS)[0m. [[0;32m OK [0m] Started [0;1;39mPostfix Mail Transport Agent[0m. [[0;32m OK [0m] Reached target [0;1;39mMulti-User System[0m. [[0;32m OK [0m] Reached target [0;1;39mHost and Network Name Lookups[0m. Starting [0;1;39mRecord Runlevel Change in UTMP[0m... [[0;32m OK [0m] Finished [0;1;39mRecord Runlevel Change in UTMP[0m. Linux Update 6 intel-x86-64 ttyUSB0 <Login prompt print expected here ("intel-x86-64 login:"), but not displayed on the log> bootloader-interrupt timed out after 280 seconds end: 2.3.2 bootloader-interrupt (duration 00:04:40) [common] case: bootloader-interrupt case_id: 185 definition: lava duration: 280.00 extra: ... level: 2.3.2 namespace: common result: fail health check yaml file snippet: - boot: timeout: minutes: 5 method: ipxe commands: nfs failure_retry: 3 prompts: - '[a-zA-Z0-9\-\_]+@[a-zA-Z0-9\-\_]+:.*?#' auto_login: login_prompt: "intel-x86-64 login:" username: root password_prompt: "password:" password: root login_commands: - cat /proc/cmdline - cd / - ls - test: failure_retry: 3 timeout: minutes: 5 definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: smoke-tests-basic description: "Basic system test command for Linaro Ubuntu images" run: steps: - printenv Thanks Rajesh

1 year, 2 months

1
0
0 0

How to start docker container with privileged mode which used for postprocessing artifacts

by Sai Ashrith Sathujoda

We have a requirement for some test cases that the artifacts need to be modified (negative scenario) in order to verify some scenarios. we are currently using postprocess under deploy action for this purpose. As of part postprocess, we need to mount partitions of disk image and make some changes based on the requirement. However, we are unable to mount the partitions using losetup command. docker: image: debian:bookworm steps: - apt-get update - loop_dev=$(losetup -P -f --show image.wic) - echo $loop_dev The job failed with below error, losetup -P -f --show image.wic losetup: cannot find an unused loop device It looks like the docker container needs root privileges. It works when we run locally with "privileged:true". Is there any way we can pass "privileged: true" option to docker image through LAVA job definitions? Any inputs regarding this matter are highly appreciated.

1 year, 2 months

1
0
0 0

How to verify secure boot negative test cases

by shivanand.kunijadar＠toshiba-tsip.com

Hello, I'm trying to verify secure boot on LAVA lab, I've verified the positive case of secure boot and confirmed the secure boot is enabled from the boot logs (kernel-start-message = "EFI stub: UEFI Secure Boot is enabled."). Next, I would like to verify the negative test case as well by tampering the secure boot image where I expect the image boot shall fail. Is there way in LAVA to make the job complete(e.g. waiting for specific message in boot logs like "Can't find image information. ") even though the image fails to boot on the DUT? Ref. https://lava.ciplatform.org/scheduler/job/1236667#L81

1 year, 3 months

2
1
0 0

concurrent job execution

by sweta.ghosh＠nagarro.com

Hi, I am not able to submit multiple jobs on my single node lava server, currently even 1 job is running the second job waits till the 1st job is completed. I have tried to increase the job limit of my worker to 3 via lava Admin UI but still I cannot run 2 jobs in parallel. Could you please let me know how can I increase the no. jobs.

1 year, 3 months

1
0
0 0

Securing LAVA Setup

by Michael Peddie

Hi, I have been looking into how to secure LAVA, as so far it's just been learning how it works. There's the obvious http/https stuff, and in the documentation mentions firewalls for remote workers (which we'll be using anyways). We already have things in place for physical security and plans on how to improve, so no worries there. Is there anything not covered in the documentation that should be considered and how could we go about putting that into place? Best regards, Michael

1 year, 4 months

1
0
0 0

What does the gdb boot method do?

by 叶轩杨

I saw a file 'lava-dispatcher/actions/boot/gdb.py' in the LAVA source code, which seems to define a gdb method in action->boot. But I didn't find a description of it in the LAVA documentation. The document I checked is `https://docs.lavasoftware.org/lava/actions-boot.html?highlight=boot#boot-action-reference`. Reading the source code may lead to some misunderstandings and is time-consuming. I hope someone who knows its function can explain to me what this new method does. Thank you very much!

1 year, 4 months

1
0
0 0

How to add custom data in the metadata field of notify callback json?

by 叶轩杨

I want to add some key:value pair in metadata of json that POST by notification of job, but I can not figure it out. I had tried add metadata in job defination and test defination, but the back json did not have my data in metadata field, it was just a []. Then I read the lava source code, I find the metadata data are produced by method `get_metadata_dict()` in class `TestJob`. This method is called at method `create_job_data` in class `TestJob`, it creates the json data which will post to callback url. `get_metadata_dict()` check whether there is an attribute `testdata`, but I can not find the place where this attribute‘s assignment. Could someone give me a way to do this? Thanks you very much! 🙏

1 year, 5 months

2
4
2 0

Auto login fails after adding banner message before login prompt

by sweta.ghosh＠nagarro.com

Hello, We are testing our custom debian image and have added banner messages before login and after login and after the implementation LAVA job is failing on auto-login. Please find example banner messages: |_ _/ | | | | _| | |_____\ Info: Example.....: ABC Example......: ABC IP: etho.......:123 Info: 1. some info Some Info: 1. ### Some more info: #### login : It looks like due to a huge banner message before the login prompt it is getting confused and due to this, it gets stuck at auto login and timeouts. Please suggest if we can ignore this in LAVA to login and execute the test case Please find the job details: device_type: qemu job_name: banner timeouts: job: minutes: 30 action: minutes: 25 connection: minutes: 5 priority: medium visibility: public context: arch: amd64 lava_test_results_dir: "/home/lava-%s" actions: - deploy: timeout: minutes: 15 to: tmpfs images: rootfs: url: file:///home/image image_arg: XXXX - boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'login:' auto_login: login_prompt: "login:" username: abc password_prompt: "Password:" password: "abc" - test: timeout: minutes: 15 definitions: - repository: git(a)test.git from: git path: test.yaml name: basic Thanks, Sweta

1 year, 5 months

4
23
0 0

Error Msg : "Connection aborted." when test monitor is used

by andy.sabathier＠nxp.com

Hello, I'm currently using version 2024.09, and when running the test monitor, I encounter the following error at the top of my job: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). I don't notice anything unusual in the test logs. Could this be a bug with the test monitor? Best Regards, Andy

1 year, 5 months

3
4
0 0

job timeouts

by Milosz Wasilewski

Hi, I think there is something terribly wrong with job timeouts in 2024.05 release. Total job timeout seems to be ignored. This was working fine on 2024.01 release. I'll try to debug, but IMHO this is a pretty critical bug that should be addressed immediately. I'll report it in gitlab as well. Best Regards, Milosz

1 year, 6 months

2
2
0 0

Copy local package for testing

by sweta.ghosh＠nagarro.com

Hello, I am spinning up debian image using Qemu and want to copy a custom package from my lava server to debian vm for testing. I could see that support for tar is planned https://docs.lavasoftware.org/lava/actions-test.html#inline-test-definition… . But is there any way I can include my custom package in tarbar created at Deploy Action to ship to my VM easily?

1 year, 7 months

4
4
0 0

Execute script while running ssh device type

by sweta.ghosh＠nagarro.com

Hello, I have multiple machines over which I test using LAVA job using primary ssh connection. As my devices OS gets installed every now and then I have to manually update the ssh public key before executing the job so that LAVA can connect to DUT over passwordless SSH. Is there a way I can include a script in device dictionary to copy the keys automatically before requesting a ssh connection? As I understand password option is not there. Kindly suggest.

1 year, 7 months

4
3
0 0

How to execute preboot command in qemu

by sweta.ghosh＠nagarro.com

Hello, I want to spin up an image with tpm enabled using qemu, and I need to execute swtpm socket command before image boot-ups via qemu. Please suggest if I could execute this command or script in qemu.jinja2 file or any other file. Thanks, Sweta

1 year, 7 months

2
1
0 0

Not able to set variable

by sweta.ghosh＠nagarro.com

Hi, I want clone a gitlab repo with id and token during job execution. To hide the token I am passing the value in my job. How can I export the value of GIT_TOKEN in environment file of LAVA? So that whenever job is executed it will read the value from its environment file. I tried to use export and tried to add "GIT_TOKEN=abcd" in env.yaml file. But I guess its not correct format. I couldn't find what kind of values we can add in env.yaml in documents. I also added GIT_TOKEN value in /etc/profile of the LAVA server still the job didn't pick it. Could you please suggest how can we clone the repo in LAVA job without exposing its password. Also I understand that ssh key of root will work but I want to avoid using ssh key of root. Kindly suggest - test: timeout: minutes: 15 definitions: - repository: https://gitid:$GIT_TOKEN@gitlab.com/lava-tests.git from: git path: tests/cisscan/cis.yaml branch: pipeline name: cis-benchmark

1 year, 7 months

3
6
0 0

Confusion about job IDs reported by lavacli for multi node tests

by Florian Bezdeka

Hi all, I run into some unexpected behavior when it comes to LAVA job IDs reported by the "lavacli jobs sumit" command for multi node tests. What I get is a list of newline separated job IDs like: lavacli jobs submit <my-job-description> 132.0 132.1 To my understanding this tries to express that 132.1 is a sub job of 132. OK, but this is a problem if I now try to download the junit report for the second job (=132.1). It seems 132.1 is not a valid job ID (API wise), it has to be translated to 133 to work properly. This is still doable, but I'm unsure if I can relay on 133 really being the 132.1 job in case there are several job submissions at the same time. Is that a bug of lavacli? What is the reasoning for returning a sub job ID which seems to be invisible to the rest of the LAVA infrastructure / API? I'm currently running lavacli 1.2-1, as shipped by Debian 12. Any input welcome... Best regards, Florian

1 year, 7 months

2
2
0 0

bootup a server and deploy debian bookworm over serial connection

by sweta.ghosh＠nagarro.com

Hello, I am new to LAVA, I want to boot up my hardware(NET-I System)which is a x86_64 and install debian bookworm image over it and execute test cases. My h/w is connected to my LAVA server via a serial device.Below is the high level job that I am planning to use. But where should I specify the serial device information and other information?Could please share any example job and device .jija2 that I can refer regarding what all information needs to be provided for LAVA server to properly connect to H/W and also is u-boot correct for my usecase? or if there is any other suggestions : device_type: your_device_type job_name: debian_boot_test timeouts: job: minutes: 15 action: minutes: 5 connection: minutes: 2 actions: - deploy: timeout: minutes: 10 to: tftp images: rootfs: url: <Debian Image> - boot: method: u-boot commands: your_device_boot_commands prompts: - 'login:' parameters: boot_options: root=/dev/ram0 - test: timeout: minutes: 5 definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: debian-check-test description: "Test to verify Debian OS" os: - debian run: steps: - echo 'Checking if OS is Debian' - 'if [ "$(lsb_release -is)" = "Debian" ]; then echo "OS is Debian"; else echo "OS is not Debian"; fi' from: inline name: debian-check-test path: inline/debian-check-test.yaml

1 year, 8 months

3
2
0 0

Change path of lava_test_results_dir and lava_test_dir

by sweta.ghosh＠nagarro.com

Hello, I would like to change path to /mnt from below: "lava_test_dir": "/lava-%s", "lava_test_results_dir": "/lava-%s", I am using an immutable image and due to which don't have permission to write, so cannot create directory eg /lava-34 due to which cannot execute test. We have partition /mnt which we can mount and run the test cases over our os. But for that to work I need to change path. I tried changing it in /usr/lib/python3/dist-packages/lava_dispatcher/deployment_data.py , but jobs fails once I change it. I have only changed the debian section as the image we are using is a debian image. Could you please help me if there is any option to make these changes while executing job or any other file for devices. Thanks, Sweta

1 year, 9 months

2
6
0 0

RecursionError: maximum recursion depth exceeded on lava-scheduler service

by sweta.ghosh＠nagarro.com

Hi , I am trying to setup LAVA on Single Master Worker node. When i am trying to submit job it is running but no device is getting selected nor I am getting any output on UI. When I checked the lava-scheduler it is failing with below error: root@debian:/etc/apache2/sites-available# systemctl status lava-scheduler × lava-scheduler.service - LAVA scheduler Loaded: loaded (/lib/systemd/system/lava-scheduler.service; enabled; preset: enabled) Active: failed (Result: start-limit-hit) since Tue 2024-07-02 18:52:36 IST; 7min ago Duration: 1.472s Process: 10345 ExecStart=/usr/bin/lava-server manage lava-scheduler --level $LOGLEVEL --log-file $LOGFILE $EVENT_URL $IPV6 (code=exited, status=0/SUCCES> Main PID: 10345 (code=exited, status=0/SUCCESS) CPU: 1.431s Jul 02 18:52:35 debian systemd[1]: lava-scheduler.service: Deactivated successfully. Jul 02 18:52:35 debian systemd[1]: lava-scheduler.service: Consumed 1.431s CPU time. Jul 02 18:52:36 debian systemd[1]: lava-scheduler.service: Scheduled restart job, restart counter is at 5. Jul 02 18:52:36 debian systemd[1]: Stopped lava-scheduler.service - LAVA scheduler. Jul 02 18:52:36 debian systemd[1]: lava-scheduler.service: Consumed 1.431s CPU time. Jul 02 18:52:36 debian systemd[1]: lava-scheduler.service: Start request repeated too quickly. Jul 02 18:52:36 debian systemd[1]: lava-scheduler.service: Failed with result 'start-limit-hit'. Jul 02 18:52:36 debian systemd[1]: Failed to start lava-scheduler.service - LAVA scheduler. root@debian:/etc/apache2/sites-available# tail -f /var/log/lava-server/lava-scheduler.log {% extends 'qemu.jinja2' %} ^^^^^^^^^^^^^^^^^^^^^^^^^ [Previous line repeated 977 more times] File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1494, in is_up_to_date return self._uptodate() ^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/loaders.py", line 212, in uptodate return os.path.getmtime(filename) == mtime ^^^^^^^^^^^^^^^^^^^^^^^^^^ RecursionError: maximum recursion depth exceeded Also when I am accessing /admin/lava_scheduler_app/device/ I am getting " 500 Internal Server Error maximum recursion depth exceeded" I have registered my worker node as debian,hostname as debian and device type is qemu. Could you please what I am missing.

1 year, 9 months

4
4
1 0

lava test doesn't wait for command to finish running

by Michael Peddie

Just as the subject says. I am using lava-test-case to confirm whether this particular command is done running and is successful or not, because the test has previously always ended prematurely and not done what I needed it to do. When I run the command manually outside of a lava test job it works just fine, no errors and doesn't end early. It does take a while to complete, but I don't think that's the issue as I use wget to download some large files and those have taken several minutes longer than this one command is supposed to. How do I get lava to wait for this command to end? Or to change how it checks for command failure? I get "Received signal: <ENDTC>" almost immediately after running the command, and subsequently the result=fail signal. I really don't know why it won't work, so if there is a change I can make to the files or a config somewhere I would like to know. Best regards, Michael

1 year, 9 months

2
15
0 0

MultiNode to run script->reboot->test

by Michael Peddie

Hi, I was wondering if it is possible to write a multinode test to boot a device, run a script to do some stuff then have it reboot one way or another, then run tests. There is intention for pipeline integration down the line, so if it isn't possible to do that with multinode I have considered the idea of setting it up so one regular test runs and runs the script, turns DUT off, then another test runs to boot and actually test the device. This (obviously) has its drawbacks, but it's something of a backup. If I understand correctly, where certain actions are placed in a multinode test actually matters, so it (in my mind) is possible to have a boot->test to boot a DUT then use test commands to run a script, and then after that another boot->test to boot the device or just login again and then actually run tests. So the final sort of action set up would be boot->test->boot->test, where the first two are for running the script and rebooting, and the latter two are actually testing. If there are other ideas that are better or equally viable, then Im open to those too, but multinode was the first that came to mind cos of secondary connections and the like. Best regards, Michael

1 year, 11 months

1
1
0 0

lavasoftware.org is unaccessible from gandi.net Paris SD6 datacenter

by Jacopo Mondi

Hello I am experiencing troubles connecting to apt.lavasoftware.org from a VPS located in the "Paris France SD6" datacenter hosted by gandi.net, The problematic VPS IP address is 46.226.105.174. Testing from another VPS, still hosted at gandi.net but located in a different datacenter (Luxembourg) presents no issues. Both VPS runs Debian 12 Bookworm. The symptoms I'm experiencing seems to suggest the TLS handshake gets truncated as suggested by the 'gnutls-cli' output below reported ------------------------------------------------------------------------------- # gnutls-cli -d9999 -p 443 apt.lavasoftware.org ... |<5>| REC[0x56498f5d7740]: Sent Packet[1] Handshake(22) in epoch 0 and length: 402 |<11>| HWRITE: wrote 1 bytes, 0 bytes left. |<11>| WRITE FLUSH: 402 bytes in buffer. |<11>| WRITE: wrote 402 bytes, 0 bytes left. |<3>| ASSERT: ../../lib/buffers.c[get_last_packet]:1185 |<10>| READ: Got 0 bytes from 0x3 |<10>| READ: read 0 bytes from 0x3 |<3>| ASSERT: ../../lib/buffers.c[_gnutls_io_read_buffered]:593 |<3>| ASSERT: ../../lib/record.c[recv_headers]:1195 |<3>| ASSERT: ../../lib/record.c[_gnutls_recv_in_buffers]:1321 |<3>| ASSERT: ../../lib/buffers.c[_gnutls_handshake_io_recv_int]:1467 |<3>| ASSERT: ../../lib/handshake.c[_gnutls_recv_handshake]:1600 |<3>| ASSERT: ../../lib/handshake.c[handshake_client]:3075 |<13>| BUF[HSK]: Emptied buffer *** Fatal error: The TLS connection was non-properly terminated. ------------------------------------------------------------------------------- Could you kindly check if my VPS is maybe in a range of IP addresses blacklisted by the service provider that hosts lavasoftware.org ? Thanks j

1 year, 11 months

1
0
0 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users