lava-users

lava-users@lists.lavasoftware.org

2 participants
918 discussions

Dependencies between master and worker when restoring from backups

by Tim Jaacks

Hello everyone, I have written a backup script for my LAVA instance. While testing the restore process I stumbled upon issues. Are there any dependencies between the master and workers concerning backups? When the master crashes, but the worker does not, is it safe to restore the master only and keep the worker as it is? Or do I have to keep master and worker backups in sync and always restore both at the same time? Restoring my master as described in the LAVA docs generally works. The web interface is back online, all the jobs and devices are in consistent states. Restoring the worker is relatively easy, according to the docs. I installed the LAVA packages in their previous versions on a fresh (virtual) machine, restored /etc/lava-dispatcher/lava-slave and /etc/lava-coordinator/lava-coordinator.conf. The worker has status "online" in the LAVA web interface afterwards, so the communication seems to work. However, starting a multinode job does not work. The job log says: lava-dispatcher, installed at version: 2018.5.post1-2~bpo9+1 start: 0 validate Start time: 2018-12-18 12:25:14.335215+00:00 (UTC) This MultiNode test job contains top level actions, in order, of: deploy, boot, test, finalize lxc, installed at version: 1:2.0.7-2+deb9u2 validate duration: 0.01 case: validate case_id: 112 definition: lava result: pass Initialising group b6eb846d-689f-40c5-b193-8afce41883ee Connecting to LAVA Coordinator on lava-server-vm:3079 timeout=90 seconds. This comes out in a loop, until the job times out. The lava-slave logfile says: 2018-12-18 12:27:15,114 INFO master => START(12) 2018-12-18 12:27:15,117 INFO [12] Starting job [...] 2018-12-18 12:27:15,124 DEBUG [12] dispatch: 2018-12-18 12:27:15,124 DEBUG [12] env : {'overrides': {'LC_ALL': 'C.UTF-8', 'LANG': 'C', 'PATH': '/usr/local/bin:/usr/local/sbin:/bin:/usr/bin:/usr/sbin:/sbin'}, 'purge': True} 2018-12-18 12:27:15,124 DEBUG [12] env-dut : 2018-12-18 12:27:15,129 ERROR [EXIT] 'NoneType' object has no attribute 'send_start_ok' 2018-12-18 12:27:15,129 ERROR 'NoneType' object has no attribute 'send_start_ok' It is the "job = jobs.create()" call in lava-slave's handle_start() routine which fails. Obviously there is a separate database on the worker (of which I did not know until now), which fails to be filled with values. Does this database have to be backup'ed and restored? What is the purpose of this database? Is there anything I need to know about it concerning backups? Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks(a)garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS! Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun

6 years, 2 months

Re: [Lava-users] Lava job always exits when running a long duration test case without outputs

by Neil Williams

Please make sure you include the mailing list in all replies so that others know when a problem has been fixed (and how it was fixed) On Tue, 18 Dec 2018 at 12:00, Chuan Su <lavanxp(a)126.com> wrote: > > According to your comments , we checked our setups and we found that we utilized ser2net & telnet to communicate with DUT , however , ser2net set default timeout parameter as 600 seconds . When DUT runs a long duration case (more than 600 seconds ) without any log outputting , the connection is usually dropped by ser2net , and telnet program always prints logs as 'Connection closed by foreign host ' . Anyway thanks for your help ! See https://git.linaro.org/lava/lava-lab.git/tree/shared/server-configs/ser2net… The Linaro lab in Cambridge sets all the ser2net configs to have a zero timeout. > Sincerely, > Chuan Su > > > > > > At 2018-12-18 15:59:00, "Neil Williams" <neil.williams(a)linaro.org> wrote: > >On Tue, 18 Dec 2018 at 06:16, Chuan Su <lavanxp(a)126.com> wrote: > >> > >> Dear all, > >> We are encountered with an issue that our job always exits halfway when running a long duration test case (around 20 minutes) which outputs nothing , and lava server reports an InfrastructureError error and prints as below : > >> Connection closed by foreign host.Marking unfinished test run as failed > > > >Connection closed by foreign host means that the serial connection > >failed at the DUT - this is not a problem in the LAVA test job, this > >is an infrastructure failure at your end. The foreign host (the DUT) > >closed the serial connection. There is nothing LAVA can do about that. > >The serial connection to the DUT has simply failed. > > > >If the serial connection is USB, check for logs on the worker like > >/var/log/messages and /var/log/syslog for events related to the serial > >connection. Check that the DUT didn't simply kill the serial > >connection - maybe the DUT went into some kind of suspend mode. > > > >> definition: lava > >> result: fail > >> case: 0_apache-servers1 > >> uuid: 597_1.4.2.4.1 > >> duration: 603.53 > >> lava_test_shell connection dropped.end: 3.1 lava-test-shell (duration 00:10:05) [ns_s1] > >> namespace: ns_s1 > >> extra: ... > >> definition: lava > >> level: 3.1 > >> result: fail > >> case: lava-test-shell > >> duration: 604.55 > >> lava-test-retry failed: 1 of 1 attempts. 'lava_test_shell connection dropped.'lava_test_shell connection dropped. > >> > >> And we just test it with a very simple python script as below: > >> #!/usr/bin/env python3 > >> import time > >> print('Hello,world!') > >> time.sleep(1200) > >> print("Hello,Lava!") > >> We can see 'Hello,world!' string outputs , but there's no more output of this program found on webUI! > >> We just don't know what's wrong , so we have to mail to you for help! > >> Sincerely, > >> Chuan Su > >> > >> > >> > >> > >> _______________________________________________ > >> Lava-users mailing list > >> Lava-users(a)lists.lavasoftware.org > >> https://lists.lavasoftware.org/mailman/listinfo/lava-users > > > > > > > >-- > > > >Neil Williams > >============= > >neil.williams(a)linaro.org > >http://www.linux.codehelp.co.uk/ > > > > -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 2 months

Default patterns and fixup dicts in Lava-Test Test Definition 1.0

by Neil Williams

https://git.lavasoftware.org/lava/lava/issues/179 If your Lava-Test Test Definition 1.0 YAML files explicitly use a parse: block (like: https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ltp/ltp…) then this will remain supported in Definition 1.0. If you use the monitors or interactive test actions, this does not affect you at all. If you rely on LAVA to create a TestCase based on a command in the Lava-Test Test Definition just echoing "pass" or "fail", then this is the Default Pattern and this change will directly affect those test jobs. The current Default Pattern and Fixup are lifted directly from V1 (https://git.lavasoftware.org/lava/lava/blob/master/lava_common/constants.py…): # V1 compatibility DEFAULT_V1_PATTERN = "(?P<test_case_id>.*-*)\\s+:\\s+(?P<result>(PASS|pass|FAIL|fail|SKIP|skip|UNKNOWN|unknown))" DEFAULT_V1_FIXUP = { "PASS": "pass", "FAIL": "fail", "SKIP": "skip", "UNKNOWN": "unknown", } We've recently updated the documentation to drop mention of the default pattern support for the following reasons: * It has always been problematic to encode a Python regular expression in YAML. Failures are difficult to debug and patterns are global for the entire test operation. * The move towards more portable test definitions puts the emphasis on parsing the test output locally on the DUT using a customised parser. This has further advantages: * The pattern does not have to be mangled into YAML * The pattern can be implemented by a language other than Python * The pattern can change during the operation of the test shell, e.g. a different pattern may be required for setup than for the test itself. We are now starting to plan for Lava-Test Test Definition 2.0 with an emphasis on requiring portable test scripts and removing more of the lava_test_shell Test Helper scripts. Full information on 2.0 will be available early in 2019. As a first step, the generally unhelpful Default Pattern and Default Fixup dict are likely to be removed. If you need this support, the pattern can be added to your Lava-Test Test Definition 1.0 YAML files. In the next release, it is proposed that unless an explicit pattern is specified in the Lava-Test Test Definition 1.0 YAML file, then no pattern will be implemented. Processes which echo "pass" or "fail" would be ignored and no TestCase would be created. Let us know if there are any thoughts or problems on this proposal. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 2 months

Lava job always exits when running a long duration test case without outputs

by Chuan Su

Dear all, We are encountered with an issue that our job always exits halfway when running a long duration test case (around 20 minutes) which outputs nothing , and lava server reports an InfrastructureError error and prints as below : Connection closed by foreign host.Marking unfinished test run as failed definition: lava result: fail case: 0_apache-servers1 uuid: 597_1.4.2.4.1 duration: 603.53 lava_test_shell connection dropped.end: 3.1 lava-test-shell (duration 00:10:05) [ns_s1] namespace: ns_s1 extra: ... definition: lava level: 3.1 result: fail case: lava-test-shell duration: 604.55 lava-test-retry failed: 1 of 1 attempts. 'lava_test_shell connection dropped.'lava_test_shell connection dropped. And we just test it with a very simple python script as below: #!/usr/bin/env python3 import time print('Hello,world!') time.sleep(1200) print("Hello,Lava!") We can see 'Hello,world!' string outputs , but there's no more output of this program found on webUI! We just don't know what's wrong , so we have to mail to you for help! Sincerely, Chuan Su

6 years, 2 months

Git authentication

by Axel Lebourhis

Hi everyone, Is it possible to handle git authentication in a test job ? I need LAVA to clone a repo that can't be set to public, and obviously it won't work because of the authentication step. So is it possible to specify a password or a token ? Best regards, Axel

6 years, 3 months

LAVA webUI hangs when test case outputs long string in a line

by Chuan Su

Dear , all We found that when lava executed a script which may output a long string (more than 30000 bytes) in a line (only one line break), lava web UI always hung and there was no more lava log outputting and devices under test (short for DUT) were still powered until Lava Job time-out function triggered , however, after checked the whole log file we found that cases behind the hanging case were executed (there's new files generated) . So the problem is that when lava encountered those cases lava web UI always hangs and DUTs may not be powered off when all the cases are completed ! best wishes, Chuan Su

6 years, 3 months

LAVA webUI hangs when test case outputs long string in a line

by Chuan Su

6 years, 3 months

Re: [Lava-users] Steps for restoring a backup

by Tim Jaacks

On Mon, 11 Dec 2018 at 11:30, Neil Williams <neil.williams at linaro.org> wrote: > On Tue, 11 Dec 2018 at 11:28, Tim Jaacks <tim.jaacks(a)garz-fricke.com> wrote: > > > > Thanks, the CLI operations are very helpful for automating the process. > > However, the docs say that all devices in "Reserved" state have to > > have their "current job" cleared. I can use "lava-server manage devices details" > > to check whether this field is actually set. There is no command to > > modify it, though. Seems like using the Python API is the only way to > > go here, right? The same applies to setting "Running" jobs to "Cancelled". > > https://git.lavasoftware.org/lava/lava/merge_requests/273 > > This should get into the upcoming 2018.12 release. Thank you very much for your quick help. The "lava-server manage jobs fail" command takes care of clearing the "current job" field of the associated device, do I understand that right? Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks(a)garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS! Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun

6 years, 3 months

MultiNode Tradefed-runner with session retry

by Karsten Tausche

Hi folks, We at Fairphone have developed a variant of the Tradefed-runner in LAVA test-definitions that is meant to run complete Tradefed test suites on multiple devices by making use of the shards feature in Tradefed. The runner is currently in “staging” state. We still want to share now what we are using and developing to see if there are more people with interest in it. Feedback on the general approach taken would also be much appreciated. On the higher level, our setup works as follows: • Use MultiNode to allocate multiple devices for one test submission. • One “master” runs the Tradefed shell, similarly as in the existing runner. • The master connects to the workers’ DUTs via adb TCP/IP. These DUTs are transparently available to Tradefed just in the same way as USB-attached devices. • Workers ensure that their respective DUTs remain accessible to the master, especially in case of WLAN disconnects, reboots, crashes, etc. Major features of our runner: • Support for Android CTS, GTS and STS. • Test run split into “shards” in Tradefed to run tests in parallel on multiple devices. This allows for a major speedup when running large test suites. • Tradefed retry: Rerun test suites until the failure count stabilizes. • No adb root required. • Based on the original Tradefed runner, having at least parts of the common code moved to python libraries. Current limitations: • Test executions are not always stable. This needs further investigation. • Test executions produce more false positives than local test runs. This needs further investigation but is at least partially due to using adb TCP/IP instead of a local USB connection. • Android VTS not implemented (would require only minor changes) Our current changes have been pushed to the tradefed_shards_with_retry topic on Gerrit[1]. Besides the two major changes to add MultiNode adb support and then Tradefed support on top of that, a couple of smaller changes that could be useful on their own have also been pushed. We are looking forward to your feedback and to joint efforts in automating and speeding up Tradefed test executions! Best regards, Karsten for the Fairphone Software Team [1] https://review.linaro.org/q/topic:%22tradefed_shards_with_retry%22+(status:…

6 years, 3 months

Re: [Lava-users] Steps for restoring a backup

by Tim Jaacks

On Mon, 10 Dec 2018 at 20:16, Neil Williams <neil.williams at linaro.org> wrote: > Yes, there is a problem there - thanks for catching it. I think the > bulk of the page dates from the last stages of the migration when V1 > data was still around. I'll look at an update of the page tomorrow. > Step 7 is a sanity check that the install of the empty instance has > gone well, Step 9 is to ensure that the newly restored database is put > into maintenance as soon as possible to prevent any queued test jobs > from attempting to start. The critical element of Step 9 is to ensure > that the lava-master service is stopped. > > The emphasis of the section is on ensuring that the instance only > serves a "Maintenance" page, e.g. the default Debian "It works!" > apache page, to prevent access to the instance during the restore. Thanks for pointing that out, Neil. I got the point, that the Apache server has to serve a static site during the restore process. > Accessing the UI would involve having an alternative way to serve the > pages. If that can be arranged, just for admins, (e.g. by changing the > external routing to the box or redirecting DNS temporarily) then the > UI on the instance can be used with the change that the > lava-server-gunicorn service does not need to be stopped (because > access has been redirected). Other services would be stopped. However, > this would involve a fair number of apache config changes, so is best > left to those admins who have such config already on hand. > > The operations can be done from the command line and that's probably > best for these docs. > > Step 7 can be replaced by: > > lava-server manage check --deploy > > Step 9 can be replaced by looping over: > > lava-server manage devices update --health MAINTENANCE --hostname ${HOSTNAME} > > or, if there are a lot of devices: > > lava-server manage maintenance --force > > (This maintenance helper has been fixed in master - soon to be 2018.12 > - so older versions would use the first command & loop.) Thanks, the CLI operations are very helpful for automating the process. However, the docs say that all devices in "Reserved" state have to have their "current job" cleared. I can use "lava-server manage devices details" to check whether this field is actually set. There is no command to modify it, though. Seems like using the Python API is the only way to go here, right? The same applies to setting "Running" jobs to "Cancelled". > I'll look at changing the page to use CLI operations for steps 7 and > 9. Some labs can do the http redirect / routing method but the detail > of that is probably not in scope for this page in the LAVA docs. I'll > add a note that admins have that choice but leave it for those admins > to implement. Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks(a)garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS! Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun

6 years, 3 months

← Newer
1
...
51
52
53
54
55
56
57
...
92
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users