Hi All,
There have been discussions about virtio-camera before and more recently I've heard the term virtio-sensor used. I think using "sensor" eludes to the fact that there are a whole class of devices that provide some sort of 2d plane view of the world (cameras, fingerprint readers, LIDAR?) that would benefit in being consumed by a workload in a standard non-bespoke way.
Why not virtio-video? =====================
There is already a specification and various implementations of virtio-video in various states of up-streaming. It is tempting to think of a camera as a simplified subset of processing video streams. However while virtio-video allows the consumption and display of various video formats it offers no direct control of the source itself.
Complex control plane =====================
Modern cameras are more than a simple CCD recording photons. Aside from controlling things like f-stop/exposure/position there are also more complex computational photography aspects. The camera soc might be capable of doing edge or object detection or even facial and feature recognition. Cameras are no longer simple webcams and have long since gone past the relatively simple API that V4L present (c.f. libcamera).
Cloud native ============
One of the drivers for these virtio devices is the concept of cloud native development. That is developing your workload in the cloud and feeding it data through standardised VirtIO interfaces. Once you are happy with its behaviour you can take that workload and run the same binaries in your edge device but this time with data being provided by a real sensor which is exposed via the same VirtIO interface.
Competing Requirements? =======================
I've heard about use cases across a wide range of deployment scenarios including:
Virtualised Mobile Devices
Here the backend containing the vendors secret sauce exists in its own isolated VM with access to the real camera HW and exports virtio-camera to a standardised main OS.
Desktop Virtualisation
Here the aim is to expose host camera devices (such as webcams) to guest system which would be an otherwise sandboxed VM that needs access to the system camera for a particular task.
Automotive
Cars are rapidly growing cameras both as driver aids and for more advanced use cases such as self driving. The cloud native case is particularly strong here as a lot of validation and iteration will be taking place in the relatively limitless resources of the cloud before being run in the carefully controlled and isolated safety critical environs of the car itself.
Do these use-cases have competing demands? Can a solution be found that satisfies all of them?
So for the next Stratos sync-up call I'd like to discuss virtio-camera and if there is enough interest to specify a work package to define and upstream the device. I'm casting a wide net for people who are interested in the topic so we can talk through the issues and see if we can arrive at consensus for a minimal viable product.
To help with that I would welcome people submitting ahead of time any:
- use-cases and user stories - concrete requirements
and also any:
- previous work and RFCs
Please forward this email to anyone else who you think might find this discussion relevant.
The next Stratos meeting will be on the 14th October, 15:00 UTC / 16:00 BST @ https://meet.google.com/rct-xgqm-woi
The meeting notes will be:
https://linaro.atlassian.net/wiki/spaces/STR/pages/28771778789/2022-10-14+Pr...
And as ever the project jumping off page is at:
https://linaro.atlassian.net/wiki/spaces/STR/overview
Thanks,