Libcamera

This is Google Summer of Code 2021 project under Libcamera aimed at achieving:

1. Add support to Libcamera’s gstreamer element to support multistreaming.
2. Add unit test to test gstreamer element functionality in Libcamera.

Project Details

• Name: Vedant Paranjape
• Mentors: Paul Elder, Nicolas Dufresne
• Organization: Libcamera
• Code: https://git.libcamera.org/libcamera/libcamera.git/
• Project Page: https://summerofcode.withgoogle.com/projects/#6332672818806784
• Progress log: https://ve0x10.in/gsoc2021/
• Midway Blog: https://ve0x10.in/blog/2021/07/19/gsoc-libcamera.html

Brief overview

The GStreamer libcamera element allows libcamera to be used in GStreamer pipelines. Libcamera supports simultaneous streaming, for example having lower quality one stream for preview, while another higher quality one is used for recording. This project extends support for the GStreamer libcamera element to support this multistreaming.

Usage

Multistream is supported on devices with IPU3, rkisp1 and raspberrypi camera pipeline. This will only work on those devices.

1. Install libcamera and gstreamer using this guide: https://libcamera.org/getting-started.html

2. If gstreamer element isn’t installed system wide, set the environment variable specifying the location:

   export GST_PLUGIN_PATH=$(pwd)/build/src/gstreamer
   

3. Install gst-plugins-bad as follows

   sudo apt install gstreamer1.0-plugins-bad
   

4. Run the following command on the terminal

   gst-launch-1.0 libcamerasrc name=src src.src ! queue ! videoconvert ! autovideosink src.src_0 ! queue ! videoconvert ! autovideosink
   

If it runs correctly, you should see such two windows with camera output as follows:

Examples

This functionality can be done programmatically, the example codes are here

1. Single Stream
2. Multi Stream
3. Using Device Provider

Work Summary

All the patches submitted by me that merged can be seen here

1. gstreamer: Add error checking in gst_libcamera_src_task_enter()
2. libcamera: Add OV5647 sensor properties
3. gstreamer: Added virtual functions needed to support request pads
4. libcamera: Fix the V4L2 pixel format for formats::XBGR8888
5. libcamera: Add pixel format BGRX8888
6. gstreamer: Store group_id in GstLibcameraSrcState
7. gstreamer: Update format specifier in Request Pad template
8. test: gstreamer: Add test for gstreamer single stream

Notes

• Use the following command to verify working of multistream on Raspberry Pi 4B+ using Raspberry Pi Cam v1

    gst-launch-1.0 libcamerasrc camera-name="/base/soc/i2c0mux/i2c@1/ov5647@36" name=src src.src ! queue ! videoconvert ! autovideosink src.src_0 ! queue ! videoconvert ! autovideosink`
  

• Use the following command to compile gst code

    gcc basic-tutorial-3.c -o basic-tutorial-3 `pkg-config --cflags --libs gstreamer-1.0`
  

• Use the following to view camera using libcamerasrc

    gst-launch-1.0 libcamerasrc camera-name="Camera 1" ! videoconvert ! autovideosink
  

• Use the following to export gstreamer libcamera element, execute this in libcamera root path

    export GST_PLUGIN_PATH=$(pwd)/build/src/gstreamer
  

Conclusion

All goals that were originally proposed have been completed. I will continue to support this project in the future, take care of any bugs, issues and will continue implementing additional features whenever required.

It has been a really great experience for me. I have learned so much in these 10 weeks, which will surely help me achieve my future goals in life. I would like to thank libcamera and Google for giving me such an amazing opportunity. Furthermore, I would also like to thank my mentors Paul and Nicolas for helping me out and supporting me throughout the project, also the members of Libcamera community, namely Laurent, Kieran, Umang and Jacopo.

On an ending note, if my words or actions hurt anyone, I am extremely sorry !