sony-dev-board

Spresense camera with GPS overlay

Using a Spresense main and extension board together with a camera and display to fetch GPS signals and overlay the GPS data over the camera video stream. The video stream is then showed on a TFT display.
Components below are available through RS online.
Product page link.

Parts List

QtyProductPart number
1×Spresense Main Board178-3376
1×Spresense Extension Board178-3377
1×ST7735 TFT display
See more products

This example uses Sony’s Spresense board with the new multicore microcontroller CXD5602GG featuring 6 ARM Cortex-M4 cores and an integrated GNSS receiver among other features. In this project the GNSS module is used to acquire the current position simultaneously as the camera is streaming video to the display. To view the video a 160×128 TFT display with a ST7735 SPI display driver is used. This example is written using the Spresense board package in the Arduino IDE.

The display

The display driver used is initially the standard driver provided as a library in the Arduino IDE that has been ported for Spresense. It also got a new function enabling most of the functions to operate on a framebuffer instead of directly writing operation directly to the display. This method saves a lot of overhead writes over the SPI bus and in the end increases the frame rate from 6-7 fps to +20 fps. Using this framebuffer method removes all flickering that can occur when writing shapes and text to the display which are overwritten with a new image every frame.

Display schematics

Image: Display connection schematic.

Camera

The camera is initiated during the setup phase to use two buffers. Using two buffers instead of one has shown to increase the frame rate. The tradeoff is of course that twice amount of memory is used. Each time a frame has been captured by the camera a callback function will be called in the sketch. This callback is specified in the startStreaming() function. Pushing the raw captured buffer directly to the display have show a distorted picture so a few things have to be done to get the captured data in a compatible format.

First, the captured data is stored in YUV422, so this has to be converted to 16 bit RGB format. Function for this are provided by the camera library: convertPixFormat(CAM_IMAGE_PIX_FMT_RGB565);

Secondly, the captured image of 340×240 is too large for the 160×128 pixel display. Instead of rescaling the image I have taken a shortcut and just re arranged the pixel data for the new smaller size, this means that the exceeding pixels are not shown.

Thirdly, seems that dumping this data to the display still looks funny. Could this be an issue with endianness? Simply swapping the endianness for the uint16_t type does the job.

Fourthly, the initial display driver configures the display for RGB, changing this to BGR will give the result we’re after. Now the stream from the camera comes out just like it should!

GNSS

The GNSS library is polled. It has to be initiated and started but after that it is up to the user to ask for information. There are a few points that are important to get good result.

During the setup phase, it is necessary to specify what satellite types to acquire signals from. This depends on where on the world you are at the moment. The table below lists where the supported systems are available.

GPSGLONASSQZSS-L1C/AWAASQZSS-L1S95% AccuracyEffective place
x<5munder open sky
xx<5min East Asia and Oceania
xx<7.8min the city
xxx<7.8min the city of East Asia and Oceania
xx<2min North America
xxx<2min Japan

Libraries

Three libraries are used in this project.

  • GNSS
  • Camera
  • TFT

GNSS and Camera are provided in the Spresense package so no installation is needed.

But the TFT drivers have to be downloaded and installed into the Arduino IDE build system. One way of doing this is simply to download the package directly from GitHub:

https://github.com/TE-KarlKomierowski/TFT/zipball/master

In the Arduino IDE click [Sketch->Include library->Add .zip library]. Select the zip file you just downloaded. This method will work, if you’re lucky. If you already have a TFT library installed you might run into problems since the currently installed drivers might be selected prior the downloaded ones by the Arduino IDE.

Renaming the folder where the new drivers are stored might help the Arduino IDE locating the files.

In Ubuntu renaming the folder could be done with the following command:

mv -v ~/Arduino/libraries/TE-KarlKomierowski-TFT-*/ ~/Arduino/libraries/TFT

Post author: KarlKomierowski

Share on facebook
Facebook
Share on twitter
Twitter
Share on reddit
Reddit
Share on linkedin
LinkedIn
Share on email
Email

Leave a Comment

Your e-mail address will not be published. Required fields are marked *

five + 12 =