I almost had my Nintendo AI agent working, but the memory leak problem of Torch7’s image.display() API really bugged me. This memory leak not only caused my program to crash after a while, it also increased system (CPU) loading because the system needed to spend extra effort to manage its swap space. I decided to spend time to implement a solution to this problem before I really start to train the Nintendo AI agent.

I’m not familiar with graphics libraries on Linux, but I happen to know OpenCV pretty well. OpenCV offers a really simple API for displaying image/video, cv::imshow(). And the good thing is that OpenCV 2.4.13 is readily available on Jetson TX1 (installed by JetPack-2.3.1). So I think I could just wrap this OpenCV API with Lua FFI to solve my problem.

I’ve only tried Lua FFI binding to C (not C++) so far. So my first try is to use OpenCV’s C API and program the corresponding Lua FFI glue layer. Most of the OpenCV example code are in C++, and there are really few examples in C. It took me quite a while to figure out all the right OpenCV C API’s for my purpose, e.g. cvShowImage() instead of cv:imshow(), and CvMat instead of cv::Mat.

The resulting C code, imshow.c, looks like the following.

#include <stdlib.h>
#include <string.h>
#include "opencv/highgui.h"

static char imshow_name[128];  /* name (title) of the display window */

#if 0
int  imshow_init(const char *name, int len);
void imshow_display(unsigned char *buf, int w, int h);
void imshow_cleanup();
#endif /* 0 */

static void bye(void)
{
        cvDestroyAllWindows();
}

int imshow_init(const char *name, int len)
{
        atexit(bye);
        if (len > 127)  len = 127;
        strncpy(imshow_name, name, len);
        cvNamedWindow(imshow_name, CV_WINDOW_AUTOSIZE);
        return 0;
}

/* Display 1 image frame (grayscale) */
void imshow_display(unsigned char *buf, int w, int h)
{
        CvMat mat = cvMat(h, w, CV_8UC1, buf);
        cvShowImage(imshow_name, &mat);
        cvWaitKey(1);
}

void imshow_cleanup()
{
        bye();
}

As usual, the full implementation could be found in my GitHub repository: https://github.com/jkjung-avt/dqn-tx1-for-nintendo/tree/master/imshow.

My current implementation only works for 8-bit grayscale image/video with ‘progressive’ data layout. But I think it should be easy to extend the code to support other image/video formats including, say, RGB24 or UYVY.

Based on my testing, my imshow FFI module not only solves the memory leak problem, but also reduces memory usage (a dozen MB or so) comparing to Torch7’s image.display(). I’m sticking with this OpenCV imshow() solution for displaying image/video under Torch7 from now on.