Hey Kurt,
In playing around with the vslam_system a bit, I found the source of a big memory leak in frame_common (was leaking an image size buffer on each frame).

The destructor for the DenseStereo object was not being called since the pointer to was of type FrameStereo

I've attached a diff for the changes to the file
wg-ros-pkg-unreleased/vision/frame_common/src/frame.cpp
that fixes the memory leak

The memory usage still seems to grow slowly over time when running just the VO (no place recognition or pose graph optimization), so there may be a small leak somewhere else but this one was the major one...

-=Abe