[ros-users] image_proc crop and decimate (binning) nodelet and bayer

Tue Oct 25 22:58:39 UTC 2011

Hi all

Al 12/10/11 06:33, En/na Patrick Mihelich ha escrit:
> Hi Joan,
>
> Thanks for the contribution, and sorry for the delayed response. I've 
> only now found time to digest everything. If you're interested in the 
> image_pipeline, you should consider joining the Perception Pipelines 
> SIG <http://www.ros.org/wiki/fuerte/Planning/Perception%20Pipelines>, 
> or at least its mailing list.
>
I was not aware of that list. Thanks!
> I created a ticket to track your code and suggestions: 
> https://code.ros.org/trac/ros-pkg/ticket/5201. Further comments below.
>
If it is better to discuss it there, let me know.
>
>     If the input image encoding is bayer, this nodelet performs
>     crop/binning and debayering.
>     We would like to discuss with the ROS community if it would be
>     more convenient perform only the crop/binning, and do NOT perform
>     the debayering. That debayering process, with possibly other
>     image_proc tasks, could be performed by another image_proc node in
>     the subsampled image topic namespace. From our point of view, it
>     will have two main advantages:
>
>        1. It will allow more complex debayering methods to be aplied 
>           to the scaled images by image_proc, with zero mantainment
>           cost of the crop/binning nodelet.
>
> I have problems with this. If you're doing 2x2 decimation of a Bayer 
> image, you have all the information to transform a 2x2 Bayer cell into 
> one RGB pixel right there. The "debayering" is trivial, no 
> interpolation needed whatsoever, and you lose very little information 
> (just from averaging the 2 green pixels in each cell).
>
> With your approach, you throw away 3/4ths of the information up front, 
> and hope that a complex (hence expensive) debayering algorithm can 
> fill some of that back in. You burn a lot more CPU and, no matter how 
> sophisticated your algorithm, you won't get results as good as the 
> trivial downsample-to-RGB approach.
I am aware of that and agree on the lost of information, but that it is 
what decimate means. I mean that from my point of view it is an issue of 
keeping concepts separated. A 2x2 rude subsampling of a Bayer image into 
an RGB image is a very particular case of image processing, not a 
decimate process. And that no interpolation is needed whatsoever is 
arguable.
>
>        1. It will allow subsampling bayer images to reduce its size
>           and send them through the network without losing the color
>           information (if sended in mono) nor triplicating the
>           required bandwith (if sended in color)
>
> True, full RGB takes 3x the bandwidth of Bayer. Then again, imposing a 
> Bayer pattern is a pretty simplistic compression method. We have 
> image_transport for applications where bandwidth is limited.
The scaling question in our case arose from the fact that we could not 
use compression to send images through the network. Our camera was 10 
frames/sec bayer, the bandwidth allowed only 7 frames/sec at full 
resolution without compression, and only 3 frames/sec with compression. 
We supposed that this was due to the limited power of the CPU compared 
to the demands of the compression method.
>
>     The subsampling is quite easy to implement. It just changes
>     depending on the encoding. Subsampling of mono and color encoded
>     images is quite intuitive. In Bayer encodings, the idea is to
>     consider each set of 2x2 pixels as a color cell, and these color
>     cells are the subsampling units. The resulting image has the same
>     encoding than the original one.
>
>
> As implemented using decimation, you'll get bad aliasing artifacts 
> when downsampling Bayer images, even in the 2x2 case where you have 
> the information to do it perfectly. For each set of 4 rows/columns you 
> throw out 2 adjacent ones.
>
> Cameras that support "color binning" in hardware do something like 
> this, but using binning/subsampling (averaging blocks of Bayer cells) 
> instead of decimation (throwing away all but one). In that case the 
> aliasing is mitigated.
I agree completely. The discussion on that follows from the original 
post mail in the next lines:

>     It should be said that this subsampling is the crudest one, and it
>     can introduce artifacts in the resulting image. Another completely
>     different approach will be to use OpenCV resizing functions to do
>     the job. In that case, as in the debayering process, a set of
>     methods could be supported with an extra dynamic_reconfigure
>     parameter.
>
>
> That's true. The current decimation approach is basically equivalent 
> to cv::resize using the INTER_NEAREST (nearest-neighbor) method, which 
> is fast but indeed crude. We could get much better output from 
> INTER_AREA, which gives Moire-free results. A parameter to switch 
> between at least those two options would be useful.
>
>     I think that last OpenCV versions only understand images as bgr8
>     encoded cv::Mat's (even mono images), so the approach will be:
>
>        1. subscribe to the color image
>        2. subsample it to a color scaled image
>        3. from this color scaled image build the mono scaled image if
>           necessary
>
>     And it won't be possible to scale a bayer image, debayer it first
>     will be mandatory. Is there any OpenCV expert that can confirm or
>     refutate these statements? Any information about the resizing
>     functions will be appreciated, since the documentation is quite poor.
>
>
> The very latest documentation for cv::resize is here 
> <http://opencv.itseez.com/modules/imgproc/doc/geometric_transformations.html#cv.Resize> 
> and seems adequate to me. I believe it handles one-channel and 
> three-channel 8- or 16-bit images just fine.
>
> For Bayer, I would still do the 2x2 decimation to RGB (almost 
> lossless), and then use cv::resize for any further decimation.
>

I am working on that version. My first approach is to create a nodelet 
that, depending on the subscribers, takes as inputs a camera_info topic 
and an image_mono, image_color, image_rect and/or image_rect_color topic 
using a TimeSynchonizer. Then the needed images are subsampled with 
cv::resize from OpenCV, and published all together with the adapted 
camera_info under a different namespace (the scaled camera namespace). 
The TimeSynchronizer trick is needed to provide just one camera_info 
message for all the scaled images. Think of it as simulating a whole new 
camera with ROI/binning in a new namespace.

Does anybody know how to handle the ROI with the scaling in OpenCV? Just 
defining an ROI over the original image before scaling?

It should be noted that in this approach Bayer images are not handled, 
and that an image_proc should be running on the original camera 
namespace to provide the needed full size images. Another image_proc in 
the scaled namespace will NOT be needed. Alternatively, rectification 
and RGB to mono nodelets may be launched only in the scaled namespace, 
provided that a debayering node is running in the original camera 
namespace if the camera is bayer. Any idea of what is more 
accurate/efficient?

Right now it is a work in progress. The synchronization of all the 
topics based on the number of subscribers is a little bit tricky. As 
soon as it is complete I will attach the code too, if you think it is worth.

-- 
Joan Pau Beltran
Grup de Sistemes, Robòtica i Visió - DMI
Universitat de les Illes Balears
Ctra. Valldemossa, km 7.5 07122 Palma
Campus Universitari, edifici Anselm Turmeda
Telf (+34) 971 17 28 13
Fax (+34) 971 17 30 03

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ros.org/pipermail/ros-users/attachments/20111026/656558b8/attachment-0004.html>