> For the special case of two threads belonging to the same class, it
> should be relatively simple to hack some "fairness" into the mix. A
> more general solution would be hard, I think.

I wasn't thinking of hacking fairness into the mix, but rather using
some extra signaling between the threads. One thing I have done in
wpa_supplicant_node (wpa_supplicant_node.cpp), in which I had a
similar situation is the following:

- There is a state variable that indicates what state the system is
in. For you that would be Streaming, ReconfigurePending,
StreamingThreadPaused.

- There is a condition variable that the threads use to wake each other up.

- Each thread, instead of just blindly grabbing the lock would do the
following: Possibly change the state, grab the lock, check the state,
and if the state indicates that the other thread should be running,
notify that thread and wait on the condition variable.

In your case, since there are clear bounds on what the time between
frames can be, sleeping for 1ms is probably just fine.

Here are some snippets:
private:
  void waitForMainThreadState(boost::mutex::scoped_lock &lock,
main_thread_states target)
  {
    while (main_thread_state_ != target && !shutting_down_)
    {
      main_thread_cv_.notify_one();
      main_thread_cv_.wait(lock);
    }
  }

  void mainThreadWait(boost::mutex::scoped_lock &lock)
  { // If somebody waiting for us, let them make progress.
    if (main_thread_state_ == MAIN_THREAD_WAITING)
    {
      main_thread_state_ = MAIN_THREAD_PAUSED;
      waitForMainThreadState(lock, MAIN_THREAD_PAUSED);
    }
  }

public:
  bool waitForMainThread(boost::mutex::scoped_lock &lock)
  { // Call with mutex_ held. Returns with main thread waiting.
    main_thread_state_ = MAIN_THREAD_WAITING;
    waitForMainThreadState(lock, MAIN_THREAD_PAUSED);
    main_thread_state_ = MAIN_THREAD_NOT_WAITING;
    return !shutting_down_;
  }

  bool addNetwork(wpa_supplicant_node::AddNetwork::Request &req,
wpa_supplicant_node::AddNetwork::Respons
  {
    ROS_INFO("addNetwork");

    boost::mutex::scoped_lock lock(g_ros_api.mutex_);
    if (!g_ros_api.waitForMainThread(lock))
      return false;


    return true;
  }

  void doWork()
  {
    char buffer[16]; // Any non-zero size is fine here.

    ROS_INFO("doWork()");

    {
      boost::mutex::scoped_lock lock(mutex_);
      mainThreadWait(lock);
    }

    int bytes;
    do {
      bytes = read(pipefd[0], buffer, sizeof(buffer));
      ROS_INFO("doWork read %i bytes from the dummy fifo.", bytes);
    } while (bytes == sizeof(buffer));

    while (1)
    {
      WorkFunction work;
      {
        boost::mutex::scoped_lock lock(mutex_);
        if (work_queue_.empty())
          break;
        work = work_queue_.front();
        work_queue_.pop();
      }
      work();
    }
  }

Cheers,
Blaise