This paper describes joint separation and tracking of multiple moving speakers using distributed microphone arrays and cameras. A promising approach is to use a spatially-regularized extension of multichannel nonnegative matrix factorization (MNMF), a blind source separation method, that allows the spatial covariance matrices (SCMs) of speakers to vary over time according to inertial trajectory priors. This audio-only method, however, typically fails to track speakers when they are silent. To overcome this limitation, we extend the model to a unified audio-visual probabilistic model for reliability-aware integration of visual cues. More specifically, our method treats the 2D position of each speaker, estimated from camera images by a pretrained object detector (YOLO), as a noisy observation. The visual likelihood is a Gaussian mixture on the 2D image plane centered at the projection of the 3D speaker position, with two components representing inlier and outlier position estimates, respectively. The audio signals, time-varying SCMs, and speaker positions are jointly estimated under the maximum-a-posteriori principle, while inferring the reliability of object detection. Experiments on simulated audio-visual data of two moving speakers with crossing trajectories showed that the proposed method achieved successful separation and tracking in the challenging scenario where the audio-only and cascade baselines failed.