This paper describes audio-visual indoor scene analysis aiming to estimate the speech signals and 3D shapes of an arbitrary number of speakers from a multichannel mixture and multi-view images. An existing vision-driven approach to this task uses 3D Gaussian splatting (3DGS) for visual scene reconstruction, followed by a spatially-regularized multichannel nonnegative matrix factorization (MNMF) for source separation, with the 3D Gaussians treated as sound-emitting primitives. This cascade approach critically relies on clustering the estimated primitives into an appropriate number of speakers for MNMF in advance, making the whole pipeline suboptimal. To overcome this limitation, we propose a statistical framework for joint source separation and shape estimation through primitive clustering, in which the audio and visual modalities mutually inform each other. Specifically, we formulate a unified hierarchical probabilistic model comprising visual and audio likelihoods based on 3DGS and MNMF with a shared prior based on an infinite Gaussian mixture model that jointly defines the primitive positions and source assignments. All parameters are jointly estimated under a maximum-a-posteriori principle, with the effective number of sources automatically determined. Experiments with up to four speakers showed stable separation as the source count increased.