The steered response power with phase transform (SRP-PHAT) is widely used for acoustic source localization because of its robustness in reverberant environments. It is typically implemented in the time domain (TD) via generalized cross-correlation (GCC). However, TD implementations suffer from coarse delay quantization in compact arrays, while frequency-domain (FD) SRP-PHAT improves resolution by applying fractional-delay steering at each frequency bin for every candidate direction, at the expense of substantially higher computational cost. We propose a grid-free direction-of-arrival (DOA) estimator that uses per-bin PHAT cross-spectra and replaces analytical steering and SRP-map evaluation with a lightweight depthwise-separable convolutional neural network (CNN) that directly regresses azimuth and elevation. The proposed architecture processes local frequency structure while treating microphone-pair phase features as channels, thereby avoiding an explicit spatial grid at inference time. Experiments on the TAU-NIGENS Spatial Sound Events 2020 dataset show lower mean absolute angular error than FD-SRP-PHAT, higher accuracy at stringent thresholds, competitive performance at intermediate thresholds, and substantially lower computational cost.