We present DECAF, a privacy-preserving neural speech codec that obfuscates a speaker's voice while preserving linguistic content while maintaining automatic speech recognition (ASR) performance at very low bitrates, inspired by decaffeination. At the transmitter end, speech is encoded into speaker-independent content embeddings, which are compressed using residual vector quantization and transmitted without any speaker-related information. At the receiver, a canonical speaker embedding, shared a priori between endpoints, is used for waveform reconstruction, enabling deterministic and consistent obfuscation of a speaker's voice. The proposed framework leverages an information bottleneck applied to self-supervised representations, along with a separate speaker embedding branch, to achieve effective speaker–content disentanglement. We further incorporate a CTC-based auxiliary objective, encouraging content representations that are well aligned with downstream ASR tasks. We show that DECAF operating at a bit rate of 0.5~kbps achieves an Equal Error Rate (EER) of up to 43.5\% for a speaker verification system, while maintaining competitive ASR performance, yielding a relative reduction in word error rate of 33.2\% compared to a state of the art method.