The speech perception abilities of adults exhibit a curious property. On the one hand, it is well known that the acquisition of certain non-native phonetic contrasts is very difficult for adults, suggesting that mature phonetic category representations are somewhat inflexible. On the other hand, listeners appear to be able to adjust to acoustic variability in their own language. These sources of variability are substantial: slips of the tongue produce imperfect or ambiguous speech tokens, environmental noise may occlude the speech signal, and different talkers, for reasons of idiolect or dialect, implement speech sounds in ways that consistently deviate from ‘standard’ pronunciations. In the present line of research, we use fMRI to assess the degree to which the neural systems that underlie phonetic processing demonstrate evidence of both flexibility and stability. Specifically, we ask whether the same system is implicated in the acquisition of non-native speech contrasts and adjustments to variability in the native-language speech signal. These results suggest a mechanism by which stable phonetic representations may be modified at both short and long time scales.