There is a substantial literature describing how, over the course of development, infants become more sensitive to differences between native phonemes (sounds that are both present and meaningful in the native language) and less sensitive to differences between non-native phonemes (sounds that are neither present nor meaningful in the input). A more difficult problem is what to do with allophonic contrasts, dimensions that vary in the input but are not recruited for lexical encoding. For example, the vowel in "I'm" should map to the same underlying category as the vowel in "I'd", despite the fact that the first one is nasalized and the second oral. Because such pairs of sounds are processed differently than those that map onto different phonemes by adult speakers, the learner must come to treat them differently as well. Interestingly, there is some evidence that infants' sensitivity to dimensions that are allophonic in the ambient language declines as early as 11 months. I summarize behavioral research, corpora analyses, and computational approaches that shed light on how monolingual and bilingual infants achieve this feat at such a young age. Collectively, this work suggests that the computation of complementary distribution, the calculation of phonetic similarity, and protolexical knowledge could operate in concert to guide infants towards a functional interpretation of sounds that are present in the input, yet not lexically contrastive.