Models of category perception can generally be broken into two different types. Under rule-based models, categories are the result of a list of rules along the lines of a checklist. Under exemplar-based models, meanwhile, categories are the sum total of the experiences with the members of a certain category. In this study, listeners heard phonetic tokens on a continuum of palatal to velar fricatives, which were paired with one of either two or three categories. The pairings of continuum members to categories varied from condition to condition. In the end, though listeners could learn certain pairings of phonetic tokens to categories, they found learning several of the mappings from speech sounds to categories impossible. Instead, most ignored their experience and defaulted to a division of the continuum similar to that used by German speakers who do have the sounds in their native language. This indicates that listeners may come to the task of categorizing phonetic tokens with strong expectations for how the sounds should be categorized. This contrasts with the predictions of exemplar-based models, under which listeners may only use their direct linguistic experience to assist in categorizing sounds.