Working memory refers to a system involved in the online maintenance and manipulation of information in the absence of external input. Due to the importance of working memory in higher-level cognition, a wealth of neuroscience studies has investigated its neural basis. These studies have often led to conflicting viewpoints regarding the importance of the prefrontal cortex (PFC) and posterior sensory cortices. Here, we review evidence for each position. We suggest that the relative contributions of the PFC and sensory cortices to working memory can be understood with respect to processing demands. We argue that procedures that minimize processing demands lead to increased importance of sensory representations, while procedures that permit transformational processing lead to representational abstraction that relies on the PFC. We suggest that abstract PFC representations support top-down control over posterior representations while also providing bottom-up inputs into higher-level cognitive processing. Although a number of contemporary studies have studied working memory while using procedures that minimize the role of the PFC, we argue that consideration of the PFC is critical for our understanding of working memory and higher-level cognition more generally.