The prefrontal cortex (PFC) is proposed to be the source of top-down signals that can modulate extrastriate visual processing in accordance with behavioral goals, yet little direct causal evidence for this hypothesis exists. Using theta burst transcranial magnetic stimulation, we disrupted PFC function in human participants before performing a working memory task during fMRI scanning. PFC disruption decreased the tuning of extrastriate cortex responses, coinciding with decrements in working memory performance. We also found that activity in the homologous PFC region in the nonstimulated hemisphere predicted performance following disruption. Specifically, those participants with greater homologous PFC activity and greater connectivity between this region and extrastriate cortex were the most resistant to PFC disruption. These findings provide evidence for a compensatory mechanism following insults to the brain, and insight into the dynamic nature of top-down signals originating from the PFC.