It is proposed that feedback signals from the prefrontal cortex (PFC) to extrastriate cortex are essential for goal-directed processing, maintenance, and selection of information in visual working memory (VWM). In a previous study, we found that disruption of PFC function with transcranial magnetic stimulation (TMS) in healthy individuals impaired behavioral performance on a face/scene matching task and decreased category-specific tuning in extrastriate cortex as measured with functional magnetic resonance imaging (fMRI). In this study, we investigated the effect of disruption of left inferior frontal gyrus (IFG) function on the fidelity of neural representations of two distinct information codes: (1) the stimulus category and (2) the goal-relevance of viewed stimuli. During fMRI scanning, subjects were presented face and scene images in pseudo-random order and instructed to remember either faces or scenes. Within both anatomical and functional regions of interest (ROIs), a multi-voxel pattern classifier was used to quantitatively assess the fidelity of activity patterns representing stimulus category: whether a face or a scene was presented on each trial, and goal relevance, whether the presented image was task relevant (i.e., a face is relevant in a "Remember Faces" block, but irrelevant in a "Remember Scenes" block). We found a reduction in the fidelity of the stimulus category code in visual cortex after left IFG disruption, providing causal evidence that lateral PFC modulates object category codes in visual cortex during VWM. In addition, we found that IFG disruption caused a reduction in the fidelity of the goal relevance code in a distributed set of brain regions. These results suggest that the IFG is involved in determining the task-relevance of visual input and communicating that information to a network of regions involved in further processing during VWM. Finally, we found that participants who exhibited greater fidelity of the goal relevance code in the non-disrupted right IFG after TMS performed the task with the highest accuracy.