Aging is accompanied by declines in executive control abilities and changes in underlying brain network architecture. Here, we examined brain networks in young and older adults during a task-free resting state and an N-back task and investigated age-related changes in the modular network organization of the brain. Compared with young adults, older adults showed larger changes in network organization between resting state and task. Although young adults exhibited increased connectivity between lateral frontal regions and other network modules during the most difficult task condition, older adults also exhibited this pattern of increased connectivity during less-demanding task conditions. Moreover, the increase in between-module connectivity in older adults was related to faster task performance and greater fractional anisotropy of the superior longitudinal fasciculus. These results demonstrate that older adults who exhibit more pronounced network changes between a resting state and task have better executive control performance and greater structural connectivity of a core frontal-posterior white matter pathway.