Sexual dimorphism in immune responses is well documented, but the underlying mechanisms remain incompletely understood. Here, we identified a subset of corticotropin-releasing hormone (CRH) neurons that express androgen receptors (ARs) as key mediators of sex differences in restraint-induced immunosuppression. Mechanistically, androgens directly activate AR-positive CRH neurons, enhancing the hypothalamic-pituitary-adrenal axis activation. This results in elevated corticosterone levels in response to restraint stress, leading to increased immune cell apoptosis and immune organ atrophy in male mice. Conditional knockout of ARs in CRH neurons eliminated this sexual dimorphism, highlighting ARs in CRH neurons as pivotal regulators of sex-specific immune responses to stress.