Testosterone influences the brain via organizational and activational effects. Numerous relevant studies on rodents and a few on humans focusing on specific behavioral and cognitive parameters have been published. The results are, unfortunately, controversial and puzzling. Dosing, timing, even the application route seem to considerably affect the outcomes. In addition, the methods used for the assessment of psychometric parameters are a bit less than ideal regarding their validity and reproducibility. Metabolism of testosterone contributes to the complexity of its actions. Reduction to dihydrotestosterone by 5-alpha reductase increases the androgen activity; conversion to estradiol by aromatase converts the androgen to estrogen activity. Recently, the non-genomic effects of testosterone on behavior bypassing the nuclear receptors have attracted the interest of researchers. This review tries to summarize the current understanding of the complexity of the effects of testosterone on brain with special focus on their role in the known sex differences.