Background: Although aggression is conceptualized as a dimensional construct with violent behavior representing the extreme end of a spectrum, studies on the involvement of personality traits in human aggression have typically only included data representing a restricted spectrum of aggressive behaviors. Methods: In the current study, we therefore examine whether trait aggression is associated with five-factor model personality traits in an enriched sample of 259 men with a broad continuum of trait aggression, ranging from very low to very high including 39 incarcerated aggressive violent offenders. All participants completed the NEO Personality Inventory-Revised (NEO PI-R) and the Buss–Perry Aggression Questionnaire (BPAQ). The association between each of the five NEO PI-R personality traits and trait aggression (BPAQ) was investigated using five linear regression models, covarying for group status, age and educational level. Results: Higher BPAQ scores were positively associated with Neuroticism and negatively associated with Agreeableness and Conscientiousness. Conclusion: Our results indicate that those high in Neuroticism and low in Agreeableness and Conscientiousness are at higher risk of exhibiting aggressive behavior, underlining the relevance of these higher order personality traits in understanding aggressive behavior. We argue that studying individual personality differences should be offered a greater attention within violent and criminal behaviors.