There were never many girls at all at the football games.
The year is pretty unspecified throughout the novel, but we can infer from the text as a whole that the book takes place sometime between 1940 and 1949. Single-sex education was very popular in private prep schools around this time period.
To help improve the meaning of these lyrics, visit "The Catcher in the Rye (Chap. 1)" by J.D. Salinger and leave a comment on the lyrics box