The OECD's Programme of International Student Assessment known as PISA, including the Gender Report published in 2015, is a well-received project and its findings keep being transferred into educational policy development. Its quantitative methodology and the sheer volume of data gathered from around 5,100,000 students from 65 countries seem to guarantee scientific rigour and hence, credibility. In this article, we critically examine six topics: (1) The gender concept the authors employ, (2) implicit hypotheses of compensation (historical, internal, and external), (3) descriptive and correlational statistics and its usage for causal explanations, (4) questions of external validity, (5) the concepts of stereotypes and role models and (6) the authors' concept of innate sex differences. We will show that the PISA Gender Report is based not on well-defined notions of sex and gender, but on what might be called an 'intuitive' gender concept, thus leaving room for theoretically unfounded and mostly implicit hypotheses and subsequent ad hoc conclusions. This has led to an interpretation of the findings that sustains a gender dichotomy and reinforces stereotypes of the helpless, social female on the one hand and the self-orienting, natural male on the other.