The radioligand binding characteristics of 125I-R-(-)4-iodo-2,5-dimethoxyphenylisopropylamine [125I-R-(-)DOI] and 3H-ketanserin were compared in rat and bovine cortical membranes. In rat cortex, 125I-R-(-)DOI labels a relatively low density of binding sites (Bmax = 2.5 +/- 0.2 pmol/gm tissue) with high affinity (KD = 0.63 +/- 0.09 nM). In bovine cortex, specific binding of 125I-R-(-)DOI represents less than 20% of total binding at radioligand concentrations above 0.6 nM, and, therefore, the data cannot be analyzed adequately by Scatchard transformation. By contrast, 3H-ketanserin displays saturable, specific high-affinity binding in both rat cortex (KD = 1.0 +/- 0.1 nM; Bmax = 11 +/- 0.4 pmol/gm tissue) and bovine cortex (KD = 1.2 +/- 0.2 nM; Bmax = 5.3 +/- 0.4 pmol/gm tissue). Ki values for 30 drugs were determined for 125I-R-(-)DOI-labeled sites in rat cortex and 3H-ketanserin-labeled sites in bovine cortex. 5-Hydroxytryptamine (5-HT) displays 250-fold higher selectivity for the 125I-R-(-)DOI-labeled sites (Ki = 3.0 +/- 0.7 nM) than for the 3H-ketanserin-labeled sites (Ki = 750 +/- 50 nM). Structural congeners of R-(-)DOI display 80- to 160-fold higher affinity for the 125I-R-(-)DOI binding site than for the 3H-ketanserin-labeled binding site. d-LSD and putative 5-HT2 antagonists are approximately equipotent at both sites. Significant correlations were found between drug affinities for 125I-R-(-)DOI-labeled sites in rat cortex and putative 5-HT2A sites labeled previously by 77Br-R-(-)DOB (r = 0.93, p less than 0.01), putative 5-HT2B sites labeled by 3H-ketanserin in bovine cortex (r = 0.63, p less than 0.01), and 5-HT1C binding sites that have been characterized by other investigators (r = 0.78, p less than 0.01). No significant correlations were found between drug affinities for 125I-R-(-)DOI-labeled sites in rat cortex and 5-HT1A, 5-HT1B, 5-HT1D, or 5-HT3 sites, as determined by previous investigators. We conclude that 125I-R-(-)DOI labels a novel 5-HT binding site subtype (tentatively designated the 5-HT2A binding site) that is present in rat cortex but is either absent or minimally present in bovine cortex. By contrast, 3H-ketanserin labels both the putative 5-HT2A site in rat cortex as well as a separate, distinct recognition site that is present in both rat and bovine cortex, tentatively designated the 5-HT2B site.