Species differences in the recognition of a series of ergolines by the 5-hydroxytryptamine2 (5-HT2, serotonin2) receptor were investigated in four species, the rat, pig, squirrel monkey and human. In pig frontal cortical membranes the initial studies showed that the ergolines gave shallow displacement curves against [3H]ketanserin binding. The component of [3H]ketanserin binding having low affinity for the ergolines was determined to be the result of [3H]ketanserin binding to alpha-1 adrenergic receptors. Thus, in all subsequent assays prazosin was used to mask [3H]ketanserin binding to alpha-1 adrenergic receptors. Examination of a series of ergolines revealed a distinct pattern in the species selectivity. Compounds that were unsubstituted at the N1 position of the ergoline nucleus showed higher affinity for the pig, squirrel monkey and human 5-HT2 receptors than for the rat. Conversely, compounds that had an N1-isopropyl substituent showed higher affinity for the rat receptor compared to the pig, squirrel monkey and human 5-HT2 receptors. For example, LY53857, a widely used 5-HT2 antagonist, has an isopropyl substituent at position N1 of the ergoline nucleus and exhibited a 4- to 5-fold higher affinity for the rat 5-HT2 receptor, whereas its N1-unsubstituted homologue, LY86057, had more than 10-fold higher affinity for the pig, squirrel monkey and human 5-HT2 receptors. Similar results were seen with three additional ergoline pairs, each having different substituents at the C8 position compared to LY53857. Even an N1-substitution on LY53857 as small as a methyl group, LY108742, resulted in the compound having higher affinity for the rat 5-HT2 receptor compared to the other species.(ABSTRACT TRUNCATED AT 400 WORDS)