[3H]Serotonin (5-hydroxytryptamine, [3H]5-HT) was used as a radioligand probe of brain 5-HT receptors in homogenates of human cortical tissue. Two binding sites were detected in the presence of 1 microM pindolol (to block 5-HT1A and 5-HT1B receptors), and 100 nM mesulergine (to block 5-HT1C and 5-HT2 receptors). One of these sites demonstrated high affinity for 5-carboxyamidotryptamine (5-CT) and ergotamine, consistent with the known pharmacology of the 5-HT1D receptor; the second site demonstrated low affinity for 5-CT and ergotamine. Computer-assisted analyses indicated that both drugs displayed high affinities (Ki values of 1.1 nM and 0.3 nM for 5-CT and ergotamine, respectively) for 55% of the sites and low affinities (Ki values of 910 nM and 155 nM for 5-CT and ergotamine, respectively) for 45% of the sites. To investigate the non-5-HT1D component of the binding, 100 nM 5-CT (to block 5-HT1A, 5-HT1B, and 5-HT1D receptors) was coincubated with [3H]5-HT, membranes, and mesulergine. The remaining [3H]5-HT binding (hereafter referred to as "5-HT1E") displayed high affinity and saturability (KD, 5.3 nM; Bmax, 83 fmol/mg) in human cortical tissue. Competition studies with nonradioactive drugs indicated that, of the drugs tested, 5-CT and ergotamine displayed the highest selectivity for the 5-HT1D site versus the 5-HT1E site.(ABSTRACT TRUNCATED AT 250 WORDS)