The Src homology 2 domain containing protein tyrosine phosphatase-2 (SHP2) is an oncogenic phosphatase associated with various kinds of leukemia and solid tumors. Thus, there is substantial interest in developing SHP2 inhibitors as potential anticancer and antileukemia agents. Using a structure-guided and fragment-based library approach, we identified a novel hydroxyindole carboxylic acid-based SHP2 inhibitor 11a-1, with an IC50 value of 200 nM and greater than 5-fold selectivity against 20 mammalian PTPs. Structural and modeling studies reveal that the hydroxyindole carboxylic acid anchors the inhibitor to the SHP2 active site, while interactions of the oxalamide linker and the phenylthiophene tail with residues in the β5-β6 loop contribute to 11a-1's binding potency and selectivity. Evidence suggests that 11a-1 specifically attenuates the SHP2-dependent signaling inside the cell. Moreover, 11a-1 blocks growth factor mediated Erk1/2 and Akt activation and exhibits excellent antiproliferative activity in lung cancer and breast cancer as well as leukemia cell lines.