AMSTeX is a TeX macro package, originally written by Michael Spivak for the American Mathematical Society (AMS) during 1983-1985 and is described in the book "The Joy of TeX". It is based on Plain TeX, and provides many features for producing more professional-looking maths formulas with less burden on authors. It pays attention to the finer details of sizing and positioning that mathematical publishers care about. The aspects covered include multi-line displayed equations, equation numbering, ellipsis dots, matrices, double accents, multi-line subscripts, syntax checking (faster processing on initial error-checking TeX runs), and other things.
As LaTeX increased in popularity, authors asked to submit papers to the AMS in LaTeX, and so the AMS developed AMSLaTeX, which is a collection of LaTeX packages and classes that offer authors most of the functionality of AMSTeX. The AMS no longer recommends the use of AMSTeX, and urges its users to use AMSLaTeX instead.
This question on the Web: http://www.tex.ac.uk/cgi-bin/texfaq2html?label=AMSpkg