Anthony Rolfe Johnson was born in Tackley in Oxfordshire. As a boy, he demonstrated musical ability and sang as a boy soprano, making a record with HMV. Despite his ability, he did not consider singing as a career and instead went to study for an agricultural degree. He worked as a farm manager, and would sing church hymns to his herd of cows. He joined a choral society in Crawley, West Sussex, and sang regularly with the choir of St Nicholas' Church, Worth, and was encouraged by another member to pursue a professional singing career.
Rolfe Johnson studied with Ellis Keeler and Vera Rózsa at the Guildhall School of Music and Drama. He was also tutored by Peter Pears.

He first appeared in opera in the chorus and in small roles at the Glyndebourne Festival between 1972 and 1976. His major operatic debut was in the role of Count Vaudémont in Tchaikovsky's opera Iolanta in 1973 with the English Opera Group. The same year, he held his first professional recital at the Purcell Room at the Southbank Centre. In 1975, Rolfe Johnson made his Glyndebourne debut, singing the role of Lensky in Eugene Onegin, for which he won the John Christie Award. In 1978 he made his first appearance with English National Opera, as Tamino in The Magic Flute.
In the course of a long and varied career he performed in Handel's oratorios, sang the role of Evangelist in J. S. Bach's St John Passion and St Matthew Passion, and sang solos in Haydn's The Seasons and The Creation. Operas he recorded include Mozart's The Magic Flute, Gilbert and Sullivan's The Mikado, Mozart's Idomeneo and La clemenza di Tito, and Britten's Peter Grimes as well as appearing in the latter's War Requiem, amongst many others. Many of these recordings were made under English conductor John Eliot Gardiner, including Monteverdi's Ulysses and Orfeo and Mozart's Idomeneo.
He performed at the world's major opera houses, including the English National Opera, Royal Opera House, Covent Garden, La Monnaie in Brussels, La Scala, Milan, the Metropolitan Opera in New York City, the Vienna State Opera, and the Paris Opera.
He sang Polixenes in the world premiere of Wintermärchen, Philippe Boesmans' adaptation of Shakespeare's The Winter's Tale.

Aside from opera, he appeared in concert with the world's major symphony orchestras, such as the New York Philharmonic and Boston Symphony, and conductors, such as Mstislav Rostropovich and Seiji Ozawa. He also gave song recitals with Graham Johnson, many of which were recorded; he was a founder member of Graham Johnson's The Songmakers' Almanac. In 1988 he re-launched the Gregynog Music Festival, remaining its artistic director until 2006. In 1990, he was appointed Director of Singing Studies at the Britten-Pears School for Advanced Musical Studies. Rolfe Johnson began to tutor singers, taking on the operatic tenor John Mark Ainsley as his first pupil.

He was made a Commander of the Order of the British Empire in the 1992 Queen's Birthday Honours.
Around the turn of the 21st century, Rolfe Johnson began to suffer from Alzheimer's disease and was forced to retire. On 21 July 2010 he died from complications related to Alzheimer's. Rolfe Johnson was cremated and he is commemorated with a memorial at Golders Green Crematorium in London.