Born in Battle, East Sussex, England, Olivier was educated between 1875 and 1877 at Sherborne School, a public school in Dorset and in 1922 gave his painting 'Easter Morn' to the school. The painting was originally intended for a church in Italy but it was put in such a bad light that he refused to leave it there. His father Henry Arnold Olivier was a clergyman, and his brothers were Henry (1850–1935), who had a military career, ending as a colonel; Sydney (the father of Noël and Daphne), who became Governor of Jamaica and later Secretary of State for India; and Gerard (1869–1939), a clergyman (the father of Laurence Olivier). He also had four sisters. He studied at the Royal Academy Schools beginning in 1881, where he won the Creswick Prize in 1882.
Olivier exhibited extensively, including the Royal Academy starting in 1883, the R. P., the R. I. and the Paris Salon. He taught at the Bombay School of Art in the 1880s. He went to Kashmir with the Duke and Duchess of Connaught in 1884. In 1885 he showed 66 of the paintings from his trip to Kashmir at the Fine Art Society. These works were considered "effective, though hard and coarse in colour" by critics. He had a one-man exhibition at the Grafton Galleries in 1908.
In 1917, Olivier was appointed an Official War Artist and in 1924 he presented to the nation, for display in 'the new War Museum at South Kensington', a number of paintings, including 'The Supreme War Council' (the original of which was given to the French Government and displayed in the Palace of Versailles), 'The Armistice Meeting', 'The Military Representatives in Conference', 'The Peace Signature Table', and various portraits. The paintings now form part of the collections of the Imperial War Museum. Olivier was elected to the Royal Society of British Artists in 1887 and to the Royal Institute of Painters in Watercolours in 1929 where a major retrospective exhibition of his work was held in 1935.
He is mentioned in Mallalieu's British Watercolor Artists and Davenport's Art Reference. He may have been the H. A. Olivier whose work was reproduced in 20 colour plates for The Royal Botanic Gardens, Kew: Historical and Descriptive, Cassell & Co. Ltd, London, 1908. His work and biography are published in The Modern British Paintings, Drawings, and Sculpture, Chamont, London 1964. In later life his work tended towards large ceremonial works using oils. He gave his painting, Lord Selborne and Bishop Gore-Browne to the Athenaeum Club in 1937 where it remains prominently displayed in the Morning Room opposite Darwin.
He died in Hayling Island, Hampshire