The Hermitage is known for its beautiful grounds, which include twelve acres of semi-formal gardens, forest and wetlands. The Lafayette River wraps around the Hermitage on three sides, providing an elegant backdrop to walks through the gardens, wedding ceremonies, and outdoor events. Both the wetlands and woodlands provide a natural habitat for hundreds of plants and animals, including a variety of waterfowl and migrating birds.

In 1908, when the Sloane family began construction on The Hermitage as their summer retreat, Florence Sloane began to beautify the grounds, determined to incorporate the design of the house into her gardens. For the next thirty years, she created a picturesque landscape, extending her artistic vision and the artwork of others into the design of the gardens. The gardens feature 105 millstones, many of which reside in the Millstone Courtyard that greets visitors at the Museum entrance. Along the south side of the house is a colorful English-style perennial border coupled with a pathway leading into the waterfront lawn. A large southern magnolia tree stands on the east lawn overlooking a stone terrace highlighted by Edward McCartan’s bronze fountain, Girl Drinking From a Shell.

The gardens feature a mixture of flowering plants, shrubs, and trees. Roses, daffodils, peonies, Japanese anemones, Russian sage, Black-eyed Susans, and other sun-loving perennials surround bronze sculptures, quiet benches, brick and cobblestone footpaths, and stoned terraces. The gardens are accented by various unique plant containers, two of which sit on the tiled terrace outside the Gothic Music Room. From a shady alcove filled with camellias and ferns, a large wisteria covers the entrance at the Millstone Courtyard.