In 1799, the town voted to divide into school districts, and 15 were established. This system continued until 1870, when the town system was adopted. However, this caused dissatisfaction among those living in the outlying districts, who complained that their scholars would have to travel long distances to school. Thus, in 1873, the town voted to return to the district system, and this continued until the town system was mandated by the state in 1893. This school was built about 1878 to replace the one room brick school formerly on this site. Besides serving as a school, the upstairs was used for many public meetings, lectures, etc. From the Chittenden Reporter of May 27, 1885, "The popular Guy Family Troupe will give an entertainment in Academy Hall next Thursday evening", and from September 13, 1923, "Forest Camp, M.W.A. [Modern Woodsmen of America], will give an entertainment and dance at school house hall Thursday evening... Ice cream and cake will be sold, and a drawing for a quilt will take place. The Richmond band will furnish music." It was used until 1958, when the new school opened; increasing enrollment forced its use for a few more years.
With the election of Marion Phillips as town clerk in 1961, the town clerk's office moved into one room of the building in 1961, although the school continued to use other parts of the building. In 1972, the town finally purchased the building from the school district for $1, and in 1976 the first floor of the building was renovated. this created a room for the town clerk, along with a fireproof vault on both the first and basement levels to store town records, replacing the three large safes formerly used for this purpose. In addition a room was also created on the first floor for use of the planning and zoning personnel and for selectboard meetings. The basement housed the Saxon Hill Preschool and the Brown's River Library at various times. During this renovation, the stairs to the upper floor were removed, and access was closed off, the only access being via the outside fire escape. But there were still problems with heating and basement flooding, as well as lack of space. So in 2007, the town voted to renovate the building again, and this was done in 2008, when the interior was completely gutted, leaving only the outside walls, and a modern office building was constructed within the original historic walls. An addition on the rear housed a much larger, modern vault for records storage.