When first granted by Massachusetts in 1735, the area was the most northern New England outpost on the Connecticut River. Its 1744 log fort, called No. 4, was a strategic military site throughout the French and Indian Wars. Several settlers were ambushed and captured by the Indians, and the fort itself was besieged by an attacking force of 400 in 1747.

The town, named for Admiral Sir Charles Knowles by New Hampshire in 1753, shared county seat honors with Keene until 1827, when it was incorporated into Sullivan County. It is a busy and prosperous river town. The village street contains beautiful old homes, a fine Gothic revival church built in 1842 and an Episcopal church designed by Richard Upjohn in 1863.

Sixty-three buildings on Main Street are listed on the National Register of Historic Places.

A nearby reproduction of historic Fort No. 4 offers displays, crafted demonstrations, musters and tours of its stockaded parade ground and pioneer-style houses.