In 1870, the building was purchased by Robert Knight Remington, a wealthy Fall River industrialist, who had the entire structure dismantled and moved three-quarters of a mile north to its present location on Rock Street. Enlarged and redesigned in the then-fashionable French Second Empire style, the structure was to serve as home to the Remington family for the next eight years. Due to the prominence of Mrs. Remington's social position and the family's elegant style of living, the residence quickly became the scene of a number of glittering social affairs. Of particular note was the October 10, 1877, wedding reception for the Remington's daughter, Harriet, and George Henry Hills, to which 500 privileged guests were invited, chosen from the over 2000 invited to the ceremony. In order to accommodate their guests, the family had erected a 20-foot by 50-foot pavilion off the circular dining room, only to have it dismantled the following day. Covered extensively by the press, it was noted that "the array of beautiful dresses was probably never equaled in this city" and that "the display of diamonds was extensive."

Financial reverses in the following year caused Remington to sell the house to David Anthony Brayton, Sr., one of Fall River's most prominent textile magnates and a member of the city's wealthiest family. The mansion was to serve as home to the Braytons for the next fifty-seven years. Following the death of Miss Elizabeth Hitchcock Brayton in 1935, the home was inherited by her nephew, David, who in turn donated the structure to the Fall River Historical Society. The house is one of the few surviving examples extant in Fall River, Massachusetts, of mansions built by the affluent mill owners during the period of the city's greatest prosperity.

In addition to the Remington-Brayton residence, the Historical Society also maintains the Alexander Dorrance Easton house, constructed in 1870. Situated on property adjacent to the museum building, the structure was renamed the Florence Cook Brigham Annex in honor of the Society's beloved late curator.