The Freedom House Museum was once part of the headquarters for the largest domestic slave trading firm in the United States, Franklin and Armfield. Enslaved Africans were brought from the Chesapeake Bay area and forced to the slave markets in Natchez, Mississippi and New Orleans either by foot or ship. The building has a long history. In 1828, it was leased by Isaac Franklin and John Armfield and used as a "Negro Jail" or slave pen for slaves being shipped from Northern Virginia to Louisiana. During the American Civil War the museum and its surrounding sites were used firstly as a military prison for deserters, then as the L'Ouverture Hospital for black soldiers, and finally as the barrack for contraband-slaves who fled the confederate states and sought refuge with Union troops.

The building is currently owned by the Northern Virginia Urban League but the museum is managed by the office of Historic Alexandria. It operates as a reminder to the people of Alexandria of the city's role in historic slavery.

In the basement of the building there is a powerful exhibition which depicts the harsh reality of the domestic slave trade and Alexandria's role in it, through the use of first person narratives from enslaved men and women. These are complemented with a range of contextual text panels, artefacts, images and maps.

Previously called the Robert Robinson Library, the museum was opened as the Alexandria Black History Research Centre in 1983. In 1987, the Alexandria City Council placed the operation of the museum under the office of Historic Alexandria, providing a large increase in funding which allowed for the building to be completed in 1989. Further expansion followed in 1995, when the Watson Reading Room, with books, documents, and periodicals on African American culture, was added. The museum's mission is to inform and enrich the lives of Alexandria’s residents and visitors about the diversity of the African American experience in Alexandria, Virginia. The museum also operates the Alexandria African American Heritage Park, a nine-acre park, which contains a one-acre nineteenth-century African-American cemetery that was buried under a city landfill in the 1960s.

The museum has several exhibitions, displaying collections of African objects, including wood carvings from the west coast of Africa, as well as collections from African American churches, photographs, and documents. The Museum also runs events related to African cultural and heritage such as guest lectures. The museum curates a range of temporary exhibition covering a variety of topics. For example, the Sharon J Frazier and Linwood M. Smith Dollhouse collection has featured in one such exhibition with miniatures of buildings and rooms capturing the forgotten businesses and people who were important to Alexandria’s development in the last century. A particular emphasis was also placed on African American culture and important institutions such as family, church, and school.