The stones that never became a castle