Monty Don moved into a "derelict" home with his wife Sarah, admitting their Herefordshire abode was "ruined" when they bought it.