A North Carolina man was building his castle for 30 years — after a romance turned sour, he was almost forced to sell