Lee and the Custis family slaves
Lee owned slaves because his wife and her family did. As the original post implies, the slaves in question probably included *her siblings*, and doubtless other relatives as well.
By framing his relationship with the Custis slaves as purely economic, a matter of labor, Lee was wilfully ignoring the fact that the slaves should have been treated as blood relatives, his wife's siblings and his children's cousins. The Custises had "indulged" their slaves because on some level they recognized that they were literally family; Lee was a stern and legalistic master IMHO because he was denying that they were.
Remember, Lee marriage to Mary Custis made many people consider him to be George Washington's heir -- but the Arlington slaves were, by that same count, also Washington's heirs. By treating them rigorously, Lee was making sure that his children had no competition for Washington's legacy.
In Martin Chuzzlewit (1843), Dickens describes the Southern American as a man
who dreamt of freedom in the arms of a slave, and woke to sell her children and his own in the public marketplace.