In fact, according to this, on a proportion basis, blacks were the preponderate owners of black slaves:
In the rare instances when the ownership of slaves by free Negroes is acknowledged in the history books, justification centers on the claim that black slave masters were simply individuals who purchased the freedom of a spouse or child from a white slaveholder and had been unable to legally manumit them. Although this did indeed happen at times, it is a misrepresentation of the majority of instances, one which is debunked by records of the period on blacks who owned slaves. These include individuals such as Justus Angel and Mistress L. Horry, of Colleton District, South Carolina, who each owned 84 slaves in 1830. In fact, in 1830 a fourth of the free Negro slave masters in South Carolina owned 10 or more slaves; eight owning 30 or more (2).
According to federal census reports, on June 1, 1860 there were nearly 4.5 million Negroes in the United States, with fewer than four million of them living in the southern slaveholding states. Of the blacks residing in the South, 261,988 were not slaves. Of this number, 10,689 lived in New Orleans. The country's leading African American historian, Duke University professor John Hope Franklin, records that in New Orleans over 3,000 free Negroes owned slaves, or 28 percent of the free Negroes in that city.
Granted the difference is, and it's a BIG difference, no whites were ever legal slaves, but how does this affect the asinine claims of slavery restitution?