Let's not forgot that the South didn't want slaves to be free because they did not want them to eventually vote.
1860 Population Distribution of the the Eleven Future Confederate States
WHITE SLAVE FREE BLACK ** TOTAL
ALABAMA 526,271 (54.6%) 435,080 (45.1%) 2,690 (0.3%) 964,041
ARKANSAS 324,143 (74.4%) 111,115 (25.5%) 144 (0.1%) 435,402
FLORIDA 77,747 (55.4%) 61,745 (44.0%) 932 (0.7%) 140,424
GEORGIA 591,550 (56.0%) 462,198 (43.7%) 3,500 (0.3%) 1,057,248
LOUISIANA 357,456 (50.5%) 331,726 (46.9%) 18,647 (2.6%) 707, 829
MISSISSIPPI 353,899 (44.7%) 436,631 (55.2%) 773 (0.1%) 791,303
NORTH CAROLINA 629,942 (63.5%) 331,059 (33.4%) 30,463 (3.1%) 991,464
SOUTH CAROLINA 291,300 (41.4%) 402,406 (57.2%) 9,914 (1.4%) 703,620
TENNESSEE 826,722 (74.5%) 275,719 (24.9%) 7,300 (0.7%) 1,109,741
TEXAS 420,891 (69.7%) 182,566 (30.2%) 355 (0.1%) 603,812
VIRGINIA 1,047,299 (65.6%) 490,865 (30.8%) 58,042 (3.6%) 1,596,206
TOTAL 5,447,220 (59.9%) 3,521,110 (38.7%) 132,760 (1.5%) 9,101,090
Hell, Southern whites still try to keep blacks from voting by demanding voter ID laws and making it harder to vote in cities. Just another reason to dislike racist rethuglicans.