Settlement of Johannesburg began in 1886, when gold was discovered in the Witwatersrand by an Australian prospector named George Harrison. The discovery spurred a feverish gold rush as fortune hunters from all over the world descended on the area. Blacks from all parts of southern Africa came to work the gold fields either permanently or temporarily as contract laborers. The government of the Transvaal, then a Boer republic, established a city at the site, and in the space of three years it became the largest settlement in South Africa. By the 1890s, several large mining companies had taken control of the area's gold mines, creating huge fortunes for their owners. Tensions between the mine barons, the English-speaking newcomers to the area, and the Transvaal's Boer government—fed by British colonial aspirations in the region—led to the Anglo-Boer War of 1899–1901. By its end, the Transvaal and the Orange Free State were under British control.

At the start of the twentieth century, the population of Johannesburg had reached 100,000. Early in the century, the British colonial government began forcibly relocating blacks from the central city to areas on its outskirts, inaugurating the principle of racial separation that became entrenched in the administration of the city and eventually led to the system known as apart-heid. The substandard conditions in which most of the city's black majority lived led to protests and strikes, including a 1920 strike by 70,000 black mine workers. There was agitation among Johannesburg's white miners as well, culminating in the general strike and "Rand Revolt" of 1922, in which over 200 people died.

The growth of manufacturing in the 1930s and 1940s brought an even greater influx of blacks into the city, especially during World War II (1939–45), when many white workers were serving in the military. The city's black population doubled, with many of the new arrivals crowded into squatters' camps. The beginnings of a black nationalist consciousness that arose during this period led to a white backlash in the 1950s when the conservative National Party came to power and implemented the policy of apartheid, banning all black opposition movements. Beginning in the 1960s, hundreds of thousands of blacks were relocated from Johannesburg to remote "homelands," and their movements were regulated by strict enforcement of pass laws.

The milestone event in the black resistance movement that eventually overthrew apartheid and white dominance came on June 16, 1976, when South African police opened fire on a student protest in the black township of Soweto. The shooting sparked a months-long popular uprising that spread to dozens of other cities in South Africa, and unrest continued through the 1980s, with massive violence erupting in Johannesburg's black townships again in 1984. Black militancy, combined with the effects of international sanctions, finally toppled the apartheid system in the early 1990s and led to South Africa's first democratic elections in 1994.

With the removal of discriminatory laws, Johannesburg's black townships have slowly been integrated into the city's municipal government, and blacks have moved into formerly white districts in the central city and inner suburbs. The city still faces many challenges, including a serious crime problem and de facto segregation as many whites retreat to the northern suburbs.