The tradition of giving misbehaving children lumps of fossil fuel predates the Santa we know, and is also associated with St. Nicholas, Sinterklaas, and Italy’s La Befana. Though there doesn't seem to be one specific legend or history about any of these figures that gives a concrete reason for doling out coal specifically, the common thread between all of them seems to be convenience.
Santa and La Befana both get into people’s homes via the fireplace chimney and leave gifts in stockings hung from the mantel. Sinterklaas’s controversial assistant, Black Pete, also comes down the chimney and places gifts in shoes left out near the fireplace. St. Nick used to come in the window, and then switched to the chimney when they became common in Europe. Like Sinterklaas, his presents are traditionally slipped into shoes sitting by the fire.
So, let’s step into the speculation zone: All of these characters are tied to the fireplace. When filling the stockings or the shoes, the holiday gift givers sometimes run into a kid who doesn’t deserve a present. So to send a message and encourage better behavior next year, they leave something less desirable than the usual toys, money, or candy—and the fireplace would seem to make an easy and obvious source of non-presents. All the individual would need to do is reach down into the fireplace and grab a lump of coal. (While many people think of fireplaces burning wood logs, coal-fired ones were very common during the 19th and early 20th centuries, which is when the American Santa mythos was being established.)
That said, with the exception of Santa, none of these characters limits himself to coal when it comes to bad kids. They’ve also been said to leave bundles of twigs, bags of salt, garlic, and onions, which suggests that they’re less reluctant than Santa to haul their bad kid gifts around all night in addition to the good presents.