The observable color of an object has to do with the light wavelengths that are reflected from that object. This reflection, in turn, depends on the atomic and molecular properties of the object, its surface structure and the angles at which the light hits it and at which someone observes it, among other things. An apple is red because when white light, which is made up of all visible wavelengths mixed together, hits it, its atomic innards reflect all the red wavelengths more than the other colors and bounce them at our eyes.
A black object, like a t-shirt, looks black because it absorbs all the wavelengths in white light and reflects none. As the shirt absorbs all the light coming from the sun and, say, a desk lamp, the energy carried by that light doesn't just disappear into the shirt, never to be seen again. Instead, as the light is absorbed, it gets converted to other forms of energy, usually heat, and then emitted by the shirt. The darker the object, the better it emits heat, because it’s a better absorber of light.
On the other hand, a white object appears white because it reflects all the different wavelengths and absorbs little to no light. It doesn’t absorb much energy, then, and puts off little to no heat