Most clothing is made from fiber, whether natural (wool, cotton, linen) or synthetic, which does not by itself have a bright white appearance. Even after washing and bleaching, most of these fibers have a slightly off-white appearance, usually on the tan or gray side. Additionally, fabrics tend to aquire a yellow or tan appearance with repeated use and laundering.
Bluing really does add a little blue to this mix. Many white fabrics are blued during the manufacturing process. Traditional bluing uses a suspension of blue iron particles, with a bit of other stuff to keep the product stable and fresh. Some laundry detergents use synthetic agents instead. That blue is the complimentary color of yellow, and serves visually to offset the yellowed appearance and make fabric look whiter. The blue, however, washes out over time. A bluing agent is used to restore it.