I have always had mixed feelings for American Apparel. I grit my teeth at its semi-pornographic advertising, gag at licentious stories surrounding sleazebag-CEO Dov Charney and have never approved of leggings taking the place of pants; particularly blindingly fluorescent ones. On the other hand, how many clothing companies have become international successes while paying their factory employees—most of whom are immigrants—almost double the federal minimum wage? Few other retailers have taken strong, progressive stands on issues like gay marriage and immigration, not to mention turned them into successful fashion statements. And I challenge you to find another deep V-neck T-shirt cut quite as well as American Apparel’s quintessential piece.

When news came out that the hipster-beloved clothing brand famous for its colorful cotton basics will be posting a projected $5 to $7 million loss for the second quarter, I was initially thrilled. Finally, that perverted puppet master Charney—just as famous for three sexual harassment suits filed against him by employees as he is for his casual designs—is going down. It’s time Charney loosens the reins on his creation. The company could benefit from a fresh face; a less hands-on leader.

But my vindictive jubilation over Charney’s fall from the top waned when I thought of the factory employees in LA who could lose the affordable family health care American Apparel provides them. It is nearly impossible to find competitive wages and benefits for seam work elsewhere.

With such a labor-centric business model, it’s an admirable feat that Charney has been able to expand American Apparel to its current scale within our capitalist market.  The brand’s biggest competitors, Urban Outfitters and Hanes, have products sewn overseas where labor costs are much lower. When American Apparel arrived, however, Charney’s marketing and message was strong, and it effectively made customers want to pay a few extra dollars for clothes made in the U.S. It opened many young Americans’ eyes to international labor issues.

The original message of fair labor Charney spread has been drowned out, though, deafened by the company’s rapid, sprawling expansion. Originally started as a wholesale T-shirt business, American Apparel went from three original retail stores in 2003 to 280 today. It became a publicly traded company in 2007. Charney remains the majority shareholder and continues to have a heavy hand in every facet of the business.

As American Apparel grew under Charney’s direction, billboards right and left were plastered with silhouettes of split-legged women whom we could only presume were wearing those slinky gold lamé unitards. Low-budget, Polaroid-style photos of women in colorful briefs and thin T-shirts became American Apparel’s signature advertising style.The models, some of whom are employees or friends of Charney’s, are thankfully, not rail thin or of supermodel height. Their familiar, healthy figures and plain styling, however, add just enough realism to those deep-set, seductive stares and twisted hips that Charney’s number one seller became sexual accessibility.

It’s a mantra Charney has since become infamous  for. He freely admits some of the provocative poses come from pictures he takes in his own apartment. The company’s job interviews and casting calls have a notorious reputation for considering everything but a résumé. Claudine Ko, a reporter for Jane magazine, carried on an interview with Charney while he received oral sex from an employee.

Some critics shrug at his antics, saying every fashion icon has his or her own eccentricities. More conscious customers are torn, though, over the compromise of supporting a company that treats workers fairly, but misrepresents many women’s personal dignities. American Apparel exploits not just female bodies, but female decency, to sell its colorful cottons.

Although there will always be demand for quality cotton T-shirts, do circle scarves and lace catsuits really have the longevity the brand needs to stay afloat? It is clear from the financial investigations, rumors of bankruptcy and huge slump in sales that American Apparel’s brand of sexy is losing its appeal, but Charney knows no other way. Take a step back, Dov. Let someone else take a whack at it.