Wall Street’s Most Reviled Investors Worry About Their Fate