WASHINGTON—A Securities and Exchange Commission judge ruled that a former Maryland banker perpetrated a short-selling fraud aided by one of the biggest stock-options brokers in the U.S.
Jonathan Feldman, who was accused by the SEC of trading billions of dollars of stock and options in ways that misled other investors, was found by the judge to have engaged in a practice regulators say has grown more prevalent in recent years: "naked short selling."
The decision makes it more likely the SEC will proceed with other enforcement cases involving similar activity, say people familiar with the SEC's thinking.
An SEC administrative law judge—an independent judicial officer who rules on SEC allegations of securities-law violations—late Friday ordered Mr. Feldman to disgorge $2.7 million in profits from his alleged trading scheme and to pay a $2 million civil fine.
The SEC is probing abusive trades. Bloomberg News
The judge also ordered optionsXpress Inc., a brokerage firm owned by Charles Schwab Corp. SCHW +0.31% , to disgorge $1.6 million and to pay a $2 million civil fine for allegedly violating laws prohibiting naked short selling.
Thomas Stern, optionsXpress's former chief financial officer and primary regulatory liaison, was fined $75,000 and barred from participation in the securities industry.
Mr. Feldman's lawyer didn't respond to requests for comment. The decision said Mr. Feldman was 56 years old and was employed as a senior vice president at a Maryland savings bank from 1993 until April 2012. It said he was now self-employed.
A lawyer for optionsXpress disputed the decision. "There was no naked short selling in this case," Stephen Senderowitz, a lawyer at Winston & Strawn LLP, said in a statement.
"We think [the judge] got it wrong with respect to Mr. Stern," said Vincent Schmeltz, Mr. Stern's lawyer, who said he plans to appeal the decision.
Messrs. Feldman and Stern and optionsXpress have 21 days to request a review of the decision from the full commission. Mr. Senderowitz said the company is "reviewing the decision and deciding whether to appeal."
If the request is denied, defendants could appeal to the U.S. District Court for the District of Columbia.
The SEC accused Mr. Feldman of using optionsXpress from late 2008 until early 2010 to profit by trading billions of dollars of stock and options without following through in his obligation to borrow and deliver shares on stock he sold short. The SEC said Mr. Feldman engaged in so-called naked short selling of stocks such as American International Group Inc., AIG +0.41% Citigroup Inc. C +0.38% and Sears Holding Corp. SHLD -0.65% The SEC also alleged that optionsXpress facilitated the trades and that it broke rules designed to curb naked short sales.
In a short sale, a trader borrows stock from a third party and sells it in the hope of buying it back later for a lower price. Traders are typically allowed to sell the stock before they borrow it, as long as they do so within several days after the sale. In naked short selling, a trader never follows through on the promise to borrow. Regulators say the practice distorts markets and can harm investors.
Brenda Murray, chief administrative law judge for the SEC, found Mr. Feldman "executed a trading strategy that he knew, or was reckless in not knowing, perpetrated a fraud on the participants in the securities markets," and that optionsXpress officials aided the scheme, according to the decision.
The optionsXpress case is seen within the SEC as a potential landmark for its market-abuse unit, since it involves trading strategies that in the past evaded the agency's radar, say people familiar with the matter. The unit is one of five specialized groups within the SEC's enforcement division created in 2009 to let its agents focus on specific areas of the market.
"This is the kind of case the market-abuse unit was set up to do," said Daniel Hawke, head of the unit.
The case is one of several continuing SEC probes that involve complex trades and ties between traders, brokers and exchanges.
SEC investigators have been looking into whether CBOE Holdings Inc., CBOE +1.39% which operates an options exchange, properly policed optionsXpress's trading, say people familiar with probe. CBOE said in an SEC filing that it expects to be fined as much as $10 million to resolve the issue. A CBOE spokeswoman declined to comment.
Write to Scott Patterson at scott.patterson@wsj.com