In a major upset, Fillon had more than 43 percent of the vote to 26.7 percent for former prime minister Alain Juppé and just under 23 percent for Sarkozy, according to the official tallies.
Only the top two candidates will progress to next Sunday's run-off.
If the results are confirmed, Fillon has pulled off a remarkable come-from-behind victory in the first round after trailing Sarkozy and Juppe in all but the final days of the two-month campaign.
Far-right leader Marine Le Pen has cast a shadow over the contest because whoever wins the nomination is tipped to face -- and beat -- her in the decisive presidential run-off next May.
With France's ruling left bitterly divided, the campaign has been heavily influenced by the populist forces behind Britain's vote to leave the European Union and Donald Trump's shock victory in the US election.
Voters appear to have chosen the more understated style of Fillon over the brashness of Sarkozy, who still deeply divides France four years after being turfed out of office by the Socialist Francois Hollande.
Juppe was the early frontrunner in the campaign but the pro-business Fillon made stunning progress, surging up the opinion polls in recent days.
It appears that the participation of leftwing voters in the first ever rightwing primary to be held in France could have been a significant factor in dragging down Sarkozy.
Anyone who paid two euros ($2.1) and signed a declaration of adherence to "the values of the centre and the right" could take part.
Many leftwing voters are thought to have voted against Sarkozy.
Turnout was high with more than four million voters casting ballots, according to a preliminary count.
One Socialist voter, a sports teacher in his 50s who identified himself only as Eric, said he was taking part to vote "against Sarkozy".
"I'm fed up of that guy, he thinks he is all-powerful and he has been involved in too many scandals. Juppe, despite everything else, is the opposite," he told AFP as he cast his vote in the Paris suburb of Pantin.
In a final TV debate among the seven candidates on Thursday, Sarkozy angrily ducked a question about fresh claims that he received millions in funding from the late Libyan leader Muammar Gaddafi towards his 2007 campaign.
The case is one of several investigations to dog Sarkozy since he left office after what was dubbed a "bling-bling" presidency because of his flashy lifestyle.
Sarkozy, Juppé and Fillon have broadly similar programmes, underpinned by pledges to reinforce domestic security in a country still under a state of emergency following the deadly jihadist attacks.
They also share a desire to reinforce European borders and reduce immigration, while tax cuts also loom large.
Ultimately, the choice came down to style.
Sarkozy emphasized his tough-guy credentials, saying it made him a "better choice" than the mild-mannered Juppé to safeguard France's position in an uncertain world following Trump's election.
Fillon, who is popular in the business world, has promised "radical" economic measures but is the most conservative of the three on social issues.
The nomination of the right-wing candidate on November 27 is expected to trigger an announcement from Hollande on whether he intends to bid for re-election despite the lowest popularity ratings of any post-war French president.
On Wednesday, Hollande's former economy minister Emmanuel Macron, 38, announced he would stand as an independent, further confusing the picture on the left.
