The arrests were made after Prime Minister David Cameron urged the country to remain vigilant against potential reprisals following the killing of al Qaeda leader Osama bin Laden in Pakistan by U.S. forces.
Police said they were unaware of any link between the arrests and bin Laden's death.
The men were arrested on Monday after officers conducted a stop check on a vehicle near the Sellafield site in Cumbria.
The men are all aged in their 20s and from London.
A spokesman for the police's North West Counter Terrorism Unit declined to comment on media reports the men had been filming the site.
"There were suspicions from the Civil Nuclear Constabulary that led to some arrests. There were some suspicions about them near to the perimeter fence," he said.
A police counter terrorism source said the arrests were not preplanned. "The local officers felt there was enough to arrest them. It's a case of seeing if there is anything to it."
Police held the men under section 41 of the Terrorism Act, which allows officers to arrest people suspected of terrorist offences and hold them for 48 hours without charge.
The men were being transferred to Manchester to be questioned by counter-terrorism officers.
Local police were alerted by officers from the Civil Nuclear Constabulary, a specialist force which provides protection for civil nuclear licensed sites.
Fifty-two people were killed in London in 2005 when al-Qaeda inspired suicide bombers blew up underground trains and a bus.
The Sellafield site, in operation since the 1940s, includes a number of nuclear fuel reprocessing facilities and waste treatment plants.
Sellafield is owned by Nuclear Management Partners, a consortium of American engineering company URS, British engineering firm Amec and French nuclear reactor maker Areva. It was operating as normal on Tuesday.