The attack bore all the traits of Islamic State of Iraq and al-Sham (ISIS). A small unit of militants, armed with Kalashnikov rifles and suicide belts, hit the Tripoli prison just before sunrise. Blasting though a wall, four fighters worked their way through the heavily guarded compound before firing a rocket-propelled grenade to breach the cells inside. Their target, security sources say, was a jailed Libyan ISIS militant. Clashes erupted. Two of the attackers, a Moroccan and a Sudanese, detonated suicide belts and shortly afterwards all four, and the militant, were dead. The prison break failed. But it was another illustration of the tactics employed by an ISIS front determined to emulate the success of the group's founders in Iraq and Syria. "When we see them fighting, they are well trained. There were only four, but they destabilized the whole base," said Muaad Khalil, a spokesman for forces at the Maitiga base. "Who would have thought to attack this base, but they did."

Four years after the overthrow of Muammar Gaddafi, Libya is locked in a conflict between two rival governments, an official one in the east, and a self-declared one controlling Tripoli, and the many armed factions that back them. Far from the battlefields of Iraq and Syria, ISIS has steadily grown in Libya's chaos, controlling the city of Sirte, and worrying Western governments who fear it can only become stronger in the post-revolution mess. They have left their mark on the North African state. They have massacred Christian Egyptians on a Libyan beach, publicly flogged criminals in Sirte, stormed oilfields, and attacked a five-star Tripoli hotel. But while Libya's turmoil and history of extremism offered fertile ground, ISIS has run up against the heavily armed factions and rival Islamists already in place. Even as they lay claim to Sirte, Libya's ISIS followers have been ousted from Derna city by local fighters, and have shown they cannot hold ground or muster the finances and oil resources they benefit from in Iraq. "They clearly want to expand from Sirte," one Western diplomat said. "They continue to maintain the ability to carry out one-offs outside their main area, but they are still small."

Although Libya has a long history of extremism Derna showed their limits. Even after ISIS leaders arrived to recruit in July, fighting erupted as Derna Islamists and citizens fed up with foreign jihadists drove ISIS out. But in Sirte, security sources and residents say, they found a more suitable base by tapping into frustrations in Gaddafi's home town, where many felt sidelined after the revolution. "ISIS saw Sirte as the perfect place. Some Gaddafi followers are now ISIS members," said local military commander Ismail Al-Mjaree. Targeted assassinations of rivals and security officials began last year, and ISIS forces moved in force on Sirte in February and March, taking a radio station and other important buildings. Forces from Misrata, a port city that is home to one of the country's more powerful military factions, arrived to take back Sirte. But fearing large-scale casualties and blaming a lack of support from Tripoli, they retreated, although hostilities may be resumed at some point.

Security sources in Tripoli and Misrata estimate ISIS has at least 500 fighters inside Sirte, and numbers are growing thanks to the arrival of foreign recruits. Most of ISIS's leaders are Libyan, some who spent time in Abu Salim prison during the Gaddafi era or had ties to Ansar al Sharia. One top leader, security sources said, is Hassan al-Karami, a Benghazi native once jailed for militancy. The SITE extremism monitoring service, citing Libyan media sources, reported that Abu Ali al-Anbari, believed to be ISIS governor for Syria directly under ISIS leader Abu Bakr al-Baghdadi, had arrived in Sirte by sea. "They control the whole city and the suburbs. No one can fight back there now," one Libyan security source said. "They are Sudanese, Egyptians and Tunisians in large numbers."

The group has also targeted the oil industry. It attempted to breach the defences of the Es Sider oil port, but failed. Its fighters overran oilfields south of Sirte, kidnapping several foreigners, but without taking control of the fields. "ISIS in Libya doesn't have comparable revenue streams, no oil sales, no sizeable population to tax and extort," said Geoff Porter, a North Africa expert at West Point's Combating Terrorism Center. "In Libya, there is a superabundance of armed groups. For the most part they are busy fighting each other, but they could potentially be harnessed to eliminate ISIS." Events in Sirte are hard to verify. But as in Iraq and Syria, ISIS appears to be slowly imposing its vision on the city, carrying out punishments such as crucifixions and public floggings, residents say. Taking over city institutions and banks, ISIS forces merchants and shopkeepers to pay a tax that would normally go to the state, and a court is in place. Barbers are banned from shaving off beards and smoking Shisha pipes in cafes has been stopped. Female students have been forced to wear one-piece robes.

A former bank employee and other residents said the local ISIS organization appeared to be struggling to manage the city financially. Prices of local goods are rising and other products are disappearing. "Life in Sirte is almost non-existent," one resident said. "People do not have enough money to flee the city." Twice in the last month, though, residents in Sirte say they saw unidentified warplanes attacking districts controlled by ISIS.