The detentions mark the first major backlash by authorities since a campaign was launched by Saudi women nearly two weeks ago to challenge the driving restrictions.
Dozens of women have since driven through the capital Riyadh and other cities.
Saudi-based rights activist Eman al-Nafjan told The Associated Press that police detained one woman on Tuesday while driving in Jiddah on the Red Sea coast. Four other women accused of driving were later detained in the city, al-Nafjan also said, adding that there has been no new information on the status of the detainees.
"This is the first big pushback from authorities, it seems," al-Nafjan said. "We aren't sure what it means at this point and whether this is the start of a harder line by the government against the campaign."
About 40 Saudi women got behind the wheel earlier this month, saying they were launching a campaign to lift the restrictions in the ultraconservative Muslim country, where women can only appear in public when escorted by a male relative.
Saudi Arabia has no written law barring women from driving - only fatwas, or religious edicts, by senior clerics following a strict brand of Islam known as Wahabism.
The group, Saudi Women for Driving, said their campaign was inspired by the Arab uprisings against autocratic rulers and appealed for high-level Western backing.
U.S. Secretary of State Hillary Rodham Clinton lent her support to the campaign against the ban on female drivers in the Islamic kingdom, a close U.S. ally.
Clinton praised the protesters, but stressed they are acting on their own, on behalf of their own rights, and not at the behest of outsiders like herself.
The Saudi protests have put the Obama administration, and Clinton in particular, in a difficult position.
While Clinton and many other top U.S. officials personally find the Saudi ban on women drivers offensive and contrary to a modern and just society, the administration is increasingly reliant on Saudi authorities to provide stability and continuity in the Middle East and Gulf amid uprisings taking place across the Arab world.