The crash, involving two cars, happened at around 6.50am in the Tuesday morning rush-hour.
A blue Peugeot left the road and hit a pillar outside retailer Vinegar Hill - moving the pillar "an inch" off its plinth, according to police at the scene.
Two significant chunks of stone have also been dislodged from the top and bottom of the pillar.
No injuries have been reported, but the Peugeot is yet to be recovered - blocking one lane of The Borough between Castle Street and the Royal Deer crossroads.
The incident has added to congestion in the town centre and surrounding roads including the A31 Farnham Bypass.
The Town Hall Buildings, designed by the great town architect Harold Faulkner, were constructed between 1930 and 1934.
The pillar forms part of the building’s iconic pedestrian arcade.