Planners were asked to comment on the proposals as part of the consultation process for the Surrey County Council plan.
"There is no way that this flat-roofed, ugly 1960s building could add anything to the town and I feel very strongly about it," declared the chairman of the southern area development control sub-committee, Ann Mugford, at last week's meeting.
She branded the scheme as an "appalling compromise".
On the library move to its proposed location she said: "Many traders were unhappy to be losing what they considered was the focal point of Wey Hill.
"We all know the sale of assets has been 'ring-fenced' and it just doesn't equate to the sum of money SCC would spend.
"We have waited so long that I almost believe people want to wait longer to get a better library."
To make sure voices are heard over the issue, the leader of Waverley Borough Council and committee member, David Harmer, requested that the matter be referred to Waverley's decision makers - the executive committee rather than be discussed under delegated powers.
And added Mr Harmer: "It is a very poor quality late 1950s or early 1960s building" which he also believed did not meet planning guidelines for the retention of existing community facilities in the town.
Jean Mansley told the meeting: "The feedback I am getting is why spend all this money on tarting up a pretty grotty building?
"A lot of people are very unhappy about the dual mix of having a youth centre and library together," she added.
"This town has wanted a library for years and years and since 1947 has had to put up with a shop. It is about time that Haslemere had a decent library. Clearly people in the town are not happy," said Mrs Mansley.
She refuted the planning policy which considered "the scale of the development is appropriate to the needs of the community".
"I don't think it is appropriate at all," she said.
The committee also believed that the proposal did not fulfil a requirement that "where buildings will house significant community uses, they are of high-quality design and create a landmark for the community which they serve".
While agreeing with the committee, Neil Jackson also felt that the proposals would attract more traffic in nearby St Christopher's Road where new flats and houses had been built, combined with a lack of car parking.
There was unanimous agreement to object to the proposals, overturning the recommendation by Waverley planning officers.