How to Handle a Cost Obsessed Client

Brief Intro: The problem I have with freelancing is that I always get clients that come right out the gate wanting to know how much I charge.

For example:
Potential Client: “I need a website. How much will it cost?”
Me: “Well it depends on what you plan to accomplish with the website and how soon you need it up.”
Potential Client: “So…$50? $100?”
Me: *headdesk*

How would you solve this problem? Just quote a flat rate?