Salesforce REST API invocation via Third Party App

Salesforce REST API invocation via Third Party App

I am currently in process of invoking Salesforce REST API from my own server.
Following are the two options I have already implemented,
Use OAuth 2.0 User-agent model for querying Salesforce REST API
User OAuth 2.0 User-password model for querying Salesforce REST API.
In both the above case, I needed to setup Remote Site in the Salesforce
Org. How can I bypass this. i.e. Without setting up Remote Site, I can
directly query Salesforce API. I have seen a popular app integrated with
salesforce already done similar thing.
Thanks for your help.