API Session Id

I've been trying to find a work around for this issue. Basically, if I use the merge field {!$Api.Session_ID} on a button, the session Id is value and works just fine. If I attempt to use the same Api.Session_ID within either a formula field on a sObject, or within the APEX VisualForce page as a standalone merge field the session is no longer valid and fails when an outside source attempts to use that session Id. Would anyone know why and/or a work around for this?