Connected App: Modified Selected OAuth Scopes does not update "Allow Acces" dialog

I wanted to reduce our list of selected oauth scopes to the ones we actually need. I create a test app with several scopes, then removed most of them, but the "Allow Access?" dialog still lists removed scopes more 20 mins after the scope change.

Is there someway to force this dialog to refresh?