Currently the "Expiry Date" on the URL Token record is not enforced.
As a potential option, you could do a Flow or similar that changes the "Where Used" to Internal. This would then remove access to this record from the public Force.com Sites. Please see the URL Parameters Sharing rules in the installation guide.
As this is not available with the standard product, any further information will require subscription to our Premium Support.