a clarification with this topic, since when using:
ALTER DATABASE con MODIFY FILE
to change the size of a datafile only works if you want to increase the size of the datafile ALTER DATABASE File and Filegroup Options .
If what you want to do is reduce the size of the datafile, what you should do is use the command
DBCC SHRINKFILE
As long as inside the data file you have free space, here is the reference for use DBCC SHRINKFILE
Anyway, what happened in your case if you checked a decrease in the size of the datafile after restarting the instance, is that since it is the base of the tempdb system, it is recreated on each restart of the instance of SQL server , so maybe the datafile of this database is very large due to some expensive operations of ordering, group by, etc .., you executed the command
ALTER DATABASE tempdb MODIFY FILE
(NAME = 'tempdev', SIZE = 10240)
to shrink it, nothing happened and when you rebooted this base recreates the original size, but what the shrink was the behavior of the tempdb and not the execution of your command.
I hope you served.
Greetings