Web Site is a type of independent project unrelated to Visual Studio. In other words, if you rename a file that has reference in another file, that reference will not be updated. For example, reference to a class in a view. If you rename the class, it will not update in your view. To put it this way, Visual Studio has no control over it.
While Web Application is a special type of Visual Studio project where you are aware of the changes you make to the files. If you rename a class, it will be updated in all the places where Visual Studio has reference to it. This makes it easier to manage. This type of project also, unlike WebSite, compiles all the code into a single dll
which makes it more organized than the WebSite and less heavy.