One of our clients informed us that the application that we did for them and that is hosted on AWS was falling constantly. We started to review the logs and discovered that the server was falling and throwing the following exception:
"too many open files"
Then we raised the limit of files allowed by the server and started monitoring how many files the application opened. During 2 days the number did not stop increasing and we decided to find out what files were being used by the Java process. After a long investigation we rule out that the problem is base connections that did not close and we discovered that in the folder "/ opt / tomcat / temp" infinite temporary files are being created with 0 in size but that the application for some motive keeps them open.
Examples of these files are the following:
oew974702517498114636.hop oew975038640901567654.hop oew975419048649710675.hop oew976498632463201934.hop oew97661135001107305.hop oew97786752219273069.hop oew978096979722662729.hop oew979934500917317891.hop oew980223348673981682.hop oew980401664060722912.hop oew982077290584656345.hop oew982911046120251661.hop oew984241948479464632.hop oew984817715497603618.hop oew987522880029695756.hop oew989321970952129224.hop
And of that type there are around 7,000 and they keep going up.
What could be the problem? Why are these files generated? How could the problem be solved and avoid creating so many files?