Process large CSV files

1

Good morning folks, I have a query to ask you. I have to upload and process large CSV files with many records. Currently they are uploading and processing via AJAX but the browser hangs while the server is running correctly but does not give a response as a term. It is a process that someone does a lot of ago is pure PHP, does not use framework.

Can you think of something to improve this process? I had planned to do it in the background and log in when it starts and ends.

Thank you very much already.

    
asked by Mauro Suarez 15.03.2017 в 15:23
source

4 answers

1

You can also modify your js If you are occupying jQuery.ajax () In the configuration parameters you can increase the waiting time

$.ajax({
    url: "test.php",
    error: function(){
        // will fire when timeout is reached
    },
    success: function(){
        //do something
    },
    timeout: 30000 // sets timeout to 30 seconds
});

Documentation: link

    
answered by 15.03.2017 в 18:22
0

hello there are several solutions for that:

* You must increase the size headers in the php script that processes the data and also where the file goes up. set_time_limit (0); This is placed on the first line of the file.

* Also if you have access to your server increase the maximum upload time of files in the php.ini file, here it says as link

* There is also an option from the .htaccess file, this one I have not tried

    
answered by 15.03.2017 в 17:15
0

Which side is better?

It's a question that needs to be asked sometimes, when we talk about JS (Ajax and jQuery included) and PHP.

As we already know, everything that is Javascript works on the client side, while PHP works on the server side.

In the case of large files, reading and managing your data is faster and more secure if we do it on the server side. Then if we want to use Javascript, jQuery or Ajax to update something in the DOM, we can combine the two strategies.

By experience, sometimes it happens that one is blocked somewhere in a program, when this happens it is convenient to ask am I focusing things on the right side ?, would not it be convenient to change sides?

Although, as I said in a comment, everything depends on the context, the scope of the application, who will use it, etc., in a case like this I would not hesitate to work my CSV on the server side.

In addition, the wheel is already invented! if we choose this option, since there is a PHP Class that is responsible for doing almost everything.

I refer to CSVReader, which can be obtained at this address from PhpClasses.org :

  

This is a class that can be used to parse CSV (Comma Separated Values)   files.

     

The class can also handle CSV files uploaded to a Web server using   forms.

     

The parsed files can be outputted as an HTML table.

     

This is a class that can be used to read a CSV (file   separated by commas). The class can also handle the rise of   files to a web server using a form. The files   Uploads can be presented as an HTML table.   * Or in any other way that we want, adding supplementary methods to the same class.

Having our class, which is nothing more than a php file that we can download at the address indicated above, we can upload and read any file using a code snippet like this:

<?php 
include_once('csvImp.class.php'); 
$csv = new csvImp(); 
 if (isset($_GET['action'])){ 
     $action=$_GET['action']; 
     if($action = 'upload'){ 
           $csv->fileUpload(); 
     } 
     if($action = 'form'){?> 
<form method="post" action="csv.php?action=upload" enctype="multipart/form-data"> 
         <input type="file" name="txtfile"><br> 
         <input type="submit" name="Submit" value="Submit" /> 
         </form> 
       <?php } 
} 
?> 

As you can see:

  • having a file called csv.php with a form to get our csv from our local environment
  • the class is included (we are supposed to have it on our server): include_once('csvImp.class.php');
  • the method fileUpload() of the class is invoked, which is responsible for reading the file and presenting it as a table

If we want to give the csv another use, we can simply add another method to the class that does what we need.

I leave this answer as another possibility, trying to open a little more the horizon:)

So ... Which side is better? That is the question!

    
answered by 15.03.2017 в 19:31
0

You need two things:

  • Your AJAX call must be async so that the browser does not "hang up".
  • Your server is taking a long time to process the response and several elements may be sending timeout; the PHP processor already has set_time_limit so it can be the browser, use timeout in your AJAX call to enlarge this value. Finally the web server can be the cause of this timeout, you can enlarge the timeout of the web server but it varies if you use IIS, Apache, NginX, etc and if you use proxy in reverse, CGI, FPM, etc.

I recommend that you check the browser console to see what happens with the HTTP REQUEST regardless of what you see in your application so you can make better decisions.

    
answered by 15.03.2017 в 20:12