PHP - Browse large CSV file with function

2

I've been looking for how to find a value on one line and return another value from another column on the same line in a CSV file.

This is my function and works perfectly on small files:

function find_user($filename, $id) {
    $f = fopen($filename, "r");
    $result = false;
    while ($row = fgetcsv($f, 0, ";")) {
        if ($row[6] == $id) {
            $result = $row[5];
            break;
        }
    }
    fclose($f);
    return $result;
}

The problem is that the actual file I have to work with weighs 4GB and the search time for a value is tremendous.

Navigating in Stackoverflow I found the following article: link

There it provides a function that in theory facilitates the search of values inside huge CSV files (it seems that it does it in parts), this is the function:

function file_get_contents_chunked($file,$chunk_size,$callback)
{
    try
    {
        $handle = fopen($file, "r");
        $i = 0;
        while (!feof($handle))
        {
            call_user_func_array($callback,array(fread($handle,$chunk_size),&$handle,$i));
            $i++;
        }

        fclose($handle);

    }
    catch(Exception $e)
    {
         trigger_error("file_get_contents_chunked::" . $e->getMessage(),E_USER_NOTICE);
         return false;
    }

    return true;
}

And the way to use it seems to be this:

$success = file_get_contents_chunked("my/large/file",4096,function($chunk,&$handle,$iteration){
    /*
        * Do what you will with the {&chunk} here
        * {$handle} is passed in case you want to seek
        ** to different parts of the file
        * {$iteration} is the section fo the file that has been read so
        * ($i * 4096) is your current offset within the file.
    */

});

if(!$success)
{
    //It Failed
}

The problem is that I do not know how to adapt this function to my search function that I initially put, my knowledge of PHP is not very high.

    
asked by Kokox 28.08.2017 в 06:03
source

1 answer

0

The above function runs a file by parts, but these parts are divided into fragments of 4096 bytes and not by lines.

The solution I recommend is that you save your csv in a mysql table at once and then you can execute more specific queries.

To save a csv in a mysql table you can use several ways:
-mysqlrestore - > Native command of mysql
-phpmyadmin - > Has a function to import
-php - > You will create a script that will delete the rows already saved from the csv file and thus shorten the file until it is completely processed.

If your server gives you errors that it can not process a request with a 4GB file to save it at once (phpmyadmin option or php function itself) it removes all the runtime and memory limits.

    
answered by 16.09.2017 в 21:54