Importing a CSV via PHP and treating duplicate lines as 2 entries
So is it correct to say that duplicates 'within a single response' keep, duplicates 'across responses' remove?
Correct. duplicates within I keep, duplicates across I remove
If that's the case, do not insert the rows until you have parsed the whole response. That way you only query your database against previously inserted rows.
$new_lines = [];
while ( ($data = fgetcsv($handle) ) != FALSE ) {
// ...
$exists = money_line_exists($timestamp, $data[1]);
if(($exists == 0) && ($amount != "0.00") && (is_numeric($data[1]))){
$new_lines[] = [
$timestamp,
$data[1],
$amount,
$commission,
$data[4],
$signup_date,
$data[6],
1
];
}
}
foreach ($new_lines as $line) {
add_new_money(...$line);
}