SSIS Performance drop when adding parameters

SSIS Performance drop when adding parameters

I'm using an OLE DB Source in SSIS to pull data rows from a SQL Server
2012 database:
SELECT item_prod.wo_id, item_prod.oper_id, item_prod.reas_cd,
item_prod.lot_no, item_prod.item_id, item_prod.user_id, item_prod.seq_no,
item_prod.spare1, item_prod.shift_id, item_prod.ent_id,
item_prod.good_prod, item_cons.lot_no as raw_lot_no, item_cons.item_id as
rm_item_id, item_cons.qty_cons
FROM item_prod
LEFT OUTER JOIN item_cons on item_cons.wo_id=item_prod.wo_id AND
item_cons.oper_id=item_prod.oper_id AND item_cons.seq_no=item_prod.seq_no
AND item_prod.lot_no=item_cons.fg_lot_no
This works great, and is able to pull around 1 million rows per minute
currently. A left outer join is used instead of a lookup due to much
better performance when using no cache, and both tables may contain
upwards of 40 million rows.
We need the query to only pull rows that haven't been pulled in a previous
run. The last run row_id gets stored in a variable and put at the end of
the above query:
WHERE item_prod.row_id > ?
On the first run, the parameter will be -1 (to parse everything).
Performance drops between 5-10x by adding the where clause (1 million rows
per 5-10 minutes). What is causing such a significant performance drop,
and is there a way to optimize it?