Working directly with Excel spreadsheets has always been very slow in PowerShell. Its due to the way PowerShell works with the Excel COM objects. I've always recommended doing as much work as possible in a CSV file then reading that into Excel rather than trying to write directly into Excel.
PowerShell 5.0 has much improved COM support and is much faster. I'd try using that version if I was you