I could not find anything except for this one video that explained which quests to do for Idol slots. So after making a new character and running through every single quest I could find I have compiled this list for so hopefully less people have to struggle to find that one obscure quest to get those last two slots.
A bit of a warning for those that care, there will be spoilers.
Main Story Quests: Saving Last Refuge, The Admiral’s Dreadnought, The Lance of Heorot.
Side Story Quests, An Ancient Hunt (time rift found in the Shattered Valley near the waypoint), The Corrupted Lake (time rift found in Lake Liath near the waypoint), The Sapphire Tablet (Quest given in the Oracle’s Abode by the Shrine Maiden), A Heoborean Cure (Given in Heoborea by an old man), Liberating the Nomads (Given in the Nomad Camp by the Nomad Survivor).
Those are all you need to do for the full 4X4 slots available currently.