So I’ve had this issue a few times and I’m pretty sure it’s reproducable (at least for me).
Step 1 - have a few summons
Step 2 - go to town
Step 3 - go back to the zone without your summons
The only workaround is to take the skills off the skillbar, because they for some reason still count as summoned and when coming back to town, they are still there - just not in the map.