I haven’t attempted respeccing passive tree points, but what I would suggest for that is to use a variant of the current system (if my proposed idea IS the current system, then I have no complaints). Essentially, you still pay gold to respec one point, with the gold cost scaling up every time you respec a point up to a cap (let’s just say 5,000 gold as an arbitrary potentially-reasonable number). When you remove a point from the passive tree, it reduces the overall level of your tree until you put the point back in. If you reduce the level below the point where you have skill points invested in a node, those skill points stay invested but you gain no benefit from the invested nodes until you put your points back in and your passive tree level reaches them again. If you want to remove points from a node that acts as a pre-requisite for another node that you have put points into, the points stay in the other node, but you lose access to all of that node’s benefits until you put enough points into the base node that you meet the pre-requisites for that other node.
This version of the system would allow for minute changes to the passive tree at any level, while still making it difficult to significantly alter your tree. If you realized that your core build concept was fine but you’d misplaced 10-15 points in bad places here and there, this system would allow you to correct that regardless of where those points ended up being. However, ideally the system would also be prohibitively expensive for trying to completely change from one build concept to another, in order to suggest rerolling as an alternative to respeccing, while still ultimately allowing it for players who really hate rerolling and who are fine with painstakingly grinding out enough money/resources to eventually buy a full respec.