My brother who was left out of dad's will is challenging it. What can we expect if we settle?

In a settlement, instead of getting nothing and having to pay his or her own legal bills, your brother would get something, which would deplete the size of the estate that would otherwise go to the other heirs. Parties can always settle a dispute any way they want, but you can start by looking at what your brother might get if he wins. If a Will is thrown out because it is invalid, say because the deceased was incompetent when it was made or because there was fraud or someone was exercising undue influence, then your father’s property would pass under the laws of ‘Intestate Succession’ of your state (the laws that apply when there’s no Will). Those laws are different in every state, but let’s say that your father’s wife is dead and you and your brother would get everything under your state’s laws. If he wins the Will Contest, then, your brother would get 50% of the estate less legal costs (assuming that you two are your father's only children).

This doesn’t mean that you have to give him 50% of the estate as a settlement. You can consider how likely it is that he will win the challenge. Since it is both difficult and expensive to win a Will Contest, your brother may be willing to settle for a lesser sum in exchange for not having to bear the cost or take the chance of losing a challenge to the Will. Even if he wins, he will have to pay his own legal costs and the amount he receives will have been decreased by the legal costs the estate spent defending the Will. A moderate estate can be consumed by the costs of a Will challenge pretty quickly.

Because of all this, if the estate isn’t really large you might offer as little as the lesser of $10,000 or 10% of the estate as a settlement.

What you offer as a settlement will depend on what you feel is fair and might involve the reasons why your father cut your brother out of the will in the first place. If your father had good reasons, you might not feel that you want to settle for giving your brother 50% of the estate. On the other hand, parents sometimes vacillate between kids, changing the Will repeatedly with one favored, then another. It thus becomes a matter of chance who is "in" the Will and who is "out" at the time of death. In such cases, even if lack of mental competence cannot be proven, the "moral" settlement may be something far closer to a 50-50 split.