A mentally ill parent will have difficulty obtaining sole custody of their children if it's contested as compared to an otherwise equivalent non-mentally ill parent. How difficult depends on how mentally ill the parent is; having the blues occasionally is not a substantial impediment to caring for your children while hallucinations or threatened suicide or homicide are quite worrying to any reasonable judge (and your local child protection agency).
Optimally a parent seeking custody is not mentally ill at all. Unfortunately, life doesn't work that way; people who are (more or less) blameless for their mental illness often need help to overcome it.
If you are fighting with the other parent or a guardian (or whoever) for custody of your child(ren) and you are alleged to be a person in need of psychiatric treatment, then you should probably seek it. If the allegation is false or is greatly overstated then your counselor can come testify to that fact or offer a report to the court and thereby undermine an important part of the argument against you. If a professional concurs that you do need treatment you can obtain the treatment and undermine an important part of the argument against you.
Generally, in my experience with family court, if you are, in fact, mentally ill:
- Everybody who's going to testify thinks you're mentally ill, and will generally, under cross-examination, admit to those elements of your behavior and to past incidents that reinforce that narrative, even your mother; your attorney will attempt to get them to "explain away" the behavior in a way that doesn't undermine your case, but it won't be as effective as you want because the objective facts are damning while the "explaining away" is self-serving;
- Lots of your past behavior, behavior that seems reasonable to you and which you can explain to your own satisfaction and that of your close friends, can be cast in a way that will illustrate your illness even if the behavior or incident is actually unrelated to your mental illness;
- There is usually an audio recording (usually a message left on a phone, but very often a video) of you saying something that sounds mentally ill or threatening, and the judge will hear that, and;
- Your denials will generally be ineffective since your even your mom admits to your crazy behavior.