Looking for Open Source Math Software with Poor Documentation
Solution 1:
Consider Maxima, the open-source version of the computer algebra system Macsyma (similar to Mathematica and Maple). If you post a similar question to their mailing list then I suspect that some folks would quickly point you at various parts of the system with little or weak documentation. Since Maxima is a general-purpose system you are more likely to find some area that intersects your interests and expertise.
Solution 2:
R software actually has good documentation in many areas, but it is becoming quite vast with many add-on packages which are not part of the base. I would target a few contributed packages which have bad documentation and offer to improve. Cannot suggest any particulars but refer to all of the contributed packages at: http://cran.r-project.org/web/packages/
Solution 3:
I imagine there's room for improvement in the documentation of any open source math project. So instead of looking for a project that's poorly documented, you might look for a project that interests you and then see how you can help.
Most open source documentation is fairly low-level, one step above reading code comments. Often what's missing is high-level introductions or case studies. When a projects as a good "quick start" guide, it is much appreciated. Maybe you could find a project in need of a quick start guide.
Solution 4:
If it is also allowed that the software itself is poorly written ;-) then you might be interested in my matrix-calculator MatMate. Here we have much less functionality to describe than in Maxima et al. - just a simple, but nontrivial matrix "pocket" calculator.
Because there are many errors you have even the possibility to include criticism concerning technology/ implementation and concerning the syntax/ semantic-paradigms (which may be interesting for someone who is beginning a writer's carreer). http://go.helms-net.de/sw/matmate/index.htm The exisiting documentation is in german language but possibly gives an idea how to approach a not-too-big introduction/documentation.
(well, even if this all sounds a bit funny/sarcastic when I reread my own msg - that was not intended but -perhaps- unavoidable. Instead maybe such a tiny software is indeed a better start than a full-featured one)