On "deepness" and complexity of IPython documents

This post argues about using an outliner interface for interactive documents wrote as IPython Notebooks, exploring why this is convenient, from my personal perspective as a casual user of IPython, but mostly as a teacher, researcher and consultant, a place where I think that IPython has a lot of potential.

Fernando Perez, first author and project co-leader of IPython, has told about the explorative nature of scientific computing and how this holds also for a lot of computer users. I agree. Most of time, (scientific) users doesn't have a strict set of predefined rules to guide/constraint their interaction with computers. A question then is how this explorative nature of computer interaction will start to show progressive structure when complexity of exploration and writing increases. This is a problem that any writer confront and is even more important/visible if you have interactive documents. To deal with it, most of the time the document starts to split on several minor documents which structure some part of the writing (in the form of section, chapters, tables and so on) with references to external data, like bibliography and images in some kind of embedding technique. I used this approach for writing my undergrad and master thesis' using TeXmacs, an interactive scientific structured document processor which honors its motto, "writting is a pleasure", in the case of scientific interactive docs. I used TeXmacs extensively and was one of the main translators of the Spanish manual, but my lack of knowledge/time to create templates and to extend/modify it using Scheme and my necessity to publish more fluently on the web made me look other solutions, mainly for using them with undergrad students in linear algebra and calculus and post-grad students on education and design who don't use/need the mathematical interactive stuff so much. As them, I was writing not so much mathematically oriented documents, but more the kind of technical docs that you find in community wikis for LUGs (Gnu /Linux User Groups). So math stoped to be a common writing necessity in a niche and I switched for wider interactions like web publishing. To make a long story short, I found Tiddly Wiki and Leo Editor and used the first for the most of my web writting (see my online microwiki) and the last for organizing my off-line stuff, from todo list, to reading/modifying/writing scripts, to note taking and progressively Leo started to hold memories of my most important "textual interaction" with the computer. I sacrificed interactivity and typography of TeXmacs for the integration ("one tree to rule them all", including, by textual reference, any data on the computer) and programmability offered by the Leo tree structure. This "discontinuous environment" of Leo plus Tiddlywiki plus Unix shell works well for most of my workflow.

When I start to wrote more and more complex documents, mainly with my PhD Thesis, I started to increase my use of Leo, mostly because writing is a non-linear activity and having trees to organize it let me see the big picture and the detail at the same time. I started to write several parts of my thesis in the order I found them and use the deepness of the tree to put some "internal" notes, that on the pdf exportation, where invisible for the reader, but let me organize the internal structure of the writing and justify the "external" surface of the text (you can see a screenshot of my thesis Leo tree on the figure above). In fact I saw how having part of the "outlining writing experience" with students and coworkers increased their understanding of what was being done in other contexts, but because of the techie/geek nature of Leo and self-fulfilled "prophesy" of being a tool by developers for developers [1], difficult to install and use for the casual user on most platforms, students and coworkers have read-only access to this structuring experience.

I created then exporting scripts from Leo to wikis and pdf, using self-referential programming features of it (a theme for a post on its own), but the lack of the tree was a suboptimal experience in digital format, compared with the original Leo experience. I started to explore the possibility of using dynatree and wikis/etherpads to recover that experience of interaction with a complex multilayered and deep document. And then I rediscovered IPython with its amazing notebook interface experience, in fact I liked so much that I started to explore writing about this ideas on my new bliki (blog + wiki) using IPython instead of microwikis. It worked really well but I think that complex (scientific) interactive documents which "emerge" from exploration need a tree interface for progressive structuration, for the reasons already mentioned in the case of Leo. In fact I would argue that Leo and IPython share the profound interest for introspection and having this feature implement in the tree would make IPython notebook really powerful. You could think of an IPython notebook as cells organized/chunked in subtrees which enable another level of aggregation to cells and I think that trees and cells are almost all of what users would need to organize IPython documents of the complexity of a thesis. Even with this metaphor of interaction, users could build complex web applications made on IPython using internal subtrees for internals of the apps and the external part of the trees for what the web user can interact with, in a similar way of hiding internals of my writings to my thesis reader (but from now this goes beyond what this post want to propose). I agree with Brian Ganger, IPython's co-leader, about featurism on free open source software and I hope the IPython community think that a proper metaphor for writing progressively structured complex and deep documents is needed if we want IPython to be the tool for a continuous writing experience in this context and trees are the way to go in that sense. Of course experimentation would be needed and hopefully I will not be writing the code alone to probe my thesis and this idea would be sound and interesting, even coming from a non-programmer ;-)