Research tools, Emacs, SQLite, and self-hosting briefing notes

For more posts on Emacs, check out the Emacs Carnival 2025-07: Writing Experience

The tools researchers use are developed over years of seeking efficiency, trialing new things, re-building workflows, creating templates, and upgrading to "new" tools that come along. We also tend to fall back to what is familiar and easy during busy times. However, some of the trialled process stick each time if they provide a benefit. The end collection of tools tend to be unique to each workflow, history, and skill of the researcher.

I have been a researcher for most of my adult life. My current workflow is very specific and I like to think tailored to the work I do quite well (who doesn't feel this way about their tooling?). That workflow heavily involves data imported into SQLite databases, writing briefing notes using Org Mode in Emacs (where I like to live), exporting using Pandoc, and building charts with the old, but still usefully simple Chartbuilder. Of course, I use several other tools such as Gnumeric, LibreOffice, and Pandoc to deal with products sent to me. Hugo and MkDocs are an essential component of sharing information. And, the nature of the workplace means that the majority of time I have the ubiquitous Firefox and Mutt running and open to resources and email.

I have previously outlined some of the programs I use in posts on Linux, GNU, and Open Source. I do not claim to be a power user of any of these tools. I say this because I find watching seasoned programmers use Emacs to build software and modify their tools is as inspiring as watching any craftsperson wielding tools I have not mastered. But, over time I have come to feel so comfortable with these tools that my efficiency using anything else drops to near zero. As such, my evangelism for the Free Software movement and the principles that set the foundation for the development of all my tools has only grown over time.

A decent familiarity with Emacs, Mutt, i3wm, and SQLite has made my work feel empowered, fun, and efficient while giving me a weird sense of being part of a movement every time one powers on. My contribution to the software development itself is extremely limited of course as I am a user, not a developer. But, the hope that learning those tools which can advance your work, modifying them deliberately and specifically to suit your unique ways of thinking is part of the whole point of Free Software ethos. That modification and sharpening of tooling is also an under-appreciated aspect of any work environment and job satisfaction.

The desire to use my own set of programs has also had an outsized influence on the places I have worked. From grad school supervisors to activism to jobs I have accepted (and left), software choice has been an important part of the decisions. If I am forced to use bloated, slow, and restricted Microsoft products on weirdly locked-down hardware I am not going to last long. Frustration of using this kind of garbage software, firms forcing people to work in unproductive and constrained ways may make IT's jobs easier, but I think it destroys the will to produce and undermines quality output.

Two recent additions to my software list are SQLPage and Datasette. Both offer some very nice features for those of us who live knee deep in data. While SQLite (and the great SQLite Browser) are most of what is needed to refine data, SQLPage and Datasette bring the data to life for those not satisfied with the static report/graph alone. They both support the production of good data products in a way that reminds me of my Emacs Org Mode workflow with Pandoc.

While developing a final product using data, forms, and graphs are rather easy using SQLite, like with Org Export via Pandoc, SQLPage and Datasette remove entire steps to get a workable output. This makes a wonderfully quick (and not so dirty) way to template ideas, collect data, and show output to users.

At the moment, I think the uniting of Org export, MkDocs, Hugo, SQLPage, and Datasette—depending on the product needed—essentially completes the researcher toolkit.

Below are some ways I develop for an audience for briefing notes, blog posts, slide decks, data presentation, and structuring data presentation.

Self-hosting with the static site or SQLPage is easy and extremely cost effective. Briefing notes have limited broader interest, so putting them on a small box has been easy for me without dealing with complicated overhead and upkeep time.

While one site was put behind Cloudflare because it was being harassed by AI bots, others see a rather steady access well below the need for bot intervention. I have seen the option of using obscure URLs, subfolders, an explicit robots.txt file, and not posting links on social media makes it easier to avoid the AI crowded internet.

For more secure password protecting needs, it has been easy with SQLPage, datasette, and older style webserver password protection. Fossil SCM has also allowed easy user account creation for access to things like threads for discussion and versioning for development.

Most of my hosting has been done on two systems:

1. Linode VPS using Arch or Debian (depending on my mood) for personal projects.
2. Home servers based on Pine64 using (again) Arch or Debian based distros depending on the device and ease of install.

NGINX as a webserver has treated me very well over the years for static, Datasette, and SQLPage sites. Moving off PHP and MySQL to static sites, SQLPage type binaries, and SQLite has reduced work time to maintain and upgrade services.

Basic server security such as using key-only login for SSH, Fail2Ban, UFW, Lets Encrypt, and non-standard ports along with CloudFlare when necessary have kept intrusion attempts to a minimum.

Running on bare metal and not using Docker or other packaged installs and secured local user environments are low level security, but I have not faced anything that could not be mitigated through basic tweaks.

Fossil SCM has been a dream for set-up compared to Git. All-in-one basic versioning, easy distributed backups, and a nice level of security has been great for upkeep. It even gives a basic interface for edits, wiki, and discussion for the few who collaborate on some products.

Datasette and SQLPage provides basic authentication and while I do not subscribe to security through obscurity, there is consistently near zero activity on my semi-obscure URLs and non-standard ports.

The benefit of Datasette running is you can drop SQLite files in a folder and they just show up. Simple views can be created for custom look-ups and links to specific queries bookmarked even with forms for filtering with maps and graphs where appropriate.

Taking it up a notch to SQLPage makes it even better. Using the same SQLite files, a best of SQL and you get dashboards, lists, export buttons, forms for entering new information, and even permission level access. Most of what I need for data exploration, collection, and displaying can be accomplished here. With the FTS5 function in SQLite and the SQLPage front end, you can easily set-up search to meet most needs.

I have toyed with using the text component in SQLPage for longer text, but my workflow with Emacs, Fossil, and Hugo/or/MkDocs cannot be beat at this point. The programs are both great at what they do, it would be hard to get SQLPage to mimic it. Search in MkDocs is great and for my research blog and briefing note set-up, search in Hugo is sufficient. Stand alone Emacs Org Mode HTML export with some small CSS tweaks makes longer notes that need to be stand alone easily readable on a mobile phone a dream.

Org mode for writing, save to folder, sync to Fossil, script to sync and build Hugo/MKDocs via a simple SSH script. All the fixings, redundancy, security built in and essentially free.

I think most folks do not realize that you can write in Org Mode style using Hugo and it will publish without problems.

Hugo and MkDocs also allow easy WebApp setup so those following the sites can install it on their phones. RSS is baked in.

Push notifications of updates is possible through free tiers of things like OneSignal and Signal groups are also an excellent way to get folks to subscribe for notifications of updates. Mass email is harder to setup these days which is annoying, but mass mail providers for low traffic political blogs and internal briefing notes are easier to establish.

Even for Fossil, notifications and updates can be automated in useful ways for folks.

Finally, I have found sticking public sites like CPress.org that get more visitation than nothing behind Cloudflare for an easy fix for bots and annoyances.

Research notes and data are part of any organisation doing serious work. It is important to have that data easily findable and readily available when people need it. This is not usually when it is produced. Having self-hosted solutions that allow for app installation, forums, notifications, search, and data exploration and collection brings the research to life. Allowing me to do this while using my own workflow, tools, software, and set up is priceless.