1. e4 { [%clk 1:30:52] } 1... d6 { [%clk 1:30:53] } 2. d4 { [%clk 1:30:02] } 2... e5 { [%clk 1:31:17] } 3. Nf3 { The Philidor. I wouldn't play this as black without significant preparation because of the space disadvantage that black accepts in the centre. } { [%clk 1:29:06] } 3... exd4 { [%clk 1:31:41] } 4. Nxd4 { [%clk 1:28:38] } 4... Nf6 { [%clk 1:32:05] } 5. Nc3 { [%clk 1:28:58] } 5... Be7 { [%clk 1:32:31] } 6. Bf4 { [%clk 1:27:25] } 6... O-O { [%clk 1:32:53] } 7. Qd2 { [%clk 1:27:23] } 7... Nc6 { [%clk 1:33:07] } 8. O-O-O { White has developed well now, has control of the centre. All of white's pieces (apart from the Kingside rook) have mobility and are ready to get involved. } { [%clk 1:27:41] } 8... Nxd4 { Black lags in space. The dark-squared bishop is not doing much. Black prefers trading off a good knight to compensate for that. Black must now plan to gain space / improve mobility of their pieces. } { [%clk 1:33:27] } 9. Qxd4 { [%clk 1:28:08] } 9... a6 { Planning to gain space on the queenside. } { [%cal Gb7b5,Gc7c5] [%clk 1:33:51] } 10. f3 { [%clk 1:26:55] } 10... b5 { [%clk 1:32:42] } 11. g4 { [%clk 1:22:00] } 11... c5 { I like this plan for Black. } { [%clk 1:30:05] } 12. Qe3 { [%clk 1:20:58] } 12... b4 { [%clk 1:30:27] } 13. Nd5!? { This is an exchange sacrifice. } { [%clk 1:21:19] } 13... Nxd5 { Black has to trade. Allowing white to take the e7 Bishop is not an option because then the d6-c5-b4 pawns become easy meat. } { [%clk 1:30:22] } 14. Rxd5 { [%clk 1:21:48] } 14... Be6 { [%clk 1:29:30] } 15. Bc4! { [%clk 1:22:04] } 15... Bxd5 { [%clk 1:17:29] } 16. Bxd5 { With Black's light-square bishop gone, white's light-squared bishop has no competition. It is peerless. All the light squares are belong to white. I would prefer to play white in this position. } { [%clk 1:22:06] } 16... Rc8 { [%clk 1:16:58] } 17. g5 { [%csl Rf7][%cal Gh2h4,Rd5g8,Gh4h5,Gg5g6] [%clk 1:15:01] } (17. Bb7?! { Unnecessary complications and gives Black counterplay. } 17... Rb8 18. Bxa6 g5 19. Bg3 Bf6 20. Rd1 Bd4 21. Rxd4 cxd4 22. Qxd4) 17... c4 { [%clk 1:17:22] } 18. Kb1 { [%clk 1:11:46] } (18. h4 { Is also good for white because h5-g6 comes fast and white's pieces are capable of guarding the king. } 18... Qa5 19. Kb1 c3 20. h5 cxb2 21. g6 Rc3 22. Qd2) 18... Qa5 { [%clk 1:08:31] } 19. h4 { [%clk 1:09:07] } 19... Rc5 { [%clk 1:07:02] } 20. Qd4 { [%clk 1:04:36] } (20. h5 { was also okay } 20... Rxd5?? { does not give black anything in return. } 21. exd5 { [%csl Ge7][%cal Ge3e7] }) 20... Rfc8?! { Inaccuracy. Rd8 was best. } { [%clk 0:33:04] } (20... Rd8) 21. h5 { [%clk 0:59:35] } 21... Rxd5 { [%clk 0:22:08] } 22. exd5 { White's earlier plan was centred around the light squared bishop. With that gone, there needs to be a new plan. White has a lot of space on the kingside. With Black's dark bishop unable to enter the game, I don't see a queenside threat for White. If I was white, my plan would center around making use of the kingside pressure and the open e-file. } { [%clk 0:59:59] } 22... Bf8 { [%clk 0:22:02] } (22... Qc7? { If not ...Bf8, then } 23. h6 Bf8 24. hxg7 Bxg7 25. Qe4) 23. h6?! { Inaccuracy. g6 was best. } { I don't like this move because it solidifies the kingside structure. The d6 pawn is defended by the Bishop. The Queen and Rook can come in and White cannot add pressure there. I like g6 more. } { [%clk 0:48:02] } (23. g6 fxg6? 24. Qe4 Qd8 25. Qe6+ Kh8 26. hxg6 h6) 23... Qc5 { [%clk 0:17:24] } 24. Qe4 { [%clk 0:48:26] } 24... g6 { [%clk 0:17:50] } 25. Be3 { [%clk 0:45:48] } 25... Qb5 { [%clk 0:17:48] } 26. Bd4? { Mistake. Qd4 was best. } { White's idea here must have been Bd4, Qf4-Qf6 to create deadly threats. White must've underestimated Black's counter-play paired with the defensive idea of f5. The d5 pawn is important but weak. } { [%clk 0:45:14] } (26. Qd4! Rc5 (26... Re8 27. Bf4 Qc5 28. Qxc5 dxc5 29. Rd1 $16) 27. Qf6 Rxd5 28. Bd4 Rxd4 29. Qxd4 $16) 26... Re8 { [%clk 0:17:15] } 27. Qf4 { [%clk 0:40:18] } 27... f5 { [%clk 0:17:40] } 28. gxf6 { [%clk 0:36:41] } 28... Qxd5 { Maybe what Nodirbek missed in his calculations? Taking the pawn also defends the f7 square against f7+ } { [%clk 0:17:54] } 29. b3 { [%clk 0:37:01] } 29... Kf7 { [%clk 0:13:06] } 30. Qg4 { [%cal Gg4d7] [%clk 0:35:53] } 30... Qf5 { [%clk 0:11:40] } 31. Qg2 { [%clk 0:27:30] } 31... c3 { [%clk 0:09:09] } 32. a4 { [%clk 0:25:55] } 32... bxa3 { [%clk 0:08:52] } 33. Ka2 { [%clk 0:24:17] } 33... a5 { [%clk 0:06:56] } 34. Rh4?? { Blunder. Qf2 was best. } { [%clk 0:14:41] } (34. Qf2 Rc8 (34... a4 35. Bxc3 axb3+ 36. cxb3 Qd3?? { The same idea does not work in this line because White gets counter-play with the queen able to participate in an attack against the Black king. } 37. Qa7+ $18) 35. Ka1 a4 36. bxa4 Rc4 37. Rd1 Qd5 38. Qe3 Rxa4 39. Rd3 Qb7) 34... a4 { [%clk 0:04:15] } 35. Bxc3?? { Blunder. Re4 was best. } { [%clk 0:15:05] } (35. Re4 axb3+) 35... axb3+ { [%clk 0:04:39] } 36. Kxb3 { [%clk 0:12:38] } (36. cxb3 Qd3 { Double attack } { [%csl Ge2,Gc3][%cal Ge8e2,Gd3c3] }) 36... Rb8+ { [%clk 0:03:36] } 37. Rb4 { [%clk 0:11:52] } 37... Qe6+ { [%clk 0:03:51] } 38. Kxa3 { [%clk 0:12:14] } 38... d5 { [%clk 0:04:04] } 39. Qf1?? { Checkmate is now unavoidable. Qg4 was best. } { [%clk 0:08:21] } (39. Qg4 Bxb4+ 40. Bxb4 Qxg4 41. fxg4 Kxf6 42. Bd2 Ke5 43. Be3 d4 44. Bd2 Kd5) 39... d4 { [%clk 0:04:02] } 0-1

About raga

This is the proposal I had drafted in June 2025: project proposal. The project was initially called pushpush, but I've since decided to rename it to raga.

The idea is to make a static site generator in OCaml that is capable enough to generate & deploy my own website + some other ideas that I have. I also want to use this as an excuse to write non-trivial software with OCaml--and in the process find out what is easy / hard to do for me in OCaml.

The project was planned to have 2 parts for me to build:

1. [done] A Handlebars templating engine
2. [done] The actual static site generator that would make use of the templating engine

Since then, a third got added and completed, which is a parser for the huml configuration language.

Links

1. raga:
   • nikochiko/raga: GitHub repository
   • This website was built with raga.
2. handlebars-ml:
   • nikochiko/handlebars-ml: GitHub repository
   • opam:handlebars-ml: opam package
   • handlebarsjs.com: Original (JavaScript) implementation of handlebars
3. huml-ml:
   • huml-lang/huml-ml: GitHub repository
   • huml.io: huml-lang official website
4. Project Proposal: Google Docs

23 November 2025

raga

Usability testing

Adding chess games to website

I implemented a PGN template for my own website to test the capabilities of raga. I had to allow front-matter in non-markdown files to make this work, but it works quite well. For example, this is the chess games list on my website: /chess.

I'm not particularly pleased with this approach because I need to replicate metadata information that is already there in the PGN into the front-matter. This is not strictly needed because it can be done from JavaScript, but this helps SEO bots and RSS feeds.

Building template with Unicode website

I did some more testing with trying to implement a website template with Unicode in the configuration. That case also works well. I ran into some problems with setting up the default case. I've added GitHub issues to track my work on this task.

The setup task is also slightly long in the sense that I need to setup the directory structure. I also discovered a bug here that expects the partials directory to always be present. I want to change that to throw a warning in that case.

Feature - Build summary

I added a feature to the CLI to display useful build stats at the end of every build. This gives some visibility into what was generated:

It also shows how long it took to build something. I like the current build times too.

publishing to opam

raga has now been published to opam. I will start talking about it and provide a single binary for installation after this round of fixes and some testing.

handlebars-ml

I'm quite happy with how stable this has been. I want to refactor the helper functionality to:

1. have more power, so that for example one helper can be used to add the boilerplate HTML for enclosed markdown content.
2. give better errors and warnings when the arguments passed are incorrect (for example, when they have wrong types).

huml-ml

I recently worked on the BNF for this and we have a new spec with a few changes that I want to update the OCaml and other parsers for.

TODOs:

• latex support for math notation (find a page that needs this)
• update huml parser for 0.2 spec
• only throw warning when partial directory does not exist
• throw warnings for incorrect helper arguments
• refactor helpers to be more powerful

03 November 2025

raga

I got started and have a working raga implementation. This website now runs on raga instead of hugo. I used the same content files for the most part, with some minor changes, and I was able to build a somewhat large website with it. This gives me confidence in its power. While working on it, I found some bugs/improvements in the handlebars implementation that I then fixed.

I think I've come up with an interesting config solution that works kind of like a pipeline-definition language. I define the source files, the processing to be done on them, and the place to output them. I'm working mostly with markdown and have frontmatter support for exclusion rules. I also want it to be a general-ish pipeline tool that can let me build arbitrary files. For example, I want to take my PGN files that contain chess analysis, and publish them as HTML pages with the correct transformations. Adding new functionality is done with OCaml code at the moment. I might want to change that to be able to generalise to bash scripts at some point. I'm also getting the feeling that perhaps s-expressions might have been a nicer, powerful format for the config and that I'm trying to emulate them into huml for the config syntax.

I have been able to add all the features from the previous wishlist, and more. I'm happy about that.

State so far:

• working raga static site generator implementation
• config parsing is strict--will error for bad key/values.

TODOs:

• compile a non-markdown file with raga (e.g. PGN with chess files)
• add support for latex

18 October 2025

handlebars-ml

I have been able to get to a version of handlebars-ml that I feel comfortable using. I have also published it to the opam package manager. I have added enough tests for me to feel comfortable with the featureset that is in there. There is support for helpers, partials, and all the basics. Inline partials are not yet supported but I am okay with that because the same usecase is possible with workarounds.

Some areas where I feel like improvements are possible are:

• better support for date & time parsing / rendering. I don't want to bloat the templating engine, so I'm thinking this can instead be part of raga.
• the CLI is not enough on its own to support adding new helpers. As of now, the workflow for that use case is to use the library, write OCaml code for the helpers and compile that. I think I'm okay living with this for the moment. I have been considering small scripting languages for this use case though.

huml-ml

Fresh off writing a new parser for handlebars, I got tempted into writing another one for a new & strict configuration language called huml. I find it interesting and useful. I think I want to add support for parsing frontmatter and configuration in huml. Maybe even start with it as the default.

I wrote about it to Kailash, a maintainer of this project, who was happy to see it. Since then, I have moved the implementation upstream to the official huml org. I am also working on a BNF grammar for it in addition to some bug fixes in other implementations. I have been enjoying this. I enjoy writing parsers and this kind of working with the community makes me feel useful.

(I have oddly enough written >3 parsers, which I think is more than the number of parsers that most engineers are likely to write).

raga

Per the schedule, I think I'm running ahead on the handlebars-ml work and just-on-time on raga. I have a design architecture in mind. I think the goal I have in mind--to be able to compile my own website--keeps me grounded in terms of what I need to build.

For the first version, I have in mind that it should work somewhat similarly to hugo, which is the generator that I currently use.

Specifically, it should be this featureset:

• Parse frontmatter in huml and pass that to the theme/template
• Parse config in huml and pass that to the theme/template
• Allow generic templates in a theme (e.g. _blog.html that will work as one template for all blogs).
• Translate markdown content to html
• Make a template for the RSS feed

I plan to work on this in the next couple of weeks.

My personal celebration this week is that I did released the OCaml handlebars templating library that I was working on. I felt happy when I read the "Thanks" and "Your PR ... has been merged" emails for publishing to the opam repository. This can now be installed with opam install handlebars-ml.

I think this is software that I am proud of. I have tested it enough for me to feel confident about relying on it. I should likely do a more detailed write-up about this because I did learn several things in the process of making it: ocamllex, menhir, sedlex, when claude code was useful, how opam publishing works, yada yada. Even the opam CI had an interesting, thorough testing process--for example that the project gets built & tested on different OCaml versions and with the latest & lower-bound versions of the dependencies. This is a new testing model that I will carry with me.

I feel more fondly about handlebars than I did at some point during the making of it. My confusion/distaste plotted against time would be a monotonously increasing graph with a sudden drop from the peak close to completion. It has pitfalls and semantics that silently ignore where things should probably crash soon. That said, I feel comfortable enough with those to be able to make use of the convenience features that sit on top. Perhaps I should add optional warnings in these cases to compensate. In any case, both the good & bad of this deserve a more detailed write-up. In summary, I feel happy about having completed this. I am also happy about the error-reporting & CLI interface that I built in, and so I would in fact like to make use of it in my own projects.

huml

I am procrastinating on the category of activities that this writing would also fall into. I am bike shedding on the static site generator that the templating engine is supposed to be a part of. My latest side quest after being satisfied with the handlebars parser was to perhaps write another OCaml parser--this time for a configuration language called huml. huml stands for HUman-oriented Markup Language, and is an experiment by kailash to make a replacement for YAML. I agree with the dislike of YAML. I do sometimes find TOML less than easily readable. I think that is because the hierarchy is not obvious from the visual representation of the text as it is in YAML. I have made some progres on this. If I do get to a version that works with the tests that are put up, then I will very likely consider using it for my static site generator and writing to kailash about it.

I also really like that there is a language-neutral test set for it, so I can actually verify that my implementation works. This is great for times when the specification is ambiguous. I think a formal grammar specification (a BNF) would also perhaps be useful towards that end.

life

I had fun making a short video in the format of old good-morning photos with hindi songs for a friend's birthday. I did that in p5.js and took help of the silicon-family techbros to get there. I am amazed again about how powerful p5 is. I will likely consider making a tool to make it easier to get GIFs of animations or fix the saveGIF method they had because it did not work for me. I ended up doing a screen recording at the end and then stitch that with an audio file with ffmpeg.

I do enjoy playing around in the command line. It is satisfying to write an ffmpeg command and have it do the thing you want. I think of the SICP-ian analogy of the wizard's incantations.

I also visited the Indian Music Experience museum this week which was absolutely beautiful. I would like to share some pictures from the visit in a post.

website

• I fixed my website's RSS feed thanks to Abhinav's nudge.
• should absolutely get started on the static site generator
• need to fix my infra. i have setup comments but they don't show up on my posts because my laptop is turned off and i need to change the boot order to get it back up and runnning as a homeserver.

more to be added?

September 2025

1. Notation as a Tool of Thought, Iverson (1979)

I want my personal website to mirror - to the extent that I can - the experience that you would have if we met in a warm setting. That warm setting can be imagined as the range between me hosting someone at mine to us sharing a common interest; perhaps better described as an interaction that is somewhat friendship-shaped.

Depending on what is the intersectionality in the venn diagram of our interests, I could be taking about one of a few things (non-exhaustive):

• I self-host all my websites from home and actually from that laptop there
• I've been working on this really cool project. It's still a work-in-progress but I can give you a demo here.
• I ran into this problem at work recently. This is how we fixed it.
• I did something funky in my coffee making process and the result was interesting. I want you to try this out.
• I analyse chess games sometimes and I would like you to go through them too and let me know if you have thoughts somewhere. I can also analyse one of your games.

I see that just text is often not the right medium. I want my web space to be more dynamic. I want to be able to render chess puzzles and analyses of chess games here too. I want to share opinions on coffee, tech, travel, etc. and the more meta aspects of them. I also want it to be such that when you are on my website, you can navigate to the interactions and conversations that are interersting to you - and filter out the rest.

Creative Programming / Creative Coding

Here's what's been on my mind:

• making a programming language for origami is hard.
• string art is cool and can be replicated in ocaml-joy.
• comparisons with music

This is something I've been fascinated by since a young age. This was my first experience with programming that I liked and wanted to do more of. Of relatively late, I have been working with ocaml-joy (hosted on kaustubh.page/joy) where I had last explored tessellation art.

While in a fit of boredom, I remembered that @anandology had once described his explorations with string art. I took inspiration and started hacking on OCaml Joy's online editor. The result was this starter code that comes with a few abstractions to emulate a nailboard and threads. It's rough around the edges, but I was able to take an instructables post and reliably see it in my canvas (see the heart photo at the top).

Along similar lines, I have Wolfram's A new kind of science on my reading pile. That book is full of interesting visualizations that I want to get to.

I can't help but see the similarities between string art and music. There's a period with which patterns repeat themselves (a bar). There is a play on how big each step is and that changes the output you will find. Two interesting explorations come to mind here:

1. conversions for these art steps to music exist and must surely sound good (time to test the hypothesis)
2. a program that uses code to represent a certain visual animation is a program that represents a certain musical composition. what combination will look and sound the best?

All of this is another way to say that visual art, when abstracted enough - has similarities to musical art, when abstracted enough. In math, we do this all the time. Abstract something in one place (say, counting -> numbers) and then play with that abstraction until we end up with natural language (LLMs) or even utter nonsense such as a complex plane and the Mandelbrot fractal.

Besides this, I also (briefly) felt the motivation to bulid a programming language for origami. A way to describe the folds unambiguously, such that an interpreter will know for sure the state of the paper and be able to create a 3D visualization.

The more I thought about it, the more I realised the difficulty of this problem. How do we describe a fold in a way that feels more or less natural to a human and is at the same time unambiguous? An instruction as simple as "Fold such that corner P aligns with corner Q" can have multiple ways of being done in 3D space. It gets more complicated with more folds. How do we keep track of the state? How does the user identify one edge from another? A single fold can create multiple new edges and corners in a way that isn't easy to track without an origami REPL.

This is a problem that, nonetheless, is interesting to me. It feels like a challenging exercise in a way that is comparable to a hard, multi-day trek.

• tixy.land
• similar to the origami language, can there be an unambiuguous language to describe crocheting?
• spirographs
• how do i extend the string art abstractions to accommodate more complex nail-shapes - such as the flower hole pattern?

Think about Rust, OCaml, even typed-Python. One is easier to implement and one is more robust to bugs. You don't want to launch a software 100% free of bugs after it has become obsolete. At the same time, you also don't want to launch a software so premature that it is as reliable as the flip of a coin.

There are two kinds of software here that change which side of the trade-off you would rather be on:

• The fresh new startup kind: you want to be the first to market. It's probably okay if you have a few bugs on the edge cases, but the fear of becoming obsolete before delivery is stronger. You can also improve the product from its users after the launch. You should be more afraid of not having any users by the time you launch.
• The incremental featureset kind: you're building a new component to a software that already exists and has users. You won't lose all your revenue if you are late by a few weeks. You will, however, lose revenue if your new commit causes the website to crash for some paying customers.

My hypothesis is that when starting out, you want to build for speed. Once built, you want to focus on making it better. An approach that gives you bug-free software from the ground up will take more time from the beginning. Instead, the ideal approach would let you start loose and then make your software more and more robust as your company matures. No programming language factors this into its construction. Most companies will just rewrite at some point.

I guess you want the best of both worlds and the ability to move slowly from one edge to the other. I'm starting to appreciate Python for this.

I completed the exercises. The idea of using a keyboard to do complicated editor actions - things like skipping through the start and end of English words, going to the beginning and end of lines, skipping to the next empty line - felt powerful. I have since then, through recommendations, been a full Neovim user. I would hate being in a world that doesn't have Vim navigation. I felt compelled to install Vimium in the browser too.

I have however also been bad at keeping up with all the extensions. I don't want to concern myself with what treesitter does and what extension I need to lookup the signature of a function. For over 2 years, I used a Neovim with only syntax highlighting and nvim-tree as my primary editor. I would use third-party tools from the terminal for any checks. This put me at a disadvantage and I realised this when I looked at the workflows of my friends.

About 5 months ago, I made the switch to VSCode with all the important extensions. I started using the mouse too often. I was using Copilot's Chat, inspecting the function signature when writing code, a shortcut for auto-formatting on close, and vim mode of course.

I convinced myself to like it, and for a while I really did. I could do so much more with my editor. I definitely was hopping around different files less. I found the codebase-wide grep with ctrl+shift+F too useful. I did not have (or know?) that in Neovim. Yet, everytime I moved my hand to use the mouse, and everytime it was stuck, I got the feeling that isn't made for me. It didn't take my personality into account. I felt distant. The Vim mode was not enough. Me setting keybindings was not enough. The final blow came when I moved houses and had a new desk setup. I did not have a mouse yet and had elevated my laptop. It was too inconvenient to use the touchpad. Around the same time, I was also being frustrated regularly with VSCode being stuck. That was probably some extension that was using too much compute. In any case, and as you might guess by now, I am too lazy to find out. It's easier to go back to Neovim.

This time, I installed LunarVim - a project that comes with all the extensions installed. The transition from a vanilla Neovim to VSCode to a fully-setup Neovim gave me so much more context on what the extensions are doing. I am still unwilling to dig into the specific extensions, but I am more aware of the kind of workflow that would make me faster. I loved LunarVim. I loved being back at the terminal to write in Vim. I love not needing to use my mouse.

I got the feeling that I was at home. I'm not switching back to VSCode now. This is good. I'm adding more extensions. I'll read up a bit on the extensions (I'll hesitate before I do that, only not as much as before). This is good!

Last updated 23 Novemeber 2025

raga (static site generator)

I've been working on raga, handlebars-ml, and huml-ml. Three OCaml projects that are all used in building my website now. I'm writing progress notes about it here.

kaafi hai

I've been talking with some friends about kaafi hai (translation: is good enough) technology. This class of technology is old, well-tested, stable, boring tech components that work and scale. They have a low innovation token cost. Think: PostgreSQL, Ruby on Rails, Django, RabbitMQ, Redis, Docker, Ansible. Counter-examples are SaaS-run, highly-marketed tools: MongoDB, graph databases, serverless functions, (to some extent) Kubernetes, GraphQL API gateways.

To be fair, I've played with all of these tools and sometimes there are use cases where they work better than old tech and are also worth the effort of maintaining it. At Gooey.AI, we use Kubernetes so that we can autoscale on a k8s cluster and have it scale down when usage is low. GPUs are expensive and this ends up saving us a lot of money. We don't use k8s for anything else in our deployment story.

But the use cases these tools are marketed to are far more than the use-cases they were built for. They end up costing the adopting org in money & effort in the future. Most of their user base is hype-driven. I want to write, talk about, and evangelise the other idea that does not get VC funding.

I absolutely love Postgres. Considering doing something with that information.

Content below was written on 4 October 2025.

handlebars-ml

My personal celebration this week is that I did released the OCaml handlebars templating library that I was working on. I felt happy when I read the "Thanks" and "Your PR ... has been merged" emails for publishing to the opam repository.

I think this is software that I am proud of. I have tested it enough for me to feel confident about relying on it. I should likely do a more detailed write-up about this because I did learn several things in the process of making it: ocamllex, menhir, sedlex, when claude code was useful, how opam publishing works, yada yada. Even the opam CI had an interesting, thorough testing process--for example that the project gets built & tested on different OCaml versions and with the latest & lower-bound versions of the dependencies. This is a new testing model that I will carry with me.

I feel more fondly about handlebars than I did at some point during the making of it. My confusion/distaste plotted against time would be a monotonously increasing graph with a sudden drop from the peak close to completion. It has pitfalls and semantics that silently ignore where things should probably crash soon. That said, I feel comfortable enough with those to be able to make use of the convenience features that sit on top. Perhaps I should add optional warnings in these cases to compensate. In any case, both the good & bad of this deserve a more detailed write-up. In summary, I feel happy about having completed this. I am also happy about the error-reporting & CLI interface that I built in, and so I would in fact like to make use of it in my own projects.

huml

I am procrastinating on the category of activities that this writing would also fall into. I am bike shedding on the static site generator that the templating engine is supposed to be a part of. My latest side quest after being satisfied with the handlebars parser was to perhaps write another OCaml parser--this time for a configuration language called huml. huml stands for HUman-oriented Markup Language, and is an experiment by kailash to make a replacement for YAML. I agree with the dislike of YAML. I do sometimes find TOML less than easily readable. I think that is because the hierarchy is not obvious from the visual representation of the text as it is in YAML. I have made some progres on this. If I do get to a version that works with the tests that are put up, then I will very likely consider using it for my static site generator and writing to kailash about it.

I also really like that there is a language-neutral test set for it, so I can actually verify that my implementation works. This is great for times when the specification is ambiguous. I think a formal grammar specification (a BNF) would also perhaps be useful towards that end.

life

I had fun making a short video in the format of old good-morning photos with hindi songs for a friend's birthday. I did that in p5.js and took help of the silicon-family techbros to get there. I am amazed again about how powerful p5 is. I will likely consider making a tool to make it easier to get GIFs of animations or fix the saveGIF method they had because it did not work for me. I ended up doing a screen recording at the end and then stitch that with an audio file with ffmpeg.

I do enjoy playing around in the command line. It is satisfying to write an ffmpeg command and have it do the thing you want. I think of the SICP-ian analogy of the wizard's incantations.

Content below was written on 19 July 2025.

July 2025 Update

I am writing this from the IndieWebClub meetup in Bengaluru, which is about personal websites. That finally gave me enough push to update this website. In the past few months, I've been wanting to write and update this website. I've wanted to make my website taste more like me.

I've also been working on a static site generator that I'm pretty excited about. I'm building it from its core - starting from the lexer for my templating engine. It's a good technical exercise. Until then, most of my writings are postponed.

Besides the tooling update, I've also been having ideas about what I want the website to look like - in a way that impacts the content and design. It's probably best if I make a separate post out of it. That also makes more sense to me because I'll have a written philosophy to drive my building. ~~Please threaten me if you don't see a link here.~~ This here is the link.

Content below was written on 26 November 2024.

back

It's been a while since my last update. So much has changed since.

I fixed the date formats now, because I noticed I had inconsistenly used the full and shortened month names on this page.

Today I need to almost force myself to feel free about writing here - that there's no need to keep it interesting. I can write exactly what I want - there's no criticism, no feedback, no rewriting. No offense to you, the reader. I do value your time too. It's only that this page is very personal to me and I want to keep it as unfiltered, as original, as I can.

Much has changed in life. I took inspiration from a friend's workout routine and decided that it was finally time to drop the delay and get off my ass. I had put up ~10kg in the past 1 year. I was not eating healthy - both in what and when. I used to feel I had unbeatable health. I have hit the gym on and off multiple times - each time lasting more than 3 months. My lifestyle - out of habit and circumstance - had always involved at least a little bit of moderate working out. That changed in the past year. To put it poetically - I carried that nostalgia of being healthy for too long.

That said, the last month has been great for my body. Going back home for Diwali - surprisingly - actually helped me get healthier. I had never put it this way before, but now that I have seen and lived the contrast, my family is actually rather diet conscious. Serving only half a sweet laddoo is the norm. Growing up in school, my tiffin box every day was some cooked vegetable and 2 chapatis. No funny treats. No excess fried snacks. I used to feel sad about it for some time. Now though, I am extremely grateful to them for that. That was probably why I felt like I could eat and do whatever and still be reasonably healthy.

No Gym

No Gym. Not yet. I have had memberships before and been a regular. I will join one eventually. For me though, what I really care about so much more than lifting weights is to be consistent. I am a believer of persistence. I like woodpeckers. The last time I had a gym membership, I did not continue beyond 3 months because I was moving. Then I didn't join a gym because I was once again not sure about how long it'll be until I move. The time before that, I think I was regular for 6-7 months, maybe a year. And then I stopped staying healthy when I stopped going to the gym. A few months after that, I had lost all benefits I gained from hitting the gym.

This time, I want to stick to a goal that I can do every single day. That I don't get an excuse out of. For me, that is 10 squats and 10 push-ups. That takes less than 5 minutes, maybe 2. That's what I set out with around the beginning of this month. It worked out really well for me because I also started walking 5K steps a day. A week later, I felt like doing more while I was at it. Skip to today, I am able to do 3 full sets of 20 squats->20 lunges->10 pushups->20 calf raises / bicep curls->1 min plank->30 jumping jacks. I want to do more and now cut down on break times between these and properly do what they call circuit training.

I am seeing benefits already. That + making my diet better is already making me lose weight and I just feel better.

Content below was written on 23 July 2024.

Ummm

I haven't written in a very long time. I see it as I see the dates. I have so much to write about. There have been moments when I have thought to myself - "yes, this happened. i should put it on the blog , yes. my internet nerds will like this."

okay - i won't try to remember everything. I have to be recent.

Homeserver!

I think I would consider this the most fun thing I've done this year. I am off away from the capitalist cloud compute machines that plague this planet with their unholy subscriptions and have instead moved to supporting capitalist machine companies that I only need to pay once. That is only partially true because I still have subscriptions for other things but not for cloud.

I now host this blog and all of my private software services on a 5 year old laptop that is, in a utilitarian sense, stripped of all its components but those that make it a computer. It's like ship of thesseus but with the components never put back in place except for those that barely make it a ship.

And yet - it is a mighty beefy machine for a cheap server. It has 20 GB of RAM now (I upgraded) and a 500GB SSD (also, upgraded by me). This, by cloud machine standards, would have cost me maybe a $100 each month? To me, zero-ing the sunk cost of this old laptop, was about a $100 in one-time upgrade costs + Rs 300 each month for a static IP. That's all you need.

It also feels wonderful. I feel like a bit of a sysadmin now. One of those real ones. I have a private network where I can host whatever software I want and put exactly the authorizations and restrictions I need.

Any description of this magnificence other than a dedicated blog would not be doing it justice.

Tailscale

This was an accidental revelation for me as part of the homeserver setup. It's wonderfully simple to setup. I had an overly complicated perception of it. And while it is probably extremely complicated and something that only a subset of 4chan veterans and seasoned network engineers (or a venn diagram intersection of the two) can fully comprehend, the user experience for it is so simple a 5 year old kid could do it.

Some setups are painfully difficiult. Tailscale was painfully easy because it made me curse myself for not doing it before.

Anyways, with this VPN setup, I can enforce access restriction to only allow this private access to myself. Not only that, all of my devices - not only this homeserver - can act like servers with a domain name and IP address. This means I can setup a private notes server, setup Obsidian + Syncthing, and SSH from any device into any other without shedding any sweat.

It's so good. I can also always choose to move to a self-hosted headscale server and get the same benefits. I chose to not do it, at least right now, because I like the folks at Tailscale. I trust them more with my private network's security than I do my (still very good) sysadmin skills.

Health

I need to take care of my health. More than I had to earlier. I realise I'm getting unfit. I used to go to the gym and I hadn't fully internalised that it has been a very long time since then. It's time I need to start from zero. Lose weight. Walk more. Eat better.

Content below was written on 13 February 2024.

Hi Again!

I'm coming back to writing on this blog after... 4 months now. A lot has happened. I am very excited. I thought this might be the right place to start because this has been my favorite part about my website, in all honesty.

I am happy to have the people I have in my life right now. That set of people has changed slightly over the last 4 months. My lifestyle, too, has changed now.

It has been almost 5 months now working at Gooey.AI - and I like it. I am working on hard problems and I like that I'm good at it. I'm learning new shit. LLMs, AI deployments, Kubernetes challenges, GPUs, costs, and more.

I have also been working on ocaml-joy - which is rather fun. I, Sudha, and our Outreachy intern Fay, are building a creative coding library for OCaml - inspired from joy, the Python original that Anand built at FOSS United.

There's books and blogs on my reading list that I am so eager to get to. There are some new ideas I learnt that I can't wait to write about. I guess micros section would be the right place to put it.

I also went on a trip to Banaras recently - that was so many new India things that I had not seen earlier. It is so rich with politics and culture and I couldn't help but notice the social difference that is present there. It feels unlike any place I have visited earlier - though I have to admit that I haven't travelled North India enough. I loved their food and I'm glad I had a friend who was brought up there who showed me around. That was a beautiful and a rather striking trip for me.

Content below was written on 12 October 2023.

nix and nixOS

I made a friend at PyCon who was into nixOS, and then few days later -- actually today -- I again met a couple of weirdos who were into nix. I take it as a sign that I should also try it out. One sentence I remember from the conversation I had today was "functional is fruity". I don't know if I have it in me to be able to forget it anytime soon.

Gooey.AI

Kinda big update -- I have joined gooey.ai as an engineer. I think my first day was 18 September. It's been almost a month, and I like it here. I had to choose between Gooey and a couple of other offers, and I was very confused when I made the decision to join Gooey. It's interesting that this is another job that I got through my network. In fact, all of the offers I had this time were through my network.

I was thinking about this -- along with this very interesting book I had been reading (called Chance by New Scientist) -- that serendipity is weird like that. You keep making friends here and there and then some day you bump into them when they are hiring and you're looking for a job. I had met Dev from Gooey at IndiaFOSS last year and then bumped into him at a meetup recently.

There's a lot more about this book called Chance that I would like to talk about -- and maybe also talk about why that was even relevant in this context -- but first let me talk about this new job.

I like it. It's challenging in a couple of ways. First, it is in a fast domain where I have some context but there's always a lot more to learn. Second, it is a different kind of company than any I've been in before. It is further in terms of product-market fit than companies I've worked with before and it also has different processes. There are things to learn here.

My priority this time, my personal KPI here, is to just get (important) things done. Something I learnt over the break is that good (senior-ish?) engineers don't yap about it when they get stuck -- due to let's say, ambiguity, or some tool not having X feature, or some vendor not offering exactly what was expected -- no they take that into account and find a reasonable path ahead. That often takes a lot of domain understanding and communication with stakeholders. Sometimes, you might be the best person available to take a call on something. In that case, it might be good to suggest a solution based on your understanding and talking about it with your stakeholders. Sometimes it could be a compromise on the scope or UX, or it could just be that you simply need more time. Communicating all of that with stakeholders boldly is important. That's what I want to get better at. Getting things done. The rest of it is only a means.

Taking rest

A hard truth is that focus is a limited currency. The last time I tried doing too much of focused activities, I couldn't ship quality stuff on time because I was left with so little energy. Often this went into a cycle of procrastination and guilt and terrible routines. I am more cautious about it these days. I do set aside some extra time besides work to do other interesting things, but I'm taking it super slow these days. If anything, I would spend maybe another hour or two on a weekday on it. No more than that. Most other time is spent hanging out with friends or just doing something fun.

I immediately feel it when I'm out of energy. I once spent more time than I intended on an open source project I'm working on and I had no energy the next day to do any meaningful work. So far, this has worked very well for me.

OCaml

I've been doing a bit of OCaml again. It's a wonderful language. I, Sudha, and Aryan are mentoring an Outreachy project in the OCaml org to port joy from Python to OCaml. I'm excited by the idea and I feel that I'm learning more about open source as I do this. It is the contribution period right now -- when all applicants accepted into Outreachy contribute to different projects. The intern is chosen based on how they did in the contribution period. For a new project without a lot having been decided on, it is sometimes a bit overwhelming to need to create new issues on demand, help everyone trouble shoot different development issues, and keeping up with code reviews. I've been setting aside a limited amount of time each day for this.

It reminds me of the time I participated in Google Code-in. It must've been similar or worse for my mentors then. Who had their own things to do and were volunteering for this -- and then kids like me would keep pushing code, incessantly ping them for reviews, and ask for new tasks to work on.

PyCon India

I attended PyCon India recently in Hyderabad. It was fun. The highlight for me to be honest was just catching up with old friends, making new friends, and the discussions I had outside the auditoriums. Lots of interesting chats. No name dropping here but if I met you at PyCon India, I loved hanging out with you. The other highlight was that I also caught up with another friend of mine in Hyderabad. We went around the good cafes, did some gokarting, talked shit, and just had so much fun.

I namedropped Gooey to folks, showed them awesome QR codes, and tried to hire folks (it's hard!).

Content below was written on 11 September 2023.

SolapurFOSS and creative coding

I spoke at SolapurFOSS on 27th August about creative coding in Python. This was by far my most OK talk. I was relaxed, I knew what to say, and I slowed down to read the room throughout the talk, repeatedly. The slides were also not all filled with boring text and I feel like brought out some emotion and connected with the audience. I felt validated when multiple people in the audience then asked me about some details that I had mentioned at some point in the talk. They were listening to me.

I filled the slides with a lot of visuals and very litle text, and that seemed to work well. These are the slides, in case you want to have a look.

This also encouraged me to try and make some sketches in preparation for the conf. I made this one sketch for a waving Indian flag on the moon. It took me some time to figure out the wavy part but it felt satisfying to do it. That however got too complicated to explain in an Introduction to Programming kind of talk, so I made another one! This one I was proud of because it actually looked pretty.

I also went back home during this time and looked at a randomly generated rectangle/line-pattern that Rasagy was exhibiting at JSConf as bookmarks and which I had bought from him. That reminded me of how beautiful this can be and that once in a while I can do that if I'm feeling artsy..

I also watched this talk by an engineer at Brilliant.org about a tool called Diagrammar. As I was running into the limits of an SVG backend that mon.school was using (won't render if too many transformations), I really wanted to use this but it wasn't open source :(

As context, the prior inspiration for me to explore all this were 3blue1brown's videos. Such as those on the Mandelbrot set with its beautiful fractals! I can't have that on mon.school (I tried). But I did learn that the library with Grant himself created the animation videos with was actually open source and it is called manim.

I haven't started learning manim yet (that's a lie, i have watched one youtube explaining its API), but I am very excited to try that out.

IndiaFOSS

I'm volunteering on the website team for IndiaFOSS. We had some problems with the designs and I couldn't give enough time early on in the volunteering season so we lagged a bit, but my team mates have done a great job and I added the finishing touches to whatever we had built. It's still lacking some info, but when that happens we will go live immediately and I (like everyone) will be happy.

Summer of Code

My Google Summer of Code season has come to an end. I liked it. I got some learning and got to contribute to something that will be used by a lot of DBAs around the world on a large volume of data (scary!?).

I don't feel too good about doing it though, because I didn't need it. GSOC is a great way for folks new to FOSS to get involved with open source projects, and make those first commits easy because you have a mentor with you. I can manage without this help, and could've even mentored some participants had I gotten involved with one of the orgs earlier. Moreover, the project turned out to be too easy for me. I probably only spent about 40-50 hours or much less during the actual coding period to finish off the entire scope of the project, over 3 months. Of course, this was also because I had done good research about what to do before I started and I stuck to the initial plan, but still that's too little effort to run away with it.

Why did I participate in GSOC then? To be very frank, I got scared about my career and the future opportunities I would have open to me if I didn't have either a college or something prestigious such as GSOC on my resume. I did have Google Code-in, but I guess people hardly know about it. It's fair because even those who know about it might not be able to guess the difficulty or significance of it.

I don't have that fear anymore that I have started applying and interviewing with companies, but perhaps that's already because of gsoc? I don't know what the alternate timeline would've looked like, or it was better or worse.

Content below was written on 31 July 2023.

Bangalore flat

I have finally moved into this new flat. I'm the new person in a sharing flat and I like my company. The room is small I think I've put in enough effort to make it feel homely and I like it. It's actually just about sufficient for me. I did a small celebration with a couple of friends and kept a momento for it. I don't know how long it will be with me and then thrown out. It's a special feeling this, though, having my own space for the first time. It's not a PG, not a hostel. My own flat room. I was not expecting it to feel this special but it does. The beauty of simple things :)

Reading

I went to a popular bookstore here today, called Blossoms', and got a couple of books. I wasn't intending to get any at first and I only went because of a friend, but then I started reading something with interest and after reading a few pages I thought that I should after all buy something. Let's hope it's not one of those times that I buy something and then never read it again.

I got a different book from the one I had started reading, and I understand that it sounds like a stupid thing to do, but I had my reasons. I am too lazy to explain, but I did get a new scientist book and then I saw one at the counter titled "The Beauty of Everyday Things" and it attracted me. I want to find beauty in everyday things. It's my kind of philosophy. That there is beauty in everyday things. There are things to feel excited about which aren't out of the ordinary. I want that, so that I don't trivialize the ordinary.

I wanted to hear out the author's take and explore my own beliefs more deeply. I read a couple of pages and so far I've not been disappointed. I am so excited that I'll read a few more this night.

I'm done filing my ITR after all ehehe.

Life's good. I need a job or a convincing reason to startup is all. I'm in some talks and experimentation. Might have an update on this at some point.

Content below was written on 14 July 2023.

Bangalore

I'm in Bangalore again and I like it here. It feels homely. I found a place finally in a good location and in good rent. I took a few days to think about whether or not to take it but I think I should take it. Having my own place would let me do more. Like setup a self-hosting server, have a better home-office setup and more. It's a small room only and perhaps it won't be everything I want yet but it is good enough to get me started.

Rust

I've been liking Rust. It's very different from other programming langauges in terms of its memory ownership model and I'm probably doing a few things wrong but I'm getting by with documentation and ChatGPT. I'm learning to write more and more idiomatic Rust.

I started with the Cryptopals challenge after building an OS in Rust.

Cryptopals

Cryptopals challenge: https://cryptopals.com.

I had a look at this after seeing it on some chat group and then thought that doing it in Rust would be good to learn Rust and also to do something hard (a blog I found on HN about proving you can do hard things).

Content below was written on 7 July 2023.

Too many ORs

I was reading the below things I wrote. I use way too ORs, and always get sidetracked by some possibility. Too many maybes. It shows that I'm okay with not resolving them into something certain. Agnosticism, when I don't care enough to resolve that confusion. It makes sense to be honest. If I didn't know myself better I would have even said that it's smart.

Solo trip

I'm on a solo trip to do a retreat of sorts and complete this series about building an operating system in Rust. I wanted to learn both about operating systems and to write Rust and this has worked out really well. I'll consolidate my notes into this blog tomorrow (actually today, it's midnight right now).

I've learnt a whole lot and I can't really complain about it. It does feel sort of lonely and I do get bored very often but I came to some important realisations in the time that I had to ponder about myself and the world.

I saw this page once again after some time. This is one of my more honest pages. I feel like this is way more honest than the about-me page I have written. Of course everything is factually correct but I always don't know what to say when asked to tell about myself (like in the about-me page). It's more of the kind of page where I try to sell myself to readers by throwing at them every interesting thing I have done. I should refine it some day haha. Last thing I want to do is say haha in a written post.

Let's talk about the philosophical realisations though, because I'm more interested in that than the technical stuff right now. Not philosophical, but let's say, one of peace and calm.

I felt like I wasn't able to contribute as much as I had wanted during my time at Pipal Academy. I feel that a bit. In hindsight, I think I was over-spending my focus and leaving little room for it to be spent on thinking creatively. What I mean is that on the same day, I would work on something challenging at work, then spend a few hours studying something as intensive as theorem provers or lockfree data structures where I had to put in a lot of effort to understanding it. I felt like I was setting out focused time, taking fewer leaves than I was supposed to, and doing what I had to do to get things done but things were still going slowly. I wasn't able to have a good opinion on the product because I didn't leave myself enough focus to spend on creative thinking like that. I don't know if that's what is called burnout, but I had normalised something like that. Now, after taking some break and feeling much more creative about things, I'm starting to notice it. On days that I do something focused in the evenings (such as building a snake game) (in a new language with a strange runtime in which I didn't even have heap allocation), I have a hard time doing focused things the next day. I felt like this was the norm for people my age, who study at college, do more things in evenings, and sleep late. Maybe it's a bit different or maybe it just doesn't work for me, but I need my breaks. I have decided to take note of it in the future as well, and say no when asked to work on something and I don't want to spend any more focus on that day.

I feel like work could've gone better if I had learnt it earlier. Even Anand told me that if I am excited about something I should take a day or two off and just get done with it. It makes a lot of sense now and I can understand why. Or at least why I should do that.

An adjacent thought was that I should also think about what it takes for me to be satisfied, in the long run, and take interest in it. I feel like I've tried too hard to do things to prove my worth, maybe even to myself. I don't have to do it, or I think I have done enough. I must stay true to myself and think of things that genuinely make me happy. This website for example, shouldn't be about me flashing my tech skills (like I do in the about page), but about things that get me excited. Of course, at certain times, it doesn't have to be told and I'll only write about things that do get me excited. That still doesn't mean that it was my intention when I came across it. Or maybe whatever I do has already been what gets me excited and what I genuinely do like. In any case, what I do in the future, or write about, should come from a good place. As in, it shouldn't be because I'm trying to establish something about myself. That happens naturally and it's not up to me. It is like in writing, when the focus should be on clarity and strength and style follows.

I've been thinking about life too, and what philosophical idea of it I feel home with. I don't know what's the right way to approach this. Maybe you catalogue through many philosophies and then pick one or something, but it doesn't matter. I have long believed that life is meaningless and whatever anyone does has no extra meaning other than to us. But then in that meaninglessness, one should make up meaning and be true to it. As an example, all relationships end some day, ultimately because of the the transientness of life itself. They are also void of any profound meaning because there will come a day when no one will be around to appreciate it. But that doesn't mean they aren't to be experienced. Life too is meaningless and transient, but it can be beautiful. That is because it is subjective. The things that make a life beautiful too are often transient. If they aren't experienced then all we are left with is dullness. I don't care much for dullness. I know I sound too philosophical and dreamy, like a drunk poet, but in that moment of thought, it all just felt right. Like I had figured out life. Of course, 30 year olds will be quick to jump at me to point out my naiveness, but then what can you do. You think, you write, you get critiqued and judged.

It's a good principle. It makes me live to stand up to my opinions and take chances. Because if I don't know what it's like to live the way I feel I should live, then what else is there to live for?

I think I might be becoming way too honest on this now section. It's like a journal entry, quite literally. To be very honest, I had tried to maintain a journal in the past and I have never been as good writing there as I am here. Maybe there's a certain refined quality that comes out due to something here.

Rust OS

I really don't feel like going much into detail about this. I will consolidate my notes tomorrow anyway and write a bit about what this was like. Let me say a few things though. When I look back at my notes at the end of a day, I realise that I learnt a heck of a lot. The knowledge gap between the start of my day and the end is so noticeable. I had never observed this before. It's a principle I should keep in mind that you can learn a lot in a single day if you do it right.

I recently looked at Partick Collison's blog titled "Fast". Things that were built very fast. I was amazed that the v1 of Unix was built in 3 weeks by Ken Thompson. That Linus only took a few days to write Git and was already hosting it by day 4 or something. I'm quite amazed.

CFP for PyCon India

I thought a bit about the CFP for PyCon India. I had a conversation with an experienced community member about topics for a talk proposal and he had suggested the topic of things like best practices that are actually useful to a large audience. I thought about it, broke it down, and came up with a rough outline (https://memos.kaustubh.page/m/190). I titled it "Writing idiomatic Python in 2023 and ahead". When I thought about it, I actually felt that it could help a lot of people. match-case will catch on, but then there are things like asyncio and typing which are often not used in the best way. asyncio because the API isn't that intuitive, and it's easy to not take full benefit of it. For example, people will await coroutines in a loop instead of creating background tasks where they can be executed concurrently. I myself have fallen for it when I started and it should be more widely known. typing because it is moving so fast and we have new patterns that are lesser known. Even things such as not needing to import annotations for List and Dict are not as widely known. Now we are getting template types too.

Content below was updated on 27 June 2023.

Reflecting

These past few days I've mostly not been doing anything work related, except for one consulting gig that I had taken up that kept getting stretched. Finally, I was able to work around some problems I faced and come to something that was acceptable for the client. I'm happy about it because it started out as something that was very technically challenging and a lot of parts that were new to me. I kept running into different problems, and limitations of tools, but finally I worked around things and got it done. Now my deep knowledge of how the internet works or how pip install works came to good use.

For example, we were using an open source tool for a use case that we didn't want to spend engineering effort on (too niche and complex) and it was giving us some problems with its limitations. It was running as a reverse proxy and we wanted to run some script on its page without modifying its content. It wasn't very extensible by default. We needed to run some javascript on that page, and there was no reasonable way to change the output of its HTML without changing its inner code. What I did was added an Nginx rule for one of its javascript assets, and made that also include the script that we wanted to run.

The other was about knowing how pip install works and how the modules are stored. Then I got what --target option was and was able to fix a few issues properly because of that.

Of course, linux and bash scripting as always helps a lot. When working with other people, it helps that you can write scripts for them to run rather than explain something step-by-step.

That was on my mind on some of the days, but besides that I am reflecting. I want to make a new section for the next part.

Ownership and usership

I've been thinking about what I had been working on at Pipal Academy. It's a neat tool to build courses. When I was working on it full-time, I would try to think about a user and how they would like it to be but I think that my standards then fell short. Now that I think about the same software as FOSS software that I would use for something myself, I am much much more opinionated and I want it to be at a much higher standard and I would in fact be ready to contribute that to the project.

It's strange how a simple change of perspective like this changes how effective I can be at building something. I think this distance from the product, this lack of a feeling of ownership, brought out more effective ownership from me. It reminds me of one of the chefs from the anime "Food Wars". I can't remember the Japanese name of it now, but it's one of the few animes that I've watched and liked. There is one character in it, one of the seniors of the protagonist, and one of the elite 10. Maybe the first seat of the 10 elite chefs at this super elite culinary school. He treats his ingredients with finesse and builds his dishes with elegance, and yet all the emphasis is on perfection as if it is something objective.

I feel that distance brings out more objectivity and is important to build a good project.

Content below was updated on 19 June 2023.

Applying again

I started applying to companies and looking at ones that do interest me. I got a quick callback from X big tech consulting company and I got rejected because I don't have a degree and background check or travelling outside India would be a problem. I don't really buy that line of reasoning. I mean, larger companies than them have people from India working without a degree, and they don't have a problem. But it is what it is.

Now I am looking at a few other companies and starting up as my options. I don't want to be in a purely consulting role because I want to be in a team and learn from others, hopefully not in a remote setting.

I know that I can get away with it, and that I have a bottom line of consulting or working at a startup that will sustain me, but I can't say the same for everyone in tech who would skip college. Moreover, I don't want "sustenance" to be what people who skip college (or me) aim or settle for. Maybe there is actually scope for career services for talented techies without a degree. I don't know how much business sense it makes, but it surely would change lives for the better. And like I had mentioned on the now page before (i hope so?), more people would feel comfortable not going for a traditional degree if options like this existed. I feel a certain conviction about it but I'm not sure of its business potential.

Maybe it's actually worth building a startup around it. I am taking a break as well. This might be a good thing to do. At least it will do something about my curiosity to know whether something like this has business potential. No, I don't want to do it like bootcamps and suck out of salaries from my own people. I'm not even sure if a direct hiring model is the best way to go about it. If I am to take money from companies instead of people, what can I provide them that will make them want to pay for it?

I'll think more about it from different perspectives and with different assumptions. I like what I'm thinking and I want to do this.

Content below was updated on 16 June.

Thinking about what to do next

I am back at home in Latur, from Bangalore. I've been relaxing since my exams last week. Winding down and letting go of all the stress from work and exams. Now I'm thinking about what to do next. One thing I know for sure is that I want to work at the edge of my abilities. I want to do something challenging. Ideally, I want it to be something that is technically challenging, but I am also okay with starting up because I know that I'll get the challenges anyway. Moreover, I think there's a lot to be learnt there in terms of interpersonal skills and the potential upside is huge.

Google Summer of Code

Back to GSOC after a break last week. Writing this parser is tricky at times. We didn't have a catchup call this week yet. I'm trying to complete this one thing but I feel stuck. I should communicate yes.

Content below was updated on 9 June 2023.

Leaving Pipal Academy

Today is technically my last day at Pipal Academy. I'm slightly emotional about it because I had a great time working here and I loved learning from Anand.

Technically it's midnight and 9 June is behind me. But okay.

Lambda Retreat was one of the best things that I have attended and it got me into programming languages.

JSConf

JSConf was okay. I enjoyed Rasagy's session on making art with code, and even made my own version of it on mon.school with python. I liked that, and meeting friends, enjoying Naren's coffee, and talking about FOSS United there. But besides that, I did not particularly like it. It felt a bit weird that speakers were treated very different. All were put up at five star hotels (not me) with special dinners and arrangements made. Someone I randomly met asked me "What qualifies you to be a speaker?". Maybe I didn't understand their question correctly, but it pissed me off. Of course, I was polite and decided to answer the better question "What qualifies you to speak about the FOSS United community?", but then I was thinking about it for some time after that. You don't have to be high gods to speak about something, or get validated by working for one of the big tech companies. It made me sick to think that it might be the baseline to have a voice. If something is interesting (and being presented nicely, which comes with practice), I don't care who is delivering it. Grr.

I don't know, maybe it was that I didn't gel that well with the community or that I was too tired or something, but I felt like I had been to better conferences.

Feeling like an impostor

I was feeling like an impostor. Just now. About an hour ago. Then I started writing this, and I remembered while writing about JSConf that I had made a sketch for a name generator inspired by Rasagy's talk, then I felt like wanting to tweet about it so I made a few changes and I started feeling confident again. There is something wonderful about being able to do this. I like it. My routine has changed so much I stopped doing things like this and then feel bad for myself.

In any case, I do remember what I was thinking when I was feeling like an impostor, so I'll go ahead and write about it anyway. I build a lot of cool projects, but then they either have very little scope or they don't do much. At Pipal Academy, I wasn't able to contribute as much as I would've liked to in the span that I was there. I also wasn't living up to my expectations in terms of productivity and speed. This was a bit of a surprise to me but I couldn't fix it even though I tried. It felt weird because a lot of people I meet tell me that I'm smart. I know that I take time to understand things deeply and ask questions when I am interested and there's more to take from a conversation. I build a good model of things in my head, but then what if my problem is that I can't deliver on it? In my head, I think of it as the immature guru problem, when you have knowledge that you never put into use, but tell that to others. Spineless gyaan. I've been this immature guru when I was a child and was just good enough to make my school cricket team but not good enough to be an important player. I still had opinions and lots of useless trivia that I would just tell other people. Sometimes it worked, sometimes it didn't. But I don't want to be that person. Programming is the one place I feel confident enough to be an elite player and I want to do that.

I feel like I have a bias towards action when I get excited about something, which is not that infrequent. I made the sketch thing right after I saw Rasagy's talk in half an hour or so.

Content below was updated on 21 May 2023

Reflecting on the "Now" Page

This experiment of adding a Now page is actually going quite well. It's a place for me to put things that aren't significant enough to blog about, or what I'm just thinking about and haven't done anything about yet. I think the main point of this is that it makes me think about What I'm doing Now, and that often leads from one point to another to another, which I wouldn't have thought or written about otherwise.

An RC alum, who happened to hang around in the same communities as me, also saw my tweet and reached out on Telegram. I had seen this person on the FOSS United Telegram group before but because I hadn't met them in person, I didn't know much about them. That was definitely interesting and I'm happy that they reached out after seeing this.

One more person I met at the FOSS United meetup (yesterday - 20 May 2023), very randomly, mentioned that I appeared on their timeline and they knew me from my posts after I told what my name was. That was new and felt really nice. We got talking and this person also knew someone I've been good friends with. Small world?

I think I should thank Tushar because probably both of them saw my tweets because of him (for one person, I know for sure, the other one I suspect). The serendipity of the internet, twitter, and communities. Even when twitter is not in a great state.

Travelling to Bangalore

I travelled to Bangalore the day after my last update, checked in at a hostel where I've previously stayed. I want to find a place but I haven't been satisfied with anything yet, and the one that I was eager to take I didn't get because the dude didn't vacate on agreed-upon date. We had a handshake deal and the owner was okay with me moving in to his room, but I should have made it a bit more formal and make him commit the same to the owner as well. It might also just have been a good thing because to be fair, I've been more productive back at my home away from the city, and this gives me a bit more time to decide how I want to live. I can also come visit hostels when I feel like I need to get out.

Bangalore has been fun though. Fun, yes. Not productive enough yet. I'm doing some GSOC thing that I have been putting ahead for about a week. The coding period starts on the 29th and I want to stick to my commitments. But I want to catch up with people and go to meetups now that I'm here, that's part of why I come to Bangalore. I met some of the hostel regulars with whom I've become friends from my previous visits, caught up with Megh and Param, attended the FOSS Meetup, and then had a farewell night for a friend who was leaving Bangalore.

I could go on in more detail about each, because each one of these interactions lasted a while and was interesting. Maybe that'll go to my microblog.

Running into self-hosting problems

This was the first time I can remember that I have ever run into some self-hosting problem that came as a surprise to me. Miniflux, which usually runs without problems, suddenly stopped working when I tried to add someone's RSS feed to it. I was doing some experiments to setup a working two-way email on my droplet and ran some scripts directly for that, and then uninstalled it with some pre-made scripts. I suspected that this must've broken the database. The authentication was working and I made sure that the database was there too. The problem was that one of the scripts had changed the authentication method from peer (the one I used) to password for all users except the default "postgres" user. My experience with pg_hba.conf came in handy and all was good within an hour.

Setting up self-hosted email

I tried to setup self-hosted email. Setting up incoming mail was straightforward. I had done it before too. This time I looked at Sivers' post on tech independence and decided to try out Mutt. Mutt looks good. I like that you can use shortcuts to do things quickly, but getting used to the shortcuts will take some time. The other thing is that actions immediately happen when I hit keys in Mutt. I prefer Vim in that sense because if I accidentally press some keys it either doesn't do anything significant in the default normal mode, or it is undoable. I'll give it some time though. Setting up outgiong mail was not at all straightforward though and I realised that there's really no way around Google's restrictions intended to avoid spam.

As I was writing this, I realised that I hadn't dived deep enough into self-hosting email and that I should try a little more. I'm doing that now, I learnt about PTR records and DKIM until now and it's interesting.

It's the next day now. Actually it's 2:30 AM on the 23rd, the original thing was written on the 21st. I was doing DKIM but did not complete it yet. I'm waiting for the CI tests to run on capstone. I can do this in the meanwhile.

Content below was updated on 17 May 2023.

Working on Capstone

Capstone is moving ahead. We have a demo planned. My sleep routine is still bad, but I feel more relaxed doing work now.

Applying to Recurse Center

I applied to Recurse Center last week. I've been anxiously checking my email since then. I'm very excited about it. Since I read their about page, I feel like it's the place for me to be at. Recently I've also been thinking about my career choice to not go for a college. I wonder if there would be more poeple not going through college if there were career services and strong communities with smart people who were as ambitious and willing to learn as me. Something that will make sure you don't miss out on opportunities by not going to an MIT or a Stanford or an IIT. I have a strong intuition that the answer is yes. Lots of kids must have thought about it but then not gone for it because of the uncertainty. Lots of kids wouldn't even think about it because they've worked to get into a good college for years and they won't consider another option. If there were good career support for kids like that, who are smart but still don't go to college, we would see a lot more people not going for the traditional route. There are good reasons to not go for the traditional route. I'm proud of what I've done. I don't think I would've thrived as much had I been in a college, even a top tier 1 college.

I think I might present about this at Barcamp this weekend. I'm not sure. In any case, I think a good first step is to curate resources and make it known that my contact is open for anyone interested in not doing the traditional route. I'm not much of a big shot myself, but I'm sure I can help younger people like me in some way.

Learning Rust

I finally picked up Rust. I like it. Borrowing is clever. I like the guarantees I get from it. It's neat. I love the pattern matching, range, and mutability being separated from referencing. It feels like a better version of C and OCaml, with some sugarings of Python. I don't take it lightly when I say that it's better than OCaml. I still see some of the same patterns. One cool thing that not many people might notice is that a Rust function with multiple lines must have semicolons on all lines but the last one to indicate that the function goes on. This is like OCaml. Straight out of it.

Writing a tutorial on Interactive Theorem Prover

I got the idea today morning, to write a detailed post on interactive theorem provers (with Coq), such that it is much more approachable to normal programmers. I call it "Interactive Thereom Prover for Normies" for now, and I have some content for a draft. I like the informal lingo I've put in there. I think it might help programmers, or at least it will help me better my writing or my Coq skills.

Raytracer in a weekend?

I saw this in a post by Julia Evans, in her DNS server in a weekend post. I had already built my own DNS server from scratch a while ago, but I liked the other posts that she mentioned to, and one of them was this Raytracer in a weekend. I ignored this then but then I came across this again while looking at Carol's blog on projects beyond webdev. Carol is a Google Code-in Winner like me, and while I haven't been in touch with her that much, I have read some of her blogs and admire what she does. I also learnt that she, like me, skipped college. I did not know this. I also came across her one other post on skipping college, and I love it. I relate so much to this. I don't think I would've thrived as much in a college, even a top tier one. I suspect that being able to build things with programming whenever I learn something is actually what keeps me from getting bored of it. It is so relatable that I think we could be friends (👉👈). I should tell her that I liked her blog and that I too skipped college.

I wonder if she has been to recurse center.

Meeting old friends

I'm in my hometown and many of my school buddies are home too. We are meeting today and I am quite excited. I feel like I haven't seen some of these people in years. Maybe we had met less than a year ago. I want to know what they've been up to and how they're doing.

For easy access

• GitHub Repo: the4thdoctor/pg_chameleon
• GitHub Diff: 50ed941...bdc376a
• Branch with my changes: gsoc-2023 (This can be installed and run manually)
• Changes after GSOC: Pull Request

This post is to mark the completion of my participation in Google Summer of Code 2023 with the PostgreSQL organisation. I worked on a database replication tool called pg_chameleon, to rewrite the SQL parsing library. My mentors were Federico -- the original author of this library --, and Andreas -- a PostgreSQL contributor.

Previously, this library was built with regex that was hard to understand and consequently to modify. It had to be made simpler so that new contributions could be added without so much difficulty. I did most of my research into this before and during the community bonding period, talked to Federico -- my project mentor and the library maintainer -- about a solution, and we settled on replacing the regular expressions with a Python library called parsy.

The first half went smoothly, but then life got in the way in the second half, for both the mentors and me and we couldn't make much progress. We discussed the scope of what was left to be done after the initial goal was reached in the first half at the earliest date we could, but I couldn't make time for it later. ~~I'm relatively free now and I can make time for the rest. My goal is to add all of that in the next couple of weeks.~~ I made a pull request with those changes in the week after the official completion of GSOC.

The first part was to port all existing parser functionality to parsy. This was the brunt of the project and where we were solving the pressing problem of too much complexity. This got completed on, or in fact slightly before the time we had allocated for these changes. It got easier because of the research and prototyping we did early on -- in the commmunity bonding period and the first week. This is the diff for the changes I made during the first half of GSoC: 50ed941...bdc376a.

I started with adding a parser for the CREATE TABLE statement in parsy and replacing the regular expressions with this. This was the first prototype because we wanted some additional validation that this tool is powerful enough to meet our purpose of achieving simplicity while getting things done for us.

I also had to make some changes to how the tool is packaged, because it was written when Python 3.5 was still supported. That is no longer the case and we decided to support 3.7 and up, because that was the oldest Python version compatible with parsy.

Later on, I added more changes to convert the rest of the regular expressions to also use parsy instead. This was a big change and I felt a relief when it was done and working correctly. I was a bit cautious with my changes, so I also added some automated tests with pytest to make sure that there was no regression. The reason regression was possible was because with parsy, the order in which we combine the sub parsers also does matter. If we combine sub-parsers with the OR combinator (| or alt(...) in parsy) for example, it will parse the statement with the earliest sub-parser that matches successfully. This big change converted a lot of Python if-else / find-replace logic to declarative parsers. It also made each parser more complete and accurate, because now we could match with the entire SQL statement rather than checking for patterns here and there, and extracting smaller patterns step by step. For example, the different subcommands (ADD COLUMN, DROP COLUMN, ADD INDEX, ...) that are available after the ALTER TABLE. Having separate end-to-end parsers for these subcommands helps because that makes it easier for us to add parsers for other subcommands (e.g. for adding / altering index that I'll get to later on).

Alter table and create table statements benefitted the most from this rewriting because they are so complex and there be so many different operations that they can entail. Also because SQL is very lenient and there are many different ways to do the same thing. Previously, they were handled with multiple regular expressions and functions with if-else chains. For example, the alter table statement was being handled by over 100 lines of function code, complex regular expressions, and naive find-and-replace for normalization. This could for example break when strings inside the statement contained whitespace. There were also some redundant find-replaces that were hard to spot because of the complexity. All this could now be split into declarative parser chunks that were easy to read on their own, and could be combined in simple albeit powerful ways to give us the complete parser. To complete the example, this is the code of the final parser of the alter table statement:

alter_table_statement = seq(
    command=seq(ci_string("ALTER"), whitespace, ci_string("TABLE")).result("ALTER TABLE"),
    __space=whitespace,
    __schema=seq(identifier, string(".")).optional(),
    name=identifier,
    alter_cmd=optional_space_around(
        alt(
            alter_table_ignored,
            alter_table_drop,
            alter_table_add,
            alter_table_change,
            alter_table_modify,
        )
    ).sep_by(string(",")).map(lambda ls: [x for x in ls if x]),  # remove none values
)

Each of alter_table_ignored, alter_table_drop, and so on, were their own little parsers that were easy to read through. The result of parsing a valid statement is a Python dict with the provided kwargs, with the keys having the __ prefix getting conveniently discarded. Even though there is some convention and prior knowledge required to understand it, the result is something much simpler and readable at a reasonable cost for the abstractions we chose.

The second part, which was to add support for some newer commands, is yet to be done. I'll be doing that now in the next couple of weeks. This includes adding support for parsing the CREATE INDEX statement and variants of it in the ALTER TABLE form. We will also be adding support for parsing a functional index so that it can be logged and alerted to the user. The same goes for other special case indices that should be manually handled such as fulltext and spatial indices. Another case to be handled is partial indices and the type of the index used.

Adding parsing capabilities for all of this is much easier and simpler because of the changes we made earlier. Regular expressions would need a lot of fore thought, verification, and probably automated tests to make sure that they capture the input correctly. All of this is very tedious and the resulting code is far from obvious to a reader. In comparison, the statements written with parsy can be understood only with familiarity of a few primitives and basic operators.

2 Sep 2023 Update I created a pull request with the changes from part 2 and they are now being reviewed. Pull Request. I liked the idea of composability here, as compared to regular expressions. Here while building parser for ALTER TABLE ... ADD INDEX statements, I could re-use the index definition parser I wrote for CREATE TABLE. Then when I added features for parsing a bigger set of index definitions, those readily started working with CREATE TABLE too. This way we can re-use features but also keep parts of our code consistent. The trade-off is that bugs could pop up at other locations because the behaviour of their utilities is changing. I think a good next step would be to add more tests for the parser so that we can be sure of it for all the cases.

5 Sep 2023 Update I have finally, successfully completed Google Summer of Code! 🎉

If you think about it, math is formal language we created to help us describe the world. It is extremely rigorous and in the end, and very importantly, it seems to make sense to us.

In the early 1900s, Bertrand Russell worked on Principia Mathematica, a set of rigorous proofs of what we consider fundamental truths in math. For example, their proof for 1 + 1 = 2 was 360 pages long! This is way too rigorous for something that we would consider obvious. But this now is the perfect segue for me to introduce the terms syntactics and semantics.

Semantics is anything that deals in meanings -- such as interpretations or observations -- as opposed to deriving one truth from another purely through mathematical rules. The opposite, deriving new truths from established ones using rigorously checked math rules, would be to deal in the syntatics.

This here highlights the contrast between what we find obvious and what Russell devoted 360 pages for. It may be obvious to us -- semantically -- that 1 + 1 = 2, but to a mathematician, a pressing question is whether that truth can be arrived at by taking some even more fundamental truth and using mathematical rules to transform that -- that is, syntactically.

We should want the semantics and syntactics to agree with each other. If we are able to disprove 1 + 1 = 2 syntactically, it would mean that our mathematical rules are broken and that possibly some other propositions we thought to be true are not. We also want to be able to conclude decisively, only with fundamental math truths that 1 + 1 = 2. If we cannot, it means that our language is not powerful enough to model the real world.

These properties of a system are called soundness and completeness respectively. We want math to be both sound and complete. We want to be able to only prove true statements, and we want to able to prove all true statements.

A related idea is one of syntactic equivalence and semantic equivalence. Two statements are syntactically equivalent if we can reach either one from the other using only mathematical rules and not interpreting the statements at any point. Parallelly, if two statements are semantically equivalent, we can reach either one from the other by interpreting their meanings.

We want semantic equivalence to entail syntactic equivalence (completeness) and vice versa (soundness). Even though they are different ideas, they can be confusing because in a good syntactic system such as math, both do happen together. Math feels like it is complete and sound.

I just said that math feels like it is complete and sound, but mathematicians are crazy and they've proved that it isn't complete and it can't prove its own soundness. I first heard about these ideas in this Veritasium video. As smart as mathematicians are, I feel like they created a footgun for themselves with this.

In any case, math isn't just hocus pocus. It gives us so many interesting things, such as the idea of syntactics and semantics. Knowing the distinction between the two gives us words to describe systems and reason about their correctness. The system in question need not always be math. It could be a programming language, a state machine, computer hardware, etc.

It makes me think about the consequences of my own work. I'm new and young, and I don't think much of it when I do my job, but probably some day my actions too will have consequences for people. I hope I am ready for it when that day comes and that I am aware enough to see it. My actions have consequences even now, just that they aren't serious enough for me to ponder over like an ethical dilemma. Either that or I'm too dumb to see it. I'll keep an eye out though.

It makes me think about journalism too. I respect them, but I don't think I'd ever want to be in such a position. There are winners and losers in the game of politics, and I don't want to play it for the fear of being a loser. The stakes are too high if you lose. I wonder if this system can be made fair. I would love to believe in it but my pessimism gets in the way. After all, it's justified given all that we humans are capable of.

I don't know how it helps. I guess it doesn't. Not at all, really. But then I thought of it and it was one of those "aha, that actually isn't half bad" moments that was actually very very bad.

But in any case, with everything that is being built in the blockchain world right now, I'm sure this use case easily beats a lot of ventures.

I started with learning about Coq through the Logical Foundations book and I have done about 4-5 chapters from it. After that I got busy and it is still at that point. I also started learning Rust and I haven't moved ahead much in the past few days on the Rust OS book.

I felt like there would be more but these are the only two that I have started and not kept going with. I was tempted to start a few more things but did not because I had this in mind. I should learn how to keep removing things out of that todo bucket.

There are some more things that I should do and do quickly, but I haven't gotten around to. Maybe I should make an app for me to keep track of all this. Would this be another thing that I build halfway? Maybe, I'll bite anyway.

I'd much rather like a phone app and a desktop app, but I would need to learn how to build either, so I'll instead start with a web app. I'll start with an API and then a simple frontend in Svelte. That should do well.

I hope I get this done tonight.

For now, I am thinking that it should have a todo list (or todo bucket?) and a journal for sure. Rest can come later.

Fingers crossed.

I've had trouble understanding burnout before. Because there were times when I was working too much and thought I had burnt out but I didn't feel it that much. Sometimes, even when I hadn't worked as much, there would just be a few days when I felt zero motivation to do any work and it wasn't that I had done too much before. I think it's not the quantity of work that causes burnout, but the stress or anxiety of not getting your work done on time. I had never before felt this close to burning out and I'm sure I would be if I had to go on with it for another week or more. I like programming and solving problems, but only when I am excited to do it and not because I'm taking money for it.

This is an important realisation for me, and an important consideration before I decide what I do next. I think a break lasting a few months where I do something challenging would be what I need. I'm also considering starting up because that would take me back into this area where I have to do work because I take money, but I'll learn how to deal with it. It's weird that I am considering it but I don't feel comfortable having this weakness which would be problematic for me because I would inevitably get in situations where I am in tight deadlines. At least then it would be for me and I might feel more comfortable doing it.

I have a conference talk coming up tomorrow and I'll be preparing for it from 3pm onwards today. I didn't work on Capstone yesterday because I met with my parents and then had to work on the TML project. I have a GSOC catch-up coming up tomorrow, the first one after the start of the coding period and I haven't done anything about it yet. It's not that I'm not working, but I'm not able to give enough focus to all of them.

I also have a term 2 examination for my online college next week and I haven't done anything at all. I feel very bad about this. I'm sure I'll be able to catch up in a few weeks, but I shouldn't have been in this position in the first place.

Too much on my plate, spreading myself too thin, wearing too many hats.

I missed a Ted Lasso episode on a Wednesday for the first time in Season 3.

How's this for a micro post, past myself?

It has felt a bit weird to me before although I have done the same thing in some other form, like while pair programming and code review.

This was the first time that I remember consciously practicing rubber ducking. And it worked very well. I didn't even complete one paragraph and I figured out what was wrong.

This was the whole thing that I wrote:

I've run into this weird bug which started when there was some change with the rendering. Initially it failed because there was some 500 error when project couldn't be fetched. Now it errors because the create user project page gives 200, but then the user project doesn't actually exist. Ah, now that I say it like this, the solution is obvious. It was probably because the code for the view was changed. Of course. The logic to create the user project was removed completely.

A lot of context is missing and I was just thinking out loud, but it helped me arrive at the solution.

The interesting thing I think is that I had already spent about half an hour to one hour looking at the logs and trying out different things to fix this issue but writing about it made the problem immediately clear. My intuitive thinking is bad at such things and I should remember that explicitly thinking about things like this works better.

But while all of that is OK, the more interesting thing is that I did work from a train. The internet connectivity has gotten a lot better in remote areas. I had to SSH into a machine to be able to work. Yes it dropped often, but I could live with it. In some places, I was also getting 5G internet. This is so good to see. Besides, I also tried to do some learning which didn't need internet. I completed some more of the Logical Foundations book I am referring to. The neovim interface is lovely. I also learnt some more Rust. The more I read about it, the more similarities I see to OCaml. It feels somehow more appraochable though. I wonder what can be done in OCaml to make it as approachable. I am also doing this.

Back to the train journey because I'm finding it very interesting and more things keep coming up. I can't see the window from here. That's a downside. I also spilled some tea on myself because I wasn't seated and with a side table like the grouped seats have. The curtains, yes. I spoke about privacy but didn't mention curtains. There were curtains. One person woke me up from sleep unintentionally by opening the curtains to find their seat. I was startled and angry because the numbers weren't up anyways, but I was too sleepy to say anything. Then one more person tried to open the curtain a bit at night while I was awake, but it was only someone from the other side trying to get down. I don't know why I remembered these things, they seem very insignificant. It's a bunch of sentences from me, but I guess a skilled writer or a Jerry Seinfeld could be able to make something creative out of it. Maybe I should give it to ChatGPT to see if it make something creative out of this seemingly random collection of sentences.

I did give it to ChatGPT. I asked it to write about the journey in the style of Jerry Seinfeld. Make some observational humour. It wasn't bad what it came up with :

Imagine me stepping onto the stage, waving a hand, grinning. “Hey folks! How are you all doing today? You know, I recently took a train ride to Bangalore. Yeah, you heard it right, a train ride. Not just a ride, I tried working while on it. Not making chai, but coding in Rust and OCaml. But let's not get into that, it's scarier than ghosts for some people in here."

Laughter fills the room.

"So, I booked a 'side upper seat'. Sounds fancy, doesn’t it? It’s like the train's way of saying 'Congratulations! You're alone but elevated.' It’s an introvert's paradise, I tell ya! Sure, I could have gotten a group-of-four or six seat, but nothing quite beats the intimacy of a solo seat. In the group seat, you've got so much company, it's like a mini family reunion where you're the odd in-law nobody likes.”

Giggles ripple through the audience.

"And here's the fun part: you know how they have these light bulbs and charging points dedicated to your seat? I was half expecting a mini-fridge and a foot massager next. Train rides have evolved, folks! I mean, soon they’ll be asking if you want a hot towel and an in-flight meal!”

The audience chuckles.

"But the surprises didn’t stop there, no sir! The internet connectivity, guys! I had better reception in some remote field than I do in my own living room. I was coding, learning, even got 5G! That’s faster than my neighbor's Ferrari, and he lives in the city!"

The laughter builds.

"However, everything wasn’t as smooth as my internet connection. You see, the side upper seat lacks a few, let's say, comforts. Ever tried having tea without a table? It's like playing hot potato with yourself, only the potato is a hot cup of tea. And the loser is your shirt.”

I pause for the laughs.

"But the best part was the 'curtain surprise'. Oh yes! The curtains. I was sleeping like a baby, dreaming about my code compiling without errors, when suddenly, BOOM! Curtain slides open! I swear, I jumped so high, I could have switched places with the guy in the upper berth! And the perpetrator? Just another passenger, fumbling in the dark. I tell you, it's funnier than a horror movie!"

The room erupts in laughter.

"In all, folks, the journey was like coding in OCaml, unexpected, surprising, at times frustrating, but never boring! I hope you enjoyed my little train escapade."

And as the laughter fades, I walk off the stage with a grin, leaving the room still echoing with chuckles and applause.

And just like that, it's no longer so much of a micro post. That's fine, let me sleep and dream about my code compiling without errors.

I was aware that I was nearing the completion of a year but I wasn't sure whether I had started in May or June. I had a conversation with Anand about this and what's ahead. Let me put that all in this blog.

It won't be much of a micro blog then. Based. I'll do it somewhere else and keep it to myself for the moment.

OK but let me say a few things anyway. It's the first time I've ever completed more than a year besides Hyperlog. Well, Hyperlog. That's for another day. I've grown a lot as an engineer. I finally feel more confident socially, like I fit in. It took me one whole year at Pipal, and >5 years since I started using real languages for building things, for that. I feel like I'm a slow mover socially. Or maybe something else that I don't know. I don't get free (as in freedom) with people until I know them for a very long time and even then I sometimes struggle, but at least the intent is clear in those situations. I think I'm getting better at that but I don't understand it one bit.

For now, let's go Kaustubh. Long live you.

This makes me wonder whether it is the end of web UIs (for mass consumers). It isn't too difficult to hook up a speech-to-text to ChatGPT, and then use plugins to get our services across. Plugins can do visualization, provide links, and (I anticipate) other richer ways to interact in chat such as choice buttons and structured forms. Sure there are security problems and jailbreaking hasn't been solved yet, but it hasn't even been a whole year since ChatGPT was released. When we do find robust ways around it, would the web economy become redundant and switch to plugins? I can already see how specialised plugin developers would be helpful to a lot of businesses. I think I might learn it too.

In any case, the web will surely look a lot different in a few years. Maybe people will use ChatGPT-esque applications instead of web browsers. Maybe there will be bodies similar to IETF that make standards for APIs that ChatGPT and company must follow for their plugins. Maybe there's something even beyond plugins.

Maybe I'll fix my sleep schedule, but that's not on the top of my priorities. I know, I complain that this is a problem, but then I'm not willing to do anything about it. That's because maybe it's not as bad. I still get time to do things I like and work. I don't hate it not having to be all sweaty from the summer heat. I'll take it if my schedule fixes itself when I travel, but I don't think I will explicitly act against it.

There is no true interpretation of anything; interpretation is a vehicle of human comprehension. The value of interpretation is in enabling others to fruitfully think about an idea.

- Andreas Buja

I'm quoting this from the book "The Elements of Statistical Learning."

This sounds fascinating, and seems prescient now that we are in the age of ChatGPT and LLMs, huge statistical models that interpret things for us.

Besides using Coq in the Coqide or with Proof General in Emacs, you can also set it up on Vim/Neovim with the Coqtail extension. I had some difficulties while setting it up and there weren't that many resources for it, so I'm documenting the steps here.

We'll have to:

1. Set up Vim / Neovim with Python support
2. Install coq for command line
3. Add the coqtail plugin / setup colors or colorscheme

1. Setting up Vim / Neovim with Python

On Vim

If you use Vim, you will have to compile it yourself with the --enable-python3interp flag. I haven't tried this myself because I use Neovim, but this seems to be the way. I found out about the flag from this stackoverflow answer.

On Neovim

On Neovim, it is much easier. You must have the pynvim package installed on your system's Python3.

$ python3 -m pip install pynvim

Once done, check that the Python interpreter setup in your Neovim is same as the one for which you installed pynvim. You can check that in Neovim by running :checkhealth provider. Make sure that the python3_host_prog mentioned in the logs is the same as the interpreter where you installed pynvim ( $ which python3). If not you can set let g:python3_host_prog = "/path/to/python3" in your init.vim or init.lua.

2. Install coq for the command line

If you install directly from the website, you'll get CoqIDE and an interactive command line program. But you need the coqc command to be there in $PATH for coqtail to use. You can set that up by installing Coq with a package manager.

For me it was brew.

$ brew install coq

And then check that coqc works.

$ coqc --version

3. Install whonore/coqtail plugin

I use vim-plug with an init.vim, so for me it was adding this line to init.vim:

Plug 'whonore/Coqtail'

and running nvim +PlugInstall.

You can find instructions for other plugin managers in coqtail repository's README.

coqtail should now be setup. Open any Coq file (.v extension), try entering <leader> cc. This should start Coqtail in a split view. These are some shortcuts that I found useful:

| Shortcut | Description | | -------- | ----------- | | <leader>cc | Start coq | | <leader>cj | Go ahead by one step | | <leader>ck | Go back by one step | | <leader>cl | Run all steps until the current line |

One more thing: if you use dark mode, Coqtail's highlighting color probably won't match well with the background. I simply switched to light mode when using Coq (relax, only for Coq). But, if you want to put in some more effort, you can set that up and it is mentioned in the repository's readme here.

I think it helped because I didn't have to consider all the cases myself. To go into some specifics of how lockfree structures are implemented, at the core of it is the compare-and-swap instruction. It takes a pointer, an expected current value, and a new value, and sets the value at location pointer to new value only if it is equal to the current value before the update is done. The whole operation is completed atomically, using a special hardware instruction that performs both the compare-and-swap in one CPU instruction.

When building a concurrent data structure, we must take into account how different interleavings of the data structure's methods affect the correctness of our program. This is complicated. Seemingly trivial things such as computing the size of a data structure are not trivial, and accepted size algorithms that are shipped with libraries could have a negative return value with a particular interleaving with other methods.

That is exactly the problem that my PR focused on. It took me 5-8 hours of work to do that. The big range is because I was mid-travel and did it in short sprints. Now that I mention it, that is another advantage of programming with ChatGPT. I would have otherwise forgotten my chain of thought after a gap, but I had it both in my chats and queryable with English language :). I had read through the mentioned paper before though, and I think reading the paper and implementing it in one go would've taken longer. This is the pull request.

I used the GPT-4 model. I don't think the stock ChatGPT model would've been useful in this case.

Note: An mdbook version, organized into chapters, can be found here. Its content is licensed under CC (Creative Commons), and if you want to modify it to fit it for your own use, you can use the GitHub repository.

Outline

• Concurrency in a single thread
• Generators
• Coroutines
• Scheduling coroutines

Concurrency in a single thread

Traditionally, concurrency has been a responsibility handed down to the operating system. A single process can create multiple threads and the OS will run these threads across its (multiple) cores as it sees fit. It will interrupt a running thread, store its state, and load another previously interrupted thread for the computer to work on.

It's often not worth doing this. Moreover, writing correct multi-threaded programs is quite hard as data races and deadlocks are easy to miss. We can solve this problem by adding another level of indirection.

Generators

Python has generators that can be created with the yield statement. They are commonly used to build iterators of some sequence.

def abc_generator():
    yield "a"
    yield "b"
    yield "c"

abc = abc_generator()
for letter in abc:
    print(letter)

Output:

a
b
c

You can also use generators to make infinite sequences, which you can stop when you want.

def natural_numbers_generator():
    i = 1
    while True:
        yield i
        i = i + 1

natural_numbers = natural_numbers_generator()
print(next(natural_numbers))
print(next(natural_numbers))
print(next(natural_numbers))

Output:

1
2
3

You can similarly also build a Fibonacci sequence with this.

def fib():
    a, b = 0, 1
    while 1:
        yield b
        a, b = b, a + b

# sequence: 1, 1, 2, 3, 5, ...

@anandology took infinite sequences one step further, adding specialised operations for them, and making Python evaluate lazily. The result is something very interesting, more like Haskell than Python. It is out of scope of this blog, but you can check it out in this GitHub repository.

Coroutines

Coroutines are functions that can be paused and resumed. Generators do just that.

yield keyword is called so because the function yields control of the CPU. This control is given back to the caller of the generator.

All Python generators have a .send() method. In addition to continuing the generator like next() does, this method accepts an argument that is sent to the generator. This value can be accessed as the returned value of an yield statement.

We can build a worker that computes the square of its input.

def square(x):
    return x * x

def get_worker():
    x = yield
    while True:
        x = yield square(x)

def main():
    worker = get_worker()
    worker.send(None)  # "prime" the coroutine
    tasks = [1, 2, 3, 4, 5]
    for task in tasks:
        result = worker.send(task)
        print(f"square({task}) = {result}")

main()

Output

square(1) = 1
square(2) = 4
square(3) = 9
square(4) = 16
square(5) = 25

Aside: A real-world example

I used this pattern recently to build a ChatGPT chatbot that takes prompts from my terminal.

import openai

# configure openai
openai.organization = ...
openai.api_key = ...

def get_chatgpt():
    messages = []
    user_input = yield
    while True:
        messages.append({"role": "user", "message": user_input})
        response = openai.ChatCompletion.create(messages=messages)
        chatgpt_message = response["choices"][0]["message"]  # chatgpt api detail
        messages.append({"role": "assistant", "message": chatgpt_message})
        user_input = yield chatgpt_message

def main():
    chatgpt = get_chatgpt()
    chatgpt.send(None)

    while True:
        # user_input need not come from "input()" only.
        # for example, it could be over a websocket connection
        user_input = input("Prompt:")
        result = chatgpt.send(user_input)
        print("ChatGPT says:", result)

Coroutines with return values

Coroutines (and generators) can return values too. This value can be accessed with the .value attribute of the StopIteration exception that is raised when the coroutine completes (the function returns).

We have two methods that translate a number to either its English word or its Tamil word. There is do_io_work() at the start where this coroutine yields.

def number_to_tamil(x):
    yield do_io_work()
    match x:
        case 1: return "onnu"
        case 2: return "rendu"
        case 3: return "moonu"
        case _: raise ValueError("x must be in [1,3]")

We can get the return value of each like this:

tamil = number_to_tamil(1)
tamil.send(None)  # "prime" the coroutine

try:
    while True:
        tamil.send(None)
except StopIteration as e:
    print("1 in Tamil is", e.value)

Output:

1 in Tamil is onnu

Running two coroutines concurrently

Let's spice up do_io_work() a bit. Instead of letting it be a normal function, we'll have it be a generator itself, which yields a random number of times. This will give code the uncertainty of real-life I/O operations.

import random

def do_io_work():
    # yield a random number of times
    for i in range(random.randint(1, 5)):
        yield

Let's use this with our number_to_tamil coroutine, along with the number_to_english coroutine.

def number_to_tamil(x):
    yield from do_io_work()
    match x:
        case 1: return "onnu"
        case 2: return "rendu"
        case 3: return "moonu"
        case _: raise ValueError("x must be in [1,3]")

def number_to_english(x):
    yield from do_io_work()
    match x:
        case 1: return "one"
        case 2: return "two"
        case 3: return "three"
        case _: raise ValueError("x must be in [1,3]")

I used yield from, which takes a coroutine, then propagates all the yields from it to the caller.

yield from coro is the same as for x in coro: yield x

We'll hook all three up in our main() function, to print the English and Tamil translations of numbers from 1 to 3.

def main():
    for x in range(1, 4):
        tamil_coro, english_coro = number_to_tamil(x), number_to_english(x)
        tamil_word, english_word = None, None
        while True:
            if tamil_word is not None and english_word is not None:
                break

            if tamil_word is None:
                try:
                    tamil_coro.send(None)
                except StopIteration as e:
                    tamil_word = e.value
                    print(f"{x} in Tamil is {tamil_word}")

            if english_word is None:
                try:
                    english_coro.send(None)
                except StopIteration as e:
                    english_word = e.value
                    print(f"{x} in English is {english_word}")

main()

Output (could have a different order):

1 in English is one
1 in Tamil is onnu
2 in Tamil is rendu
2 in English is two
3 in Tamil is moonu
3 in English is three

Notice the different order of English/Tamil translations in each iteration. We just ran two coroutines asynchronously.

Aside: Compatibility with asyncio

The code that we wrote is not a metaphor for what asyncio does. It is exactly what how asyncio works. We need to only wrap our coroutines in an asyncio.coroutine decorator to make it work with async/await.

import asyncio

@asyncio.coroutine
def number_to_tamil(x):
    ...

@asyncio.coroutine
def number_to_english(x):
    ...

async def main():
    for x in range(1, 4):
        tamil_word = await number_to_tamil(x)
        english_word = await number_to_english(x)

        print(f"{x} in Tamil is {tamil_word}")
        print(f"{x} in English is {english_word}")

asyncio.run(main())

Output:

1 in Tamil is onnu
1 in English is one
2 in Tamil is rendu
2 in English is two
3 in Tamil is moonu
3 in English is three

This is still sequential, because we are waiting for number_to_tamil(x) to return before we start executing number_to_english(x). We can change that by converting each coroutine to a task.

While the await coro statement will add coro to the queue and also yield, asyncio.create_task(coro) will only add the coroutine as a task to the queue, without yielding control. That's why we can keep adding more tasks until we call an await on something.

But to see this in action, we will need to modify our code slightly to print values as soon as they are computed. Let's add this logic in a new coroutine we'll call translate.

import asyncio
import random

...

async def translate(language, number):
    match language:
        case "English":
            result = await number_to_english(number)
        case "Tamil":
            result = await number_to_tamil(number)
        case _:
            raise ValueError(f"Language {language} is not supported")

    print(f"{number} in {language} is {result}")

Now let's modify our main() coroutine to use this.

async def main():
    for x in range(1, 4):
        tamil_task = asyncio.create_task(translate("Tamil", x))
        english_task = asyncio.create_task(translate("English", x))

        # wait for both tasks to finish
        await tamil_task
        await english_task

asyncio.run(main())

Output

1 in Tamil is onnu
1 in English is one
2 in English is two
2 in Tamil is rendu
3 in English is three
3 in Tamil is moonu

Scheduling coroutines

We saw a pattern when we were running the two coroutines concurrently. We tried running the first one, tried running the second one, and returned only when both had finished running.

We can setup a scheduler and put in a queue to run tasks one by one.

import queue

class Scheduler:
    def __init__(self):
        self.tasks = queue.Queue()

    def add_task(self, coro):
        self.tasks.push(coro)

    def run(self):
        while not queue.Empty():
            task = self.tasks.pop()
            try:
                task.send(None)
            except StopIteration:
                pass
            else:
                # task didn't raise StopIteration, push it back to queue
                self.tasks.pop(task)

Notice that we aren't storing the returned value anywhere. We'll come to it later. To test this out now, we will have to print the results of the number_to_tamil and number_to_english coroutines as soon as they are returned.

We'll create a new translate function to do this.

def translate(language, number):
    match language:
        case "English":
            result = yield from number_to_english(number)
        case "Tamil":
            result = yield from number_to_tamil(number)
        case _:
            raise ValueError(f"Language {language} is not supported")

    print(f"{number} in {language} is {result}")

Notice how the yield from syntax makes the code a lot more straightforward.

Let's use this in main() now.

def main():
    scheduler = Scheduler()
    for x in range(1, 4):
        scheduler.add_task(translate("Tamil", x))
        scheduler.add_task(translate("English", x))
        scheduler.run()

main()

We can also add all the tasks first, and then run all the tasks at once:

def main():
    scheduler = Scheduler()
    for x in range(1, 4):
        scheduler.add_task(translate("Tamil", x))
        scheduler.add_task(translate("English", x))
    scheduler.run()

main()

Output (order may vary):

3 in English is three
1 in Tamil is onnu
1 in English is one
2 in Tamil is rendu
2 in English is two
3 in Tamil is moonu

We have built a scheduler that runs concurrently!

Keeping track of return value

We did it, but we aren't keeping track of the return values of tasks. To fix this problem, we add another level of indirection.

All problems in computer science can be solved by another level of indirection.

We build a Task wrapper for our coroutines. This will have add two things on top of coroutines:

• a .status attribute which can be None, queued, running, or finished.
• a .value attribute which will store its return value

task.value is only vaild when task.status is "finished".

The scheduler will be responsible for setting .status and .value.

class Task:
    def __init__(self, coro, status=None):
        self.coro = coro
        self.status = status
        self.value = None

    def get_value(self):
        if self.status == "finished":
            return self.value
        raise Exception("Task has not finished")

Change scheduler to work with tasks instead of coroutines:

class Scheduler:
    def __init__(self):
        self.task_q = queue.Queue()

    def add_task(self, task):
        assert isinstance(task, Task), "task must be an instance of Task"

        task.status = "queued"
        self.task_q.put(task)

    def run(self):
        while not self.task_q.empty():
            task = self.task_q.get()
            task.status = "running"
            try:
                task.coro.send(None)
            except StopIteration as e:
                task.value = e.value
                task.status = "finished"
            else:
                self.task_q.put(task)
                task.status = "queued"

Let's see this working in an example:

def translate_big_number(language, n):
    match language:
        case "English":
            translator = number_to_english
        case "Tamil":
            translator = number_to_tamil
        case _:
            raise ValueError(f"Language {language} is unsupported")

    digits = [int(d) for d in str(n)]
    result = ""
    for digit in digits:
        result += yield from translator(digit)
        result += " "

    return result

We added a new function translate_big_number, which translates a number with multiple digits.

def main():
    scheduler = Scheduler()

    for x in [12, 133, 120]:
        task1 = Task(translate_big_number("English", x))
        task2 = Task(translate_big_number("Tamil", x))

        scheduler.add_task(task1)
        scheduler.add_task(task2)
        scheduler.run()

        print(f"{x} in English is {task1.value}")
        print(f"{x} in Tamil is {task2.value}")

Output:

12 in English is one two
12 in Tamil is onnu rendu
133 in English is one three three
133 in Tamil is onnu moonu moonu
121 in English is one two one
121 in Tamil is onnu rendu onnu

Bonus Can you build a coroutine sleep(duration) that sleeps, and allows other tasks to run for a certain duration?

In asyncio, the scheduler is known as the event loop. Besides tasks, it also has "Futures", which are analogous to JavaScript promises. Read more about it here.

Conclusion

We looked at how to build coroutines, how they can be run concurrently, and how tasks can wait for other tasks to complete.

To get the most out of this, we need to be able to actually perform I/O asynchronously. This can be done purely in Python with the selectors module. This is out of scope for this blog.

Programming language-level support for lightweight threads has a lot of benefits to offer. Besides better performance on blocking I/O, it simplifies writing concurrent code which is notoriously hard to write correctly and debug. Coroutines need not run on a single OS thread. In other languages like Go and OCaml, they can and do run on multiple OS threads as well.

I was impressed by this setup, and I hope that writing this post inspires confidence in others to self-host their own software.

VPS Information

I have a single $6 droplet and a 50GB volume mounted to it that costs $5. I reckon most people would be able to do without needing that extra 50GB, but I have to mention it for the sake of completeness.

I have allocated 4GB of disk space for swap. I did that while attempting to host a minecraft server (and not succeeding), but I left it that way. My RAM usage usually doesn't exceed 2GB in total, but I host apps like Plausible (Website Analytics) and Synapse (Matrix server), which recommend having 2-4GB of RAM.

• Memory: 1GB
• CPU: 1vCPU (Regular, not the Premium Intel/AMD offering)
• Disk: 25GB Disk + 50GB mount
• Datacenter: Bangalore
• OS: Ubuntu 20.04 (LTS)
• Cost per month: $6 (Droplet) + $5 (Extra disk space) = $11

Software that I self-host

| # | App | Description | Deployment method | | - |-------|-----|-------------|-------------------| | 1 | My blog | Static website built with Hugo | Static with Caddy | | 2 | Plausible | Website analytics | docker-compose | | 3 | Trilium | Notetaking | docker-compose | | 4 | Memos | Notetaking | docker-compose | | 5 | Miniflux | RSS Reader | supervisor | | 6 | Komga | Manga Reader | supervisor | | 7 | Synapse | Matrix homeserver | docker-compose | | 8 | Mautrix-Telegram Bridge | Matrix-Telegram Bridge | docker-compose | | 9 | Otto | Cute doggo discord bot that plays music and chess | supervisor | | 10 | Otto-web | Web app for otto | supervisor | | 11 | Minesweeper | Fully client-side minesweeper | Static with Caddy | | 12 | Self-hosting 101 Slides | Notes from my workshop, built with mdbook | Static with Caddy | | 13 | ... more slides | More static sites from other talks | Static with Caddy |

Deployment tools

• Caddy: Webserver for reverse_proxy and serving static sites.
• supervisor: For running services that I need to define (not systemd)
• docker-compose: To deploy more complex apps. I use restart: unless-stopped or restart: always to automate restart on rebooting crashes
• Makefile / rsync: To write deployment scripts
• PostgreSQL: used by some of the apps
• Redis: Used by some of the apps

supervisor configuration is simple enough that I can define all my services in the same /etc/supervisord.conf file (unlike systemd). This is a huge help because I can copy paste configuration from the same file. It is easy to understand so I know what to change.

The same is true for Caddy. I configure all my public sites in /etc/caddy/Caddyfile. Most website entries only need 3-4 words of configuration. This file also acts as a registry that I can refer to, to know which sites I have deployed and disabled. As an added plus, I don't have to manually configure SSL for sites deployed on Caddy. I like this much more than nginx and apache, where configuring new sites takes up multiple steps, and the Certbot SSL routine has to be repeated each time.

I use docker-compose instead of supervisor when a project has messy dependencies. It is easy to setup and I don't have to add supervisor entry for it. I find this convenient. I use docker ps to know which containers are running. I use volume mounts so that apps that use more disk space can use the larger 50GB disk.

rsync is very handy for deployments. I first heard about it after I had already had some experience hosting apps by hand on servers, and I felt stupid about all the hacks I had used before that to transfer files between my computer and the VPS. You can read more about it here.

While this is already a cost-effective setup, I want to move to actually hosting apps on my own hardware in the future. I've been putting it off because I've been on the move a lot recently. I recommend that even more if you can do it.

It's been a rewarding journey for me. I have rotated different tools before settling on these, and I'm sure I'll continue to explore more.

Ciao! 👋

It changed the way I think about programs, added new perspective, and got me interested in the design and implementation of programming languages. I have heard varied opinions from people who have read this SICP. Some feel that it is overrated and for some it changes how they think about programs.

To some extent, it was the latter for me. This post highlights what I had to take away from the book that was positive for me.

Controlling complexity with abstractions

This, and the following point, were the biggest take aways for me. It started with the exercise problem on Ackermann function. First I tried simplifying it the dumb way by expanding all the values. I noticed a pattern building up but it took a while to see what it was doing. The expansions were getting harder and harder for higher values of the first argument. Then I abstracted out A(m-1, n) as a function f(n) and it was straightforward after that.

Abstractions play a big role in the cognitive complexity of something. It would have taken a very long time for me to see the pattern if I had not created this abstraction. We encounter this all the time in programming. If we don't keep two simple chunks separate from one another, they form one big complex thing. Next time we want to make a change to this monstrosity, we have to wade through the details and redo all the thinking that we did the first time.

Building composable software with layers of abstraction

Any non-trivial software will have layers to it. For example, a database system could have a file system layer, an engine layer, and a user interface layer. If we want to add a new operation to the engine, we shouldn't have to implement something at the file system layer. It is much better if we can re-use some part of the existing file system operations in our new operation. This way we don't have to switch contexts between the engine logic and the details of file system manipulation. It is easier to focus only on one.

+--------------+
|  interface   |
+--------------+
|    engine    |
+--------------+
|  filesystem  |
+--------------+

This is only possible if our software doesn't have circular dependencies between layers. An explicit choice has to be made so that our file system layer logic is limited in scope only to file system level manipulations. It shouldn't implement procedures that are specially meant to perform everything that engine operation X will do. It should rather be the responsibility of the engine level code to pick the right tools from the file system layer and build its logic.

For example, an insert operation in the engine might need to 1) find the right place to write the new contents, 2) serialize data into that format, 3) write it there. The file system part should provide an API to do 1 and 3 separately, rather than combining the logic of all three into one single procedure.

That way, each procedure has one responsibility and the dependency is well-defined. The same file system primitives can also be re-used to implement other operations. We don't think of the specific logic in the engine layer when writing logic in the file system layer. We build primitives that any engine-level operation could use.

SICP looks at this in a more programming languages oriented manner. In any programming language, there are three parts: primitives, means of abstraction, and means of combination. Each layer of abstraction should, in a way, provide these things to the next layer of abstraction.

Eval-Apply: the wizardry of the wizard book

The cover image of SICP shows a wizard holding the eval-apply globe. It is how programming is done. Evaluation happens as a combination of applications, and applications happens as a combination of evaluations. Application refers to the application of functions, and evaluation refers to the evaluation of primitive statements. Combination happens through application of functions. That the core of programming can be described with only two procedures is mind-blowing.

Function application is a two step process: evaluate the parameters, and then evaluate the body with the obtained values as arguments. Evaluation is a series of applications.

By the end of the book, when I had implemented the evaluator, it felt very simple. A whole programming languages implemented with only some functions. I think the implementation seemed simple because we were guided through building good abstractions and layers of abstractions in a bottom-up manner. Doing the same thing without that context would have been much more complicated. It shows the power of breaking down the problem in terms of primitives, the means of combination, and the layers of abstractions.

On Feb 11, I conducted a 3 hour introductory workshop on Self Hosting at FOSSMeet '23 at NIT Calicut. This post has my notes on teaching Self-Hosting to a crowd and an easy-to-find link for people who land on my personal website.

My slides for the workshop are available here.

Audience

The audience for the event was mostly smart students who were relatively new to the Linux terminal. Rather surprisingly, I got to know after the workshop that some of the attendees were not even coming from a tech background.

I think this is what should be expected for future workshops too. The people who will benefit the most from an introductory workshop to Self Hosting are people who have some familiarity with the Linux terminal but things like SSH, tmux, nginx configuration might be new to them.

Structure / Infra

The structure of the workshop was that each participant would get a DigitalOcean droplet allotted to them for the day. We had set up these droplets with a non-root sudo user, and a default password for all instances that they could use. Agreed that while even things like creating a non-root user would be useful to someone looking to self-host, it is better to include things like that in an appendix and use the workshop time on more interesting things.

The DNS entries were set for all nodes (an A record) for a subdomain and the corresponding wildcard for it (e.g. alpha.k8x.in and *.alpha.k8x.in would point to the same node). Authentication was done with a password, with SSH authentication disabled.

After this, we would go over SSH, tmux, apt, Nginx, static hosting, dynamic hosting, HTTPS with certbot, persistent services, and htop. Each of these would include a small exercise that the participants will need to do. The workshop team would then go around the participants, help whoever might be stuck and then keep going.

The highlight of the workshop was hosting a static website and a dynamic website, enabled from HTTPS.

What went well?

It was a crowded workshop and we had to say no to some interested participants because we had only prepared 40 nodes for the workshop. Participants showed interest and there were attendees with different levels of familiarity attending.

During the tasks, the three of us (me and two student volunteers from NITC) on the conducting team used to walk around different areas, help participants, and answer queries. This worked very well because a lot of little things could go wrong and having an experienced person to fix things on the spot meant that the participants weren't getting stuck anywhere.

For example, when we were using certbot to apply HTTPS, some participants made a typo in the domain and certbot wrote a new configuration to the Nginx default configuration, rather than the site it was intended to be on. This configuration needed to be restored back to normal and the website configuration had to be fixed. I had to do this for multiple participants and it wouldn't have been easy to fix if someone experienced hadn't been there on the spot.

At the end, it made me very happy that almost all of the participants had stuck through the 3 hours and had deployed both the static and dynamic websites. Some of these participants were new even to SSH, cat and nano commands when they started, and by the end they knew their way around tmux and nginx.

It helped to keep the things to cover very limited. I didn't go into docker and docker-compose, vim, Makefiles, and other things I had initially wanted to cover. That was the right call. I had my notes reviewed by my colleague Anand, who has conducted such trainings before, and that helped me set the goals accordingly.

Publishing the notes to a website helped because then participants could catch up on things they had forgotten. They could also use it later at home to remember what they learnt in case they decide to start self-hosting on their own. An easy to pronounce, short link makes it even better.

mdbook worked very well for my slides.

What could've been better?

Self-hosting also refers to using your own hardware at home to host software and exposing it to the web. A workshop on that kind of self-hosting is hard to conduct at scale, and it's easier to get started with cloud machines. Some participants came to me later saying that they expected this kind of self-hosting rather than relying on cloud vendors. I should've stated it at the start to set the expectations of participants on what kind of self-hosting we would be talking about.

While most of the content in my notes was correct, there were 2 times that I forgot to include a line of code somewhere and 1 task where I had only mentioned a step in English text and not as code. I was able to get most students to see my correction, but some still missed them. Then I had to repeat it multiple times and help some participants personally on it. Typos or forgotten lines of code like this are easily mendable in small groups but they become much more expensive as the number of participants scales.

Some participants mentioned to me later that the pace of the workshop was quite slow for them, and that it was beginner oriented. While I think that was the right pace anyway while keeping the whole crowd in mind, it might have helped to have some fun bonus tasks along with normal tasks so that the participants remained engaged.

In this post, we will go over what this implies, understand the type-system bottom-up, and then use that understanding to reason about some confusing behaviours. By the end of this post, you will have a clear picture of what classes and objects really are and what lies behind this abstraction.

Table of contents

1. The Chicken-Egg problem
   • object and type
   • relationship between object and type
   • PyObject and PyTypeObject
2. Method lookup
   • Lookup order, and setting and deleting attributes
   • self injection
   • Descriptors
   • Crafting methods by hand
3. Classes as objects
   • Creating classes with type
   • class keyword as syntactic sugar
   • Note on recreating the class keyword with functions

• Conclusion
• Footnotes

Note: I drafted an updated version of some of the explanations for a more recent talk. It drops the 2nd section on method lookup, but I believe it has a better introduction into the chicken-egg problem and creating objects as classes. The JupyterLab slides can be found here. You can scroll to move between slides.

The Chicken-Egg Problem

object and type

When people say that everything is an object in Python, they mean that very literally. There is an object class that all Python objects are instances of. This is also the implicit base class when you create a new class.

>>> isinstance(5, object)
True
>>> isinstance(int, object)
True
>>> isinstance("abcdef", object)
True

Classes themselves are objects too. This is slightly counter-intuitive when you think of an object as an instance of some class. But it is very much an object in that you can pass it around as a function argument, name it differently, set attributes and call methods on it.

All classes are instances of the type class. In that sense, type is a metaclass and other metaclasses need to inherit from type. ¹

Relationship between object and type

There are three things about this that are confusing when considered together:

1. object is a class
2. All classes are objects of the type type.
3. type itself is a class

We can verify this in the Python interpreter.

If an object is a class, it should be an instance of the type class.

>>> isinstance(object, type)
True

The interpreter agrees.

To verify #2, we will create a custom class and check whether it's an instance of object. Then we'll also verify whether it is an instance of type.

>>> class A:
...   pass
...
>>> isinstance(A, object)
True
>>> isinstance(A, type)
True

Checks out.

Now let's see if type is a class. For this, we will check whether type is an instance of type and whether type is an object.

>>> isinstance(type, type)
True
>>> isinstance(type, object)
True

See, I wasn't lying.

What's happening here? type is a type, type is also an object, and object is itself a type.

We can understand this relationship better with issubclass:

>>> issubclass(type, object)
True
>>> issubclass(object, type)
False

While type is a subclass of object, object isn't a subclass of type. Instead, only the type of object is type. That is, an object is an instance of type.

This diagram captures this relationship accurately:

Each object has a type, some other metadata, and its attributes associated to it. A type itself has everything that an object does (including a type), along with some extra fields such as the base classes that it borrows from.

This relationship can be better understood when we take a look at how it is implemented in CPython.

PyObject and PyTypeObject

In the CPython implementation, there are two basic structs for PyObject and PyTypeObject that look kind of like this. Notice that the _object struct has a pointer to its type (*ob_type), and there is a separate bases tuple in _typeobject that stores the bases for a type object.

// a simplified struct definition for PyObject and PyTypeObject

#define PyObject_HEAD       PyObject ob_base;

typedef struct _object PyObject;
typedef struct _typeobject PyTypeObject;

struct _object {
    PyTypeObject *ob_type;
    // and other metadata
};

struct _typeobject {
    PyObject_HEAD  // macro to "inherit" from PyObject
    const char  *name;
    PyObject    *bases;
    // and other things
};

The ob_type of object is manually set to type, and this object is then added to the bases tuple in type.

To make it clear, let's try to write the C code that will represent such a relationship:

// let's assume this function is defined and initialises a tuple with a single PyObject
PyObject* tuple_init(PyObject*);

PyTypeObject type = { .name = "type" };
type.ob_base.ob_type = &type;

PyTypeObject object = { .name = "object" };
object.ob_base.ob_type = &type;

// type.bases needs to be set to a Python Tuple
type.bases = tuple_init((PyObject*) &object);

wtfpython has a good explanation on this here and CPython has comments explaining this too: https://github.com/python/cpython/blob/a286caa937405f7415dcc095a7ad5097c4433246/Include/object.h#L24-L29

Method Lookup

Lookup order, and setting and deleting attributes

This section sets up the context for the next one about method lookup.

The lookup order in Python goes like Object → Class → Base classes. However, the setting and deleting of attributes only happens directly on the object itself.

Take this example where we initialise an object whose class has an attribute x, and then try to delete it:

>>> class A:
...   x = 10
...
>>> a = A()
>>> print(a.x)
10
>>> del a.x
AttributeError: x

The attributes of a class can be looked up by its objects, but cannot be deleted from there.

However, deleting directly from the class will work fine:

>>> del A.x  # is fine
>>> print(a.x)
AttributeError: 'A' object has no attribute 'x'

The same happens when setting attributes too. We can see it in this example where we set x on both the object and the class, and then delete the x on the object:

>>> class A:
...   x = 10  # set on class
...
>>> a = A()
>>> a.x = 20  # set on object
>>> print(a.x)
20
>>> del a.x  # delete from object
>>> print(a.x)  # doesn't raise attribute error because A.x is still there
10

Even though we deleted a.x, A.x was still there and that's the value we got when we printed a.x the second time.

We can observe that the lookup is dynamic and any modifications to the attributes on the class should reflect when we perform lookup from the object.

self injection

When we define methods inside a class body, we take the bound object as the first argument self. This is strange because self is defined when the method is called from an initialised object, but not when the method is called from the class itself.

>>> class A:
...   def hello(self):
...     print("Hello, world!")
...
>>> A.hello()
TypeError: A.hello() missing 1 required positional argument: 'self'
>>> a = A()
>>> a.hello()
Hello, world!

Additionally, this self injection does not happen at the time that the function is called, because Python also allows you to assign a different variable to this method and it still works:

>>> a = A()
>>> f = a.hello
>>> f()
Hello, world! 

So what's the magic here?

If we inspect the types of A.hello and a.hello, we'll see the difference.

>>> type(A.hello)
<class 'function'>
>>> type(a.hello)
<class 'method'>

They're not the same, and a.hello is a method and not a function.

The hello attribute returns different values depending on if it is called from the class or an initialised object. Python allows such behaviour with descriptors.

Descriptors

Python descriptors are classes that have __get__, __set__, or __del__ methods. So instead of the actual value, these methods are called when a descriptor is accessed as an attribute. A popular example of descriptors in use is the property decorator that can be used for dynamic attributes.

Python functions are actually descriptors with a __get__ method that returns a method with the object and function bound to it. I'll take a couple of examples to explain descriptors.

A simple Descriptor that always returns 10 would look like this ²:

>>> class Ten:
...   def __get__(self, obj, objtype=None):
...     return 10
...
>>> class A:
...   x = 5
...   y = Ten()
...
>>> a = A()
>>> a.x
5
>>> a.y
10

A descriptor has access to the object it is called with from the obj argument. We could write this descriptor to return the square of obj.x:

>>> class SquareX:
...   def __get__(self, obj, objtype=None):
...     return obj.x * obj.x
...
>>> class A:
...   y = SquareX()
...
...   def __init__(self, x):
...     self.x = x
...
>>> a = A(5)
>>> a.y
25

You can also directly try using the __get__ method on a function. Calling it with the object as its argument will return a method that binds this function and the object.

Crafting methods by hand

This is a side section, but rather interesting because we can directly observe how methods are actually represented differently from functions.

The method class isn't available as a builtin, but you can get it by calling type() on a method. This class takes a function and an object during initialisation, in that order. We can use that to create methods by hand that were never bound to the class or the object.

>>> class A:
...   greeting = "Hello, I am Foo Bar."
...   def hello(self):
...     pass
...
>>> a = A()
>>> method_class = type(a.hello)
>>> def greet(self):
...   print(self.greeting)
...
>>> greet_with_a = method_class(greet, a)
>>> greet_with_a()
Hello, I am Foo Bar.

We just crafted a method that was never an attribute.

Classes as Objects

Creating classes with type 

Everything is an object, and classes are not exempt from it. We saw earlier that classes are instances of the type class.

We can also create classes with the type class initialiser, similar to other objects. For example:

>>> def foo(self):
...   return "bar"
...
>>> A = type("A", tuple(), {"foo": foo, "x": 10})
>>> a = A()
>>> a.foo()
'bar'
>>> a.x
10

The type signature of the 3-argument type initialisation is type(name, bases, dict) → type.

The name argument will be the value taken by cls.__name__, bases can be used to achieve inheritance by providing a tuple of classes, and dict is a dict of key-value pairs that are the attributes on this class.

When a metaclass needs to be used, that class can be used instead of type.

class keyword as syntactic sugar

A syntactic sugar is a some additional syntax that makes it easier to express something but doesn't add a new feature to the language.

The class keyword can be seen as calling type with the variables defined in its scope as value of the dict argument. In that sense, class is a a syntactic sugar for creating instances of type. We'll write a decorator to show this that will mimick the behaviour of the class keyword as closely as possible.

At the end, we should be able to define classes with something like this:

@create_class()
def A():
	x = 10	
	def __init__(self, y, z):
	    self.y = y
	    self.z = z
	return locals()

a = A(5, 10)

We should also be able to perform inheritance.

@create_class(A)
def B():
	x = 20
	return locals()

b = B(5, 10)  # y and z, as in A.__init__

And use metaclasses:

@create_class(type)
def Meta():
    def __repr__(self):
        return f"Meta {self.__name__}"
	return locals()

@create_class(metaclass=Meta)
def C():
    pass

This syntax is very similar to the one used to declare classes with the class keyword, except that it is a def, has a create_class decorator, and returns locals().

The decorator gets the function object, and can access the parameters and default arguments. From there, we can resolve the variables from globals() and initialise the class.

Feel free to give it a try now before reading on with the solution.

Our decorator gets access to the function and we can inspect it to find out its parameters (base classes) and other keyword arguments (metaclass). We will only get these parameters as strings, and we need to resolve it using globals(). Then we can use the metaclass (by default, type) to initialise this with the three arguments. We can assume that the function will return the locals() dict from inside and use that for building the third argument in initialisation.

def create_class(*bases, **kwds):
    def wrapper(func):
        metaclass = kwds.get("metaclass", type)
        func_locals = func()
        klass = metaclass(
            func.__name__, bases, func_locals,
        )
        return klass
    return wrapper

Now, the examples above will work and we can recreate the class keyword without using it.

These examples should work now:

@create_class()
def A():
    x = 20
    return locals()

@create_class(A)
def B():
    y = 30

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

    return locals()

@create_class(type)
def Meta():
    def __repr__(self):
        return f"meta {self.__name__}"
    return locals()

@create_class(B, metaclass=Meta)
def C():
    return locals()

We created a simple class, a subclass, a metaclass, and a class which uses that metaclass without using a decorator that spans less than 10 lines of code.

We can even get rid of the need to return locals() from the definitions if we use something like inspect.getsource and then performing an exec on the function body and supplying a local env, but inspect.getsource will actually read the literal text from the Python file to produce the output. I didn't want to get that hacky and stick to what is always available at runtime.

Note on recreating the class keyword with functions

One point to note with this is that the scoping in classes is different from scoping in functions. Even though variables declared inside a class block are scoped to that block only, the class block can readily access global variables.

It can get slightly confusing, but this is perfectly valid Python code:

x = 10

class A:
    x = x + 5
    print("x inside A:", x)  # x inside A: 15

print("x outside A", x)  # x outside A: 10

This is not possible with our way of defining classes because functions by default don't get access to the global variables unless used with nonlocal or global.

Conclusion

• Because everything in Python is an object and all objects have a type, there exists a strange relationship between object and type
• We looked at how type is itself a type and how that relationship is defined in the interpreter
• We saw how Python uses descriptors to implement methods and how we can craft our own
• We blurred the line between classes and objects by eliminating the need for a special keyword for classes, and instead working with type.

The objects all the way down philosophy has its quirks, but it also makes Python very expressive. We can combine classes and functions in a similar manner as other primitive types. Lists and tuples need not be restricted to a single type and we can even have custom meta types. All of this is possible because there is some shared structure to all the objects that the interpreter can exploit. With the knowledge of a few rules, we can extend and combine anything in Python.

Footnotes

[1] In fact, Python allows you to define your own metaclasses. Then the class will be an instance of a different type other than the default that is type. Note that metaclasses also need to subclass from type, so the stated relationship between type and classes still holds.

[2] Example borrowed from Descriptor HowTo Guide, Python documentation.

Around the time when I was in school and had some familiarity with programming, I didn't really think it was a good analogy to think of computers as sophisticated calculators. Calculators are dumb, they can only perform operations on numbers whereas computers are able to build something as complicated as the internet.

I was younger then. Now I am older and wiser and I can see why “computers are sophisticated calculators” and even appreciate the beauty of what it implies.

To a younger me, I would explain the idea like this:

Padawan, you need to understand that computers are the epitome of applied mathematics.
Why, you ask? Look around you. Everything you perceive, you can represent as math. Colors, bodies, collisions, and even movements.

You don't believe me? Take the movement of a falling box. Can you represent this as math?
You can say there's a box that is 1m by height, width, and length. It is in free fall and its current velocity is 20 m/s. Acceleration due to gravity should be assumed to be 10 m/s2.

Now I tell you, you can represent that with math as 1112010. You see? I just put all the numbers together with the assumption that m, m/s, and m/s2 are our units. The first three digits are height, width, and length; in that order. Then there are two digits for current velocity, and two digits for the acceleration.

Now if I give you a different sequence, let's say 2343010, and tell you that this also represents a falling box in a similar way, will you be able to recreate the scenario?
Good work, Padawan. Of course it is an unusually shaped large box with dimensions 2x3x4 in m. Its current velocity is 30 m/s and acceleration due to gravity is 10 m/s2.

That is much closer to what a calculator does. These are numbers and we can combine them as we want. If we were asked to find out the time to collision if the two hypothetical boxes were thrown one after the other, we could do that with a calculator.

If the calculator is powerful enough, we can write the procedure once and use it multiple times with different values. Computer enough for you?

Not yet? No? Ah, you're not convinced how I would teach a computer to understand and answer questions such as the one about collision. Good question.

It's the same, really. No different, except the computer now understands something different. It will be easier to explain if we put words to what we did: we first encoded the motion of a box into a sequence of digits. We know how to decode it, but we now need to teach a computer that.

Once again, we will encode. This time, we will encode the procedure of decoding this falling-box format, and then the procedure for finding the collision time.

010 111 212 323 524

This is my encoding. Each group of three has 3 digits. First digit represents the starting position in the sequence (starting from leftmost digit as 0th position), second digit represents the number of digits that need to be taken (notice it is 2 for the last 2 groups, for acceleration and velocity), and the third is its reference number. The reference number will allow us to use these quantities as variables.

This first part of the procedure decodes the data into variables. Now let's solve the collision problem. The slower box is dropped first, and the faster box is dropped later. We will ignore the dimensions of the boxes in our calculations.

If you do the math, you'll arrive at the result that the time to collision would be:

\(u_1t + \frac{1}{2}a_1t^2 = u_2t + \frac{1}{2}a_2t^2\)

\(2 *(u_1-u_2) + (a_1-a_2)*t = 0\)

\(t = 2 * \frac{(u_2 - u_1)}{a_1 - a_2}\)

Let's say this is the encoding for our math language that the computer understands this notation:

• Operations:
  • 000 - addition
  • 001 - subtraction
  • 002 - multiplication
  • 003 - division
  • 004 - parenthesis open
  • 005 - parenthesis close
• Operands:
  • 1xx - with xx being a variable ID
  • 2xx - with xx being a constant number as decimal

We can encode the above equation as:

222 002 004 113 001 103 005 003 004 104 001 114 005
 2   *   (   u2  -   u1  )   /   (   a1  -   a2  )

That's how you would represent a procedure as data, in the form of sequences of digits.

Computers only understand bits because they communicate with base 2 numbers. The same ideas are applicable when they are converted from base10 to base2.

Appreciation

Isn't it wonderful that with such simple ideas of encoding and decoding we have today been able to invent things such as the internet?
We have been able to use numbers to encode our words and social interactions so that they can be shipped to remote places and decoded there. It blows my mind.

Good abstractions capture thought ideas with only as much bend as is absolutely needed to eliminate ambiguity. The ideation of a web app is in terms of the content the user can see and the interactions they can make with the website -- simple. When it goes to implementation however, the developer has to think analytically and separate what goes into the frontend from what goes into the backend. This feels very artifical and low-level.

Secondly, it's amazing how much HTML and CSS (or pseudo-CSS like Bootstrap/Tailwind/et al.) we still write. How many times has a simple Bootstrap form been created by you? By other developers? Yet we spend time doing it again, caring about its alignment and how it looks on different screen sizes. All we do care about is what content is displayed, what data is collected from a form, and what happens after submitting it. Context switching to low-level details takes our attention away from these things.

Thirdly, our database abstractions are not there yet. Databases are part of our webapps by necessity rather than choice. They bring complexity with them. To maintain them, we have to keep a schema and migrations. We have to write SQL queries that are performant and at the same time compatible with higher-level language abstractions. It's a tricky problem to solve. ORMs bring in a different kind of complexity and often end up hiding performance bottlenecks. For example when you want only the username, an ORM will likely load all other user data in one shot. The magic they add could have side effects like loading data from JOINs that won't be used later.

When I presented my thoughts to a group of developers coming in from varying backgrounds and levels of experience, I was confronted with arguments that justified this complexity. My pipe dream is one where a "web developer" without design expertise (me) is equipped with tools to build complex, maintainable production grade web apps where they can focus on the important parts only. Even with these counter-arguments, I felt that my frustration was justified and in fact even shared by other developers. I wasn't happy with any of the existing solutions that were mentioned -- Liveview, Hotwire, Isomorphic Javascript.

I've never read a novel like this before. There is only a series of events, narrated with a hint of indifference. The murder of a man, Mersault's separation from his fiance, his death sentence, and of course the death of his mother. None of these incidents were profound enough to throw the man into some odd kind of reflections on his life. He never regretted any of his actions, and as he proved to the chaplain towards the end, he didn't change his religious beliefs despite much attempt.

It was as if his every action was actually an inaction. No action was his own "wish". He didnt wish to kill the Arab, but it was the weather and the Arab's knife that made him pull the trigger. He didn't wish to write the letter to Raymond's girlfriend. That too, like Marie's marriage proposal, he just agreed to without express enthusiasm. Yet at the end it was these "actions", and his indifference for which the jury imagined motives (for the common people are obsessed with meaning and causation) and eventually condemned him to death.

On the other hand, it is just absurd that the kind of arguments made to put Mersault to death were not a crime on their own -- not wanting to see her mother before burial, not paying respects, having coffee with milk, smoking a cigarette. In fact, someone who even feigned interest in these activities would've gotten a reduced sentence. Moreover, contrary to the jury's opinion, he did in fact quote his mother several times and indeed loved her. Rather than being honest, one is supposed to play the game and should pay the price otherwise if they don't.

When we have some code that sends emails, we probably also want a stub that we can use locally -- which won't send actual emails, let us inspect the content of each outgoing mail, and let us conduct tests.

I needed something like this when we were building the skeleton of a guided project for a training [[1]]({{< relref "#references" >}}). One of the tasks was for the students to send an email notification upon some action.

We had two use-cases for an SMTP server like this here:

1. To recommend to the students for trying their code locally, and
2. To test that the emails were being sent once it is deployed.

For the first, I needed something simple to use and deploy locally that I could easily recommend to others (and not have to spend time instructing them on how to configure XYZ). For the second, I wanted something that gets the job done without having me pay for some fancy SaaS. Luckily, I found a single solution that was good enough for both.

I had to only look at the standard library (smtpd package) to find what looked like the perfect solution -- something as simple as python -m http.server. But the documentation there recommended using another library (an asyncio replacement) - aiosmtpd. I tried both and ended up going with aiosmtpd because interacting with it as a client was simple to do for humans. smtpd required ending all commands with <CRLF>.<CRLF>, at least with the default configuration. That is a tricky sequence to type for humans.

aiosmtpd though, was just perfect! This is all that needed to be done:

pip3 install aiosmtpd
aiosmtpd -n

That's it! An SMTP server is now running on localhost:8025 and it will print any received emails to console. That's brilliant if working locally, the default is exactly what we need. No need for more code or monkeypatching, just some configuration tweaks.

You can even netcat to this port and try sending emails with raw SMTP commands. It can be quite satisfying from my own experience.

Besides ease of interacting for human clients, there are some more reasons to prefer aiosmtpd over smtpd (in case you have to choose for your own needs):

1. Extending it is very simple, as we will see.
2. Its default invocation (with only one CLI flag) starts the debugging server that we want. No option values to remember.
3. smtpd is deprecated and will be removed from the standard library in Python3.12.

Extending aiosmtpd

If local setup is all you wanted to know about, you can close the blog at this point. But I'll go on and share how we solved another interesting problem -- verifying that code written by our students was correct and that emails were actually being sent.

Now the caveat is that we were deploying our students' code to a website of their own [[2]]({{< relref "#references" >}}) and the checks should use HTTP endpoints on those websites for checking results. Because we had our own deployment we had the luxury to be able to override certain environment variables on a per-request basis without making changes to the application code (CGI is powerful!). This meant we could use a different SMTP configuration when certain HTTP headers were received.

With that being sorted, I could write an extension class in less than 50 lines of code that writes the email content to a single file. We could then read from the same file to get the latest email. At our scale this was good enough. I didn't need to worry about 10 people simultaneously submitting and overwriting the files for each other, but if that were a concern for you, the Maildir convention can be used instead.

This is the code for the extension, with comments for clarity:

# email package is part of the Python standard library,
# which provides an "email.message.Message", along with
# parsers/generators for converting to/from a standard format.
from email.generator import Generator
from email.parser import Parser

# Message class handles all SMTP commands with sensible defaults
# and allows us to only implement a `handle_message` method
# that will be called with a parsed email message
from aiosmtpd.handlers import Message


class MailFile(Message):
    def __init__(self, mail_file, message_class=None):
        # we are taking a mail_file, the path that should be
        # mentioned on the CLI, that we write the mail to
        self.mail_file = mail_file
        super().__init__(message_class=None)

    def handle_message(self, message):
        # overwrite the old file
        with open(self.mail_file, "w") as f:
            # Generator is an abstraction for parsing/writing the
            # email content and the headers
            g = Generator(f, mangle_from_=True, maxheaderlen=120)
            g.flatten(message)

    def get_latest_message(mail_file):
        """Returns an email.message.Message object

        Useful accessors on Message object:
        `message["X-MailFrom"]`
        `message["X-RcptTo"]`
        `message.get_payload()` -> str | list[str]
        `message.is_multipart()` -> bool
        """
        p = Parser()
        with open(mail_file) as f:
            return p.parse(f)

    @classmethod
    def from_cli(cls, parser, *args):
        # this gets the args received on command line
        if len(args) < 1:
            parser.error("The file for the mailfile is required")
        elif len(args) > 1:
            parser.error("Too many arguments for Mailbox handler")
        return cls(args[0])

Then if we package and pip install it as mailcatcher, we can write:

aiosmtpd -n -c mailcatcher.catcher.MailFile latest.mail

The reference for extending should be the handlers.py file and the documentation.

This is the gist

Unit tests with aiosmtpd

The same class that we wrote can be used for unit testing as well. Except in this case, you might not want to have a long-running process for the SMTP server, and rather start/stop it for each test.

Here's how you can extend the same mailcatcher to build a pytest fixture which does that:

import pytest
from aiosmtpd.controller import Controller
from mailcatcher import MailFile

from app import config, function_that_should_send_email


@pytest.fixture
def mailcatcher(tmp_path):
    """Fixture to setup a temporary email server

    `tmp_path` is provided by pytest.
    """
    mail_file = tmp_path / "testing.mail"
    catcher = MailFile(mail_file)
    # you can set a different hostname/port as args to `Controller`
    controller = Controller(catcher)
    controller.start()
    config.update(smtp_hostname="localhost", smtp_port="8025",
                  smtp_username="", smtp_password="")
    yield catcher
    controller.stop()


def test_email_is_sent(mailcatcher):
    address = "eva.lu.ator@example.com"

    # procedure to be tested
    function_that_should_send_email(address)

    latest_email = mailcatcher.get_latest_email()
    assert latest_email is not None
    assert latest_email["X-RcptTo"] == address  # or a loose check like `address in latest_email["X-RcptTo"]`
    assert "expected content" in latest_email.get_payload()

I should mention that lazr.smtptest is also a good alternative with a similar API and workings, made especially for testing.

Thanks for reading! If you liked it or have some feedback, let me know on Twitter or Telegram.

References

[1] The training was delivered by Pipal Academy. The guided project in question is open source: repo link. This repo has the skeleton needed, and the tasks are mentioned on the dashboard. If you want to host this project and run in your own group, the code for dashboard site is available here: repo link. Let us know and we will help you do it.

[2] We built our own deployment platform for this, called "Hamr". We satirically call it the "next-gen" serverless platform. It is made to be as simple and boring as possible -- using CGI to make the apps serverless and deployment script being a human-readable cURL without an elaborate JSON body. There were philosophical decisions that went into its design and it deserves a blog post of its own.

Before we go on to writing our server, I want to give some context that should be treated as a prerequisite.

SMTP is the protocol that is used to send and receive emails (as the name suggest, Simple Mail Transfer Protocol). When a mail is sent by a user, it is handed over to an SMTP server to pass it on to its destination, which is a receiving SMTP server. Once the mail reaches its destination SMTP server, other protocols such as IMAP or POP3 take over and store the emails so that they can be retrieved by the receiving user at a later point.

SMTP uses stateful, long-lasting TCP connections where the client and server send data back and forth. The client can even send multiple emails on the same connection. This is in contrast to HTTP 1, where client asks for exactly one resource and the server sends back just that before the connection is closed.

The SMTP specification defines some commands that the client can use to set the headers and content. The client will send these commands in their specified format with arguments, and the server will respond with a response code that the client can understand. Optionally, the server can send some human-readable data that may be useful for debugging.

These are some of the basic commands that all SMTP servers should support:

• HELO: Hello
  • Once the TCP connection is established, this is always the first command that the client should send. By this command, the client identifies itself with a domain name (an FQDN - fully qualified domain name)
  • Syntax: HELO kausm.in
• MAIL FROM: I want to send an email, and this email is from ...
  • This starts a mail transaction (I want to send a mail), and specifies the sender address (and this email is from).
  • Syntax: MAIL FROM: sender@kausm.in
• RCPT TO: A recipient of this email is ...
  • This is used to specify a recipient of this mail. If a client wants to send a mail to multiple receiving addresses, it should run this command multiple times with each of the receiving email addresses.
  • Syntax: RCPT TO: receiver@kausm.in
• DATA: My mail data is ...
  • Starts the actual mail data. Once this command is run and an OK response is received from the server, the client should go on to send first the email headers and the email content as text. The delimiter to end this text is <CRLF>.<CRLF>.
  • Headers can be specified like Date: Mon, 14 February 2022 20:40:34, Reply-To: kaustubh@kausm.in, etc. separated by newlines
  • Syntax:
    • Client: DATA
    • Server: 250 OK
    • Client:

    Date: Mon, 14 February 2022 20:40:34
    From: sender@kausm.in
    To: receiver@kausm.in
    Subject: Hello Receiver
    Reply-To: kaustubh@kausm.in
    
    Hey there
    
    .
    
    

• QUIT: That's all I wanted to do, thanks for your service.
  • This tells the server that its work is done and that the server can now close the connection.
  • Note that the SMTP specification explicitly mentions that the server shoudln't close the connection abruptly unless the client executes this command. Even after you complete a mail transaction, this gives you the option to send addiotional mails over the same connection.
  • Syntax: QUIT

There are some more commands that can be used once the client and server agree that both support that command. For example, the LOGIN command for authentication and STARTTLS for encryption. These are extensions that do not have to be implemented by all SMTP servers, but are practically required to go past the security/spam filters of sophisticated receiving servers. The functionality of these commands can be agreed upon by using an EHLO (Extended Hello) command instead of the usual HELO command, which the server responds to with a list of supported extensions.

OK, let's start.

Here's what we need to do:

1. Write a server that can accept and read from TCP connections.
2. Support the HELO command.
3. Support MAIL FROM.
4. Support RCPT TO.
5. Support DATA.

1. Writing a TCP server

1   package main
2   
3   import (
4-7      ...
8   )
9   
10  const HOST = "localhost"
11  const PORT = 3333

Initialize a listener on the host and port, and accept connections on it. See Go standard documentation of the net pkg for more information.

13  func main() {
14  	// for now, we'll just listen on localhost:3333
15  	listenAddr := fmt.Sprintf("%s:%d", HOST, PORT)
16  
17  	listener, err := net.Listen("tcp", listenAddr)
18  	if err != nil {
19  		// for now, just panic with the error
20  		panic(err)
21  	}
22  
23  	// accept connections forever
24  	for {
25  		conn, err := listener.Accept()
26  		if err != nil {
27  			// simply log the error for now, and continue listening
28  			log.Printf("error while accepting conn: %v\n", err)
29  		} else {
30  			go handleConnection(conn)
31  		}
32  	}
33  }

In handleConnection, we'll first read from the connection into a buffer and then write that back into the connection.

35  func handleConnection(conn net.Conn) {
36  	defer func() {
37  		conn.Close()
38  	}()
39  
40  	// we'll be keeping the data here
41  	buffer := make([]byte, 1024)
42  
43  	// read into a buffer
44  	n, err := conn.Read(buffer)
45  	if err != nil {
46  		log.Printf("error while reading from connection: %v\n", err)
47  		return
48  	}
49  
50  	newlyRead := string(buffer[:n])
51      log.Printf("Got input from client: '%s'. Echoing it back now\n", newlyRead)
52  
53  	// now just echo it back with a prefix
54  	_, err = conn.Write([]byte("Server says: " + newlyRead))
55  	if err != nil {
56  		log.Printf("error while writing to connection: %v\n", err)
57  		return
58  	}
59  }

Now we can test out the server with netcat.

$ # save the Go code in a main.go file and run this in one terminal
$ go run main.go

$ # in another terminal, start netcat and connect to our server
$ nc localhost 3333
Hello there. This will be echoed back
Server says: Hello there. This will be echoed back

Great, we now have our TCP listener. We can get started with the SMTP commands.

2. Supporting SMTP commands

To support different commands, we need to read and parse the line input by the client. Let's make a list of what we'll need to do:

• Parse the text - split it into command and arguments
• If the command is a valid command, pass the conn object to its handler until it finishes its job and returns it back.
• Sometimes, we need certain prerequisite data from commands before we can start executing other commands. For example, the client has to send the HELO/EHLO command before being able to send any other command, the MAIL FROM needs to be run before RCPT TO is run, and so on. For this reason, we have to keep state of what data we have received from the client.

The HELO command

1. RFCs: RFC 883, etc.

1. d4 d5 2. c4 e6 3. Nf3 c5 4. e3 Nf6 5. Nc3 Nc6 6. cxd5 exd5 7. Be2 cxd4 8. Nxd4 Qb6!? 9. Na4 Qa5+ 10. Kf1?? Bd7 { [%csl Ra4][%cal Gc6d4,Rd7a4] } 11. Nc3 Bb4 12. Nb3 Qb6 { Sacrificing another pawn! However, White's king is uncastled and very importantly, his c1-bishop and the a1-rook are out of the game. If he takes the d5-pawn, he will have to face the consequences of lack of development, king in center, and open files leading straight to the king. } 13. Nxd5 Nxd5 14. Qxd5 O-O-O! { Dares white to take a 3rd pawn on f7. } 15. Qc4 Kb8 16. Qf4+ Ka8! { This was my idea anyway with Kb8. To tuck the king in the corner so that I can carry on with my attack without worrying. } 17. e4 { White's idea is to get the Bishop out to e3. Once that is done, the Rook covers the back rank. He can go g3-Kg2, or Kg1-h3-Kh2 to complete his development. If I wouldn't have found the next move, I would be down 2 pawns with no compensation, and would have eventually lost the game. } 17... Nd4!! { This is the only move to keep the advantage. The trick here being - if White goes Be3, I can go Nxe2 threatening his queen. If he takes on b6 and I take on f4, I have a nice idea that gives me at least a draw but probably more. Another reason I was excited about this is because at this point I was seeing lines that were quite similar to the Game of the Century played by Fischer. Where he sacrificed a queen to lead with some minor piece combinations with his 2 bishops and a knight, with discovered checks like these. https://lichess.org/study/mcmGGgdB } 18. Nxd4 (18. Be3?? Nxe2 19. Bxb6 Nxf4 20. Bxd8 Bb5+ 21. Kg1 Ne2+ 22. Kf1 { [%csl Gd8][%cal Ge2g3,Gh8d8] } 22... Rxd8 { There is no way to stop Ng3+, picking up an exchange. If White goes a4, I can simply drop the Bishop back to a6. Another cheeky threat is ...Nc1+, Kb1, Rd1#! }) 18... Qxd4 19. e5?? { White resigned after my next move. Can you find it? } 19... Qd1+!! { Taking advantage of White's backrank weakness, and my well positioned Bishops. White resigned here. 0-1 } 20. Bxd1 Bb5+ 21. Be2 Rd1# *

Hello!
I am kaustubh, erstwhile known (sometimes even fondly) as nikochiko, kubasth, bubbles, or tubhtubh.

I write computer programs at my job and sometimes for fun. I take an interest in (non-exhaustively):

• Free and Open Source Software (FOSS)
• Self Hosting / Tech Independence
• Functional Programming & Making of Programming Languages
• Chess
• Tech Policy & Politics

If you are here, chances are that I'll love talking to you. I believe I am easy to nerdsnipe and it would make me happy to receive an email about something that fascinated you recently.