Refactor my entire backend database to use InstantDB instead of Supabase Postgres.

Recently my friend Mark let me know about InstantDB, which is a client side database inspired by firebase. Their whole value prop is its super easy to build quickly, and they have 3 core tenets: 1. Optimistic updates. 2. Multiplayer, which allows for a real time experience across multiple different front end instances. 3. Offline mode. Now if you've read any of my previous blogs you know I'm a Postgres hardliner, and I do everything in Postgres. However, I really had issues building this Blog Builder application. Getting my database to update in real time as I was writing was very hacky and something I didn't enjoy. So I decided to learn something new and implement the blog builder completely in InstantDB (Except for image storage, hit me up with beta storage access plz Mark).

This one is going to be huge. I'm new to Graph database and GraphQL kind of syntax, so I'm going to have to rewire my brain a little bit to get used to InstantDB. I'm going to need to read and learn all about InstantDB and the syntax of using it, start implementation of simple things like tracking slider values, and then really work on deep implementation of some of my more difficult fields like dynamic tasks. And on top of all of that, I'm going to have to get NextJS to play nice (Spoiler: NextJS was difficult to get to play nice with it). Without further ado, here is the plan:

1. Read all about InstantDB. Go through their beginner tutorial.
2. Start by testing out super stupid simple shit, like reading/writing a single string text field.
3. Try some more complex operations, like updating a JSON field, or partially updating some keys of a JSON field.
4. Try creating Typescript interfaces that work with read operations, and experiment with their InstantCLI to update the schema for my table directly from Typescript. (and tie it to my interfaces)
5. Get a working version in NextJS that includes my introduction and reflection fields.
6. Start testing out the ability to switch blogs of different days, and start adding more than one row in my "Table". (can you even call it a table? It's called a namespace, but I think that's dumb. I like table, I'm going to call my namespaces tables from now on)
7. Start implementing the more complex Task fields into my table. It's a dynamic list of Task objects, which normally don't play well when UPDATING.
8. Smooth over a lot of nextJS issues encountered when using InstantDB. There's a lot of issues for NextJS that come along with real time updates to your data. UseMemo is going to be my friend here, so I don't murder React with re-renders every time I type a goddamn character.
9. Rework Quill to work with image uploads and code chunks again (Future will here: getting vietnam flashbacks going back and editing this shit)
10. Write today's blog in the new InstantDB refactor!
11. Experiment with ways to automatically read and insert my Postgres stored blogs into InstantDB.

So let's start with the basics of basics. In yesterdays blog I should have written about redesigning the blog builder to be a NextJS application wrapped by Flask. Converting everything to NextJS is a lot of boiler plate and took a lot of time to set up, but now I'm ready to get down and dirty with InstantDB. I started by doing a lot of research into InstantDB. I read the docs, I read the examples, I looked at showcase code. I even looked in their discord channel to see what people were building with it and how. Unfortunately, InstantDB's documentation can be somewhat lacking. It speaks to a pro that almost all of the basic operations are pretty easy, but the lack of complex examples made me want to jump right into building.

So I did start building, first by doing their basic tutorial. I got a basic example up really quick (using Vite tho ew). Cool. I really like the fact that you can start building very quickly. No schema needed to start, which can sometimes be more of a problem than a feature (says the guy whose favorite language is Python).

Initializing the app is super easy. I provide a Typescript schema and then call init with my application id, which is okay to expose to the client. Cool Im already experienced in hard coding API keys! The syntax for reading from the database is very easy. You simply provide the namespaces (tables) you want, and describe the data you want or leave it empty for all of the data. I now should take a small amount of time to describe the schema that I created for this Blog Builder. I used their suggested format of providing these schemas in an instant.schema.ts file at the root of my nextJS project. I called it dailyBlogs, which looks like the following:

As a massive JSONB fan from SQL, I started by modeling the entire reflection and introduction section as a JSON field. This would allow for rapid iteration on the design if needed. Also, only later did I find out that you could actually create the JSON schema with a Typescript interface, which I already had from my NextJS frontend. Dope dope dope. As you can see, I have two tables: dailyBlogs and task. Task needs a little explanation. Each daily blog can have 1 to N tasks dynamically, as thats how I break apart major parts of my work during the day. Now normally when I make SQL tables, I really shy away from making unnecessary complexity of multiple tables. I instead chose to use JSONB (huge shocker) to store the list of Task objects. Here's the old SQL schema:

If you remember from previous blogs, I choose to jointly design my SQL schema with Pydantic models, representing JSONB fields as submodels or in the case of task, a list of submodels. This worked well when the heavy lifting could be done in Python by the Pydantic model itself. Also JSONB in Postgres provides some really nice helper functions that I could use directly in SQL. The problem I'm now facing is that InstantDB doesn't quite have the same level of functionality with JSON. You basically have two options, overwrite the entire JSON object at once with .update(). OR you can overwrite the value of some key/value pair or nested key/value pair with .merge(). This worked really well, except for one case. Whenever I wanted to delete a top level key from my JSON, such as removing an entire Task from the task list, I couldn't. It would simply set the value to null or undefined, which wasn't very fun to code around. And using a built in array object for my list of Tasks was even worse, as I had to wholely replace the entire array every time I wanted to make an update. That means no prop passing, no React components, or if I did do that a fucking ugly display of prop drilling. No. So one thing that I really overlooked at the start of my InstantDB adventure was the power of RELATIONS. Crazy, right. InstantDB gives you the opportunity to link namespaces together, very similar to how a foreign key relationship might work in Postgres. Normally I shy away from this kind of complex relational modeling, but it seemed like it might be the best choice to refactor the blog builder. I had no problem with refactoring the introduction and reflection to JSON, as there would only ever be 1 of each for every daily blog. However tasks could very much benefit from this relationship.

This is the Schema definition for the links between my tables (namespaces). It reads very simply, a single dailyBlogs (I already named it plural I wish I could go back) has many tasks. Where each task by definition can only belong to one dailyBlog. It looks even better in InstantDBs explorer (dumb name InstantDB, who are you: Dora?)

So this is pretty much the final schema, which was made after many trial and error attempts. Honestly, it was difficult to get my head around some of the graphQL syntax and way of thinking. And also NextJS most definitely got in the way of things too. Here's the core of my root page. I initialize the db with my schema, create my query to get all dailyBlogs and receive a list of their linked tasks, read in the data, and then I have my logic for pinning the currently selected blog. The currently selected blog has dependencies on the selectedDate (duh), and also the activeTask. I need to be able to re-render the UI when the user selects another blog, or the active task changes, because that could mean the deletion of tasks or addition of new tasks.

There's some additional core logic like dealing with a blog for a new day, where I have to do some initialization. I cannot stress enough how important the useMemo function is. Currently, almost any time the data can change React will force a re render. Constraining that was one of my first hard tasks, and then figuring out which dependencies I needed to add to the useMemo. I began to discover the power of instantDB when I thought about how I wanted to make separate react components for the Introduction and Reflection. I started by erroneously creating the react components and then refetching the data from my database with a new init. WRONG. This caused re-renders galore, and was clunky. I thought about it for a moment, and theres no reason I can't just pass down the selectedBlog.introduction and selectedBlog.reflection. All I have to do is pass down the data, and any updates to the selectedBlog at the root level will re-render the page for me. Dope!

As I was working through this, I realized I still needed to provide some kind of functionality for updating individual fields of the introduction. Because instantDB has a .merge() function which allows you to overwrite keys or nested keys in a JSON field, I wrote two functions to merge the fields of any field_name, and any field_value. I wrote the mergeNumeric because one of the UI libraries I chose for my mood sliders returned the value as a number or number[] for some stupid reason, so I was lazy and rewrote it. Also, there is no need to reinitialize my database connection. Fuck you supabase, I can just pass down the db and tx to access my database. I use that inside the merge functions and like Todd Howard says: "It just works!". Honestly this got me really excited. After really figuring out the syntax and how to not get react blow up with re-renders, this is super simple and intuitive. It's kinda beautiful. Fr. Not just glazing because. I like it.

Since I was using Quill editors, I had to add an event handler to each that called the merge function whenever a field changed. This is a preview into the hell that Quill caused me, but I'll go over that later. I added some other logic to initially load the previous value of a field.

Things got a little more interesting as I move to implement the Task section, which if you remember I created as a separate table in InstantDB. I did this because I was going to have to find a way to dynamically read and update 1 to N tasks for each blog. Similarly to the introduction, I pass down each task into a Task React component, as well as the db and tx for accessing Instant. Interesting to note that I have to pass down a function to deleteTask and switch the UI view of the active task. That must occur at the root level, because I need to trigger a top level React re-render in case new Tasks are added or deleted. Also another note, Ive previously been updating Instant by calling dailyGoals.update. In the tasks, I'm calling the Task table instead.

Most of the rest of the implementation was generally boring, and had to do with polishing off React specific stuff. Not bad! However, I must warn you, the next task I'm going to write is going to go over the absolute hell of dealing with Quill editors in React Typescript.

I'm looking to clean up some of the look, feel, and usage of the Blog Builder. Some minor things have changed since I started the major refactor.

Mostly looking to enact QOL changes in the UI of the blog builder, as well as some QOL changes eradicating warnings and following some best practices in coding.

First I'm going to focus on eliminating errors and warnings, especially from Quill. Next I'm going to make some QOL changes to the UI and frontend, which coincide with the major refactor to InstantDB which I'll talk about in the next task.

Here's one of the annoying errors. It's not exactly the same error, but it's the same idea. Quill does not play well with Typescript at all. I will talk about this later today in greater detail. Whenever you initialize a Quill editor with a non-default tool format, such as Image or Code Block, it absolutely complains that you didn't provide an image handler for it. Sick. All it took was adding a dummy handler to get it to shut up. Although I debate whether I wanted to wrap Quill Initializations in code that nuked the console.warning feature, I decided this was probably bad design. Here's how I initialize the Quill options:

Future Will here: I went to fucking war with Quill's image handler today. I'm going to talk more about it later, I'll dedicate an entire task to the bullshit I went through. For now though, I can talk more about some QOL Blog Builder changes I made.

First things first, I wanted to stop the default behavior of the previous blog builder to load today's blog. I updated the initial logic to get all valid dates of previous blogs, and always require that today's date was included. Now, a user must select a date for a blog and "load" the blog. It's kind of dumb, because I load all of the blogs anyway with InstantDB. But this plays so much nicer with React. I was hitting some weird race conditions with React's DOM loading and my data loading from InstantDB, where infinite reloads were ocurring. If I make the user manually choose and load a blog, I can gurantee that all my data is loaded from InstantDB. That way I can be more careful on setting up useEffect dependencies for reloading the UI of the page.

Another QOL change is handling the naming of tasks differently. Previously, I had simply called each task "Task 1" or "Task 2", but that was stupid. And it got more of a bad idea when I started having to use InstantDB. I thought briefly about sticking to the index and 'reindexing' when a blog was deleted, but that would mean iterating through all tasks and reassigning objects in InstantDB. And this would cause a terrible problem where I could never actually delete a blog, only use InstantDB's merge functionality to set an index to null. It got messy super quick, and this led me to refactor how I do tasks. Again, I'll talk about that in the next Task which is about my InstantDB refactor. Here's a pretty image of the ability to actually set names for tasks (although under the hood they're actually governed by the ID of the task entity). QOL!

I've been beating around the bush here talking about the major refactors I have been doing with InstantDB. I thought I'd start with something simple, easy, and visually appealing. We're going to get down and dirty with IndexedDB refactor in the next task don't worry.

Quill is being a pain in my ass. I need to manage the initialization, handle destruction, and get god damn image uploading to work in NextJS!

As stated in the goal, fuck Quill in typescript. I was somehow able to get it working just fine in Javascript, but when you add the strict typing requirements of Typescript horiffic shortcomings in Quill's source code come to light. Not to mention the absolute difficulties in making Quill play nice with React's element lifecycle.

This one seems to have no planned approach. I need to figure out why the fuck things aren't working.

1. Why the fuck is it so hard to initialize a Quill editor in Typescript, and then access it later?
2. Why the fuck are there no typescript types for core parts of the Quill code?
3. How the fuck do I fix Image uploading, to allow for my custom Image override that worked just fine in Javascript?

Okay, so let's talk about Quill initialization first.

It was relatively simple in Javscript. I created divs with IDs for all of my fields, then I called initializeEditor. I would select the Div, attach a Quill instance with some toolBar options. I would set the inner HTML, and add a custom image handler for uploading images to supabase storage. Not too difficult, definitely a 3/10 on the Javascript complexity scale. However things get much more complex when we're talking about using Quill in React. Manually handling the creation, destruction, and re-rendering of Quill editor instances became IMPOSSIBLE in React. It did not play nicely at all. So after a lot of research, I found a react friendly Quill fork which allowed for me to use the useQuill hook: "const { quill: quill_personal_context, quillRef: quillRef_personal_context } = useQuill({ theme, modules });

". This worked good enough, but I soon ran into problems with updating text fields into InstantDB. In Javascript I attached an event listener, however that wouldn't work well here. Through many hours of reading through the dogshit Quill documentation, which contained no typescript Type hints for key pieces, I came up with this monstrosity:

For every one of my quill editors, I had to create separate references for the quill editor class and the reference to the DOM element. I had to create a useEffect to initialize the Quill Editor properly. To start, this useEffect has dependencies for the selectedBlog and quill_personal_context. That way, this only runs once when the quill editor is created above, and only re runs when the entire blog changes. I created a custom image Handler for handling images, because their default handler is broken as fuck. You can't override any useful functionality in typescript as all of the types are :unknown. I have to provide a function to start the chain of uploading an image, as well as provide a reference to the Quill class, so that deep within my lib functions I can actually embed the image once I save it. I attach another handler to listen for text changes, which calls the mergeField function for the correct key (field) of the task table. And finally I have to nuke the Quill instance when re-renders happen. One of the most infuriating errors was when I first created this and could not tell WHY switching tasks would overwrite the fields of the previous task. I debugged and ensured, yes I had the correct ID. Everything I was doing using InstantDB was correct. I just couldn't see that my previous Quill instance was alive, and still watching changes in the OTHER task, and updating the WRONG task with the wrong HTMl. Now I need to talk about the most painful part, which is the image handler:

I had to create a bunch of different functions for handling different parts of the upload process. selectLocalImage kicks everything off, which is called by my custom Image Handler. This creates an input button where the little icon for upload image is, and adds a listener that will wait for the button to be pressed. It starts the chain of opening a modal to upload an image, saving said file to my supabase storage, renaming the file to have the correct URL of supabase, and finally re-embedding the <img> tag into the editor. Now the entire problem with this whole chain was that Quill failed to keep track of the initial quill editor instance. The base Quill code failed horifficaly in typescript, which led to the inability to embed images. I had to override functionality to pass in a reference to the editor myself, and ensure it didn't conflict with the implementation of Quill's toolbar which for all intents and purposes is immutable.

Well enough complaining about Quill, although it was a nasty nasty challenge. Now you reading this are probably thinking, "Will that's kinda interesting. If you fixed it, and spent so much time reading the source code, you should submit a pull request and help other people out!". Great idea, I would love to, yet somehow beat me to it. And you know what's crazy, his fix is 1 line of code. I wish I was kidding:

And what's even crazier is that it hasn't been merged for a year and a half now. It's kind of funny, there's like 10-15 comments with many "upvotes" or whatever you call it of people who all had the same problem, begging the maintainers to merge it. Sad.

Today was a very long day of grinding. A day of learning about InstantDB, and most importantly building. I'm starting to love InstantDB, even though it can be a little rough around the edges. (Dave, you must make the title today about InstantDB)

Need to update my frontend to read from INstantDB. Part of that will be retroactively converting SQL schema into the InstantDB format.

I need to work on ensuring that the AI editor pipeline still works. I need to connect the Flask backend to the NextJS Blog Builder frontend. I am interested, Sean suggested I polish the site a little more and then ask Swyx his opinion on it. He's a big fan of learning in public, which is what I built this shiet for in the first place.