The site uses cookies that you may not want. Continued use means acceptance. For more information see our privacy policy.

JSON Schemas and Making a Versatile Application.

Please excuse the big wad of JSON in the middle of the post.

Things have come a long way from the previous post (diehealthy.org: 27 October 2022: “Working on a Local Single Page Application.”) about working on a local file that provides a whole data tracking. While I originally wanted it to be used for games, I’d seen other JSON-schema-based stuff online and decided it could be fully schema driven. While it doesn’t yet edit its own schemas, that’s probably feasible in the long run (if not something I’ll necessarily do).

In order to further refine it toward that schema system, I forked it and made a version to track the movies I’ve watch. The post image is of that version, though they are currently at parity aside from the schema differences.

To give an idea what I mean by schema, here’s some excerpts from the film version:

{
    "name": "filmtracker",
    "defaults": {
        "filters": {
            "recorded": 1,
            "watched": 0
        }
    },
    "props": [
        {
            "name": "name",
            "title": "Name",
            "type": "string",
            "display": "headertext",
            "displaystyle": "italics",
            "sortOn": true,
            "req": true,
            "def": null
        },
        {
            "name": "recorddata",
            "title": "Record data",
            "type": "group",
            "props": [
                {
                    "name": "added",
                    "title": "Added",
                    "hidden": true,
                    "display": "fulldate",
                    "displayprefix": "Added on",
                    "type": "date",
                    "sortOn": true
                },
                {
                    "name": "modified",
                    "title": "Modified",
                    "hidden": true,
                    "display": "fulldate",
                    "displayprefix": "Updated on",
                    "type": "date",
                    "sortOn": true
                }
            ]
        },
        {
            "name": "watchdata",
            "title": "Watch info",
            "type": "group",
            "props": [
                {
                    "name": "recorded",
                    "title": "Recorded",
                    "type": "bool",
                    "display": "iffalse",
                    "displayvalue": "Unrecorded.",
                    "summaryvalue": "📅",
                    "filter": true,
                    "def": 0
                },
                {
                    "name": "watched",
                    "title": "Watched",
                    "type": "bool",
                    "display": "text",
                    "displaydepends": [
                        "recorded"
                    ],
                    "displayvalues": [
                        "Unseen",
                        "Seen"
                    ],
                    "summaryvalues": [
                        "💾",
                        "🍿"
                    ],
                    "filter": true,
                    "def": 0
                },
                {
                    "name": "watchdate",
                    "title": "Date of watch",
                    "type": "date",
                    "display": "monthdate",
                    "displaydepends": [
                        "watched"
                    ],
                    "displayprefix": "Watched",
                    "sortOn": true,
                    "sortDep": "bought",
                    "def": ""
                }
            ]
        },
        {
            "name": "filmdata",
            "title": "Film info",
            "type": "group",
            "props": [
                {
                    "name": "releaseyear",
                    "title": "Year of Release",
                    "type": "num",
                    "display": "text",
                    "displayprefix": "Year:",
                    "sortOn": true,
                    "minv": 1900,
                    "def": 2022
                },
                {
                    "name": "score",
                    "title": "Review score",
                    "type": "range",
                    "display": "stars",
                    "displaydepends": [
                        "watched"
                    ],
                    "displayprefix": "Score:",
                    "summaryvalue": "stars",
                    "sortOn": true,
                    "sortDep": "watched",
                    "minv": 0,
                    "maxv": 5,
                    "step": 1,
                    "def": 0
                }
            ]
        },
        {
            "name": "categories",
            "title": "Tags",
            "type": "strarray",
            "display": "ulist",
            "def": null
        },
        {
            "name": "notes",
            "title": "Notes",
            "type": "text",
            "display": "longtext",
            "summary": "📓",
            "def": null
        }
    ]
}

It’s quite a lot, and messy (as is the rest of the application so far), and it mixes data definitions with their presentation, but it’s mostly serviceable. But it lets me define, with the current version:

  • groups
  • numeric inputs
  • string inputs
  • url inputs (unvalidated at the moment)
  • date inputs (though I’d prefer to move to month/year or such)
  • boolean inputs (implemented as radio buttons)
  • tri inputs (radio buttons)
  • range inputs (radio buttons)
  • text inputs (text boxes)
  • string array inputs (for tags, with autocomplete!)

I’d originally gone with checkboxes and three-state checkboxes for booleans and tris, but decided it was too much trouble and having separate buttons looked nicer. The basic checkboxes were reused instead for toggling off or on the listing filters for boolean and tri fields.

It can also search in and sort the listing.


One of the nicer parts of all this is the ability to export and import from JSON, as well as being able to use the browser console to loop through and batch-modify the data. I’m not sure how feasible it would be to implement that through UI, which would be more useful to nontechnical users. But for now this is just a little project to handle my own data.

If I can get it cleaned up, I’ll probably throw a copy up here at some point, though so far I’m still iterating on it a good bit as I go through my old list of movies I watched and fill in their details. I started keeping a list at some point last year, and that was title-only, so it’s been a lot of searching, figuring out which was the movie I actually watched, trying to remember enough to give it a rating.

But the main goal is that if I decide I need a list of something, I can copy the file over, throw a schema together, and start making the list in a way that lets me update it or review it better than a text file and not requiring the ugliness of a spreadsheet. Like when I kept a list of the legislators who objected on 6 January 2021, that’s just a text file. Next time it could be a nice little application that makes the task not necessarily easier, but more useful:

  1. Get a text list of the things
  2. Some light massaging to turn it into JSON
  3. Write a quick schema
  4. Import the data
  5. Fill in any details, be able to filter, sort, search easily.

I can only imagine what data wrangling looks like in 100 years. I hope by then it’s all AI user interfaces that look pretty are functional no friction. Until then, this project has reminded me how powerful, how versatile, and above all how fairly simple, HTML plus Javascript has gotten.

Customizing the Web

Small example of how modifying your web can be useful.

I use the web quite a bit, and one of the things I do quite often is customize my experience on sites I frequent. Most of that customization comes via Stylish and Scriptish (the latter a fork of Grease Monkey).

Some of the things I customize are minor, like page aesthetics. Some I consider more important. Take Google Search for example. Did you know some of your search results are censored? This is thanks to laws like the Digital Millennium Copyright Act (DMCA). Google is good enough to notify you when censorship has occurred.

But only if you scroll to the bottom of the search page. Which you may not do for quick, one-off searches. This is the equivalent to newspapers printing important news on page 8N which is otherwise a full page of ads.

But thanks to Stylish:

@namespace url(http://www.w3.org/1999/xhtml);
@-moz-document domain("google.com") {
  #mfr {
    position: absolute !important;
    top: -20px !important;
    right: 0px !important;
    width: 300px !important;
    background-color: #FCC !important;
    font-size: 1.2em !important;
  }
}

This adds a bit of styling to the notice and throws it at the top-left of the search results.

With most software this wouldn’t be possible. Without the open web that notice (if there at all) would be permanently stuck down at the bottom. Maybe I would adopt the practice of always looking for it, but it’s doubtful.

Instead, I can remain aware of the prevalence of censored results. I can find out more.

The challenge is opening up this technology to those who don’t know CSS, JavaScript, HTML, or how to debug those technologies (as modifying the page often requires understanding it rather than simple repositioning like above). There are ways to do that, but so far they seem limited to having programmers hand ready-made solutions to novices (eg, via sharing snippets like that above).

The problem with that approach is that it limits the idea pool to those who either can implement a modification themselves or knows someone who can.

It Came…From the Internet! MutationObserver

A brief overview of the DOM4 MutationObserver.

[Fade in Louis Armstrong singing On the Sunny Side of the Street.]

Grab your lab coat // Put on your hazmat, baby
Leave your worries on the doorstep
Life can be so sweet
On the sunny side of the street
[…]

Okay, MutationObserver isn’t really that scary. It’s part of the DOM4 specification as a replacement for Mutation Events. MDN: DOM: MutationObserver documents it.

What are Mutation Events? A DOM3 specification to allow you to watch the DOM change using the well worn events API. You know, the old

node.addEventListener('click', onNodeClick, false);

style events.

But Mutation Events made things slow. A lot of sites mutate or change the DOM quite often, and depending where you attach the listener and how frequent the changes come, it added up to a lot of event traffic. MutationObserver replaces this.

A MutationObserver doesn’t receive a callback for every change. Instead it receives periodic callbacks for a group of changes. That group may contain a single change, depending on the stability the DOM tree (if only a few changes occur at large intervals, they may be announced separately). Think of it like an announced log instead of alerts.

MutationObserver takes a bit more setup than just tagging an event listener on a node. But that setup makes it cleaner and more useful.

var observer = new MutationObserver(function (mutations) {
    mutations.forEach(function (mutation) {
        console.log(mutation.type);
    });
});

That’s a simple MutationObserver. You provide a function that receives an Array of MutationRecord objects. But that just creates the observer, you have to activate it.

var config = { childList: true, attributes: false, characterData: false, subtree: true };

So first you need an object (technically a MutationObserverInit object) specifying what types of mutations you care about (you can also specify some extra options about which data to include in the MutationRecord objects you get back). You really just worry about making this give you what you want and not give you anything else. If you want to know about node added or removed from the DOM tree, you don’t need attributes or characterData, for example.

Once you have your configuration ready, you also need a root node to observe. You can observe document, the true root, but you should attempt to observe the minimal subtree you will need.

Say the page shows a chess game, and you want to add some automatic comments or classes (for styling) to the move log. You only need to observe the part of the tree that contains the log. The page logo, the board itself, the chat, and anything else don’t matter.

var moveLog = document.querySelector('#moveLog');

We can finally turn the MutationObserver on:

observer.observe(moveLog, config);

Now when new changes happen under moveLog, the function we specified when creating the observer will be called back. But we can go a bit further. Let’s say you have two separate subtrees to observe. You could either find their lowest common ancestor and observe that whole subtree, or you can do the smart thing and simply register the observer with the second subtree:

observer.observe(oneNode, config);
observer.observe(twoNode, config);

Note that was, “register the observer with” the node, not the other way around. Although you call on the observer, this tells it to add itself to the list of observers associated with the node in question.

You can also shut the whole thing down:

observer.disconnect();

This kills all the registrations, so unfortunately you cannot easily modify the actively observed subtrees.

Also note:

observer.observe(myTarget, config1);
observer.observe(myTarget, config2);

This replaces the options for observing myTarget.

Anyway, that’s a brief overview of the not-so-scary, highly useful MutationObserver. It’s particularly nice for user scripts that want to cope with dynamic web pages. Instead of being stuck with some modification only for the nodes that were there when the page loaded, you can now easily observe changes and carry your modifications over as new nodes appear.