2020

SkySpark Tooling

Building tests, migrations, and Git workflow on a platform that shipped without them

SkySparkTestingMigrationsDeveloper Experience

SkySpark is a fantastic platform for IoT, built from the ground up to handle the problems you hit when you're ingesting, storing, and acting on streams of sensor data. It comes with a database (Folio), a query language (Axon), a web interface, and a way to package and ship extensions (pods).

SkySpark ships with a wealth of apps for different personas. When you're writing code, you spend your day in two of them: the Code app and the Tools app, side by side. Write a function in one, call it from the other, see the result immediately. As a fast-feedback loop, it's hard to beat.

The day-to-day in-browser workflow: Code app on the left, Tools app on the right, fast feedback in between

For developers used to a full IDE, though, the in-browser experience falls short. It's come a long way over the years I've worked with it, but compared to IntelliJ or VS Code, it's still an editor in a tab. Three things in particular were missing for the way we wanted to work: source control, a way to run automated tests, and a way to evolve the database schema across environments in a controlled order. We had a platform built on SkySpark, so we built that tooling ourselves. This page covers those three pieces.

Code is records, not files

When you write a function in the SkySpark editor, you're not writing a file. You're creating a record in Folio, SkySpark's document database. Functions, configuration, and the metadata that describes your data points all live there as records.

That's the first disconnect with a normal web-dev workflow: there's no Git history on a Folio record. Folio's optimistic concurrency control will stop two saves from silently overwriting each other, but the resolution is blunt: if your save conflicts, you reload and lose whatever you just typed. No PR, no review, no diff worth keeping.

SkySpark does have its own packaging system. A pod is a versioned bundle you can ship from one instance to another, which gives us a clean unit of release. But pods are for shipping, not iterating. While we were still working on a function, the work happened in the editor, against the live records.

What we wanted was both feedback loops at once: the fast in-browser one (write a function, run it from the Tools app, see the result) for iteration, and a slower out-of-browser one (build a pod, spin up a Docker container with SkySpark and the pod loaded, seed test data, run the tests) for the things that give us confidence in production code. To make that work, the same code had to exist in two places: in Folio as records, and on disk as text. And they had to stay in sync.

Bridging the editor and the repo

Each SkySpark project has a built-in io/ directory inside its install. Files written there are accessible to Axon. In dev mode we bind-mount that directory to a gitignored work/ folder in the repo, so anything Axon writes shows up locally without polluting Git. The Git-tracked source lives separately, with one .trio file per Axon function:

bolt-skyspark/
├── src/main/axon/lib/                  # Production functions, one per .trio
│   ├── dev/                            # IO sync helpers
│   │   ├── exportFuncsToIo.trio
│   │   ├── importFuncsFromIo.trio
│   │   └── autoExportFuncs.trio
│   ├── feedPoint/                      # Domain functions
│   │   ├── createFeedPoint.trio
│   │   └── ...
│   └── migration/                      # Versioned migration functions
│       ├── runAllUpdates.trio
│       ├── v1_0_063__seed_default_units.trio
│       ├── v1_0_064__add_feedPoint_uuid.trio
│       └── ...
├── src/test/axon/                      # Test functions, mirroring main packages
│   ├── feedPoint/
│   │   ├── testCreateFeedPoint.trio
│   │   └── ...
│   └── migration/
│       ├── test_v1_0_063__seed_default_units.trio
│       └── ...
└── work/                               # Gitignored; bind-mounted to SkySpark's io/
    └── <PROJ>/io/                      # .trio files land here when Axon writes

.trio is SkySpark's flat-text format for serialising a Folio record: a few key/value lines for the record's tags, followed by a src: block holding the function body. The format is symmetric: anything you can put in a Folio record, you can dump to .trio, and vice versa.

Two Axon functions and two shell scripts make the round trip between Folio and Git. The first, exportFuncsToIo, reads functions from Folio and writes each one out to io/ as a .trio file. Here's the function itself, exported:

// exportFuncsToIo.trio

name:exportFuncsToIo
func
doc:Exports all Folio funcs to .trio files in the io/ directory
src:
  () => do
    readAll(func).removeCols(["id", "mod"]).each( f => do
      filePath: "io/" + f->name + ".trio"
      f.ioWriteTrio(filePath.parseUri)
    end)
  end

importFuncsFromIo does the reverse: it reads the .trio files in io/ and writes them as records into Folio, skipping anything already present. From there, two shell scripts (copyIoFilesToSrc.sh and copySrcFilesToIo.sh) shuttle files between the gitignored work/ directory and the Git-tracked source tree. The full edit-and-commit loop: edit in the browser, run exportFuncsToIo, run the copy script, commit, PR.

For active development, a one-call setup helper takes care of the loop: it loads the latest source into Folio and starts a background job that drifts your in-browser changes back out to disk every few minutes. After that, the shell scripts are only needed when you're ready to commit.

The full round trip: source files in the repo, a manual or shell-script copy into work/io, importFuncsFromIo into Folio, exportFuncsToIo back out, and a copy back to source. Three boundaries to cross.

A testing framework on afAxonExt

SkySpark doesn't ship a test runner. The community library afAxonExt (from Fantom Factory) gives you two things: a runTests primitive that takes a list of functions and runs them, and a set of assertions like verifyEq and verifyNotNull. What it doesn't give you is anything else you'd expect from a modern test framework: setup, teardown, discovery, isolation. We built that on top.

The wrapper, runAxonTests, gave us those four things. It discovered tests by tag rather than by an explicit list; it ran package-level setup and teardown around each test; and it handled record isolation, which was the real reason we built it.

A test in this framework is a normal Axon function with a package tag declaring which group it belongs to. The function name starts with test, and the body follows the usual structure:

// testGetWorkbench.trio

name:testGetWorkbench
func
package:workbench
src:
  () => do
    code: curFunc()->package
    markerDict: getTestMarkerDict(code)

    // Arrange
    siteRec: addSiteForTest(code, false, {dis:"Test Site"})
    persistedWorkbench: {navName:"Test Workbench", jsonStr: "{}"}
      .merge(markerDict)
      .addWorkbenchForTest(testUserUuid, siteRec->id)

    // Act
    retrieved: getWorkbench(persistedWorkbench->id, testUserUuid)

    // Assert
    verifyEq(retrieved->navName, persistedWorkbench->navName)
    verifyEq(retrieved->jsonStr, persistedWorkbench->jsonStr)
  end

The interesting bit is the marker pattern. Folio is a single shared database; tests can't spin up an isolated copy the way a SQL test would. Instead, every record a test creates is tagged with a per-test marker (workbenchTester in the example above). Teardown runs a single readAll(parseFilter(markerStr)) and removes everything matching, so a failing test never leaks state into the next one. Setup and teardown functions are themselves discovered by tag, so a package can declare its own without the framework needing to know about it.

Axon is where a huge amount of our business logic lives, and we needed the same level of confidence in it as in the rest of our stack. Java and TypeScript already had test coverage running in CI; the SkySpark code needed the same. The framework was wired into that pipeline, which built the Docker image that ultimately got deployed to every environment.

Versioned migrations

Folio is a document database, closer in spirit to NoSQL than to SQL. There's no , no ALTER TABLE; "schema" is just whatever records happen to exist. The pros and cons of NoSQL versus SQL are well documented and not the point of this section. The point: we had a development environment, staging, and multiple prod environments running in parallel, and each one ended up with a slightly different idea of what records existed. The database wasn't going to enforce consistency for us across them, so we wanted a best-effort implementation that did.

The standard pattern for this in SQL is versioned migrations: each one checked into source control with a version number, and a runner that tracks which versions have already run on a given database. Plenty of tools implement this (we use Flyway in our Java stack). We borrowed the pattern for Axon.

Migrations are Axon functions tagged dataUpdateFunc with versioned filenames like v1_0_064__add_feedPoint_uuid.trio. The version string is the prefix, parsed at runtime from the filename itself; the description follows a double underscore. They live under migration/ alongside the rest of the code, so they get reviewed in PRs the same way a calculation change would. A real one looks like this:

// v1_0_064__add_feedPoint_uuid.trio

name:v1_0_064__add_feedPoint_uuid
dataUpdateFunc
doc:Adds the uuid tag to all feedPoints
func
src:
  () => do
    feedPoints: readAll(boltFeedPoint and not uuid)
    count: feedPoints.size
    if (count > 0) do
      diffs: feedPoints.toRecList.map(w => diff(w, {uuid: randomUUID()}))
      commit(diffs)
      logInfo({name: "migration"}, "Added uuid to " + count + " feedPoints")
    end
    count
  end

A runner is wired into SkySpark's startup steady-state hook, so it runs automatically on every boot. It discovers everything tagged dataUpdateFunc, sorts by name (which sorts by version, thanks to the zero-padded patch number), and runs each one. Before invoking, it checks Folio for an audit record keyed by boltAudit == "dataUpdate" and script == <funcName>. The behaviour from there depends on a runControl tag on the migration:

  • ONCE: skip if any audit row exists. The default for irreversible state changes.
  • COMPLETE: skip if the previous run reported zero work. Useful for migrations that should keep retrying until the dataset is fully cleaned up.
  • (no tag): re-run every invocation, incrementing totalRuns on the audit row. Reserved for maintenance functions you want to invoke deliberately.

Each migration returns a count of how many records it changed, which lands on the audit row alongside timestamps and the description tag. That gave us two things. First, deterministic state: every environment converged to the same schema regardless of when it last shipped. Second, a written history: when something looked wrong on prod, we could read the audit records to see exactly which migrations had run, when, and how much work each one did.

Audit records from a dev environment: one row per migration, with version, description, run timestamps, and the count of records each one touched

Sharing it with the team

SkySpark is proprietary software, so developers joining the team typically arrived with no prior exposure to it. The training programme covered both halves of that gap: SkySpark itself (Folio, Axon, the Code/Tools workflow), and the patterns we had built on top of it (IO sync, the testing framework, migrations). I ran the first sessions for the team in April 2024.

The format was a mix of walk-throughs and hands-on exercises, building up from "how do I query the database?" through to "write your first migration".

With the platform basics and our conventions covered together, new joiners could pick up full-stack tickets (Java, TypeScript, and Axon side by side) without spending weeks finding their footing in the SkySpark side.