The Developer Bottleneck
Back in 2019, our platform needed custom calculations for everything: energy savings, carbon emissions, baseline comparisons, aggregations, unit conversions, and rollups. Every customer had unique requirements. Every building had different equipment. Every report needed slightly different maths.
The problem was clear. We had only a handful of people who could write these calculations in SkySpark's Axon language, and we had dozens of stakeholders who needed calculations built: internal engineers validating measurement and verification methodologies, project managers configuring dashboards, and clients requesting bespoke metrics. Every calculation request joined the queue, waiting for scarce developer time.
The fundamental tension was familiar to anyone who's worked with business users and code: spreadsheets versus software. Our non-technical colleagues understood formulas. They could build complex calculations in Excel or Google Sheets without any help. But translating those spreadsheet formulas into production code required a developer, and that developer had become a bottleneck.
The Solution: Declarative Calculations
The core insight was to stop thinking about calculations as code and start thinking about them as data. Instead of writing functions that fetch inputs, transform them, and produce outputs, we could define calculations as structured configurations that an engine would interpret.
We built on SkySpark's native "history grid" data type, essentially a data frame for time series. Each calculation defines its inputs (columns), its formula (how to combine those columns), and its output. The engine handles the mechanics: fetching the time series data, aligning timestamps, handling missing values, performing unit conversions, and evaluating the formula row by row.
Architecture
Each calculation is defined by a set of inputs and a formula. Inputs can be time series data from sensors (pointInput), the output of other calculations (calcInput), or static values (constantInput). Each input type supports configuration options that handle the messy realities of real-world data.
The key configuration options for each input included:
- toUnit: Convert to a specific unit before use
- nullValue: Replace missing values with a default
- foldFunc: Aggregation function for rollups (sum, avg, max, etc.)
- convertFunc: Quantity conversions between related units where only the time dimension differs (e.g., energy to power, distance to speed). Since each row represents a specific time interval, these conversions are straightforward.
- asUnit: Reinterpret units without numeric conversion
The engine processes calculations in phases. First, it assembles all inputs into aligned columns. Then it applies transformations: unit conversions, null handling, and any custom functions. Finally, it evaluates the formula across the grid, producing the output series.
Version 1: Google Sheets Interface
The first iteration used Google Sheets as the user interface. This was a pragmatic choice: our users already knew spreadsheets, and Sheets provided a familiar environment with built-in collaboration. We built a sync layer that pushed calculation definitions from Sheets into SkySpark and pulled validation results back.
This worked, but had limitations. The sync overhead introduced latency. Version control was awkward, with different iterations of spreadsheets scattered across our shared drive. Most significantly, users had to context-switch between Sheets and the platform, making the feedback loop slower than it needed to be.
Version 2: In-App Editor
The second version brought calculation editing into the platform itself. We built a custom editor using a pill-based formula interface. Each input appears as a clickable pill in the formula view. Clicking a pill reveals its configuration panel, allowing users to adjust unit conversions, null handling, and other settings without leaving the formula context.
This design used progressive disclosure. The formula overview shows the high-level structure. Clicking any input pill reveals its detailed configuration. This keeps the interface clean while giving power users full control. The Bulgaria front-end team built this interface, translating our design specifications into a polished React component.
Public Presentation
In 2019, I presented this work at SkyPosium EU in Rome, the annual conference for the SkySpark community. The talk covered the developer bottleneck problem, our declarative approach, and the lessons learned from two iterations of the interface.
The reception validated our approach. Other SkySpark users faced similar bottlenecks. The presentation forced us to articulate our design decisions clearly, which improved our own understanding of what we'd built.
Evolution and Current State
The calculation engine continues to run in production today, years after its initial deployment. It processes thousands of calculations daily across customer sites, handling everything from simple unit conversions to complex nested calculations.
The system has evolved significantly since 2019. We added support for nested calculations with mixed intervals: a parent calculation might operate on hourly data while referencing a child calculation that processes minutely data. The engine handles the necessary rollups and folds to align the data correctly.
Performance requirements eventually drove a migration from pure Axon to Java. The primary motivation was horizontal scalability in our cloud environment, something we couldn't achieve with a typical SkySpark deployment. These architectural evolutions will be covered in a separate project write-up.
The most recent layer is a browser-based TypeScript component that handles final processing, performing client-side transformations, caching, and presentation logic without round-trips to the server.
Impact
The developer bottleneck is gone. Engineers define their own measurement and verification calculations. Project managers configure dashboard metrics. Clients request (and often define) custom calculations for their reports. The developers who once spent their time translating spreadsheet formulas into code now focus on improving the engine itself.
More importantly, the architecture has proven durable. Six years of production use, hundreds of customer sites, thousands of calculations. The declarative approach made the system extensible: new input types, new configuration options, and new output formats can be added without rewriting existing calculations. The abstraction worked.