A meditation / review of work in progress.
Context: I think we're at a relatively stable point in development, so I wanted to just summarise where I am, and how I got here, because I think I need to spend the next 2-3 weeks on bookkeeping and hardware repairs instead!
Architecturing the software stack at a small business
1. Table of Contents
2020-09-23, meditations on work
Design Concerns for Small Businesses
● Technology Infrastructure
○ A Brief Review of Requirements and Design
--------------------------------------------------------------------------------------
● Business Context
● 0. Standardisation of Language
● 1. Outsourcing of Machine Maintenance
● 2. Cost Granularisation
● 3. Hedging Against Vendor Lock-in
● 4. Graphical User Interfaces
● 5. Software Development Framework
○ 5. Super-structure
○ 5. Reducing Degrees of Freedom
○ 5. Normalised Form Data Handling
○ 5. Normalised JSON Handling
● Thanks
2. Design Concerns
for Small
Businesses
Technology
Infrastructure
A Brief Review of
Requirements and
Design
In the second half of 2019, I found myself piloting a four-year-old cafe
business into its fifth year of operations. Initially capitalised at around
$100,000, the balance sheet had shrunk to just $25,000 after the first year
of operations. The three years that followed required a nearly constant
effort to optimise and improve operations through research and
development.
Due to internal disagreements, this B2C brick-and-mortar business had
now been running for almost two years without social media - an
unanticipated forcing factor, which deeply shaped the company’s
operations, and thus branding. You don’t start an independent lifestyle
business with minimal funding UNLESS you have free access to marketing
channels which your larger competitors cannot fully utilise due to their
regulatory limitations.
In any event, regardless of operational weaknesses … it was high time we
spent some time on software development, in order to build capabilities
which go beyond those of other companies in our immediate spaces (hotels,
restaurants, cafes, co-working spaces, co-living spaces, and managed
facilities in general).
My estimate being that we have about 0.2 programmers - myself, devoting
about a fifth of my time to software development, meant I’d have to do
about a year’s worth of coding every five years.
3. 0
… but then what
tools should we
build on?
The
considerations
were as follows.
0. Standardisation of Language
● We’d want to be able to communicate about software using a
medium which is stable and formally defined.
● Strategy: the most politically mature, and indeed officially
standardised of the popular contemporary computer programming
languages was JavaScript, so I decided we would build on that.
4. 1
How should we
pick, design, or
build tools?
(continued)
1. Outsourcing of Machine Maintenance
● We’d want to push as much work as possible to managed clouds, to
avoid having to run our own low-level servers whenever possible.
● Strategy: the most politically mature, and feature-wise diverse
public cloud available appeared to be Amazon Web Services
(henceforth, AWS), so this was the next variable locked down.
5. 2
How should we
pick, design, or
build tools?
(continued)
2. Cost Granularisation
● We’d want to optimise for minimal incremental cost per newly
developed business service. We would optimise for a high volume of
services, each operating on small datasets.
● Strategy: I aimed to normalise work around a FaaS architecture;
specifically, the “AWS Lambda” FaaS with deployment to the Internet
via the “AWS API Gateway” reverse proxy. For persistence, the “AWS
DynamoDB” database was selected, under its “On-demand Capacity”
billing model. For federated/authentication, the “AWS Cognito”
OpenID Connect Identity Provider would be used.
6. 3
How should we
pick, design, or
build tools?
(continued)
3. Hedging Against Vendor Lock-in
● We’d want to retain the flexibility to migrate off any specific platform,
to something self-managed in the event of growth.
● Strategy: Cohesive with our language selection of JavaScript, and
preference for a FaaS architecture, I elected to use the Node.js
runtime language interpreter. AWS Lamba’s wrapper around its
hosted Node.js runtime is minimal, so we should be able to migrate
off AWS to self-hosted Node.js instances if and when the need arises.
● Tactic: likewise, the projected target for replacing AWS DynamoDB,
should the need arise, is the (supposedly compatible) open-source
ScyllaDB.
7. 4
How should we
pick, design, or
build tools?
(continued)
4. Graphical User Interfaces
● In order to minimise repetitive low-level CRUD programming, we
would benefit from some sort of WYSIWYG (popularly termed
“no-code”) front-end for data modelling and manipulation.
● Strategy: create a virtual-table (henceforth, “desk”) system, where
the GUI allows for creation of Cartesian (2D) desks defined in terms
of column-schemas on the X-axis, and rowed-data on the Y-axis.
● Tactic: given the architecture of DynamoDB, I have settled on a
rudimentary two-table backend. One table holds desk schemas, and
the other holds desk cells, where each cell is a single item. The latter
table has four indexes, allowing us to perform rudimentary
comparison queries on data in our desks.
8. 5. Software Development Framework
● Now that we roughly knew what we wanted to build, and with what
kinds of macroscopic tools, the question remained about how we
would organise our minute-to-minute work of building it. This is
generally referred to as “picking a coding style”, or “framework”. A
superficial survey of the marketplace for JavaScript frameworks
proved to be comically dismal, as anticipated from the memes about
these. So I dismally admitted the equally risky approach, of course,
to build a framework from scratch.
● Strategy: The approach to building this framework, however, was to
first seek to understand the various Internet Engineering Task
Force standards around HTTP, JavaScript, HTML, and related
technologies, and then to establish a language agnostic design
pattern which could then be implemented as needed in any future
language runtimes. The reference implementation for this would of
course be on the list of preceding list of technologies already
determined.
5
How should we
pick, design, or
build tools?
(continued)
9. 5. Super-structure
● Tactic: in order to maximise transparency, and user-maintainability
of the framework, I hoped to utilise composition or piping of data,
through a “chain of responsibility”, whenever possible.
● Anti-pattern: a popular architecture for client-server frameworks is
the “onion middleware” approach, where requests are piped from
the outside of the onion, into its centre, and then back to the outside
again. This means that the business logic walks through each
middleware twice … and commonly we find that for each time the
business logic walks through a middleware, there is are points of
intercept, typically called “callbacks”. So for each middleware there
turn out out to be three things to worry about, the “before-callback”,
the “middleware logic” itself, and the “after-callback”. This is too
complicated.
● Pattern: in reaction to this, I chose to simplify the data flow through
such that business logic walks through each middleware once .... so
that there are no callbacks, and every additional operation that we
want to run on the data is just another middleware. A
reducer/dispatcher is used to convey the output of each preceding
middleware to its successor.
5a
How should we
pick, design, or
build tools?
(continued)
10. 5. Reducing Degrees of Freedom
● Tactic: IETF standards and RFCs allow for very broad diversities of
shape in application architecture. For the purposes of this project, I
sought to narrow them down to a more rigid subset.
● Anti-pattern: a critical example of weak “standards” is the official set
of HTTP methods, whose definitions are far from
mutually-exclusive-and-collectively-exhaustive (MECE) of possible
operations on data.
● Pattern: staying within the loose IETF definitions, I am exploring the
imposition of narrower definitions upon common HTTP methods.
○ The shape this has taken so far, is that PUT and DELETE are
considered fundamental operations.
○ Constrained-POST is defined as a batch of PUTs and DELETEs.
POST is not used in any other way.
○ When batch-atomicity is added, then what results is a
PATCH-over-constrained-POST. PATCH is not used in any other
way.
○ This may reduce the number of API end-points, as you just
need to implement a robust PATCH-over-constrained-POST,
and implicity it would be able to handle any PUT, DELETE, or
constrained-POST, with or without batch atomicity.
5b
How should we
pick, design, or
build tools?
(continued)
11. 5. Normalised Form Data Handling
● Anti-pattern: Getting data from HTML to servers is complicated also
because of the weakness of the HTML form data standard.
● Pattern: staying within the loose IETF definitions, I am exploring the
imposition of narrower definitions upon HTML form data handling.
○ Establish one syntax to embed data in HTML form
input[name]s (a subset of HTML); and a library which parses
HTML form data to JSON/POJOs.
5c
How should we
pick, design, or
build tools?
(continued)
12. 5. Normalised JSON Handling
● Anti-pattern: Similarly, most server-side frameworks tend to pick
their own way of operating on data.
● Pattern: I am exploring the imposition of narrower definitions upon
JSON/POJOs.
○ Establish another declarative syntax which describes
validation rules for JSON/POJOs (a subset of JSON); and a
library which validates arbitrary JSON against valid models.
5d
How should we
pick, design, or
build tools?
(continued)
13. 5. Normalised Reactive Object Storage
● Anti-pattern: Why are there so many reactive stores, and reactive
data usage patterns but no formal definition standard from IETF?
● Pattern: In order to learn how to write a formal definition for these, I
have prototyped a reactive object store for use on either server or
client sides, however, this work was antecedent to the earlier points
(5a-5d) and is not currently integrated with these.
5e
How should we
pick, design, or
build tools?
(continued)
14. 5. Normalised Internode Communication
● There isn’t much to say here except that I feel we should just port
Erlang/OTP semantics to JavaScript. :P I’ve gotten started, and have
some proof of fundamental concepts, but the broader project is
parked and not expected to have all of OTP’s functionality for the
foreseeable future. Might have to strike the lottery first.
5f
How should we
pick, design, or
build tools?
(continued)
15. Well, that’s it for now.
That’s all I’ve managed to develop so far, and it is still a work in progress.
Please send me a message if you find any of this interesting.
I generally work by myself, and am in various states of discomfort from
that experience, but I haven’t found much else to do with my time. Coffee
and chit chat is welcome.
- Jerng
- fb.com/jerngatwork
- twitter.com/jerng
- wa.me/0162352931
How should we
pick, design, or
build tools?
(to be continued)