In this first post, we’ll be taking an exclusive look at the backend stack for data processing here at Oscar and how it continues to evolve. This is a lengthy discussion, so it will come in three parts over the next few weeks. Building an engineering program at a new tech company is an exciting and often tricky business. There are so many amazing new technologies out there, so many battle-tested and reliable services, and so many tools that could be right for the job if you just make a choice and tweak it until it works for you. It’s fun to try new things and experiment, and it’s rewarding to build rock-solid services. At Oscar, we’re all about maintaining a high degree of curiosity and technical exploration, while relying on proven methods and technologies. Does that sound consistent to you? It’s not! So, how do you reconcile conflicting engineering impulses while still having a good time? By making sure you’re addressing the stuff that matters first and foremost. Let’s establish some foundational premises from which arguments can proceed. Those premises will be the properties our systems must achieve by whatever means we come up with. We’ll get back to the conflict in a later post. Data Acquisition First, what are the properties we need? Remember, we’re talking about backend data processes alone here. Let’s start with one of the first things we had to do in engineering - connecting partner feeds. The insurance business is a complicated space, and we’re currently handling multiple data feeds from about ten different partner companies. As of this writing, we’re receiving about 60 individual data feeds and that number is likely to explode over time. Some of the feeds are trivial, requiring us to simply copy new data into our storage systems and leave it there, but that’s the exception. One of our feeds is an ASCII database transaction log representing hundreds of tables from a remote database written in Cobol. Many are fixed-width text files or CSV that must be parsed according to a schema of byte offsets and data types. For good measure, we also see EDI and HL7. Whatever the format, the mission remains the same: to create a unified and up-to-date view of the data for users - both external and internal. And of course, don’t let a single bit get misplaced! This data really matters. It affects the lives of our customers and if we mishandle it, we can create inconvenience or even add significant stress at a time when focusing on medical treatment should be top priority. Basically, we shouldn’t ever screw up our data, and when it gets screwed up (yes, it will happen), we must be able to recover fast. Now we have our properties - whatever our choices, our feed systems must: ● Respect order. ● Break on failure ● Be idempotent ● Be low-latency ● Maintain privacy That’s already a fair amount of properties. If we have several competing approaches, they all have to maintain the same general behavior and that’s more work.

2. Our objective seemed simple at first: to
create a unified and up-to-date view
of incoming data for all users, without
misplacing a single bit.
!
It started small, just a few vendors to
connect. Maybe 5 important feeds.
!
5 became 10, then grew to 50.
There were data streams
In the beginning
Formats became more varied, as did the
properties of the feeds themselves.
!
Achieving the goal was going to require
thought, but first, let’s look at the data.
2

3. ASCII blobs - schemas define character
ranges and data types.
!
Example: Transaction log from a Cobol
DB. 254 tables in single feed.
!
Aggressive parsing required! Often, data
types and NULL constraints are the only
obvious clues indicating data integrity.
ASCII byte ranges
Fixed-Width
3
00226345user.ix1 00000074
2013102112110400CHANGED
00000074IT8208IV*Z IT8208
IT8208 INFO| 100000.00
100000.00 100000.00
A20050101INITLD20120720IT8208
215 91515 91414 11114 7 0 9
814 7 1 0111414 0 010 7 0
51514 01415 41515 01415 0 714
0 01015 11114 51514 41515
0151414 0 1 3 1 3 1 3 1 3 1 3
1 3 1 3 1 3 1 3 1
3SASSSASSSASSNORM
0225DEMO
AHAUSSLER@SS-HEALTHCARE.COM
18991231
NORMNORM
A00819870000271400000002C00025
89
C00272560000050000000087

4. Since 1989 (v2), XML as of 2005 (v3)
!
Targets clinical and admin data
interchange among hospitals
!
Core reference model has been
called an incoherent standard
(Smith & Ceusters, 2006)
Health Level Seven
HL7
4
<POLB_IN224200 ITSVersion="XML_1.0"
xmlns="urn:hl7-org:v3"
xmlns:xsi="http://www.w3.org/2001/
XMLSchema-instance">
<id root="2.16.840.1.113883.19.1122.7"
extension="CNTRL-3456"/>
<creationTime value="200202150930-0400"/>
<versionCode code="2006-05"/>
<interactionId
root="2.16.840.1.113883.1.6"
extension="POLB_IN224200"/>
<processingCode code="P"/>
<processingModeCode nullFlavor="OTH"/>
<acceptAckCode code="ER"/>
<receiver typeCode="RCV">
<device classCode="DEV"
determinerCode="INSTANCE">
<id extension="GHH LAB"
root="2.16.840.1.113883.19.1122.1"/>
<asLocatedEntity classCode="LOCE">
…
</POLB_IN224200>

5. Since 1971
!
Elements, segments, loops, and hierarchy - like
XML, schema is up to the designers. Unlike XML,
structure must be derived at parse time.
!
Context-sensitive grammars! Need a parse stack
to understand all EDI document types.
Electronic Data Interchange
EDI
5
ISA*00* *00*
*12*ABCCOM
*01*99999999*101127*1719*U*004
00*000003438*0*P*~GS*PO*440519
7800*999999999*20101127*1719*1
421*X*004010VICS~ST*834*0179~B
GN*00*1*20050315*110650****~RE
F*38*SAMPLE_POLICY_NUMBER~DTP*
303*D8*20080321~N1*P5*COMPAN_N
AME*FI*000000000~INS*Y*18*030*
20*A
REF*0F*SUBSCRIBER_NUMBER~NM1*I
L*1*JOHNDOE*R***34*1*0000000~P
ER*IP**HP*2138051111~N3*123
SAMPLE
RD~N4*CITY*ST*12345~DMG*D8*196
90101*F~HD*030~DTP*348*D8*2008
0101~REF*1L*INDIV_POLICY_NO~SE
*16*0179~GE*1*1421~IEA*1*00000
3438

6. Get a feed, write a parser, model it, store it, try to
do it right, then deploy it. 50 times! In 50 ways?
Sure, as long as we stay organized
Easy, right?
6

7. Every feed has its own characteristics.
!
Every project seems to want its own solution.
!
It’s so easy at first just to implement things.
Then comes The Mess.
Insurance is hard. Let’s simplify it.
Complexity is the Enemy!
Can we identify common properties
needed by our systems, and guarantee
that those properties are satisfied in
reaching a fundamental goal?
7

8. Control latency. Integrate data as
quickly as possible.
!
And last but far from least:
!
Maintain privacy. Our jobs are often
handling user data, and they must
handle private data with great care.
To keep our data correct, we need
to ensure some fundamental
system properties
8
Respect order. You process things out
of order, you corrupt your data.
!
Break on failure. Never write bad data
or continue in an abnormal state.
!
Be idempotent. Mid-process or mid-
transaction errors are to be expected.

9. Judgment is required here, in
abundance. These goals are more
nebulous but of critical importance to a
growing engineering team.
Let’s not forget some
important higher-order
properties
9
Implement consistently. Avoid The
Mess. Consolidate best practices.
!
Deploy consistently. Every new deploy
style carries a constant increase in
operational complexity.

10. Parse and model each feed as a custom
step, but let the framework handle
common properties.
!
oscaretl: a framework for
transactional safety
10
Data streams in which data is not
independent (often the case), will
benefit from being handled as
transaction logs.
!
Monotonically increasing transaction IDs
are very useful. Try to derive them if they
don’t exist naturally in the data.

11. Parsers and schemas are custom, but
data formatting, safe writes, and safe
execution can be factored out.
!
Common schema types can be re-used.
!
Factoring out the
common elements
11

12. Good start, but we need more: at this
point, a good runtime model and a job
scheduler.
What have we achieved?
12
Strict ordering. Processing will halt on missing data.
!
Idempotence. Careful state binding allows us to
resume where we left off.
!
Break on failure. Processing will halt on error.