Financial Modeling for Real

Financial Modeling for Real
How to Debunk the Myths of
Finance and Economics
Using R Code
Steven H. Kim
Kenwave Research MintKit Institute

Contents
0. Overview, 4
1. Introduction, 7
Aspects of Novelty, 7
Lineup of Goals, 8
Layout of Contents, 10
2. Background, 12
Driving Forces, 12
Orthodox View of the Markets, 13
Degrees of Efficiency, 14
Spawns of the Efficient Dogma, 15
Flaws of the Efficient Voodoo, 17
3. Method, 18
Features of the Dataset, 18
Proper Metrics, 19
Treatment of Null Returns, 20
Choice of Statistical Test, 22
Pick of Software Environment, 23
4. Fetch and Glom of the Data, 24
5. Gauging the Monthly Returns, 30
Formatting the Input, 30
Monthly Gains Over Time, 31
6. Crafting the Models, 34
Encoding the Efficient Models, 34
Building the Sway Model, 35
Average Return by Month, 36
Outliers in a Boxplot, 37
Products of the Boxplot Command, 39
Mean versus Median Gains, 40
7. Profile of Errors, 42
Chain of Flubs, 42
Boxplots of Residuals, 43
Variance of Errors, 45
Average Size of Flubs, 45
Median Absolute Deviation, 46
Comparing Measures of Scatter, 47
8. Overall Direction of Movement, 52
Precise Peg of Ups and Downs, 53
Roundup of Directional Bias, 57

9. Global Score of Errors, 59
Size of Errors, 59
Triumph of the Seasonal Model, 61
10. Odds of Intra-year Wobbles, 63
Audit of Orientation, 63
Mettle of October versus August, 64
Recap of Intra-year Odds, 66
11. Magnitude of Intra-year Swings, 67
12. Discussion, 69
Fiber of the Binomial Test, 70
Approximate Technics in Practice, 72
Universal Scope of the Binomial Tool, 73
Frequency of Errors, 74
Constructive Plays Versus Expunging Schemes, 75
Caveat for Intra-year Results, 76
Streaks of Novelty, 77
Progression of Patterns, 78
Supplementary Materials, 78
13. Conclusion, 80
Strengths of the Binomial Test, 81
Rundown of Major Results, 82
References, 85
* * *
© 2020 Kenwave.com and MintKit.com

0. Overview
The Efficient Market Hypothesis (EMH) lies at the heart of finance and economics. According to
the entrenched doctrine, the real and financial markets are so efficient that they absorb every nub
of information with perfect wisdom and adjust the price of every asset forthwith. In response to
each flash of news, the updates occur so abruptly that no single actor can exploit the dispatch
with any measure of confidence or prowess.
One outgrowth of the EMH is the Random Walk Model that pictures the path of the market as a
thoroughly erratic process. If all the clues have already been baked into the current valuation of
each asset, then the next shift in price has to come as a complete surprise.
Sad to say, but this caricature of the real and financial markets belies the actual behavior of the
participants along with the assets. A testament to the fallacy shows up in the subtle but persistent
waves in the stock market throughout the year.
From a formal stance, the EMH corresponds to a couple of models that are dubbed here as the
Idle and Drift templates. By contrast, a pliant portrait of seasonal patterns is evinced by the Sway
Model.
The triplex of frameworks is assessed against the record of the stock market spanning 69 years
starting from the onset of 1950. The raw data consist of monthly readings of the S&P Index, the
leading barometer of the bourse among professionals ranging from fund managers to academic
researchers.
To underscore the gulf between the theory and reality, the case study employs only a minute
fraction of the wealth of information freely available to all comers at the top resource favored by
the investing public. Moreover, the quantitative assay relies solely on the most basic of statistical
tools; namely, the binomial test that is highly robust by dint of its simplicity. From a
computational stance, the attendant code invokes just a tiny subset of the plethora of functions
built into the core module of the R system: the leading choice of programming language and
software platform for data science in the context of scientific research as well as pragmatic
inquest.
Based on the Idle Model, the market should move about as often in the upward direction as the
downward heading. However, a binomial probe using hard data rejects the bland model of equal
odds at a highly significant level. Put another way, the Idle mockup is grossly inconsistent with
the reality of the marketplace.
4

By contrast, the Sway Model does a laudable job of balancing the odds of an uprise versus a
downstroke of the market. More precisely, the fraction of positive moves is 0.493 thus lying
close to the expected value of 0.500. In this way, the seasonal template bisects the prospects of
rising and falling with a great deal of verism. From a formal stance, the cyclic framework
conforms to the null hypothesis of equal odds against a two-sided test of the alternative premise.
The foregoing results focus on the direction of motion. Each assay is inherently a binary affair
dealing with a shift of the market to the upside or downside while ignoring the extent of the
traverse.
A different way to compare the candidate models involves the size of the errors that ensue. In
this context, one template outperforms another if the magnitude of the lapses on average happens
to fall below its rival’s. By this measure, the Drift Model trounces the Idle version at a high level
of statistical significance. Better yet, the Sway framework outshines both of the Efficient
mockups.
For starters, the seasonal model bests the Idle icon in 55.7% of the cases at a staunch level of
statistical heft. In particular, the p-value of 0.0006081 falls decidedly below the usual boundary
of 0.05 that marks the threshold of significance.
Moreover, the Sway motif outranks the Drift version as well. The former beats the latter in
53.9% of the cases and proves its edge with a p-value of 0.01425.
In these and other ways, the limber model of seasonal waves disproves the dogma of efficiency
at ample levels of statistical grit. The upshot is to roundly upend the dominant school of finance
and economics.
From a larger stance, this guidebook fulfills a medley of objectives. A broad-based goal is to
dispel the gospel that proclaims the markets to be so efficient as to absorb every shard of
information with perfect wisdom, thus leaving no further tip for foretelling the market. By this
argument, the price of every asset is deemed to move in a purely flighty and aimless fashion.
A second and pragmatic goal is to devise an honest profile of the temporal patterns in the stock
market. The factual template serves to debunk the myth of randomness that hampers the mass of
practitioners and observers ranging from investors and analysts to academics and policymakers.
A third aim is to present a tutorial for harnessing the R system to make sense of a thorny and
quaggy domain. In the process, a tiny subset of the basic functions built into the software
platform is enough to answer cryptic questions of great consequence in theory as well as
practice.
A fourth target deals with the power of an elementary tool in statistics for grasping a knotty
system in the form of the stock market. The latter complex has long thwarted—and continues to
stymie—myriads of actors ranging from solo investors and communal pools to academic
researchers and public officials. Remarkably, though, the plainest technique in the toolkit of
5

hypothesis testing yields decisive results at high levels of statistical clout in plumbing a murky
domain of the utmost complexity.
6

1. Introduction
According to the reigning school of financial economics, the markets are so slick and quick that
the entirety of information available to the general public is snapped up at once and put to use
with perfect wisdom. This sleek image of the marketplace is known as the Efficient Market
Hypothesis.
More generally, the doctrine asserts that the price of every asset reflects the totality of facts and
figures, skills and insights, tactics and rumors, available to the entire population. Since every
scrap of know-what and know-how is baked into the current price, no one can improve on the
valuation of assets ranging from stocks and bonds to commodities and realties.
As a corollary, the EMH trumpets the absence of trusty cues to predict the price level: if every
gob of dope has already been imbibed, then there is nothing left over for portending the market.
Ergo, every move has to come as a complete surprise. A direct fallout is the Random Walk Model
that pictures the path of each asset as a form of Brownian motion whereby the price level is
prone to shift in any direction with equal likelihood.
Against this backdrop, the main goal of the guidebook in hand is to debunk the glib model of
efficiency while using the stock market as a case study. To underscore the chasm between the
theory and reality, the work plan takes a minimalist approach.
For starters, the assay employs only a paltry fraction of the treasury of information freely
available to all comers at the most popular portal among the investing public. Moreover, the
quantitative analysis relies solely on the simplest technique in the entire arsenal of statistical
tests. From a computational stance, the program invokes a skimpy subset of the inbuilt functions
within the kernel module of the R system: the leading choice of programming language and
digital platform for data science in the context of scientific assays as well as pragmatic pursuits.
Aspects of Novelty
Since the autumn of the 20th
century, a smattering of in-depth studies has exposed a variety of
flaws in the Efficient Market Hypothesis and its spinoffs. Given the beachheads into reality
forged thus far, clearing up the morass of myths and spoofs is not in itself a novel exercise.
7

Instead, the innovative features of this handbook lie in extensions to prior work. In addition to
the concrete results rendered by the study, the experimental setup displays a medley of desiderata
ranging from simplicity and parsimony to clarity and efficacy.
From a tactical standpoint, the refutation of the Efficient dogma takes three forms. The first
mechanism relies on proof by contraposition, as the testable properties of the EMH happen to
contradict the actual behavior of the marketplace. The second type of rebuff involves proof by
existence; in particular, a textured model of the bourse displays lithe patterns that differ markedly
from the stiff and bland stencils espoused by the Efficient mantra. The third and related mode of
rebuttal lies in proof by supersedence whereby the refined model turns out to be demonstrably
superior to the kludgy icons of the fictive doctrine.
Any type of proof—whether by contraposition, existence or overclass—is by itself enough to
uproot the EMH. In that case, the three forms of confutation taken together pose a triple
whammy of overwhelming evidence against the ruling credo of efficient markets.
Furthermore, each class of proof may in turn bear a multiplicity of results. For instance, the tactic
of supersedence reveals that the Sway Model supplants the Idle template in terms of the
imbalance of moves to the upside versus downside; and likewise for the average magnitude of
the errors in profiling the monthly returns. The story is similar for the dual vantage of the Sway
framework in topping the Drift makeshift. In these ways, the method of outperformance by itself
delivers at least four distinct conclusions, each of which suffices to expunge the EMH and its
spinoffs.
To sum up, the case study presents a nested system of multiform proofs anchored in multiplex
dimensions. The meshwork of rebuffs amounts to a great deal of overkill since each of the
elemental results is by itself enough to ax the myth of efficient markets.
Lineup of Goals
A key objective of this handbook is to build a practical model of the seasonal patterns in the
marketplace. In this context, the stock market represents the epitome of a flighty system that
eludes the mass of practitioners and researchers ranging from investors and analysts to
academics and policymakers.
A second goal is to dethrone the ruling dogma of finance and economics which proclaims the
marketplace to be a model of perfection: a system so efficient that it absorbs every whiff of
information at once, thus leaving no further scent for forecasting the price level. As a direct
offshoot, each asset is deemed to move in a thoroughly random and unpredictable fashion.
A third function is to serve as a tutorial for using the R system to craft a simple but trenchant
model of a convoluted system. Amazingly, a small subset of the basal functions built into the
software platform is enough to corral a problem of mind-boggling complexity.
8

A fourth task is to demonstrate the power of an elementary technique in statistics for grasping an
enigma that has long confounded legions of actors ranging from lone investors and fund
managers to academic researchers and market regulators. In fact, the simplest prop in the entire
repertory of hypothesis testing is enough to render decisive results at high levels of statistical
grit.
The binomial test used throughout this guidebook happens to be exact rather than approximate.
That is, the criteria for testing the null hypothesis make use of clear-cut numbers rather than iffy
or fuzzy estimates.
To bring up a counterexample, consider a statistical tool which is far more popular than the
binomial probe in all manner of domains ranging from scientific research to public policy. In
deploying the t-test, the threshold for a critical region stands on the assumption that the data
spring from a normal distribution. Unfortunately, the latter constraint happens to be a mootable
and uncertain premise in modeling a system of any kind in the real world.
By contrast, the binomial routine makes no assumptions vis-à-vis the nature of the random
process other than the independence of trials governed by fixed odds for a pair of disjoint
outcomes. In our study, the duo of turnouts takes the form of a gain or loss pegged by a suitable
measure of performance. More precisely, the basic yardstick deals with the monthly return on
investment to the upside or downside. A variation on the theme involves the lead or lag in
performance notched by one model against another in profiling the market.
In the process, the binomial test does not require the observations to come from a Gaussian
process or any other constrictive function. For this and other reasons, the technique is
straightforward and versatile as well as elegant and trustworthy.
The empirical results from the case study comprise only part of the agenda. A complementary
goal is to present a friendly tutorial for building a concise model in the R language in order to
capture a sprawling system amid the din and flap of the world at large.
The topics on hand include the core issues for consideration, ranging from the design of the
overall methodology to the rationale behind specific commands. At the high end of the spectrum,
an example of a metalevel theme involves the aptness of monthly returns for tracking the stock
market; the vantage of the median over the mean as a robust measure of centrality; or the fettle of
the binomial test as a sturdy mode of hypothesis testing. Meanwhile an instance of a nuts-and-
bolts topic concerns the acquisition of the dataset required for the case study; the convenience of
a tabular setup for culling wispy patterns from noisy data; or the automatic conversion of data
types on the fly when the R system executes a given command.
9

Layout of Contents
In line with earlier remarks, a primal aim of this primer is to fashion a candid model of the
seasonal patterns in the stock market. Moreover, a kindred goal is to expose the false foundations
of finance and economics.
A secondary role is to spotlight the mojo of a basic technique in statistics for grasping a chronic
source of pratfalls in a slippery domain. A parallel function is to showcase the translation of an
abstruse puzzle into a simple program in the R language.
After this introductory chapter, the next module presents the background behind the Efficient
Market Hypothesis and its outgrowths. The survey includes a flashback of the landmark
publications in the field. A related task is to expose the humbug that has come to dominate both
theory and practice in the marketplace.
On the other hand, Chapter 3 delves into the methodology behind the case study. The topics on
hand range from the choice of data series and the gauge of price action to the starch of the
binomial test and the flair of the software environment.
The subsequent chapter presents a mélange of basic steps in acquiring and dissecting the data. An
example involves fetching the raw values then reading the entries into the R system. Another
sample concerns a qualitative expo of the dataset by way of graphic displays.
Chapter 5 starts off the initial phase of the quantitative assay. The first step is to reckon the
fractional changes in price from one month to the next throughout the entire timeline. Another
task entails a visual display along with an intuitive scan of the periodic changes in price.
The next chapter deals with the translation of the Efficient doctrine and the Sway motif into their
formal analogs. An example lies in a tally of the overall return across the entire timespan,
followed by a pin-down of the geometric mean on a monthly basis, thence to a setup of the Drift
template in precise form. Another instance concerns a simple way to format the data in order to
compute the mean and median values of the monthly returns throughout the year without
drawing on ornate schemes such as programming loops in the R language, nor add-on packages
meant to streamline the manipulation of time series data.
Chapter 7 examines the pattern of errors stemming from the triplex of pricing models. An
example in this vein is a visual capsule of each cluster of errors by way of a boxplot. Another
type of précis involves the mean and median levels of the ensuing residuals.
Next up is Chapter 8 dealing with the first phase of the quantitative workout based on the
direction of motion without regard to the extent of the movement. The attendant program makes
use of concise and handy ways to compare values and perform tests in the R language.
10

By contrast to the odds of rising and falling examined in the previous module, the next chapter
centers on the magnitude of the errors that arise. The appraisal covers the timeline as a whole
without paying heed to the movement of the market in a particular month versus another within a
given year.
If the market exhibits a cyclic pattern tuned to the seasons, then certain months are apt to fare
better than others. This phenomenon is fathomed in Chapter 10.
In particular, the buoyant month of October is compared against the downcast spell of August in
order to gauge the extent of the divergence. The grilling focuses on the odds of rising or falling
throughout the entire timeline, regardless of the amplitude of the moves to the upside or
downside on each occasion.
Meanwhile, Chapter 11 is similar but different. The contest here involves the size of the returns
in October versus those in August of the same year. In other words, the purpose of the assay is to
discern whether the former month tends to outshine the latter in a credible way within a given
year.
The next chapter digs into a posy of metalevel issues. An example involves a closer look at the
advantage of the binomial test over other techniques in hypothesis testing.
Another topic concerns the moxie of a rudimentary probe based on the odds of an upturn versus
a downthrow of the market as a way to defrock the shamans of efficiency. For starters, the rift
between the EMH and the actual data shows us how the market does not behave. Better yet,
however, is a constructive proof by way of a sound template that shows how the market does in
fact behave—witness the Sway Model.
This chapter also points up a caveat on the hazards of statistical studies in general and sampling
methods in particular. To wit, a fine line separates a mindful study of the salient patterns in a
dataset versus a slapdash focus on striking features that could well be mere artifacts of an
aberrant sample.
This pitfall highlights the need for a healthy dose of skepticism in judging the results of a
statistical assay in any niche, no matter how compelling the findings may seem at first sight.
Naturally, the need for thought and sense applies even more to flaky myths that clearly contradict
the data, as in the likes of the EMH and its entourage.
This chapter also presents a couple of supplementary materials. One adjunct is a compact listing
of the lines of code scattered across multiple chapters of this very guidebook. The second rider is
a file containing the raw data used in building and testing the candidate models.
Finally, Chapter 13 affords a medley of closing remarks. One topic deals with the universality of
the methods used in the case study along with the hearty benefits they provide. Another sample
involves a roundup of the main conclusions from the case study.
11

2. Background
To provide the proper context for the case study, this chapter surveys the crucial issues from a
conceptual stance as well as a pragmatic slant. The topics span the rainbow from driving forces
and knock-on effects in the marketplace to historical milestones and inborn flaws of the orthodox
doctrine.
Given the diversity of causal factors along with the complexity of their tie-ups, the real and
financial markets have long foiled—and continue to vex—the mass of practitioners and
observers. On the bright side, though, a dollop of quiet reflection is enough to unveil some vital
features of the landscape.
From a practical stance, a simple program in R is sufficient to construct a solid model of the hazy
patterns in the field. The concrete findings serve to refute the airy theory of perfect markets that
pervades the vale of finance and economics.
Driving Forces
The securities in the financial forum are issued by private concerns and public organs for a
variety of reasons. An example involves a bond stamped out by a government agency in order to
to raise funds for a special project, as in the case of building a dam or paving a highway. Another
type of instrument involves the equity of a company listed on a stock exchange as a way to gain
cash for expansion or boost the firm’s visibility amongst the general public.
In the financial arena, the price of each widget depends on the perceptions of the participants
regarding the security along with its issuer. To cite an example, a bond beckons the investor with
a stream of periodic dividends followed in due course by the payback of the principal upon the
scheduled date of maturity—assuming that the debtor remains viable and retains the ability to
discharge its obligations. Meanwhile a share of stock conveys a slice of ownership in the current
assets and future profits of the business as long as the firm survives the vicissitudes of the
marketplace.
In the financial theater, the owners of bonds have priority over those of stocks. If a company
were to declare bankruptcy, liquidate its assets, and close its doors, then the bondholders would
have first dibs on any of the surplus that remains.
12

By contrast, the shareholders are entitled to the ensuing scraps which may well amount to little
or nothing at all. Due to the specter of hefty losses for the shareholders, stocks tend to be more
volatile than bonds. In exchange for the greater risk in store, however, the shares offer the
prospect of juicy returns by way of capital appreciation. In this milieu, the main draw of a stock
is the potential for capital gains while that of a bond is the promise of cash dividends.
The valuation of any asset depends greatly on the conditions in the marketplace. For instance, a
spot of good news may lift the price of a stock while a stroke of bad luck might sink it. If a
company is to remain viable, however, its gross revenues and net earnings must expand over
time if only to keep pace with the bloat of inflation. Since a share of equity represents a claim on
the intake of profits, the widget ought to appreciate in tandem.
Along the way, though, the stock will gyrate with the shifting fortunes of the real economy as
well as the financial forum. An example of the former is the total volume of sales or earnings in a
broad swath of the economy. For instance, certain markets may surge in the winter as in the likes
of heating oil or ski equipment, children’s toys or tropical cruises.
The story is similar on the financial front. From a larger stance, millions of workers in the real
economy enjoy perks by way of cash bonuses as the year draws to a close. In the wake of the
windfall, a portion of the bounty finds its way into the financial tract and thus bolsters the
demand for virtual assets ranging from singular stocks to composite funds.
In these and other ways, a slew of cycles prevail in the tangible economy as well as the financial
bazaar. As a result, there is plenty of reason to suppose that the assets in the marketplace should
flow and ebb with the seasons of the year. On the other hand, we would not expect the cyclic
pattern to be highly obvious nor strictly regular. As we shall see, any conspicuous motif will
vanish in short order as a side effect of the wheeling and dealing by the hustlers who take
advantage of the opportunities for easy profits.
Despite the tricky factors, however, the meaningful question is not whether any patterns exist at
all in the financial realm. Rather, the thoughtful query concerns the nature and extent of the
motifs along with the degrees of transparency and tractability.
Orthodox View of the Markets
According to the dominant school of financial economics, the actors in the marketplace make
full use of the information available to the general public. An example in this vein is the history
of prices for a singular stock or the volume of transactions for the entire bourse. The sweeping
model of boundless rationality and flawless judgment at all times is known as the Efficient
Market Hypothesis.
By this image of efficiency, the entirety of public information informs the current price of each
asset (Fama, 1965). Since every scrap of intel is exploited to the max, no one can augur the
market beyond its latest price.
13

Like all good myths, this fable of efficiency does spring from a glint of reality. To clarify this
point, we take up a numeric example. Suppose that a stock that currently trades at $7.00 is slated
to rise to $7.50 a week hence. In that case, a host of punters would dive in and snap up all the
shares that are currently on sale at $7.00.
But why stop there? The gamers will be almost as eager to scoop up all the shares offered at
$7.01. In that case, the prevailing price shifts to $7.01 from an even $7.
The turnout is similar at the next higher price. That is, a pack of beavers will scarf up all the
shares offered at $7.02. As a consequence, the price jumps promptly to $7.02.
The same phenomenon is bound to recur at higher prices as well. Due to the ramp-up, the stock
streaks toward the target of $7.50 until there is no more room for making a decent profit. In this
way, the price soars right now rather than a week hence.
This image of zealous action coupled with perfect wisdom stands behind the Efficient Market
Hypothesis. That is, the market taps into the entire pool of information available to the populace
and forthwith vaults to the optimal price based on the panoply of hard facts and cold logic, while
taking account of transaction costs such as the registration fees payable to a stock exchange upon
the purchase or sale of shares.
Degrees of Efficiency
The origins of the EMH and Random Walk may be traced at least to the onset of the 20th
century.
The watershed was a mathematical model of Brownian motion along with its application to the
valuation of stock options (Bachelier, 1900).
The Efficient Market Hypothesis comes in three flavors: weak, semi-strong, and strong forms.
The strength of each version refers to the tightness of the constraints along with the degree of
plausibility.
According to the weak form, the prices of traded assets—ranging from stocks and bonds to
commodities and real estate—reflect all the information from the past which is available to the
general public. In that case, for instance, the historical record of stock prices is useless for
foretelling future changes in price.
Meanwhile, the semi-strong version of the EMH asserts not only that each asset reflects all the
dope available to the public but that the price adjusts posthaste to incorporate brand-new motes
of public information. Put another way, only an operator armed with confidential information
may gain an advantage in the marketplace.
14

Finally, the strong form of the doctrine asserts in addition that the price of each widget subsumes
even occult facts unknown to the general public. An exemplar involves the latest tally of global
revenues beheld only by the top executives of a conglomerate.
From a pragmatic stance, the strong form of EMH is clearly unrealistic. In fact, the unfair
advantage bestowed by confidential information has given rise to regulations against exploitative
behavior such as insider trading by the directors of a corporation.
By the weak form of the Hypothesis, current prices reflect all the public information to be found
in the past and present. In that case, for instance, there is no way to beat the market by sifting
through historical data. This variant of the doctrine is weak in the sense that it embodies fewer
assumptions than the other two versions, but is actually the strongest in terms of the scope of
application.
Put another way, the frail version of the EMH is the most plausible of all—or more precisely, the
least implausible. For this reason, the weakest in a formal sense is the strongest from a real-world
stance.
Meanwhile, the semi-strong form of the doctrine contends that fresh jots of news available to the
public are incorporated promptly into the price level. In that case, no single actor can beat the
market in a consistent fashion by reacting to the bulletins on the news wires.
To be fair, we should note that the orthodox faith does offer a concession in terms of trumping
the benchmarks of the market. In particular, the expected return on investment is deemed to rise
with the risk entailed.
In the financial community, the primary measure of risk reflects the volatility of the price level. A
popular example lies in the standard deviation of monthly returns chalked up by the S&P Index.
Based on this type of yardstick, an investor may earn lush profits by taking up a risky asset.
According to the hallowed theory, however, a surfeit of risk is the only way to garner an excess
return.
By merging the notions of perfect rationality and risk dependence, the EMH avers that it is
impossible to outrun the market after taking into account the volatility of the chosen assets. In
other words, the only way to overtake the market at large is to overflow the risk.
Spawns of the Efficient Dogma
The blarney of efficiency comes with a litter of sidekicks and spinouts. An exemplar lies in
another fanciful theory dressed up with an Ample Dose of Capital Letters. According to the
Random Walk Model, the path taken by the market looks much like the aimless meandering of a
drunken sailor. To wit, the price level at any stage is just as likely to advance as to retreat on the
next step—in which case the trajectory resembles the rudderless moves of a soused seaman.
15

The quintessence of the EMH and the Random doctrine lies in the stock market where millions
of entrants vie with each other to snap up any traces of dope in a fervid effort to rack up superior
gains. If the odds of rising and falling are deemed to be equal, then the current price is the best
guesstimate of the price level going forward. This shibboleth is a direct outgrowth of the
Random Walk yarn from a practical stance. We shall refer to this send-up of the market as the
Idle Model or Lame Theory.
At this juncture, we should take a step back and grok the big picture of the stock market over the
long range. Even a casual student of the bourse can see at once that the price level has a way of
tramping higher as the decades go by—in which case, a plausible model of the bourse ought to
take account of the overall trend.
For this reason, a variation of the EMH opines that the upward tilt represents the best guess
going forward. For instance, the optimal forecast of the price level a month hence is the current
price plus a fitting increment of the gradual migration. We shall refer to the trending version of
the EMH as the Drift Model or Limp Theory.
Since the autumn of the 20th
century, a sea of literature has sprouted in support of the EMH and
its offshoots. Along the way, the mirage of utter efficiency has established itself as the premier
model of financial economics.
Sadly, though, the picture of perfect rationality contradicts the facts of life in the financial mart
as well as the real economy. A good example concerns the challenge of survival in a harsh and
unforgiving environment. Unfortunately, death is the way of life for all manner of firms in the
real world. For this reason, the best forecast of the price level for any stock over the long run is a
breakdown followed by a rubout.
Another form of delusion shows up in a ubiquitous myth in the financial realm. In the received
wisdom of amateurs as well as professionals, a market index tracks the fortunes of the
membership over all time frames, from less than a day to more than a year.
In truth, however, a mutable benchmark such as the Dow Industrial average does not mirror the
fate of the constituent firms nor their attendant stocks over the long range or even the medium
term. Rather, a yardstick of this kind merely reflects a relay race run by a succession of sprinters
in the prime of their lives. To adapt the words of a well-known bard, each member of the
ensemble is merely a shadow, a poor player that struts and frets its hour upon the stage and then
is heard no more.
As these examples show, a plethora of fairy tales hold sway in the real and financial marts. A
balanced review of the myths and mistakes lies beyond the scope of this guidebook. Even so, a
systematic survey of the passive flubs and active muffs in the field is available elsewhere
(MintKit, 2020).
16

Flaws of the Efficient Voodoo
In the modern era, the EMH has been identified as the culprit behind numerous ills in the
marketplace. A showcase cropped up during the financial crisis of 2008 along with the Great
Recession that ravaged the global economy. The cohort of the critics against the Efficient hoodoo
continues to swell. For instance, a former chairman of the U.S. central bank openly decried the
bunkum by stating, “It should be clear that among the causes of the recent financial crisis was an
unjustified faith in rational expectations, market efficiencies, and the techniques of modern
finance” (Volcker, 2011).
In addition to flouting the facts from the get-go, the EMH suffers from a slew of internal flaws in
logic. For instance, the catechism begins with the concession that any investor in particular is
imperfect and thus unable to make flawless decisions. Even so, the argument leaps to the
conclusion that a large number of humans can—and will—in the aggregate make faultless
decisions at all times.
Sad to say, but the quackery is defective from a conceptual stance as well as a pragmatic slant.
From a logical standpoint, a finite number of imperfect beings cannot produce perfect results. To
presume otherwise is to imagine that the imperfections will cancel each other out at all times on
every aspect of every market. Put another way, the latent premise behind the EMH is a flight of
fancy that breaches the bounds of logic.
Moreover, the polemic is even more groundless from an empirical stance. In fact, the opposite of
the dozy notion usually prevails in the real world. For instance, the mass of investors routinely
violate the strictures of pure logic and even the dictates of common sense. A blatant example
pops up in the frothy prices at the height of a bubble or the pulped assets in the depth of a panic.
And in between, the market has a custom of lurching back and forth, from a state of being
overpriced to undervalued and back again.
Amid the endless cycles of blowup and meltdown, a tenacious streak shows up in the custom of
the stock market for swerving in tune with the seasons of the year. We will christen this portrait
of the market as the Sway Model or the Lithe Theory.
To sum up, the Efficient Market Hypothesis undergirds the entire edifice of financial economics.
Yet, a modicum of data screened with dispassion is enough to refute the blarney of efficiency. In
the imperfect world we inhabit, perfect rationality is a sheer figment of the imagination.
17

3. Method
As we have seen, the Efficient Market Hypothesis denies the reality of the real and financial
markets in numerous ways. For this reason, the doctrine may be refuted through a variety of
means.
A basic approach centers on the discrepancy between the surmised nature versus the actual fiber
of the markets. A case in point is the failure of the price action to follow the Gaussian
distribution entailed by the Efficient mythos. In this light, a series of impartial studies since the
summer of the 20th
century has shown that the movements of the market belie the straitjacket of a
normal density function (Mandelbrot, 1963; Mantegna and Stanley, 1997).
On the downside, though, the landmark publications to date suffer from an excess of complexity.
More precisely, the disproofs of the EMH and its spinoffs rely on loads of wizardry ranging from
occult symbols and fancy equations to arcane procedures and baroque arguments. The byzantine
nature of the exposés has doubtless played a major role in blunting the impact of the revelations.
Not surprisingly, the Efficient dogma continues to prevail in academic cloisters and pragmatic
circles despite the abundance of evidence to the contrary.
The dearth of progress to date is a good reason to adopt a fresh approach to the subject. In this
spirit, the case study at hand pursues a plain and intuitive track based on the seasonal patterns in
the marketplace along with a lean set of elementary tools from the fields of statistics,
programming, and software.
From a practical slant, the S&P 500 Index serves as the leading beacon of the stock market for
professional actors, be they practitioners in the trenches or researchers on the sidelines. The
lodestar thus serves as the benchmark of choice for tracking the bourse.
Features of the Dataset
The multitude of actors in the marketplace confronts a wide range of time frames. For instance,
the short end of the spectrum applies to the day trader who buys and sells stocks within the span
of a few seconds or a handful of hours. Meanwhile the long end of the range pertains to the
standoff player who sets up an automated system to transfer funds from a bank account to a
mutual fund for the purchase of shares every month, but otherwise ignores the antics of the
market for years or decades on end.
18

Despite the diversity of timespans, however, myriads of players pay heed to the course of the
bourse from one month to the next. For this reason, sampling the market on a monthly basis is a
common practice in the financial patch.
An example involves a review of alternate strategies for investment over the past couple of
decades. A case in point is the average payoff relative to the risk entailed by the flutter of returns
during the holding period. In keeping with the fond choice of practitioners as well as researchers,
the current study takes up a sampling period of a single month.
A second and related facet concerns the timing of observations within the monthly cycle. For
instance, the readings could be made at the beginning of the timespan, the end of the window, or
some point in between.
Once again, the financial community subscribes to a standard practice. The seasoned players in
the ring view the price level at the end of each period—be it an hour or a day, a month or a year,
or any other timespan—to be the culmination of the action in the marketplace over the entire
interval. In other words, the closing price serves as a roundup of the price activity in more ways
than one.
To sum up, a wonted practice in the professional community is to regard the S&P Index as a
proxy for the stock market as a whole. A second habit is to monitor the performance of the
bourse on a monthly schedule. A third praxis is to treat the price level at the end of each period as
a representative value for the entire interval. In keeping with these customs, the current study
makes use of the closing prices of the S&P benchmark on a monthly basis.
Proper Metrics
In the financial tract as in other walks of life, a routine practice is to reckon the value of a widget
by way of an absolute figure. An example involves a singular stock priced at $9.28 per share, or
a compound index pegged at 387.45 points.
From the standpoint of an investor, however, the absolute level by itself has little or no meaning.
Instead, the raw price is a means rather than an end in the decisions they make and the actions
they take.
To bring up an example, consider a stock named Alpha which trades at $14.83 per share. Is the
latter price high or low, fair or foul?
To cite another instance, suppose that a market index named Omega rose by 10 points over the
past year. Was that a slugger or a slacker, a windfall or a letdown?
In each of these cases, there is no way to give a cogent answer based solely on the information at
hand. To determine whether Alpha is overpriced or not, for instance, we need to obtain more
19

facts. An example of the latter concerns the total value of the company along with the number of
shares outstanding, or the price level today compared to the unit cost a year ago.
In a similar way, an uprise of 10 points for Omega happens to be a respectable showing if its
value at the outset was 50 points—in which case the payoff amounted to 20% in a single year.
But the same turnup was paltry if the starting figure was 500 points thus yielding a capital gain
of just 2%.
In line with these vignettes, the ultimate yardstick of performance for an investor is the relative
payoff over time. An example lies in the fractional change in price notched by a singular stock in
a year, or the relative gain scored by a compound index from one month to the next.
As we saw earlier, the most popular benchmark among professionals—be they practitioners in
the trenches or researchers on the sidelines—lies in a muster of large-cap stocks maintained by a
service provider named Standard & Poor’s. The beacon is known as the S&P 500 Index and
often goes by the nickname of SPX. The stalwart thus serves as the touchstone for our case study.
Treatment of Null Returns
The financial forum finds itself in a continual state of flux as every asset swells and swoons over
time. Despite the gyrations along the way, however, the price level may wind up at the same
point from one period to the next—in which case the capital gain of course amounts to a big fat
zero.
There are several ways to handle a null return in building a bare-bones model that admits only a
pair of dichotomous outcomes, whether to the upside or downside. One approach is to ignore the
datum entirely. In that case, a value of zero is treated as an aberration and excluded from further
consideration. In the parlance of statistics, such an anomaly is branded as “Not Applicable” and
ignored in calculating the mean, median, or any other digest of the sample.
A second ploy available to the analyst is to assign a nil value to both of the distinct groups. For
instance, a reading of zero could be reclassified as half an instance of a positive return and half a
case of a negative gain.
A third method makes use of domain knowledge relating to the case study. As a backdrop, we
note that the purpose of venturing onto the bourse is to earn a profit by way of a capital gain
and/or dividend yield. On the other hand, many a stock pays little or nothing by way of cash
payouts. For this reason, the prospect of capital gains is in general the main reason for traipsing
into the ring.
To bring up a counterpoint, an investor whose main goal is the preservation of wealth in nominal
terms could put their money in a savings account at a commercial bank. In that case, the saver
would enjoy a steady stream of interest payments as well as get their principal back in due course
—all with much less heartburn and far greater assurance than holding a stake in the stock market.
20

An alternative ploy is to buy a bond and hold it till the date of maturity. In that case, a
bondholder in the credit market can be much more confident of retrieving the principal invested
at the outset in comparison to a shareowner in the equity market. Moreover, the dividend yield
on a bond is apt to exceed the interest rate on a savings account at a bank.
Despite these facts of life, however, the jive of efficiency brushes aside a vital aspect of the
bourse by claiming that the price level is wont to rise or fall with equal odds and to like extent. If
the expected return were in fact zero, why would any prudent soul accept the threat of a wipeout
in the stock market when they could enjoy a steady stream of earnings from a savings account or
a government bond, as well as regain the principal in due course, without incurring any risk to
speak of?
The blindness to domain knowledge is a fundamental way in which the EMH lacks verity and
lucidity, not to mention relevance and credence. To believe in the pretense of null returns is to
deny the very purpose of investing in the stock market in the first place. Moreover, the mummery
of a Random Walk gainsays the reality of the bourse which in fact has a way of trudging higher
over the long range.
To return to the main thrust of this section, we note that the investor who encounters a stagnant
market has failed to fulfill their aim of snagging a capital gain. In other words, a null change in
price is a setback for the investor. From a pragmatic stance, then, a flat turnout amounts to a
failure, much like a negative payoff. For this reason, only a positive gain counts as a success for
the serious investor.
To recap, the stock market has a way of bouncing around over all time scales, from less than a
day to more than a year. In spite of—and due to—the tireless rambling, a widespread practice in
the financial community is to track the movements on a monthly basis.
To bring up another custom, the S&P Index is reckoned to the accuracy of two decimal places. In
such a tally, the last decimal represents a small fraction of the spread of prices that usually arise
during a single day. In that case, the proportion is of course even smaller compared to the range
of prices over an entire month.
As the market turns and churns, the chance of a null change from one period to the next is
piddling. In our dataset, there is only a single case of repeat performance. The S&P benchmark
notched 109.32 points at the end of August 1979; after bouncing around for a while, it closed at
the same spot the following month.
In short, a sober investor ventures into the stock market and bears the risk of loss in the hope of
earning an ample profit. For this reason, a null return is a letdown for the risk-taker. As a result,
we will regard a turnout of zero percent as a failure rather than an ignorable case, and certainly
not as a success.
21

From a pragmatic standpoint, however, any reasonable approach to handling a null return should
be acceptable. Given the rareness of a flat spell in the market, the special case is unlikely to have
a material impact on the overall results regardless of the particular mode of treatment.
Choice of Statistical Test
The plainest form of hypothesis testing lies in the binomial method. The technique imposes no
constraints on the nature of the process under investigation, other than the fixed odds of
observing a pair of disjoint outcomes.
Put another way, the assumptions behind the exact binomial test are minimal. First of all, each
trial or incident—also known as an experiment—gives rise to one of two outcomes. Secondly,
the trials are mutually independent. Thirdly, the probability of each outcome is the same for the
entire sample. In other words, the only source of variation is the innate randomness of a binary
process wherein the odds of the polar events remain constant.
If we look at the big picture, the binomial test is conservative in the following sense: the
technique is less likely than other methods to detect a departure from the null hypothesis even
when such an aberrance does in fact exist. Ironically, the reason has to do with the liberal nature
of the binomial checkup.
To begin with, we note that a system that yields only a single fixed outcome amounts to a certain
process rather than a chancy one. From the converse stance, a random process of any kind has to
display at least two distinct states.
The other assumptions behind the binomial procedure are also sparing and lenient; namely, the
independence of trials coupled with fixed odds of observing one outcome versus its complement.
In this way, the binomial routine adopts an expansive view that admits a plethora of flighty
systems.
In short, just about anything goes. By the same token, almost nothing is exceptional. Due to the
broad embrace, only extreme oddballs fall outside the scope of admissible candidates.
For this reason, the binomial test is often unable to detect the gonzos even when they can be
spotted by other techniques. Yet this very shortcoming turns into a rare virtue for the case study
at hand. If a binomial assay—in spite of its accommodative nature—happens to reject the null
hypothesis, then we may accept the unusual verdict with a high degree of surety.
In the current context, the random process involves the movements of the stock market from one
month to the next. If the return at a given stage is positive, the outcome is treated as a success;
and otherwise as a failure.
A binary model of this kind readily serves as a testbed for the EMH along with its spinouts in the
form of the Idle and Drift Models. The setup is similar for the Sway Model which presumes that
22

the odds of overshooting or underrunning the average return remain fixed within each portion of
the annual cycle; that is, the outcome depends only on the pertinent month and is independent of
the other months of the year.
To sum up, the binomial routine comes with a bevy of attractions. For starters, the technique is
an exact method rather than an approximate one. That is, the criteria for vetting the null
hypothesis make use of precise values rather than rough estimates.
Moreover, the binomial test is so elemental that it imposes scarcely any fetters on the underlying
process. Rather, the only restraint involves a set of independent trials, each of which results in
one of two distinct outcomes having fixed odds of occurrence.
As a consequence, no parameter has to be presumed in advance nor inferred from the data other
than the very statistic being tested; namely, the proportion of successes, or equivalently the ratio
of odds for a couple of opposing outcomes. The upshot is a nonparametric technique that can
handle any reasonable form of distribution function lying behind the binary process.
In particular, the binomial test does not require the observations to spring from a Gaussian
distribution nor any other prefixed function. For a variety of reasons, then, the procedure is
highly robust as well as trustworthy.
Pick of Software Environment
The R language is a dandy medium for crunching data and culling statistics. To begin with, the
programming lingo is backed by an open-source platform that offers a wealth of functionality.
The software kernel–also known as Base R–may be bolstered by installing one or more
extensions, each of which is called a package. For this purpose, an online resource called the
Comprehensive R Archive Network stockpiles a myriad of plug-ins; at last count, the number of
boosters surpassed 15,000 (CRAN, 2020). The add-on modules range from routines for
conducting advanced statistical tests to utilities for streamlining the analysis of time series data.
In keeping with the bare-bones approach, however, the current study refrains from using such
turbocharged tools and relies only on a scanty subset of the basal functions built into the core
platform. The entirety of data formatting, graphic output, and statistical grilling is effected
through a pithy program comprising several dozen lines of code in the R language, along with a
sprinkling of explanatory remarks which are provided solely for the convenience of the human
reader rather than the steerage of the computer system.
23

4. Fetch and Glom of the Data
You may use a Web browser to obtain a historical record of the S&P Index at no cost from Yahoo
Finance, the most popular site amongst the investing public (Yahoo, 2019). On a curious note,
though, the portal has a quirky custom of denoting the renowned benchmark by the handle of
^GSPC.
A table of monthly readings, available as a text file containing comma-separated values (CSV),
may be cloned and placed in a directory called Downloads on your local computer. The brand-
new file is automatically given the full name of ^GSPC.csv. In displaying the document,
however, your computer is likely to hide the file extension (.csv) and instead list the item more
tersely as ^GSPC.
For the sake of good form, we will make a couple of logistic tweaks by renaming the file and
moving it to a different directory. Our first step is to change the name of the document from
^GSPC to SpxMonth1950-2019.
At this juncture, we could access the renamed file via the R system after figuring out its full
address on the local computer. On the other hand, a simpler approach is to transfer the file
manually from the Downloads folder into the current working directory. To this end, you may
ascertain the latter directory by using the following command:
> getwd()
In the foregoing line, the greater-than sign (>) is a prompt from the system showing that it is
awaiting input from the user (that’s you). For this reason, you should type only the command
itself; that is, getwd().
The system responds by printing the location of the current directory. If you wish, you may
change the working directory. For instance, suppose that you are using a Windows machine and
have created a folder called Testbed which sits on your Desktop. Then you could set your
working directory to that folder by a command such as the following.
> setwd("../Desktop/Testbed")
In this decree, the double dot (..) denotes the directory above the current one. The rest of the
path description refers to the Testbed folder which resides on the Desktop belonging to the
current user.
24

We now put aside the R system for a moment. Instead, we turn to the operating system and
manually move the file into the chosen directory.
After transferring the file, we return to the R platform. The next task is to copy the contents of
the external file into an internal object within the R system. The newborn structure is a hybrid
table that contains numeric values as well as character strings. The compound object is called a
data frame.
We will christen the resulting table as Raw. The latter label features an initial capital letter to
serve as a reminder that the object is a full-fledged table rather than a simpler structure such as a
character string or a numeric vector.
On occasion, an exception to the naming convention may arise in handling data from external
sources. For instance, we shall soon see that the vector of closing prices has been tagged by the
data provider as Close. We will leave the latter label in its original form—as a character string
in camel case—rather than convert it into the lower case mode that we would otherwise use in
creating a vector from scratch.
Without further ado, we move on to the intake of data from the external file. The following
command poses a simple way to load the material into the current session in R.
> Raw = read.csv("SpxMonth1950-2019.csv")
To confirm that the data has been copied properly, we may check the first handful of rows by
using the head command.
> head(Raw)
Date Open High Low Close Adj.Close Volume
1 1950-01-01 16.66 17.09 16.65 17.05 17.05 42570000
2 1950-02-01 17.05 17.35 16.99 17.22 17.22 33430000
3 1950-03-01 17.24 17.61 17.07 17.29 17.29 40410000
4 1950-04-01 17.34 18.07 17.34 18.07 18.07 48250000
5 1950-05-01 18.22 18.78 18.11 18.78 18.78 45080000
6 1950-06-01 18.77 19.40 17.44 17.69 17.69 45660000
In the resulting output, the first column displays no name or label at the top of the stack. Even so,
each entry within the column denotes the position of the corresponding record in the table.
Meanwhile, the second column contains the Date; that is, the first day of each month starting
with the onset of January 1950.
The next four columns convey the Open, High, Low and Close values of the S&P Index
during the corresponding interval. We note that the High, Low and Close prices refer to the
extreme readings for the entire month rather than merely the first day of the period.
25

The subsequent column displays the Adjusted Close. The latter attribute is useful for
tracking a widget whose status may change due to logistic or administrative events. An example
involves a payout of dividends by an index fund, or the split of a stock whereby each investor
receives an additional share for every unit they already own.
On the other hand, the adjusted value makes no difference in our case since the S&P Index itself
incorporates the impact of stock splits, and tracks only the capital gains while ignoring the
dividend payouts. In other words, the Adjusted Close is always identical to the nominal
value of the Close in the current context.
At this juncture, we should check the entries for the last half-dozen records of the Raw array.
> tail(Raw)
Date Open High Low Close Adj.Close Volume
824 2018-08-01 2821.17 2916.50 2796.34 2901.52 2901.52 69238220000
825 2018-09-01 2896.96 2940.91 2864.12 2913.98 2913.98 62492080000
826 2018-10-01 2926.29 2939.86 2603.54 2711.74 2711.74 91327930000
827 2018-11-01 2717.58 2815.15 2631.09 2760.17 2760.17 80080110000
828 2018-12-01 2790.50 2800.18 2346.58 2506.85 2506.85 83519570000
829 2019-01-01 2476.96 2708.95 2443.96 2704.10 2704.10 80391630000
From the printout, we may infer that the Raw table has in fact been replicated properly.
Our goal is to highlight the dance of the bourse as the seasons go by. To this end, we want to
uncover the average flux of the benchmark during each month of the year. In tracking the market,
we could focus on one or more aspects of the price action. An example lies in the price level at
the beginning of each month. A second approach is to peg the mean value of the high and low
prices every month. In these and other ways, a variety of choices are available.
Even so, a standard practice in the financial community is to sample the price at the end of each
period. We will abide by this custom and monitor the closing values on a monthly basis.
As a warm-up task, we may plot the readings to get a visceral sense of the data.
> plot(Raw$Close, type='l', col='magenta', xlab='Month', ylab='SPX')
In the foregoing command, the first argument within the parentheses denotes the Close vector
of the Raw array as the source of input. Meanwhile, the second modifier prescribes the type of
marker on the graphic output as a line connecting the data points.
The third clause sets the color of the line to magenta. Finally, the last two arguments specify the
labels along the x and y axes as ‘Month’ and ‘SPX’ respectively.
The resulting object is shown in Figure 4-1. We can see that the price level grows over time
despite a series of sizable upswings and downstrokes along the way.
26

Fig. 4-1. Monthly level of S&P Index from 1950 to 2019.
On the whole, the S&P Index appears to grow at an exponential rate over the entire stretch. That
would be a reasonable turnout since the benchmark comprises a weighted average of the stock
prices for a rolling roster of the biggest and stoutest firms on the bourse as they come to the fore
then duly fade away.
Moreover, every share of stock represents a slice of ownership in the current assets and future
profits of the underlying company. At a minimum, then, the equities ought to keep pace with the
march of inflation which tends to advance by a couple of percent or more per year.
Before we proceed further, however, we will pause to discuss a couple of logistic issues. The first
item concerns the clarity of the graphics produced by the R system. All too often, an image may
display jagged lines or fuzzy characters. An example involves a curve or a diagonal that looks
ragged and disjointed rather than smooth and continuous.
To circumvent this type of snag, we can save the diagram in a scalable mode such as the
Portable Document Format (PDF). This may be done by using the “Save as” function under the
tab named “File” in the basic version of the software platform, otherwise known as Base R.
On the other hand, many a user prefers to interact with the basic engine in an indirect way. In
that case, a widespread choice lies in a friendly interface called RStudio.
27

To save a graphic in a scalable format, a user of the RStudio package may click on the Plots tab
of the pane dealing with documentation. The next step is to select the Export menu followed by
“Save as Image...”, then pick SVG (Scalable Vector Graphics) as the “Image format”. Once the
file is saved, it may be opened by a suitable app such as a drawing program then reworked into a
different format as desired.
The second item of practical concern is a tad more substantive. In the plot command we used
earlier, the vector of Close prices was denoted as Raw$Close. To shorten the names of such
variables, we may issue the following diktat.
> attach(Raw)
Thanks to the foregoing decree, we can henceforth refer to the vector of closing prices by the
shorthand of Close rather than the mouthful of Raw$Close. The same type of shortcut applies
to the other columns of the Raw array.
Our next trick is to confirm the rising trend over the long haul by plotting the logarithm of the
original values.
> plot(log10(Close),type='l',col='magenta',xlab='Month',ylab='Log of SPX')
Based on the foregoing command, the log to base 10 of the Close values shows up in the
vertical direction while time advances along the horizontal axis.
The output in the form of Fig. 4-2 confirms that the overall trend is more or less steady. We thus
infer that the original variable displays an exponential streak. Since the SPX has a way of
swelling over time, we ought to extract the conspicuous trend before trying to discern any subtler
patterns.
To bring up a different but related issue, we noted earlier that the price level happens to be a
means rather than an end for the mindful investor. More precisely, the main item of interest to the
decision maker is the relative gain over time.
For this reason, we want to focus on the fractional changes in price from one month to the next.
As a bonus, sizing up the return on investment in a stepwise fashion has the helpful impact of
incorporating the uptrend that prevails over the long range.
Following the qualitative scan of the data in this chapter, we may proceed to a quantitative probe.
For this purpose, the next step is to reckon the fractional changes in price from one month to the
next over the entire timeline.
28

Fig. 4-2. Log to base 10 of SPX by month over the entire timespan.
29

5. Gauging the Monthly Returns
As a prelude to grasping the seasonal patterns in the stock market, we ought to perform a couple
of tasks. A helpful move is to recast the data into a handy format then segue to a visual scan of
the relative shifts in price.
Formatting the Input
A key step is to pin down the average movement of the market by month throughout the year. To
this end, we will rework the original dataset so that each row carries the date along with the
closing prices for the current month as well as the subsequent spell.
Before recasting the data, however, we first pin down the number of rows in the Raw dataset. To
this end, we find the length of the Close vector then store the result in a scalar variable called
len.
> len = length(Close)
In the customized array we desire, each row includes the Close value for the current month as
well as the next period as long as the data is available. For this reason, the brand-new table will
be slightly shorter than the original; to be precise, the number n of records will be 1 less than
len.
> n = len - 1
From a pragmatic slant, n is the number of usable cases for building and testing the candidate
models.
Our next step is to convert the raw input into a format better suited to our needs. We will regard
the trimmed dataset as the adjunct table and call it the A array.
> A = cbind(Date[1:n], Close[1:n], Close[2:len])
In the foregoing statement, the second argument in the cbind command denotes the entirety of
the Close vector except for the last item. Meanwhile, the third modifier covers every datum in
Close starting with the second entry; that is, the whole shebang except for the first item.
30

We may verify the result by printing the first 3 rows of table A as follows.
> head(A,3)
[,1] [,2] [,3]
[1,] 1 17.05 17.22
[2,] 2 17.22 17.29
[3,] 3 17.29 18.07
By the same token, we should check the last trio of records in the adjunct array:
> tail(A,3)
[,1] [,2] [,3]
[826,] 826 2711.74 2760.17
[827,] 827 2760.17 2506.85
[828,] 828 2506.85 2704.10
The cbind command has produced a table. Since the latter array contains only items of a single
type—in this case, the numeric form—it is better known as a matrix in the parlance of R.
In constructing the matrix, each character string in Date has been converted into a unique
integer. Since every entry in Date differs from its peers, the resulting integers happen to
correspond to the rows of the newborn array. In this way, the first column of the table contains
plain numbers rather than multipart dates.
To expand on this theme, we recall that the columns within the Raw array are numeric vectors
except for the first column called Date. The latter vector contains a series of alphanumeric
entries which are interpreted as the categories of a factor variable.
On the other hand, the cbind() function takes a set of vectors or matrices then creates a matrix
that contains entries of a single kind. An example of the latter is a numeric format or a character
type. In line with this policy, the Date vector—which is a factor variable in the Raw table—has
been coerced into a numeric form whereby each entry happens to denote its position within the
first column; that is, the row number.
We may now massage the data further to fit our needs. In the first phase, we want to tease out the
monthly returns from the pile of noisy and chaotic readings.
Monthly Gains Over Time
To profile the seasonal patterns in the stock market, we may determine the rate of change on a
monthly basis. At each step, we calculate the relative gain—also called the forward return—over
the next period. The fractional change is then converted into a percentage rate. Next, the chain of
results is stored in a vector called ret:
31

> ret = 100 * ( A[ , 3] - A[ , 2] ) / A[ , 2]
In the foregoing line, the expression in parentheses pegs the change in price from one month to
the next based on the entries within each record in the A array. The difference in values is divided
by the current price at each time slot; that is, the entry in the second column of A.
The resulting quotient is multiplied by 100 to express the relative change as a percentage of the
current value at each step. In this way, the ret vector represents the sequence of monthly
returns.
Next, we plot the entries in ret to obtain a graphic view of the dispersion of values.
> plot(ret, type='l', col='magenta', xlab='Month', ylab='Return (%)')
The output is shown in Fig. 5-1. We see that the returns bounce around a great deal, especially
after the first hundred cases or so. Despite the jumpiness, though, there appears to be no material
change in the scattering pattern across the entire stretch.
Fig. 5-1. Forward monthly return for SPX from 1950 to 2018.
32

The ret vector may serve as the mainmast of a new array containing a few crucial vectors. The
burnished table, which we shall call the B board, will serve as a baseline or backbone for further
analysis.
> B = data.frame(Date[1:n], Close[1:n], ret)
The foregoing statement creates a table in the form of a data frame. Unlike a matrix, a data frame
may contain more than one type of object. In this case, the Date vector keeps its original format
as a series of alphanumeric values which are treated by the system as the elements of a factor
variable.
To verify the outcome, we print the first 3 rows of B as follows.
> head(B,3)
Date.1.n. Close.1.n. ret
1 1950-01-01 17.05 0.9970675
2 1950-02-01 17.22 0.4065157
3 1950-03-01 17.29 4.5112722
In the foregoing display, we see that the system has renamed each of the first two columns by
attaching “.1.n.” as a suffix. The postfix serves as a reminder that the entries in the respective
columns run from rows 1 through n, rather than the entire range covered by the source vectors—
namely, Date and Close—residing within the Raw table.
We may check the last triple of rows in the B board as follows.
> tail(B, 3)
Date.1.n. Close.1.n. ret
826 2018-10-01 2711.74 1.785936
827 2018-11-01 2760.17 -9.177689
828 2018-12-01 2506.85 7.868440
As an example, the last line above displays the record for December 2018. The first number
identifies the listing as the 828th
record of the dataset. Meanwhile the second item denotes the
starting date of the month in question while the third factoid denotes the value of the S&P Index
at the end of the given period. Finally, the last item shows that the return from the Close of
December 2018 to the end of the subsequent month was 7.868440 percent.
The latter value takes account of the price levels at the end of December as well as the close of
the following month; namely, January 2019. For this reason, it would be wrong to regard the
original dataset—along with the Raw table—as a muster of readings across a window spanning
precisely 69 years. On the other hand, it would be correct to say that the working dataset covers
exactly 69 years’ worth of monthly changes in price starting from the onset of 1950.
At this juncture, the raw data has been molded into a convenient format. The resulting table sets
the stage for molding precise models of price activity.
33

6. Crafting the Models
The purpose of this chapter is to construct a trio of pricing models. Two of the templates
represent the Efficient dogma while the third portrays the seasonal cycle of monthly returns
throughout the year.
Encoding the Efficient Models
The first task is to convert the woozy visions of the Efficient doctrine into a couple of concrete
models. We begin with the Idle canard which presumes that the market rises or falls with equal
likelihood and to similar extent. In that case, the expected move to the upside is identical to that
for the downside. As a consequence, the best forecast of monthly returns remains fixed at zero
percent at all times.
On the other hand, the setup differs somewhat for the Drift motif. The latter does rely in a small
way on the actual mien of the market; namely, the uptrend over the long range. We may obtain
the average rate of ascent, prorated on a monthly basis, in the following way.
> wax = A[n,3] / A[1,2]
> gm.wax = wax^(1/n)
> gm.r = 100 * (gm.wax - 1)
> gm.r
[1] 0.6137568
In the first statement above, the price at the end of the timespan is divided by its value at the
outset. The quotient, which represents the magnification factor over the entire window, is stored
in a variable called wax.
In the next command, the waxing factor is raised to the inverse of the number of time slots in
order to obtain the geometric mean of the magnification factor. The result goes into a scalar
variable called gm.wax.
The third dictum transforms the monthly multiplier into an incremental return. To this end, we
first reduce gm.wax by 1 in order to convert the scaling factor into the fractional shift from one
period to the next. The increment is then multiplied by 100 to express the proportional move as a
percentage value.
34

Finally, the fourth command above calls for a printout of the final result. Based on the output, the
geometric mean amounts to 0.6137568 percent per month.
The latter value forms the crux of the Drift template. On the bright side, this framework is more
realistic than the Idle version. On the dim side, though, both of the Efficient mockups still leave
much to be desired.
Building the Sway Model
Our next task is to divine the seasonal patterns in the stock market. For this purpose, we will
discern the average return by month throughout the year.
To this end, we take an elementary approach rather than invoke any fancy routines in R for
handling time series data. In keeping with this maxim, an apt move is to arrange the list of
returns into a tabular format.
More precisely, we want to reshape the vector of monthly gains into a matrix comprising 12
columns. In the resulting table, each row represents a given year and each column a specific
month. The cured table, which we dub as the C cache, will serve as the input for culling the
monthly gains on average.
> C = matrix(ret, ncol=12, byrow=TRUE)
In the foregoing statement, the second argument within the parentheses sets the number of
columns to 12. Meanwhile, the last modifier directs the system to fill up the matrix by row rather
than by column while cloning the elements of the ret vector. That is, the entries are to be
assigned in sequence to the slots within the first row; followed by the second row; and so on.
To verify the result, we start by printing the first few rows.
> head(C, 3)
[,1] [,2] [,3] [,4] [,5] [,6]
[1,] 0.9970675 0.4065157 4.511272 3.929170 -5.804047 0.847931
[2,] 0.6463481 -1.8348579 4.813084 -4.057066 -2.602235 6.870234
[3,] -3.6453978 4.7721453 -4.308580 2.315613 4.610218 1.762825
[,7] [,8] [,9] [,10] [,11] [,12]
[1,] 3.251121 5.59175353 0.41131103 -0.1024117 4.613019 6.124449
[2,] 3.928576 -0.08591494 -1.37574807 -0.2615606 3.889865 1.556580
[3,] -1.456689 -1.95765074 -0.08150366 4.6492659 3.546376 -0.715096
We now do likewise for the tail.
> tail(C, 3)
[,1] [,2] [,3] [,4] [,5] [,6]
[67,] -0.412836 6.59911146 0.2699398 1.532460 0.09109211 3.560980
[68,] 3.719816 -0.03891972 0.9091209 1.157625 0.48137751 1.934883
[69,] -3.894737 -2.68844986 0.2718775 2.160834 0.48424360 3.602156
35

[,7] [,8] [,9] [,10] [,11] [,12]
[67,] -0.12192431 -0.1234451 -1.942568 3.4174522 1.820076 1.788436
[68,] 0.05464328 1.9302979 2.218814 0.3720043 3.434256 5.617870
[69,] 3.02632115 0.4294287 -6.940336 1.7859357 -9.177689 7.868440
The data structure appears to be in order. We thus proceed to the next step.
Average Return by Month
Having formed the C matrix, it’s a simple matter to compute the mean value of each column. We
will refer to the resulting rundown of average zigzags in price as the zig vector.
> zig = colMeans(C)
We can plot the mean return by month in the following way.
> plot(1:12,zig,type='b',col='blue',xlab='Month',ylab ='Return (%)')
> abline(h = 0, col='red')
The first statement above creates a chart in which the horizontal axis displays the numbers from
1 to 12 to denote the respective months of the year. Meanwhile the values of the zig vector
appear in the vertical direction.
The third argument within the parentheses specifies the type of display as both points and lines.
The latter items should be colored in blue. Moreover, the x and y axes are labeled as ‘Month’ and
‘Return (%)’ respectively.
Furthermore, the second command in the foregoing interaction draws a straight line across the
current chart. The addendum is a horizontal stripe passing through the origin, to be rendered in
red.
The result is shown in Fig. 6-1. The graphic shows that, more often than not, the mean return had
a positive bent. On the other hand, the market hardly moved on average during January, May and
July. On a downcast note, the worst performance cropped up in August.
For a richer view of the dataset, we may generate a boxplot of the forward returns by month.
> bp = boxplot(C, col='yellow', xlab='Month', ylab ='Return (%)')
In the foregoing statement, the second argument sets the internal color of the boxplots to
yellow. The ensuing graphic is shown as Fig. 6-2.
36

Fig. 6-1. Mean return for each month of the year.
We should keep in mind that the thick bar within each rectangle represents the median level
rather than the mean value for the corresponding month. From a different slant, the thumbnail for
each month reflects a great deal of dispersion compared to the jitter of median values over the
year.
Outliers in a Boxplot
As we have seen, the trunk of a boxplot consists of a rectangle. In addition, a vertical rod—also
known as a fence or whisker—extends outward from the lower edge of the box; and likewise for
the upper end.
In many an application, the whiskers reflect the minimum and maximum values of the data
points. On occasion, though, extreme readings are plotted above or below the whiskers; such
anomalies show up as singular points detached from the rest of the boxplot.
37

Fig. 6-2. Boxplot of forward monthly returns in percent.
To expand on this topic, we note that the interquartile range (IQR) refers to the distance between
the first and third quartiles. From a graphic slant, the IQR equals the height of the rectangle that
forms the body of the boxplot.
Meanwhile, an extreme reading that lies more than one and a half times the IQR from either end
of the box is deemed to be an outlier. That is, a datum is treated as an oddity if it lies more than
1.5 times the IQR above the third quartile or below the first quartile.
In line with this convention, each of the whiskers spans up to 1.5 times the IQR. More precisely,
the low end of the lower wick extends to the smallest value lying above the oddballs placed
below the low side of the rectangle. By the same token, the upper rod extends to the biggest entry
sited below the jumbos hovering above the high end.
Most of the thumbnails in Fig. 6-2 display one or more outliers as shown by the lonesome circles
lying beyond the whiskers. The only exceptions are the sixth and twelfth months—namely, June
and December—which lack any extreme values.
As we noted earlier, the dispersion of values within each month is sizable compared to the
variation in median levels across the groups. Despite the broad scatter of readings along with the
attendant clutter, however, some differences in the average values are apparent.
38

For instance, the median return for October appears to surpass those of September and
November. Meanwhile, an example on the flip side applies to August whose average gain falls
below those of July and September. From a larger stance, the relative placement of the median
values is largely consistent with the configuration of mean returns that we saw in Fig. 6-1.
Products of the Boxplot Command
In response to the boxplot command we issued earlier, the R engine created a couple of
objects; namely, a graphic summary and a numeric structure. The pictorial component appears in
the foregoing chart.
Meanwhile, the numeric construct is a multifaceted object comprising a handful of modules,
each of which may in turn contain multiplex parts. We can get some idea of the contents by
typing bp as follows.
> bp
$`stats`
[,1] [,2] [,3] [,4] [,5] [,6]
[1,] -6.0548180 -6.420471 -6.1969875 -6.572964 -7.24553544 -7.900427
[2,] -2.1846176 -1.016711 -0.7497453 -1.888384 -1.82574613 -1.622218
[3,] 0.3548922 1.109581 1.1530644 1.067418 0.01958265 1.139964
[4,] 1.8995148 3.309475 4.0014318 2.315613 2.12785442 3.937368
[5,] 7.1489277 9.671983 9.3925080 5.857688 5.44383310 8.837026
....
For the sake of brevity, the foregoing display shows only the initial portion of the longish output
provided by the system. More precisely, the printout above represents the first half of a module
called stats within the bp object.
In the foregoing layout, each column of decimals defines the boxplot associated with the
pertinent vector in the C array. For instance, the last entry in the first column above states that the
upper whisker for the 1st
boxplot (corresponding to January) lies at 7.1489277 percent.
Moreover, the second element of the first column places the first quartile for January at
−2.1846176%. The next two numbers show that the median value for the month is 0.3548922%
while the third quartile lies at 1.8995148%.
We note that the extreme values in each column define the whiskers—namely, the lower and
upper bounds of the connected segments of the boxplot—rather than the minimum and
maximum readings within the entire cluster. For instance, the very first entry in the foregoing
printout is −6.0548180. The latter number denotes merely the bottom of the lower whisker rather
than the minimal value for January.
39

On one hand, the first month of the year contains no outliers at the high end. Even so, the same
month features a couple of extreme values on the low side; namely, −10.9931198% and
−9.22907358% (which do not appear in the stats module). The latter two values correspond to
the pair of circles lying below the lower whisker of the leftmost thumbnail shown in Fig. 6-2.
In the R statement above, we were able to invoke the object called bp because we prescribed that
very name in issuing the boxplot command a couple of sections earlier. The prior decree now
enables us to access the information contained within the bp object. We will make good use of
the lineup of median returns in the next section.
Mean versus Median Gains
At this juncture, we would like to determine whether the mean or the median statistic provides a
better fit between the model and the data. To this end, our first step is to extract the row of
median values and store it in a vector called zag.
> zag = bp$stats[3, ]
As we saw earlier, the object named bp includes a matrix called stats. The third row of the
stats table contains the median values of the dozen columns in the C table. This sequence of
numbers has been placed into the zag vector by the foregoing command.
The next step is to determine the absolute values of the errors due to the zig and zag arrays.
> aeZig = abs(ret - zig)
> aeZag = abs(ret - zag)
We now reckon the mean value of the absolute residuals for each of the zig and zag vectors.
> mean(aeZig); mean(aeZag)
[1] 3.089324
[1] 3.074057
According to the foregoing printout, the mean level of absolute miscues for the zig array is
slightly larger than that for zag.
Next up is the median amplitude of the misfits for each vector of average returns.
> median(aeZig); median(aeZag)
[1] 2.4755
[1] 2.491434
Here we find that the median value of the absolute flubs for the zig vector is slightly less than
that for zag.
40

To sum up, the zag vector outperforms the zig array by a small amount in terms of the mean
level of absolute residuals. But the converse relationship holds in terms of the median values.
Based on the mixed results, there is scant reason to choose one metric over the other.
In the larger scheme of things, however, we know that the median value is more robust than the
mean level. The same is true for an entire suite of median tallies as opposed to a comparable set
of mean readings. As a result, we will adopt the zag vector rather than its zig rival as the
backbone of the seasonal model.
In short, this section has examined a couple of ways to capsule the movements of the market;
namely, the mean versus median returns by month throughout the year. The purpose of the
workout was to determine whether the slate of mean values or median levels would better serve
as the mainstay of the seasonal template.
We found mixed results between the zig and zag vectors in terms of the average size of the
residuals. Moreover, the mean values of the miscues did not differ much between the two forms
of error vectors; and likewise when the median yardstick served as the measure of performance.
Even so, we opted as a matter of principle to adopt the chain of median values as the truss of the
Sway Model.
The next chapter takes a systematic approach to screening the errors due to the Sway framework
as well as its Efficient counterparts. The litter of residuals is an apt way to assess the candidate
models as viable portraits of the stock market.
41

7. Profile of Errors
By this stage, the raw dataset has been arranged into a couple of handy tables. We may now use
the recast data to glean patterns and gain insights, build models and test outcomes.
According to the EMH, the market is so slick and swift that it quaffs every drop of dope in a jiff.
Moreover, the current valuation of each asset subsumes any scent of future events. Ergo, the
price level cannot be foretold except perhaps by happenchance.
One spinout of the EMH is the equality of odds for the market to rise or fall from one period to
the next. Granted, the very nature of a dicey system means that the number of upturns is unlikely
to precisely match that of downthrows during any given interval. Even so, the disparity should
not be so flagrant that the premise of equality may be squashed at an ample level of statistical
significance.
In an irony of sorts, the verity of a model may be judged by the randomness of the errors that
ensue. As a counterpoint, a palpable pattern within the residuals implies that the model has failed
to capture one or more salient aspects of the target system.
With this truism in mind, we will delve into the errors borne by each model. The next step
involves an informal scan of the slip-ups through a graphic display. After the visual survey, we
proceed to a quantitative assay based on the binomial routine.
Chain of Flubs
As we saw earlier, the Idle Model presumes that the expected value of the price level going
forward is the same as the current state. Put another way, the forecast at any stage involves a null
change in price. For this reason, the actual return over the subsequent month amounts to the
mantic error for that stage.
In our software model thus far, the ret vector contains the chain of returns which also happens
to represent the string of miscues for the Idle template. When the ret array is viewed in this
fashion, we will refer to it as the eI vector.
> eI = ret
42

The eI array thus contains the slate of errors for the Idle Model.
Next up is the Drift framework. This mockup pictures the market in the same way as the Idle idol
except for a steady uptrend in price. For this reason, the ensuing error at each stage is the
difference between the actual return and the geometric mean that we computed earlier. We will
denote the corresponding vector of errors as eD.
> eD = ret - gm.r
In the foregoing statement, the ret vector carries the monthly returns for the entire sample of
828 cases. On the other hand, the variable named gm.r holds only a single element.
To cope with the mismatch of lengths, the software engine first subtracts the scalar value in
gm.r from each element within the ret vector. The resulting strand of numbers is then stored in
an array of commensurate length in the form of the eD vector.
To move on, we recall that the Sway Model captures the flitter of the market over the year. In
that case, the tacit forecast at any stage is the mean change in value for the corresponding month.
Moreover, the gap between the actual return and the foretold value represents the goof for that
period. We will refer to the resulting vector of errors as eS.
> eS = ret - zag
As with the previous command, the expression on the right side of the equation contains objects
of unequal lengths. To be precise, the ret vector contains 828 items while the zig array holds
only a dozen entries that peg the monthly gains on average throughout the year.
Despite the mismatch of lengths, the system acts deftly by recycling the values of the zig vector
as needed in order to cover the entirety of the ret column. The results are then saved in a new
vector called eS.
Having formed the train of errors for each model, we may now peruse the bloopers. The first task
involves an informal scan by way of a graphic display.
Boxplots of Residuals
To obtain a compact portrait of the data, we will generate a boxplot for each of the error vectors.
In the following command, an argument called names assigns a triplex of labels to the boxplots
for eI, eD and eS respectively. Meanwhile a modifier named col plays a similar role for the
colors to use within the bodies of the thumbnails.
> boxplot(eI, eD, eS, names=c('Idle','Drift','Sway'),
+ col=c('orange','yellow','green'))
43

The plus sign (+) at the beginning of the second line above is displayed by the R system as a
signal that it is awaiting further input from the user during an interactive session via the Console
pane. For this reason, the plus symbol should be omitted when the program is run in batch mode;
for instance when the user selects a swatch of code within the Editor pane then directs the system
to run the entire block in one fell swoop.
The output from the foregoing command is shown in Fig. 7-1. The exhibit makes clear that the
errors on average do not differ much across the three models.
Fig. 7-1. Boxplots of errors by model.
On the other hand, each boxplot displays a great deal of dispersion including large values in the
upward and downward headings. It’s easy to see why a cursory scan of such an image would lead
a casual observer to believe that there is no meaningful difference among the error profiles.
Despite the merits of a graphic layout, however, a visual survey of the data can be misleading;
hence the need for a rigorous assay. For starters, we may compare the error vectors by way of
quantitative summaries.
44

Variance of Errors
To get a quick sense of the fitness of each model, we may assess the dispersion of errors. For this
purpose, an obvious tack is to compute the variance of the misfits.
In plumbing the failings of the Efficient doctrine, it makes scant difference whether the Idle
Model or the Drift version is chosen as the archetype. The only distinction between the two icons
is a fixed level of monthly returns. More precisely, the implicit forecast due to the Idle icon at
each stage is precisely zero while that for the Drift template is a static rate which happens to be
slightly positive.
Either way, the average scatter of predictive errors around their central values will be identical
for the pair of Efficient models. In particular, the variance—along with the standard deviation—
will be the same whether the Idle makeshift or the Drift variant is employed.
Regardless of the kinship of errors between the Efficient spoofs, we may obtain the variance of
the residuals for all three models with a few keystrokes.
> var(eI) ; var(eD) ; var(eS)
[1] 16.94586
[1] 16.94586
[1] 16.58341
From the printout above, we see that the first two numbers are equal as to be expected. Even so,
it’s always reassuring to find that the derived data along with the software engine happen to
produce consistent results and thus appear to contain no glitches.
On the bright side, the variance for the Sway template is slightly less than that for the Idle and
Drift kludges. The difference is modest since the small edge in accuracy due to the seasonal
framework is largely drowned out by the noise and stew of the stock market.
Average Size of Flubs
A scrutiny of the errors may take a variety of forms. At first blush, a straightforward tack is to
take a set of residuals and compute their average value. Unfortunately, the obvious scheme can
easily lead to misleading results.
As an example, consider a heap of error measurements that contain lots of large readings to the
upside as well as downside. In that case, the average value of the entire stack could turn out to be
small or even zero due to the cancellation of positive and negative readings despite the slew of
whoppers in both directions. The runted value of the average error would imply that the model is
highly accurate—which is certainly not the case here.
To avoid the sinkhole of cancellation, we may first obtain the absolute values of the lapses.
45

> aeI = abs(eI)
In the foregoing command, we take the absolute value of each entry in the eI vector then store
the results in a new array called aeI. We may do likewise for the residuals stored in eD and eS:
> aeD = abs(eD)
> aeS = abs(eS)
We now print out the average value of the absolute errors for each model.
> mean(aeI) ; mean(aeD) ; mean(aeS)
[1] 3.214201
[1] 3.13209
[1] 3.074057
In the foregoing interaction, each tally denotes the mean absolute error (MAE) for the
corresponding vector. That is, the output represents the mean amplitude of the offsets around the
mean value of the referent sample.
The results above confirm that the mean magnitude of the misses due to the Idle Model exceeds
that of the Drift version. Moreover, the latter in turn outweighs that of the Sway template. In a
nutshell, the seasonal model does a better job of profiling the bourse than the Drift makeshift
which in turn outflanks the Idle mockup.
Median Absolute Deviation
A variation on the theme of miscues involves the median rather than the mean value as a way to
gauge the average size of the flubs. In this light, the median absolute deviation (MAD) refers to
the median value of the absolute offsets from the median level of the readings.
More precisely, the MAD routine first determines the median value of the observations.
Secondly, the procedure reckons the difference between each datum and the median of the entire
sample. The third step is to discern the absolute values of the residuals. Lastly, the recipe extracts
the median value of the resulting numbers.
The entire operation may be effected via the mad command as follows.
> mad(eI, constant=1) ; mad(eD, constant=1) ; mad(eS, constant=1)
[1] 2.604879
[1] 2.604879
[1] 2.491434
The foregoing printout shows that the MAD values for eI and eD are equal. As we noted earlier
in connection with the MAE yardstick, the Idle and Drift models display the same configuration
of errors aside from a fixed offset that reflects the global uptrend of the stock market. Due to the
46

parallel layout of the error vectors, the scatter of misfits around their median values is identical
as well. For this reason, the Efficient templates display the same value of MAD.
From a different slant, an argument called constant has been set to 1 throughout the first line
of the foregoing interaction. This clause ensures that the results of the computation are better
suited to our current assay.
As a counterpoint, suppose that the constant modifier had been omitted from the foregoing
decrees. In that case, the mad routine would have applied a scaling factor of 1.4826. That is, the
median value of the absolute deviations from the median level of errors would be multiplied by
that peculiar constant.
Put another way, a plain command of the form “mad(eI)” would be equivalent to the following
diktat:
1.4826 * mad(eI, constant=1)
In the foregoing line, the multiplier of 1.4826 serves to calibrate the MAD value against the
standard deviation (SD) of the sample.
The rationale is as follows. The expected value of the boosted version of the MAD statistic—that
is, after multiplication by 1.4826—equals the standard deviation if the sample were to come from
a normal distribution (Leys, 2013; Ruppert, 2010). From a pragmatic stance, then, the bulked-up
result would be apropos if we wanted to compare the output of the mad command directly to the
standard deviation of the sample.
At this stage, however, we wish to assess the MAD value against the mean level of the absolute
error as well as the standard deviation. For this reason, we will use the plain value of MAD
rather than its pumped-up version.
Comparing Measures of Scatter
In the previous section, we saw that the median absolute deviation is apt to be smaller than the
standard deviation if the observations spring from a normal distribution. More precisely, the
expected ratio is 1/1.4826 which amounts to some 0.67449. In other words, the likely value of
the MAD statistic is a mite over two-thirds that of the SD metric.
By comparison, the mean absolute error tends to be somewhat bigger than the MAD. To wit, the
expected value of the MAE is around 0.79788 times the SD when the sample comes from a
normal distribution (Geary, 1935). Roughly speaking, the MAE is four-fifths the size of the
standard deviation if the density function happens to be Gaussian.
The relative sizes of these statistics are of course consistent with their intrinsic nature. For
starters, we recall that the MAD yardstick makes use of the median level of the input data. The
47

median is a sturdy measure of centrality whose value is unaffected by a moderate number of
outliers to the high side and/or low side. This staunch property applies to the median of the
original dataset as well as that of the ensuing residuals around their own median point.
To bring up another facet, the MAE is a measure of the accuracy of a model in profiling a target
system. By contrast, the MAD gauges the precision of a mockup in terms of the scatter of errors
around their central point. In a similar way, the SD of a bunch of errors reflects the precision of a
model rather than its accuracy.
To clarify this issue, we take up a numeric example. Suppose that the errors for a series of three
trials come out to –3, 1 and 2. In that case, the mean value of the triad is 0 while the absolute
values of the goofs are 3, 1 and 2 respectively. The MAE is the average of the latter three
numbers; that is, 6 divided by 3 which equals 2.
We now consider a second model which yields errors of 7, 11 and 12. In that case, the mean slip-
up is the sum of these three numbers divided by 3; namely, 30/3 which amounts to 10. The
turnout reflects the fact that the second set of flubs has the same configuration as the first group,
except that the entire vector of miscues has been shifted upward by 10 units. As a result, the
mean error comes out to 10 rather than 0 as in the first case.
From a larger stance, taking the absolute value of a number is a nonlinear operation. For this
reason, the transformation may easily disrupt the relationship that prevails between a pair of
numbers at the outset; and likewise for a couple of vectors.
In our example, the MAE for the first model amounts to 2 as opposed to the 10 units stemming
from the second template. The difference between these two values is 8 rather than the offset of
10 units that separates the original pair of error vectors.
We now turn to the MAD statistic. We recall that the errors for the first model are –3, 1 and 2.
Meanwhile, the miscues for the second model amount to 7, 11 and 12.
Given this input, the median value of the first set of goofs is 1 while that of the second suite
equals 11. Moreover, the deviations from the median error amount to –4, 0 and 1 for each of the
two models. In this way, the initial pair of error vectors happen to differ by 10 units but the
separation makes no difference in gauging the distance of the bloopers from their respective
medians.
The absolute values of the deviations then come out to 4, 0 and 1 respectively. Based on the last
three numbers, the median level is 1. The latter value comprises the MAD statistic for each of the
two models.
In brief, the MAD yardstick gauges the precision of a model by the median size of the offsets
around the median point of the errors. By contrast, the MAE is a measure of accuracy in
profiling the target system while making no effort to centralize the flubs.
48

Financial Modeling for Real

Recommended

Recommended

More Related Content

What's hot

What's hot (13)

Similar to Financial Modeling for Real

Similar to Financial Modeling for Real (19)

Recently uploaded

Recently uploaded (20)

Financial Modeling for Real