A talk to beginning graduate students, Part 2. This is about the fundamentals of knowledge, understanding and science, promoting the scientific method and Karl Popper's views. The remainder outlines the practice of a thesis work, from hypothesis through proposal onwards... And I shamelessly mock pyramidology and related fields...

1. About your graduate studies, Part II:
Knowledge, Science, Predictive
Models, and the Magic Spell
Seppo Karrila
March 2015
PSU Surat Thani

2. Executive summary
• This is a discussion of the philosophy and
principles behind modern natural sciences
• As a graduate student your task is to perform and
report a scientific study, so you need to know
how science is done
• Predictive models are emphasized because
– They give testable predictions, which allow application
of the scientific method
– Using training, validation and test data is a simple
trusted approach to doing the job correctly

3. Theory of knowledge
• Has been studied for a very long time, called
“epistemology”
• Such theories have been unable to reach practical
usefulness
– The question “how do we know what we know, and
do we really know?” is important
– Time travel would also be important…
– Companies are not struggling to find and hire
epistemologists

4. What is “understanding”?
• In common language it often just means to sympathize. “I feel
for you, I understand.”
• In science there are different levels of understanding
– Forming concepts and nomenclature, and general rules: an apple
falls if you let it go
– Naming the ghost behind the action: gravitation. Being able to name
the ghost gives a “feeling of understanding it”.
• Why does the apple fall – gravitation! This is nonsense and word magic,
just another name that sounds more learned than “it falls because it falls”.
– Quantitative understanding: how fast does it fall, how hard does it
hit. Can be calculated accurately with classical mechanics and ITS
EQUATIONS FOR GRAVITATION. Now we are getting somewhere!
(Some went to the moon, satellites for GPS and communications.)
– Application models based on experiments: when the apple hits the
floor, will it break, how much is it damaged, does this affect its price
or shelf-life, how should we package apples… These are clearly
things you can’t calculate accurately from classical mechanics, you
have to do experiments AND MAKE MODELS.

5. Stages of development
• A “young” scientific discipline
– Names and documents things
– Creates taxonomies
– Is descriptive
• Becomes more quantitative with time, a “teenager”
– Finds general rules that are equations, like “conservation
of mass”, thermodynamics of equilibrium
• Strives to replace experiments with predictive
computations, becoming an “adult mature discipline”
– This transition is currently happening in chemistry…
• In other words a somethingology tries to advance to a
somethingonomy, from qualitative to quantitative

6. “Predictive” is the keyword!
• There are theories that explain everything afterwards, but
predict nothing
– Pyramidology predicts history accurately but future always
poorly
– There is a BIG difference between a regression fit and a
predictive equation. Ability to match what is known does not
equal ability to predict the unknown.
• Engineering design is based on sufficiently accurate
predictions. Design equations may be mandated in official
standards.
– You can’t know the strength of a concrete mix accurately, so
engineers use safety margins in designs
– Still a lot of computations are useful and used in mechanical
design, inaccuracies don’t make a model useless

7. Training, validation and test data
• Here is the correct way to fit predictive models to
experimental data.
• The data are split to three sets, or generated at different
times for each of the three
– All candidate models are fit to the training data
– The validation data is “predicted” with every type of model, and
the best model type is chosen
• Now it can be fit to training + validation data
– The chosen model is tested for performance
• It has not “seen” the test data before, if you test with previously seen
data you are looking at “fit”, not “prediction ability” !
– Now you can pass on the model, and make some claim about its
prediction accuracy

8. How to make pyramidology look good
• Make predictions about the starting year in January.
• Check predictions next January
– Ups, all wrong again
• Quickly update the model
– Publish how the new model fits history
– Never discuss results from testing old predictions with new data, erase
the old models quickly
• Effect: the public model is always an untested fit, there is never
embarrassing numeric evidence about predictiveness (actually
complete lack of it)
– This game is played all the time. Various companies have forecasters
who don’t want to look incompetent. So does the government. The
forecasters want to keep their jobs and maintain credibility, and the
purpose of the forecasts is not to be right but to influence decisions.
Once the decisions are made, why look back…

9. Intermediate summary
• In the natural sciences, the highest level of understanding is a
quantitative predictive model
– Then you can predict effects of choices or actions on future function
and performance, this is called “engineering design”
• Such models must be based on reproducible experiments, the
models must be validated, and their accuracies known. The
models can only use validated reproducible characterizations
whose accuracy is also known.
– Characterizing material properties of cement mixes, or steel, or
rubber, or plastics – check the official standards.
• Fundamental theories are not yet good enough
– You only learn EQUILIBRIUM thermodynamics
– Even “viscosity” becomes difficult with polymeric liquids, so don’t
expect accurate flow calculations in a complicated geometry
– Flow  mixing  reaction rates  …

10. How does science progress?
• According to Karl Popper
– there is an accepted “scientific paradigm” that scientist use,
until enough evidence accumulates to refine or update it, and
that is a “paradigm shift” (like classical mechanics  relativity,
continuum  quanta)
• Negative evidence = false predictions ! Without them there will be no
paradigm shift
– Various fields like sociology would not fit his definition of
science at all, since they seldom predict anything. There are
no testable hypotheses, but an ideology may “explain”
anything in past history.
– Thomas Kuhn became popular because he essentially
proposed that science is whatever scientists do. This made
sociologists and various others happy again, they loved Kuhn.
– Popper’s views are most appropriate in the natural sciences
and engineering that have developed to a quantitative stage.

11. The scientific method!
• Make a hypothesis or conjecture
• Design experiments that could show it is false
– In fact, the hypothesis should predict something that
is extremely unlikely without it being correct, so when
that happens it is strong support to the hypothesis
• If the hypothesis survives the tests, keep it. If it
makes useful predictions, others will adopt it and
teach it as current wisdom.
– It becomes part of the current scientific paradigm

12. The big point
• Science looks for general truths that summarize
experiences in a useful way, and that allow making
predictions.
– Instead of learning every sentence you use, you learn
grammar. Summarization and general rules enable learning
and doing your own things instead of just copying and
repeating.
– Measurements may be needed, but doing measurements
is not the same a doing science.
– The most valuable hypotheses have a wide generality. Very
specific and restricted hypotheses are “doing the right
motions” but amount to little in scientific learning. They
provide a data point, not a useful summary. However, they
are the necessary less glorious grassroots of science, and
that is where most scientists live. It is the big things that
are historic with fame.

13. Other points to note
• If the hypothesis is such that nothing can falsify it,
then it only predicts things that would happen
anyway (or nothing at all)
– It cannot usefully predict anything new and surprising
• But the testing is based on predictions!
– A theory that predicts nothing does not enable any
decision, design, or action. These are the only things
that give value to scientific theories.
– This is why “theories” that predict nothing and can’t
be tested are pseudoscience.

14. About “mistakes”
• Progress of science depends on being wrong
– Falsification of old paradigm is the only way to a new
paradigm
• If you are very afraid of mistakes, you can’t do anything
– But it hurts less to learn from mistakes of others
– Admit mistakes quickly, correct them quickly before they
can take effect, avoid repeating them. It is not honorable
to hide a mistake.
– A man who never made a mistake has done nothing
• However, don’t study mistakes, study successes
– First, this way there is less to study
– Second, repeating a success is better than repeating a
mistake

15. You can never prove that something is
true – you can prove something false
• Mathematics is based on axioms, assumed truths. Its
proofs are valid IF the axioms are valid. In other words
the absolute truths of mathematics are confined within
mathematics.
– Still, logic and mathematics are the tools of clear thinking.
We want equations, and quantitative predictions!
• However many red roses you see, it does not prove
that all roses are red.
– It only takes ONE white rose to prove they are not ALL red
– This is why a hypothesis must be FALSIFIABLE, that is the
best we can do. You cannot have a PROVABLE
experimental hypothesis.

16. Null hypothesis
• Statistical testing is based on the same ideas.
• You make a NULL HYPOTHESIS “some roses are
not red”, that means falsification of the actual
hypothesis “all roses are red”
• You show that the null hypothesis is very unlikely
to be true
• That supports your actual hypothesis as
“significant”, meaning that we can keep it for now

17. Back to the magic spell…
• Your scientific story is “good” if you have
– Issue, significance, approach, results, conclusions
– The question must have importance, the conclusions
state effects of your results on theory or on practice!
– It is really good if the results are unexpected and
surprising
• A technically useful engineering result usually
includes a quantitative model
– How do you make a model? Where can you start?
How is “science” done in practice?

18. You are not Newton
• We don’t expect you to come up with new
general principles of fundamental science
• But recall that our science is limited, we need
experimental models
– You have equilibrium thermodynamics, but kinetic
reactions, flows, multiphase materials
– Rheology of a polymeric liquid is difficult enough
• Now add solids, perhaps nanomaterials, reactive species,
electromagnetic fields, …
– Or just examine if an apple breaks when it hits the
floor

19. For modeling
• Define concepts included in the model
– Some need to link to reality through direct
measurements
• Prefer physical measurements over industry technical
standards, the former will stay as they are today
– Others can be computed intermediate variables
• These may come from physics, physical chemistry, etc.
• Dimensionless groups!
• You should use these in statistical modeling

20. The problem with indirect
measurements
• Does “happiness” relate to “wealth”?
– A sociology problem where no concept can be directly
measured. Does wealth include your future inheritance from
grandfather, possibly winning lottery, or your relative who works
for PTT where you may get a good job? A rich girlfriend? How
do you measure “happiness”? Are you feeling 6.3 happy or 8.5
happy, on average this week?
– By adjusting definitions, you can get anything you want. The
result is theories that live as long as the professor who started
them, and who sits on committees of the National Research
Fund while alive. So you better agree with his theory while he
still lives, if you want to do related research. Oh, he is also the
Editor or on board of all relevant journals…
– Stick with the real sciences, we do things better. I like Karl
Popper.

21. Assume you got a topic from your
advisor
• Were you given a hypothesis, or do you need to
come up with one?
• Learn the basics, read review articles, find out
about techniques used in experimental
determinations
– For example, nanomaterials are modern and difficult
exactly because they are too small to “see”. Some
might also have nasty effects that are slow and
delayed, like asbestos fibers, so take precautions.
Don’t rush into exciting new things unless you can also
measure and detect.

22. Hypothesis from review of literature
• Are there important gaps in knowledge?
– Turn these into hypotheses that seem reasonable
• Can they be studied with YOUR available
equipment and techniques?
– Can you estimate or guess sizes of effects?
• What are the factors affecting results?
– Which ones can you manipulate
– Which ones can you observe/measure
– Which ones can you limit or select, essentially
defining the scope of your study

23. Decisions and actions, once you have
your hypothesis
• Select scope, manipulated and observed variables
– Do you have the technical skills?
– Do you need to design and construct devices?
• Do you need preliminary or “pilot” experiments?
– Instead of gambling a long-term plan with uncertainties, can you
quickly check for some effects or phenomena?
• Design of experiments, after selecting a minimal set of
main variables
– Can be complicated, better to stay with some standard design
(e.g. Plackett-Burman), otherwise consult a statistician
– If task is to optimize something like yield, check out response
surfaces and Box-Behnken design
• Statistical analysis of results
– Significant effects AND their effect sizes !

24. About effect size
• Opening your car windows changes
aerodynamics, probably for the worse
– The top speed may go down by 3 km/h if you
open the windows
– If in experiments you would repeatedly find this is
so (using GPS to measure speed), then this effect
is statistically significant
– However, the effect size of 3 km/h is marginal, you
don’t need to care. If it dropped the top speed by
40 km/h, that would be practically important.

25. A key observation
• Statistically significant is not the same as
significant!
– Statistically significant means, it is likely there is
some detectable difference. A detectable
difference can be marginal in size.
– Significant Breakthrough Discovery: with a small
effort and at a low cost, you get a large effect (on
yield, quality, production rate, …)
– This is why you should pay attention to effect
size!

26. The arts and sciences cherish novelty
• But it is not enough. Pay attention to significance!
– If I paint with a banana, it is a novelty but nobody will
buy my painting
– If I multiply two 50-digit numbers and subtract
1234567, nobody else ever did that calculation with
the same numbers: it is a novelty!
• No insight, nothing of interest, can’t be published
– In descriptive sciences, things are published just to
document
• Asteroid number 123456 photographed through telescope –
it goes to a database as an entry, but nobody really cares.
This is their grassroot science, a data point.

27. Art, science and engineering, a
caricature
• When something is done for the first time, it can
be art or science
• When it is done repeatedly and efficiently, it can
be plagiarism, forgery, or engineering research
• In science, if you know the result you should not
do it, because you are looking to expand
knowledge
• In engineering, if you don’t know it works, you
should not do it, because most likely it will not
work.

28. Engineering and science are
intermixed
• The exploration exploitation dilemma in learning
– The caricature was about extremes in exploration and
exploitation
– If you only exploit existing knowledge, that may be convenient
and predictable but you learn nothing new
– If you only explore, you are wandering aimlessly and not
productive
– How much of time or budget should be exploration?
– Some companies have been very good at leaving explorations to
others, then quickly doing a technically reliable good job once a
new technology has been demonstrated. This is a cost
advantage.
– Similar opportunities abound in science, where you can transfer
techniques established in another field to your field. You will
not be “the first”, but if you are quick you can be the first in a
specific context. So keep your eyes open, read widely.

29. In STEM you should use precise
language
• You drive by a field and see a black bull in
profile view.
• Layman’s statement: ”See, they have black
bulls here!”
• Scientific statement: “In this region there is at
least one bull that is black on at least one
side.”

30. On evaluating your own work, or that
of others
• Use the magic spell
– Issue, significance, approach, results, conclusions
– If any of these are missing, thumbs down
• Do this also when you are planning your work
– What kind of results do you expect?
– How can the results affect theory or practice?

31. Research proposal
• Surprise:
– Issue, significance, approach, expected results,
expected effects (conclusions)
• Now approach needs to include
– An experimental design
• How many samples, what measurements, what
experiments, how many replications
– Time – how long would it take?
– Budget – how much would it cost?
– Risks – what can go wrong?

32. Dealing with risk
• Can you prevent it from happening?
– Vaccination may prevent infection
– Do you need to check for impurities of raw materials,
sterilize samples, remove suspended solids before optical
measurements or chromatography, de-aerate liquids, …
– Start with small doses of an additive, see if trouble arises.
Increase dose if all goes well. Large dose first is risky.
• What will you do if the bad thing happens?
– Go to doctor or hospital when you get sick
– Do you have alternatives or backups, for sources of
samples, for determination techniques, …

33. Conclusions
• Most likely you will run experiments and do some statistical
analysis of the results
– Planning of experiments should be based on a hypothesis, the
most “ignorant” just assuming that some factor has an effect
• After literature research, you should have a hypothesis and
an experimental approach, convert these to a detailed
research proposal
• When reporting results, or checking results of others,
statistical significance is worth little, effect size is worth a
lot
• The Magic Spell gives structure to any communication, or
analysis of communication. If an item in it is missing, then
the proposal/presentation/manuscript is incomplete.

34. One final note and warning on
science vs. pseudoscience
• We have not defined science, but it clearly seeks knowledge
and understanding that are useful and predictive
– The best we can do is accept a useful hypothesis until it has been
falsified. Useful ones predict something, so they can always be
tested.
• We can identify much of pseudoscience from this already
– Either the hypothesis can’t be falsified ever, by anything
– or its proponents bitterly oppose any such test that could falsify it,
their interest is not truth and knowledge but politics
• But… everybody wants to have the clout of science. If you point
a finger at pseudoscience, many people will be upset.
– It suffices that you know the truth, don’t waste your time in an
argument. Just keep it between you, me, and Sir Karl Popper. And
that guy in the toothpaste commercial who wears a white lab coat,
he knows, too.