Hash Functions: lecture series by Ahto Buldas

Lecture 1 on Hash Functions July 26, 2011

Lecture 1:
Hash Functions:
Short Identifiers for Large Files

We introduce a problem of giving unique identifiers to (potentially) large
files and study how short such identifiers can be. Hash functions are de-
signed for giving standard ways of solving the problem.


Hash Functions
A hash function converts a large, possibly variable-sized amount of data
into a small datum (hash) that may serve as an index to an array.

Hash functions are mostly used to accelerate table lookup or data compar-
ison tasks—such as finding items in a database, detecting duplicated or
similar records in a large file, finding similar stretches in DNA sequences,
etc.

With a hash function in use, we can check the presence of an item in a
database with just one single lookup!

The idea was proposed in the 1950s, but the design of good hash functions
is still a topic of active research.
2


Collisions
A hash function may map two or more keys to the same hash value—this
is what we call a collision. In most applications, it is desirable to minimize
the occurrence of collisions, which means that the hash function must map
the keys to the hash values as evenly as possible.

3


Birthday Paradox
Birthday paradox: For randomly chosen 25 people, very likely we have two
with the same birthday.

NB! This does NOT mean that if you are among those people, you will very
likely ﬁnd someone with the same birthday!
4


How many different hash values do we need?

How large hash values should we use for uniquely identifying N items?

Answer: for a good hash function, we need about N 2 different hash values.

If N = 2256 documents will be created all over the world during the ex-
istence of mankind, we need about 2 · 256 = 512 output bits to identify
them uniquely.

Eddington number : number of protons in the observable universe is about

1.57 · 1079 ≈ 136 · 2256 .
This is still just a small fraction of all possible 40 byte sequences!
5


Modeling Hash Functions With Random Oracles
The values of a good hash function are distributed as evenly as possible.

A good hash function is modeled as a random function (a random oracle)—
the computation of hash value is replaced with a uniformly random choice
(all hash values occur with the same probability 1/M ).

6


Bounds for Collision Probability

Say we have N ﬁles and the hash function has k output bits, i.e. there
are M = 2k possible hash values. If the hash function is modeled as a
random oracle, the probability P (N, M ) that all N ﬁles have different hash
values has the following bounds:
N (N − 1) − N (N −1)
1− ≤ P (N, M ) ≤ e 2M .
2M
Homework exercise: Show that P (N, M ) can only decrease if the hash
function deviates from random oracle, i.e. the output probabilities differ
1
from M .

7


A Useful Inequality from Analysis
We used the inequality 1+x ≤ ex that holds for all real x. This also means
that 1 − x ≤ e−x.

The inequality holds because ex is convex and 1 + x is the tangent line of
ex at x = 0.
8


Upper Bound

For the upper bound, we use the observation that
1 2 N −1
P (N, M ) = 1− · 1− · ... · 1 −
M M M
≤ e−1/M · e−2/M · . . . · e−(N −1)/M = e−(1+2+...+N −1)/M
= e−N (N −1)/2M .
Explanation: We compute the hashes of the N ﬁles one by one, consid-
ering each hash computation as a uniformly random selection of a hash
value. Before the i-th selection, we already have i − 1 selected hash val-
ues and the probability that the new hash is one of those is i−1 .
M

9


Lower Bound

To get a lower bound for P (N, M ), we ﬁrst observe that for any pair of
ﬁles, the probability that they have the same hash is 1/M . As there are
N (N − 1)/2 pairs, the probability of having a collision is upper bounded
by N (N −1) and hence,
2M
N (N − 1)
P (N, M ) ≥ 1 − .
2M

10


Examples
√
For large N ≈ M , we have
1 √
≤ lim P ( M , M ) ≤ e−1/2 ≈ 0.6065306597 .
2 M →∞
For the Birthday paradox, we set N = 25 and M = 365 and get

P (N, M ) ≤ 0.4395878005 .

11


Lecture 2:
Cryptographic Security Requirements
for Hash Functions

We study how to protect the uniqueness of identiﬁers against malicious
behavior of users. Cryptographic hash functions are designed for this
purpose. We describe the main properties that are required from crypto-
graphic hash functions: one-wayness, second pre-image resistance, and
collision resistance. We go through some examples to show where these
security properties are needed in practice.


Three Main Security Properties
Cryptographic hash functions are designed for having the following secu-
rity properties:
• One-wayness—hardness of finding a message given its hash.
• Second pre-image resistance—hardness of modifying a message with-
out changing its hash.
• Collision-freeness—hardness of finding two different messages with the
same hash.

Collision freeness is the strongest requirement of the three, because the
adversary has the largest degree of freedom.

Note that CRC32 does NOT meet any of the three requirements! It is in-
tended to fight against occasional errors, not malicious attacks.
13


One-Wayness (or Pre-Image Resistance)

Given a hash h(X) of a randomly chosen input X, it is hard to ﬁnd an
input X with the same output h(X ) = h(X). Here, X is not known to
the adversary!

14


Second Pre-Image Resistance

Given a randomly chosen input X, it is hard to ﬁnd a different input X = X
with the same output h(X ) = h(X). Here, X is known to the adversary!

15


Collision Resistance (or Collision-Freeness)

It is hard to ﬁnd two distinct inputs X = X with the same output h(X ) =
h(X).

16


Examples for One-Wayness and Second
Pre-Image Resistance

We need one-wayness in a situation, where X contains secret data. We
have to uniquely identify X but do not want that the identifier reveals the
contents of X.

We need second pre-image resistance if we want to protect a public doc-
ument X from malicious modifications. For example, if we keep the hash
h(X) of X in a safe place, it should be impossible to create a modified
document X with the same hash, because then the modifications were
undetectable.

17


Digital Signatures and Collision Resistance
We create two contracts X and X with the same hash, where X is clearly
favorable to us.

We show X to our partner who then signs the hash h(X).

Later, we show X in court with the partner’s signature on h(X ) = h(X)
and claim that he/she accepted the conditions of X .

18


Additional Remark about Digital Signatures

It may seem that in the last example, we can overcome such a situation by
stating (in law) that a signature that uses h is invalid whenever one comes
up with a collision for h.

However, this would lead to another way of deception. We prepare to con-
tracts X and X , where X is the contract we actually sign and X is an-
other contract with the same hash h(X ) = h(X) but which is much more
proﬁtable to us than X.

Our intention is to deny having signed X by showing in court the other
version X and claiming that X was actually the contract we wanted to
sign.
19


Lecture 3:
Relations Between the Security Properties
of Hash Functions

It turns out that some of the security properties for hash functions can
be derived from others. In this lecture, we prove mathematically that the
collision-resistance is the strongest requirement of the tree because it im-
plies both the one-wayness and the second pre-image resistance.


Relative Security Proofs by Reductions
To prove relations between security properties, we use reductions.

To prove that property P implies property Q, we use the contraposition:

If there is an efﬁcient A that breaks Q, we construct an efﬁcient A that
breaks P.

21


Collision Resistance Implies Second Pre-Image
Resistance.
Every adversary A that finds second pre-images for h can be modified to
a collision finding adversary A :
• A first generates a random input X and then
• uses A to find an X = X such that h(X ) = h(X).

22


Collision Resistance Implies One-Wayness ...
... if the hash function is sufficiently compressing. Every adversary A that
is able to invert h can be modified to a collision finding adversary A :
• A generates a random X, computes y = h(X) and
• uses A to find X such that h(X ) = h(X).

If h is sufficiently compressing then Pr[X = X] is small. Hence, A very
likely finds a collision when A is successful.

23


Mathematical Details

Direct computation of the success of A is not that trivial because X de-
pends on X, and we cannot view X and X as independent random vari-
ables. However, if the output value y is known and ﬁxed then X becomes
independent of X. Indeed, we can regenerate X with a uniform choice
from the pre-image of y, and by doing so we do not change the distribution
of X. Hence, the work of A is equivalent to the following (inefﬁcient!) pro-
cedure:
• Choose a uniformly random input Z and compute y = h(Z).
• Choose a uniformly random X from the h-preimage of y.
• Apply A to obtain X = A(y).

24


The steps 2 and 3 are independent random choices and hance the proba-
bility C(y) that a collision is produced for y is:
| h−1(y) | −1
C(y) = Pr[A inverts y] · Pr[X = X ] = Pr[A inverts y] ·
| h−1(y) |
1
= Pr[A inverts y] · 1 − −1 .
| h (y) |
24


The overall success δA of A is the average of this probability:
1
δA = Pr[y] · C(y) = Pr[y] · Pr[A inverts y] · 1 − −1
y y | h (y) |
1
= Pr[y] · Pr[A inverts y] − Pr[A inverts y] · Pr[y] · −1
y y | h (y) |
δA 1/N
1
= δA − · Pr[A inverts y]
N y
≤M
M
≥ δA − ,
N
where δA is the success probability of A, and N, M are the input set size
and the output set size, respectively.

24


Insufficiently Compressing Hash Functions

Insufficiently compressing function can be collision-free but not one-way.

If there exists a collision free hash function h [with (k − 2)-bit output] then
there exists a function h that is collision-free but not one-way.

We define h so that it returns a (k − 1)-bit output when given a k-bit input.

1 z, if x = 11z
h (x) =
0 h(x), otherwise

25


• Finding collisions for h is as hard as ﬁnding collisions for h.
• Inverting h is relatively easy, because with probability 1/4, a random x
is of the form 11z and the output 1z of h completely reveals the input.

25


Lecture 4: Provably Secure vs Practical
Hash Functions

We study two main goals when constructing a hash function. Provably se-
cure hash functions are more reliable but inefficient. In practice, we use
more efficient ad hoc designs, which have no guarantees against discover-
ing more and more efficient attacks. In this lecture, we present some basic
ideas how to construct provably secure hash functions. We also character-
ize the practical hash functions and summarize how secure some of them
are, considering the known attacks against them.


The Idea of Provably Secure Hash Functions
The security of a hash function h can be based on a hard mathematical
problem if we can prove that finding collisions for h is as hard as solving
the problem.

This gives us much stronger security than just relying on complex mixing
of bits.

A cryptographic hash function h has provable security against collision at-
tacks if finding collisions is reducible to a problem P which is supposed not
to be efficiently solvable. The function h is then called provably secure.

Proof by Reduction: If finding collisions if feasible by algorithm A, we could
find and use an efficient algorithm S (that uses A) to solve P , which is sup-
posed to be unsolvable. That is a contradiction. Hence, finding collisions
cannot be easier than solving P .
27


Some Hard Mathematical Problems for Provably
Secure Hashing

Provably secure hash functions are based on hard mathematical problems
that are mostly related to well-known mathematical functions.

Some problems, that are assumed not to be efficiently solvable:
• Discrete Logarithm Problem: Given ax mod p, find x, where p is a large
prime number.
• Finding Modular Square Roots: Given a2 mod n, find a, where n is a
hard to factorize composite number.
• Integer Factorization Problem: Given n = p · q, find p and q, where p, q
are two large primes.

28


Example of a Provably Secure Hash Function

Modular exponent function:

h(x) = ax mod n,
where n is a hard to factor composite number, and a is a pre-speciﬁed
base value.

A collision ax ≡ ax (mod n) reveals a multiple x − x of the order of a
(in the multiplicative group of residues modulo n). Such information can be
used to factor n assuming certain properties of a.

Such an h is inefﬁcient because it requires 1.5 multiplications per input-bit.

29


Design Principles for Practical Hash Functions
Practical hash functions are combinations of bit operations which are hard
to study, partly because of their ”ugly” mathematical properties.

The avalanche criterion requires that if an input is changed slightly (for
example, flipping a single bit) the output must change significantly (e.g.,
many output bits flip). First used by Horst Feistel (1915-1990).

The Strict avalanche criterion (SAC) is a generalization of the avalanche
criterion. It is satisfied if, whenever a single input bit is complemented,
each of the output bits changes with probability 1 . Introduced by Webster
2
and Tavares in 1985.

The bit independence criterion (BIC)—two output bits should change inde-
pendently when any single bit (not among these two) is inverted.
30


˚
Merkle-Damgard Design
A hash function must be able to process an arbitrary-length message into
a ﬁxed-length output. This is achieved by breaking the input up into blocks.

The last block processed should also be unambiguously length padded.
˚
This construction is called the Merkle-Damgard construction. Most widely
used hash functions, including SHA-1 and MD5, take this form.
31


˚
Merkle-Damgard construction is as resistant to collisions as is its compres-
sion function—every collision for the full hash function can be traced back
to a collision in the compression function.

The construction has certain inherent ﬂaws, including length-extension and
generate-and-paste attacks, and cannot be parallelized.

Many candidate hash functions in the current NIST competition are built on
different constructions.

31


Practical Hash Functions and their Known Strength

32


Lecture 5:
Time-stamping as an Application for Hash
Functions

Time-stamping is one of the areas where cryptographic hash functions are
used. We introduce the basic concepts of time-stamping and describe how
the hash functions can be and are used to create efﬁcient practical designs
for secure time-stamping services.


Motivation
Proving that a document (ﬁle) existed at certain time.

Usage examples:
• Intellectual property —knowing the earliest time stamp for an invention
would be a strong argument for the authorship.
• Revocation of signature keys—when a signature key has been revoked,
it should still be possible to distinguish between the signatures created
before the revocation and those created after.

Note that if you just having a time stamp for a document does not prove
that you knew the contents of the documents at that time. Knowing a hash
of a document does not imply the knowledge of contents.

Time stamps are not intended to prove the ownership of information but
rather the existence (i.e. that somebody or something had the contents).
34


Trivial Solution with a Trusted Server
Trivial solution may be (and is) used which involves a trusted server, who
receives hash values of documents from clients and signs them together
with current time. The main shortcomings of this solution are:
• We have to trust the server completely; the server can never be assumed
to be malicious.
• If the signature key of the server accidentally becomes public, then all
the previous time stamps become useless.

35


Hash and Publish Solution

A hash of a document (or a hash of a batch of documents) is published in
a widely witnessed way like in newspapers or in public web-pages.

This provides an irrefutable proof of existence of those documents at the
time of publishing.

36


Merkle Tree: How to publish many hashes?

37


Time Stamp = Hash Chain

For example, for Document 1, the time stamp contains:
• hash 00, to establish a connection between Document 1 and hash 0.
• hash 1, to establish a connection between hash 0 and root hash.

Such time stamps are compact because the number of hashes in a time
stamp is logarithmic in the number of leaves.
38


Efﬁcient Hash and Publish with Merkle Trees

A hash tree is created every second and the root hash values are published
in a widely witnessed way.

39


Lecture 6: Security Requirements for
Hash Functions Used in Time-Stamping

What do we require from a secure time-stamping solution and what kind of
cryptographic hash functions do we need? It turned out that it is not easy
to consistently formalize the concept of a secure time-stamping scheme.
It also turned out that the three classical security requirements for hash
functions are not exactly what we need in time-stamping schemes. We
explain why the early attempts to formalize the security of time-stamping
failed and outline the consistent deﬁnitions that are known today.


An Early Attempt to Deﬁne a Security Condition
In 1997, Haber and Stornetta proposed the following adversary A:
• A sends requests x1, . . . , xm to a server, and after the server has cre-
ated a hash tree and published the root hash r,
• A comes up with x ∈ {x1, . . . , xm} and a hash chain from x to r.

41


Inconsistency of the Condition
Adversary may send a request x1 = h(x, x ), so that x ∈ {x1, . . . , xm}.
A hash chain from x1 to r can easily be extended for x by just adding a
step that contains x and establishes a connection between x and x1.

The security condition is violated, but nothing is back-dated. The request
x should be ”new” and not having involved in pre-computations.
42


Unpredictable Adversaries

An improved security condition proposed by Buldas and Saarepera (2004)
and reﬁned by Buldas and Laur (2006) assumes that the new request x is
chosen (by the adversary) in a random way.

More formally, it is assumed that the new request should be unpredictable
even having all data that the adversary had in the ﬁrst stage of the attack.

43


This means that any efﬁcient predicting algorithm Π that hash access to all
computations of the ﬁrst stage of the adversary can guess the value of x
(computed by the second stage of A) only with negligible probability.

43


Improved Condition with Unpredictable Adversaries
Time-stamping scheme should be secure against all efﬁcient unpredictable
adversaries A:
• A sends requests x1, . . . , xm to server, and after the server has created
a hash tree and published the root hash r,
• A outputs an unpredictable request x and a hash chain from x to r.

44


As A is unpredictable, the condition x ∈ {x1, . . . , xm} is no more neces-
sary. Moreover, this would be very hard to test in practice because the list
of all requests is too large and cannot be examined by any real observer.
Also, A may just send a one single request r to the service without reveal-
ing how it is computed. Hence, the attack scenario reduces to:
• A publishes r,
• A comes up with an unpredictable request x together with a correct hash
chain from x to r.

44


Collision-Resistance is Still Insufﬁcient
Improper computation of r may help to back-date new requests without
exposing collisions.

It might be possible to compute root hash values for very large hash trees
without actually computing all nodes of the tree.

If you know that 1 + 2 + . . . + 100 = 5050 this does not mean that you
actually summed up all these integers—you might use the multiplication
instead: 1 + 2 + . . . + 100 = 100·101 = 5050.
2

Buldas and Saarepera (2004): if collision-resistant functions exist, then
there is a collision-free hash function that enables back-dating random doc-
uments.
45


Hash Chain Limitations as a Solution

The hash tree used for time-stamping should somehow be restricted (for
example, length limitations to hash chains, ﬁxed shape of the tree, etc.).

First correct security proof for restricted schemes was given by Buldas and
Saarepera (2004).

For example, tn the Guardtime system, there are explicit length restrictions
for hash chains:
• Each hash step includes a length byte that represent the number of pre-
ceding steps in the hash chain.
• Veriﬁcation procedure checks, whether the length bytes are in increasing
order.
46


Why Collision-Resistance and One-Wayness are
Unnecessary?
Say, we can ﬁnd a collision h(X) = h(X ) where X and X are two
different documents and obtain a time stamp for h(X).

The fact that we also get a time stamp for X this way should not be con-
sidered as an attack against the time-stamping scheme, because nothing
is back-dated!

The adversary can only get two time stamps for the price of one!

Buldas and Laur (2006): if there exists a hash function h that is secure for
time-stamping schemes, then there is a hash function h that is secure for
time-stamping but is not collision-resistant and not even one-way.
47


Such a hash function h can be defined as follows:
y if x = y y
h (x) =
h(x) otherwise
i.e. h is the same as h, except that h (yy) = y for any y.

The point being that whenever there is a hash chain (produced by the
adversary) that contains h -specific hash steps in the form h (yy) = y,
then we can simply delete such steps without breaking the correctness
(changing the output value) of the hash chain.

Hence, the modification h → h cannot be the cause of insecurity—if back-
dating is easy with h then it is also easy with h.

47


Lecture 7:
Security Proofs for Time-Stamping
Secure time-stamping schemes require hash functions with security prop-
erties that are not directly related to the three classical security goals for
cryptographic hash functions: one-wayness, second pre-image resistance,
and collision resistance. Hence, there are no guarantees that such func-
tions are secure for time-stamping. In this lecture, we will see how to de-
ﬁne restrictions in time-stamping schemes so that the classical security
properties of hash become sufﬁcient for their used in the (restricted) time-
stamping schemes. Based on the known research results in this area,
we draw some conclusions about the security of practical time-stamping
schemes.


Collision Extraction From Hash Chains
The proof from Asiacrypt 2004 uses the following fact on hash functions:

If there are two hash chains of the same shape that have different inputs
x and x and the same output y, then having those two hash chains, it is
easy to extract a collision for the hash function.


Two Chains with Different Shape

Two chains from different shape do not necessarily produce a collision. For
example, if they belong to one and the same tree:


Outline of the Security Proof from Asiacrypt 2004
The collision-ﬁnder A executes the ﬁrst stage of the adversary to produce
a published hash value r, and then executes the second stage twice in
order to have to successfully back-dated requests x and x with two hash
chains of the same shape.


Efficiency of the Collision Finder
If there are N allowed shapes for hash chains, and δ1,...,δN are the prob-
abilities that A succeeds with a hash chain of the corresponding shape,
then the overall success probability of A is δ = δ1 + ... + δN .

As the values x and x are unpredictable, Pr[x = x ] is negligible. Hence,
the success of the collision-finding adversary is about
2 2 δ1 + ... + δN 2 δ2
2 + ... + δ 2 = N · δ1 + ... + δN ≥ N ·
δ ≈ δ1 = .
N
N N N
As the running time t of the collision finder A is about twice the running
time of A, we have that the time-success ratio of A is:
t t
≈ 2N · 2 ,
δ δ
which means that if we want to have a t/δ-secure time stamping scheme,
we have to use a 2N · δt2 -secure hash function.


An Improved Reduction

A recently published security proof by Buldas and Niitsoo (2010) estab-
lishes a security reduction in which (for having t/δ-security), the hash func-
√ t
tion must be about 14 · N · d1.5 -secure, i.e.

t √ t
≈ 14 N · 1.5 ,
δ δ
which is much more efﬁcient than in the previous reductions.

This is the ﬁrst security reduction that implies the security of global scale
time-stamping schemes that use 256-bit hash functions.


The collision-ﬁnding adversary executes the second stage of the collision-
ﬁnding adversary m times (instead of two, as in the proof of Asiacrypt
2004), with m = N where δ is the success of A.
δ


Conclusions for Practical Schemes

What is N for practical schemes? It is not constant but grows over time.

If a new tree of size C is produced every second, then N = τ · C, where
τ is the running time of the adversary, measured in seconds.

In security proofs, however, the running time t of A is measured in compu-
tational steps.

World’s total computational power :It has been estimated that the compu-
tational power of the whole world is close to 262 operations per seconds
(from 2007). We assume that this has been grown to about P = 264.


It is natural to assume that t ≈ 264 · τ , and hence
C·t
N ≈ .
P
The efﬁciency formulae for the two reductions are as follows:
t ≈ 2C · t 2 (2004),
δ P δ

t ≈ 14 C · t 1.5 (2010)
δ P δ


Conclusions for Practical Schemes
t t
For having a δ -secure time-stamping scheme, we have to use δ -secure
hash functions, which means that if they are secure (to the Birthday barrier)
their output in bits should be
t
k ≈ 2 · log2 .
δ
t
By denoting c = log2 C and s = log2 δ , we deduce from the efﬁciency
formulae that:
k ≈ 2(1 + c − 64 + 2s) = 2c + 4s − 126 (2004)

c
k ≈ 2(4 + 2 − 32 + 1.5s) = c + 3s − 56 (2010)

Hash Functions: lecture series by Ahto Buldas

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (12)

Ähnlich wie Hash Functions: lecture series by Ahto Buldas

Ähnlich wie Hash Functions: lecture series by Ahto Buldas (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Hash Functions: lecture series by Ahto Buldas