A short summary on clickstream analysis before google analytics.

1. Clickstream Analysis
Clickstream Analysis
Clickstream Analysis what is it useful for.
Collected and composed by Stefan Ziegler,
December 2001
Definition
Clickstream is a record of a user's activity on the internet, including every web site and every page of every web
site that the users visits, how long the user was on a page or site, in what order the pages were visited, any
newsgroups that the user participates in and even the email-addresses of mail that the users send and receive.
Both ISPs and individual web sites are capable of tracking a user's clickstream.
Clickstream data is becoming increasingly valuable to internet marketers and advertisers.
Be aware of the big amount of data a clickstream generates. These 'footprints' visitors leave at a site grown wildly -
large businesses may gather a terabyte of it every day. But the abilty to analyse such data hasn't kept pace with
the ability to capture it.
The next frontier of web data analysis is better integration of clickstream data with other customer information such
as purchase history and even demographic profiles, to form what's often called a quot;360-degree viewquot; of a site
visitor.
Clickstream analysis can be seen as a four-stage process of collection, storage, analyis and reporting. The first
two concentrate on gathering and formating information, and the latter two on making sense of it.
There are two levels of clickstream data analysis: Web traffic analysis, movement related, and commerce-based
analysis, which looks at e-business-related activities.
Web traffic analysis
Web traffic analysis operates at the web server level and concentrate on how visitors navigate through the site. It
measures the number of pages delivered to the customer as opposed to pages sent by the server. It determines
how often visitors hit the browser stop button, how much of the page was delivered until they hit the button and
how long they waited before they hit it. Performance parameters are also logged, such as length of time it took for
loading a page ad determining how much data was transmitted.
Commerce or e-business analysis can use higher level information out of clickstreams, such as tracking visitors'
responses to pages and their content. One of the main reason for measuring clickstream data at this level is to
analyze the effectiveness of the web as a channel to market. Measuring the success of commerce activities is
much more difficult than evaluating web traffic, because it looks at why visitors behave in a particular way, not just
where they went.
So for high-level clickstream analysis it is possible even to see the reactions of the customers. What items do
people buy and which they take out of their shoppig basket. This provides business-level information about how
visitors interact with the site which can be helpful to aid further site development.
With clickstream data more values can be gained by combining them with information from other sources like
direct marketing or sales. A direct mail campaign may be used to encourage customers to visit the web site. Now
the effectiveness of the mail campaign can be measured by collecting clickstream data from users who have been
sent mail shots and those who have not.
Web traffic analysis cycle creates a closed loop in which clickstream data is used to drive decisions to improve the
web site.
Gathering Data
The clickstream contains a record for every page request from every visitor to the site. But the clickstream data in
its raw form only gives some of the dimenions you need for powerful analysis. A raw entry in the page event log
from the web server gives back:
• Date/time of the page request
• IP address and possible cookie ID of the visitor (if they accept cookies)
1

2. Clickstream Analysis
• Page object being requested (the whole page or an object on the page)
• Type of request (almost alwaysquot;Getquot; or quot;Submitquot;)
• Context from where the page request was made (the so-called referrer)
• Browser version making the request (usually Netscape or IE)
Cleaning this low-level data up we get the following dimensions:
• Date of the page request
• Time of the page request
• Visitor
• Page object
• Request
• Session type
• Session ID
• Referrer
• Product/service
Let's have a more closely look at a few of these dimensions that have unique requirements related to clickstream.
Date/Time Dimensions
The date and time of a page request both must be expressed relatively to a single standard time zone. Merging
web logs from pysicaly seperated servers means aligning the clocks of these servers within less than a second.
Visitor Dimension
This is challanging beause you have three types of visitors coming to the site:
First, a huge pool of completely anonymous visitors identified only by their IP addresses. The IP address is only of
moderate value because it only identifies an outbound port on the visitor's ISP (internet service provider). These
ports may be dynamically reassigned, so visitors can't be tracked from session to session, or sometimes even
from within a session.
A second and more useful type of visitor is one who has agreed to store a provided cookie. This cookie then
becomes a reliable identifier for a visitor machine. With this cookie it is pretty sure that a given machine is
responsible for a session, and you can determine when the machine will come back again, assuming the user
hasn't deleted the cookie file.
Finally, the third and most valuable level of visitor is the human-identified visitor who not only has accepted the
cookie but somtimes in the past has revealed their name and other information. Being realistic, you may not be
certain that the same human being is sitting infront of the pc, but at least you know that person's quot;representativequot;
is there.
Page Object Dimension
This is one of the most importat information getting out of the clickstream data. Having useful data at the end,
every page must be described by more than its location in the web server's file system. It is a classic mistake to try
to use a file system both for uniquely locating files and describing their content. Instead, any given page must be
associated with a set of textual attributes that describe and classify the page, regardless of where it is stored in the
web server's file system or how it is generated. The attributes should be drawn from structured lists whose rules
the data warehouse team creates, so that the attributes can most usefully drive analyses. Some group needs to
take responsibility for assigning these attributes. This could be made by the web page designers or the webhouse
team, depending of who is paying attention to the needs of the webhouse analysis.
Session Type
This is the other important clickstream dimension. The session type is a high-level diagnosis of the complete
session. Plausible types quot;Product Orderingquot;, quot;Quick Hit and Gonequot;, and even more interesting diagnoses such as
quot;Unhappy Visitorquot;, or quot;Recent, Frequent, Intens Return Shopperquot;.
Looking at the web log file you get information about the files and objects which are transferred to the user and
when they are transferred. It is now the challenge building a session diagnosis tool. It is a blend of data extract,
pattern recognition, and link analysis.
2

3. Clickstream Analysis
The major point is to provide page objects and session dimensions for the clickstream. These dimensions are the
keys to analyzing web behaviour.
Example of Implementation a Data Collection Server with Java Script
Find enclosed a simple technical workflow on how clickstream analysis could be implemented with Java Script
technology. Here, the workload of generating the logs is transferred from the content web server to the data
collection server. Having load balanced servers running this implementation is much easier to maintain.
fig 1: Generating clickstrea data with Java Script
(1) Web server delivers content with attached instrumentation
(2) Instrumentation direct hits to Data Colection Server
(3) DCS validates and generates cookie and logs the hit
(4) If ciient is new, DCS deliveres gif file and cookie, others receive gif only.
(5) DCS generated logs are analysed
(6) OLAP reports from database
E-business feedback
The e-business analysis cycle is more sophisticated. This process combines web site activity with data from other
sources, such as visitor pofile information, sales databases and campaigns that include links to the web site. It
provides higher-level information, more focused answers ad information that can be used to enhance e-commerce
activities across the business as well as improving the web site.
The e-business cycle is a continual process, involving the integration of web and other data with web-site activity
data analysis, followed by improvements.
The integration of e-business and enterprise data with web traffic and other type of data allows discoveries and
insights that cannot be gained by observing web activity alone, and increases the potential for qualitative analysis.
3

4. Clickstream Analysis
fig.2: needed technology infrastrucure for complete e-crm
with clickstream analysis included
Attempting clickstream analysis it is important to differ between the two techniques tools on the market which are
used. Some analysis tools just report actions on web sites, while straightforward reporting tools will only log
actions.
A second consideration is whether the analysis tool supports real-time data feeds or uses a batch processing
model. Batch processing can only ever analyse historical data, and this lengthens the time between customer
actions and a firm's reaction. Take into this consideration that real-time data feeds are more in tune with the move
towards dynamic web pages, customer profiling and the use of personalisation engines. Real-time data feeds do
not restrict the company generating weekly or monthly reports, but can support real-time reporting, which can
speed up the decision making process when tuning the web site. However there are performance and bandwidth
issues associated with real time reporting.
Clickstream analysis automates much of the analysis process, bt even with the best tools, some human
intervention and analysis will be necessary, especially if the clickstream data is used in conjuncion with other data
sources.
fig.3: Dataflow for real-time analysis external sources included
For example, if site visits peak at a certain time on a particular day, the tool can readily recognize the spike but will
not necessarily discern the reason, which may be that a special marketing campaign ran just beforhand.
Building a 'Visitor Relationship Management'
Combining Clickstream Ananlysis and external data sources it is possible to create a 'Lifetime Customer' while
turning a visitor into a customer. Therefore the right actions must be taken. See fig.4 for more details.
4

5. Clickstream Analysis
fig.4: Creating a Lifetime Customer
Building a fully integrated VRM a closed circulation process is necessary. Beginning with gathering data,
integrating them in a database, doing reports and mining, with the result of having campaign management and
personalization.
fig.5: closed circulation process of VRM
Technology Details of a Scalable Architecture
Using a modular and open architecture you have total control to extend the base functionality to other systems and
customize the functionality.
5

6. Clickstream Analysis
fig.6: building a scalable architecture
Information and Content Exchange
Integrating clickstream data with data warehouses, legacy systems or external business partners is an important
part of clickstream analsis and is achieved using established extraction, transformation and loading tools. Another
solution might be the emerging Information and Content Exchange (ICE) protocol.
ICE was developed in 1998 by a industry consortium, uses HTTP as underlying communications protocol and a
standard set of messages for managing syndicaion using XML. In October 1998 ICE was submitted to the W3C for
official web standard status.
Who's Job is it?
The first critical task is deciding who should own the process. Some experts believe that marketing, as the primary
user of web-derived customer data, should own the analysis process, but it will likely be the IT department that's
charged with collecting that data. According to a report from Jupiter Research Inc. in New York, usage analysis
should be performed by those who are most likely to use the information. quot;While IT resources are certainly
necessary,quot; the reports says, quot;the IT group is too far removed from the business goals of usage analysis.quot; The IT
leaders interviewed for this story paint their clickstream data-analysis function as a equal partnership between
marketing and IT. The marketing folks set the vision, deciding where the business needs to go and what
information must be gathered and understood to go there. IT's role is to implement their vision. A human interface
understanding two of the worlds and being able to translate between the two quot;languagesquot; would be very helpful.
Decide whether to Outsource
The two biggest difficulties in effectively analyzing clickstream data, analysts say, are having too much information
and not enough people to analyze it. It's no secret that the amount of customer data available from business' web-
sites continues to increase. These days larger sites pull in several gigabytes a day, maybe terabytes. Hence, it is
important having a full time employee devoted to data collection and analysis.
But having no employee for doing this job, outsourcing takes that off the plate. Of course, outsourcing any data-
related function - collection, analysis or even storage - you give a third party access to information. And when this
information is customer-related, you're playing with fire. But with careful selection of partners and rigorous
application of standard security and privacy practices this can be overcome.
Future Outlook
In the next two years we should seek to do a better job merging clickstream data with information from other
sources to form a richer picture of customers. The clikckstream is fine but its use is limited if it stands alone. The
6

7. Clickstream Analysis
wave of the future is an integrated data-snapshot of a customer that includes clickstream data; previous
purchases, if any, not only from the web site but from other channels; the consumer's customer service; and
demographic data. That 360-degree view of customers and potential customers should be a high priority in the
forthcoming years.
The Privacy Path
One problem coming up with data gatherig and analyis is the privacy question. Problems caused by inadvertently
exposed customer data skyrocketed. Customer information data-sharing is going to be a big problem for
enterprises because of the increasing amount of information shared with service providers and because of the
extended relationships in the supply chain. Therefore you need to make a choice: Do you obey the letter of privacy
laws primarily to limit corporate liability? Or do you quot;take the high roadquot; and establish tougher policies than
required? Regardless of which path you choose, somebody in the business (a chief privacy officer or CIO) must
stay current on the array of privacy-related laws and regulations that are pouring forth from the federal and state
legislatures.
It's worth remembering that taking the high road may actually provide a competitive advantage in some
businesses, particulary those that are customer-oriented. quot;If customer don't trust you, they don't want you to know
them betterquot; (Arabella Hallawell, senior analyst at Stamford, Conn.-based Gartner Group Inc.).
7