Running Internet Systems in China

Running Internet Systems in China
Details you need to succeed in Chinese Internet Operations
White Paper

Running the World's Internet Servers
February, 2013
Table of Contents
Overview.......................................................................................................................................................................... 3
Challenges.................................................................................................................................................................. 3
ChinaNetCloud......................................................................................................................................................... 3
Country Profile.............................................................................................................................................................. 4
Internet size.............................................................................................................................................................. 4
Mobile & Tablet Growth....................................................................................................................................... 4
Demographics.......................................................................................................................................................... 5
Cities by Tier............................................................................................................................................................. 6
Infrastructure & Challenges..................................................................................................................................... 7
ISP/Carriers.............................................................................................................................................................. 7
Not one company.................................................................................................................................................... 7
Data Centers.............................................................................................................................................................. 8
IDC Connectivity...................................................................................................................................................... 9
IDC Service Quality.............................................................................................................................................. 11
CDNs.......................................................................................................................................................................... 11
Hardware................................................................................................................................................................. 11
Clouds ........................................................................................................................................................................... 12
Hosting Outside China........................................................................................................................................ 13
Website Monitoring............................................................................................................................................. 13
Security Issues....................................................................................................................................................... 14
Hackers..................................................................................................................................................................... 14
DDoS.......................................................................................................................................................................... 14
IDC Local Attacks.................................................................................................................................................. 15
Physical Attacks.................................................................................................................................................... 15
Regulatory............................................................................................................................................................... 16
Licensing / ICP...................................................................................................................................................... 16
Website Development Services............................................................................................................................ 18
About ChinaNetCloud............................................................................................................................................... 19
ChinaNetCloud - Running Internet Servers in China [DRAFT] Page 2

Overview
China is today home to the world's largest Internet user population, over 500 million, mostly young,
urban, and with discretionary income to spend. They have iPhones, play games, write microblogs,
and connect with friends from all over. As a result, they are the very face of the future of the
Internet, in China and beyond.
As a result, more and more large-scale Internet sites, games, apps and services are springing up to
serve the needs of the people. While strongly local in nature, there are a large number of Western
and Global brands with a strong presence, in e-commerce, gaming, mobile, and more. And this
number is increasingly quickly as both local and global companies race to take advantage of the
growing market.
Challenges
As exciting as China's Internet scene and opportunities are, it's not quite so simple to build and
operate Internet operations here. There are a wide variety of regulatory, legal, infrastructure,
connectivity, cultural, language, and other problems that are not common outside of China. In
addition, there are many misconceptions about the very nature of how the Internet works and how it
is managed or regulated in today's PRC. All of these challenges and often do trip up or complicate
otherwise well-planned projects.
ChinaNetCloud
ChinaNetCloud can help. As China's leading Internet Server Management company and a world-
class managed service provider, we are experts on all aspects of Chinese Internet operations. We
routinely help companies of all nationalities, types, and sizes to design, build, and operate large-
scale systems successfully reaching hundreds of millions of Chinese users, every day.
This brochure describes the modern Chinese Internet, its structure and challenges, and how to
succeed on-line in the world's most populous country.

Country Profile
Internet size
According to China Internet Marketing research data by Nanjing Marketing Group, by the end of
December in 2011, There are 513 million Internet users in China, 56 million more than the previous
year. That's about the same as the total population of USA, Canada, Australia, Germany and the
UK. There are 136 million rural Chinese Internet users.
Mobile & Tablet Growth
China's mobile and tablet growth is mirroring the west, with Apple's iPhone and iPad leading the
way with millions of sales each. In addition, Android is extremely popular in China, especially with
the introduction of low-cost tablets and smartphones, putting world-class power and applications in
the hands of nearly everyone in the coming years.

Demographics
According to Nanjing Marketing Group, there are 356 million mobile Internet users in China. 69%
of China's Internet users use mobile devices to access the Internet.
27.3% of Chinese Internet users are under the age of 20, 30.8% are between 20 to 29 years old,
23.2% are between 30 to 39 years old, 11.4% are between 40 to 49 years old, and 4.8% are above
50 years old.

Cities by Tier
One important and unique feature of China is the very large number of large cities, with various
statistics placing over 150 cities over 1 million people. These cities are often described in 'Tier's
such as Tier 1, 2, 3, and 4 with some common lists such as Tier 1 cities including Beijing, Shanghai,
and Guangzhou while Tier 2 includes Hangzhou, Nanjing, Xiamen, Qingdao, Chengdu, Dalian, and
so on. Understanding where your users or customers are is very important to your connectivity
strategy in China, including if you include students, rural or western region populations, etc.

Infrastructure & Challenges
China has an interesting Internet architecture and infrastructure situation. In most ways, it's just like
the U.S. or any other large countries or regions. But in other ways, it's quite different with some
surprising challenges that can trip up the newcomer, local or foreign. These challenges are
especially a problem for the highest tier of system, such as luxury e-commerce or complex national
MMORPG, since achieving premier performance at low, or even any, price is an ongoing challenge.
ISP/Carriers
The first and the most obvious challenge which even foreigners are often aware of is the so-called
'north-south' split. Basically, most areas of China have only one single dominant ISP or carrier for
all local Internet access.
In the north, including Beijing, Tianjin, and the important provinces of Shandong, Shaanxi, plus the
whole northeast, there is China Netcom (merged with China Unicom). In the southern areas,
including Shanghai, Guangzhou, Nanjing and over 50% of the country's population, there is China
Telecom.
Other ISPs do exist, such as Railcom and others, though they generally handle the western and rural
areas, so reaching them effectively is only interesting to specialized brands or systems such as farm
information, migrant worker recruiting, and some types of video or gaming.
In addition, the university network, called CERNET, is separated and actually quite hard to get good
service to without special provisions. College students are of course a valuable demographic, so
efficiently reaching this separate network is a priority for some brands and site operators. However,
this is not easy and it takes special attention using the best-connected data centers and CDNs.
To further complicate things, China has the world's largest mobile Internet population, but one that
is also separated and difficult to reach. Most people are on China Mobile, but most iPhone users are
on China Unicom, while Telecom is making inroads in some areas. Historically these mobile
carriers were totally separated and throttled bandwidth, though recent mergers and re-organizations
have complicated this picture.
Not one company
Complicating matters is that even in a given region, such as the north or south, the Carrier is not a
single entity, but is made up of a large company for each province and often each city. For
example, in China Telecom's area in the south, the properous province of Zhijiang has a China
Telecom carrier company, but the lovely capital city of Hangzhou has its own, separate, China
Telecom company. And even though these companies really don't directly compete for business,
they don't always get along nor cooperate all that well.

Data Centers
All Internet servers are housed in data centers, called IDCs in China (for Internet Data Center).
These come in all sizes and types and are extraordinary diverse, from very small few hundred
square meters to what would be considered a small-to-medium 10,000-square-meter IDC in the
West.
Data Centers are generally very quite small and severely power-limited. In fact, the available power
feeds is almost always the determining factor in IDC size. Also many IDCs are quite urban, often
housed in part or all of a floor of an old office building, though many newer green-friend facilities
are coming on-line, so actual data center quality and environment vary widely in size, quality, and
price.
A key result of power limits is that per rack power is severely limited in China, which in turn limits
the number of servers per rack. Typical power is about 2,000 watts per rack, and 1,800-2,500 is by
far the most common. This limits racks to about 12-15 servers or 1U devices such as switches,
which is far less than it is seen in the west.
Thus racks and data centers often appear empty or under-utilized but are in fact fully-loaded. This
influences the IDC costs, since system must be spread out over more racks than many companies
expect, raising hosting costs.
In addition, the small size and limited power of IDCs severely limit expansion for any given
customer. It's very common for popular IDCs to be fully booked and adding an extra rack ranges
from not possible, to adding on another floor (and LAN), or having to move data centers.
Blades and other high-density data center technologies are simply not seen in China. Even large
storage or networking equipment such as 4U SANs or Cisco 6500 switches are often limited to one
per rack.
The vast majority of IDCs are owned by the carriers, especially China Telecom. However, you can't
easily rent space directly from them as these are mostly wholesale operations. So the carriers lease
large blocks of space to smaller IDC companies, from the large 21ViaNet and ChinaNetCenter to
very small fly-by-night one-man organizations.
Private IDCs are quite rare, and most 'IDC Companies' are in fact re-selling space in carrier IDCs,
though many have their own connectivity. This is a complex regulatory and legal area with various
dynamic requirements, so the future structure is still not clear, though the top few companies are
doing relatively well.
Note that enterprise IDCs are quite different and have very different characteristics. In this area,
foreign and Hong Kong data center companies like AT&T, NTT, and others are quite active with
high-end services focused on MNCs.

IDC Connectivity
Generally an IDC's most important asset is its connectivity to the Internet, and this varies widely in
quality and price. It is critically important to understand where and how the IDC is connected,
including to what carriers and via what type of routing.
The simplest and cheapest option is a single line IDC, which is connected to the local carrier, such
as Shanghai Telecom in Shanghai. This can work well, but users in other parts of China will
generally have poor performance, for example Beijing users will not enjoy a site connected only in
Shanghai.
The next level up is called dual-line, where the IDC has lines to their local carrier plus a purchased
line to the 'other' carrier. So if the IDC is in the south with Telecom, it will also have a 'north' line to
Netcom. This is most common in the large city of Tianjin which is popular as it seems to have good
southern Telecom connections in addition to its local northern Netcom.
There is a catch, however. Dual-line systems have two IPs, as they are not using the BGP protocol
for routing, as would be common in the rest of the world. BGP is not common in China, though this
is changing, see below. This dual-IP method, which disappeared from the West 10-20 years ago,
means that each public device such as web server or load balancer, must have two (or more) IPs and
properly handle the default routes. This is complex even on Linux systems, and not even supported
on some hardware systems. Plus, geo-DNS is needed to correctly send users to the right IP address
for their region and carrier. This type of system was very common until recently and is gradually
falling out of favor as China slowly moves to BGP.

BGP is the obvious solution to this, but is still uncommon in China, due to cost and the required
inter-carrier cooperation requirements. But this is slowly changing and in 2011-2012 the number of
BGP-based systems has greatly increased, though still only at the high end. And some systems
advertised as BGP are actually pseudo-BGP with special routing buried in the IDC core routing
system to make things look correct.
The best IDCs, such as 21ViaNet and the Aliyun cloud, offer what they call 8 line BGP, which are
western-style systems with full BGP to all eight carriers in China, including Telecom, Netcom,
Unicom, Mobile, CERNET, Railcom, and the rest. These are the best solution, but are very
expensive. So much so that they can only be used for core interactive systems and all static assets,
especially video, etc. should be handled in CDNs.
Chinese IDCs have many unusual service restrictions or challenges, such as not allowing 24 hour
access or requiring 24 hours notice before entering, almost unheard of in the West. In addition,
some IDCs don't have 24 hour staff or the staff quality is quite poor and only reached via mobile
phone off-hours.
In addition, some lower-tier IDCs have no patience for difficult or demanding customer and will ask
you to leave, especially if you suffer from hacker DDoS attacks, too many legal or other issues, etc.
Note also that a few IDCs do not allow foreigners to enter, as they host sensitive or government
systems. This can be a problem for startups or other organizations with non-Chinese engineers or
management, but is not a common problem.
Note also that during sensitive periods, such as the 2008 Olympics or 2012 government leadership
transition, all movement of server hardware and IDC access is severely restricted. For example for
six months in 2008 and about a month in 2012, no servers could move in or out of IDCs, and in
most cases, engineers couldn't make any changes to servers, cables, or even touch servers, even if
they crashed, failed, or suffered outages. While not common, such restrictions do happen.
Some people worry about power quality and reliability in China, but generally this not a problem as
power is reliable as in any U.S. city, and power-related IDC issues are very rare. In addition, IDCs
do maintain standard UPS and generator-based alternative in line with their general service and
SLA levels. Power issues are not a major concern.
Connectivity issues can be more of a problem, especially for IDCs in heavy construction zones
where cable cuts can and do occur, with occasional outages. Proper routing and HA methods can
help, but some IDCs have a bad reputation in this area.

IDC Service Quality
Many factors influence an IDC's service quality, but connectivity, 24 hour access, and sufficient
remote hands support are mandatory for a good IDC. Added value items such as DDoS protection,
dual uplink drops, added power per rack and similar items can make them even more attractive.
CDNs
Given China's infrastructural and geographical challenges, it's not surprising there are several CDN
services to help get the data where it needs to go. Foreign companies are mostly prohibited from
this type of service (Akamai works through local partners), so several Chinese firms dominate,
including ChinaNetCenter, ChinaCache, and FastWeb (recently bought by 21ViaNet).
CDN servers are fairly standard and similar to what is available in the west, though more limited in
feature set. All offer image and static object caching, whole-site-acceleration, and various video
streaming services. Other common services that are less common in the west include large file
downloads with restart (for large game downloads), anti-link stealing, and various media and video
download prevention methods.
Generally, all websites in China should utilize one or more CDNs, if only to get the images and
static objects out to users, especially for mobile and university end users. Best practices are of
course necessary for successful CDN utilization, most importantly good cacheability settings and
renaming objects on update.
Hardware
Server, network, and other Internet infrastructure hardware is plentiful in China and very easily
obtained from all the global brands. Dell dominates the server marketplace with all products
available in a few days from numerous distributors and Dell itself (which makes all the servers in
Xiamen, Fujian Province). HP is the clear number two, with IBM and others far less common in the
Internet market. Cisco is the leader in networking equipment, though Huawei and others are also
common.
As noted previously, power is a major concern, so efficient 1-2U servers are by far the most
common, and blade equipment is very rarely seen. SANs are not common in the Internet space,
though NetApp Filers and other storage systems are often used in large systems. F5 and Netscaler
dominate the high-end load balancer area. All these brands are heavily supported by distribution
networks in major cities.
Most hardware is purchased outright, though a few IDCs offer lease arrangements which can of
course reduce the impact on small company cash flow. Like all such arrangements, the terms are
important, including what happens to the equipment at the end of the lease and how easily the lease
can be ended early if things are not going well. In the best cases, you can walk away by returning
the equipment and the loss of a modest deposit.

Clouds
Cloud Computing is of course increasingly popular around the globe, though still quite young in
China. In particular, top global players such as Amazon and Rackspace do not have systems in
China and their international systems are too slow to be useful inside the country.
After a few false starts, there are now a few local players, most importantly Aliyun, the cloud
service of e-commerce giant Alibaba. Their goal is to be the Amazon of China and their services
are both patterned after, and often compatible with, Amazon's. Their services are still only a subset
of those available internationally, but they appear quite serious and are rapidly expanding and
improving.
A handful of other useful services are becoming available, usually with standard basic IaaS virtual
servers, occasionally with Amazon S3-style cloud storage. Some of these clouds have strong
backers but weak sales/marketing skills, while others are small startups but are aggressive and
offering interesting services.
Private Clouds are becoming increasingly common, where a number of powerful physical servers
are purchased and virtualized into several or dozens of VMs to run load balancers, web servers,
memcache, search engines, databases, and more. This is the dominant deployment method for
modern large-systems in 2012, though the expertise to design and build these remains in short
supply.
The whole cloud industry has grown in 2012, and 2013 will probably be the year that momentum
really shifts for new small system development, with a very significant portion heading to Aliyun
and other similar firms.

Hosting Outside China
Many global companies want to host their global content or e-commerce systems in only one place,
such as Amazon U.S., and to serve the whole world, including China.
Unfortunately, this is not possible with decent performance. China's international links are large
and well-connected, but are often overloaded and in the end, congested through a small number of
ocean cables to the U.S., Japan, etc.
Performance is often okay, almost tolerable, until the day or week it's not, for no good reason. The
trans-border performance and reliability is simply not sufficient for a real business to serve users in
China. In addition, due to the regulatory challenges, it's possible to be blocked for a variety of
reasons, including because your system is at the same foreign data center as one hosting prohibited
content.
As a result, companies that serve Chinese users must have a system inside China. If there are also
international users and the data or system can be split, then two systems are needed, one inside the
PRC and one outside.
If Chinese domestic and international users must share the same system, such as for trade or other
systems, then a single system is usually best placed in Hong Kong, which has good connections to
both. However, even Hong Kong is susceptible to periodic outages or slowdowns, though CDNs
and careful monitoring can help manage and mitigate these issues as much as possible.
Website Monitoring
A variety of services are available for monitoring a website's status and performance across China.
The global brand Gomez is available and has a variety of services, along with the local firm
Network Bench, which is especially good for monitoring local CDNs and per-province, carrier, last
mile details nationwide. These services are quite costly, so choosing monitoring level, cities, and
frequency is important for good cost control.

Security Issues
Security is a serious concern in the Chinese Internet, though probably less so than in the West.
While hackers and other malicious activities certainly happen, they are not yet nearly at the rate nor
sophistication of international attacks, yet. This is changing as the Chinese consumers and their
information, along e-commerce fraud, become more valuable.
More broadly, the consumer PC area has extraordinarily poor security and many, if not most, PCs
are thoroughly infected, often by dozens of things. Home firewalls are not that common and
consumer awareness of these issues is surprisingly low.
This is made worse by many, if not most, users using illegal version of Microsoft Windows and
applications, and never updating them, even if they could. So numerous security holes and issues
are never addressed, making these systems very vulnerable.
Privacy issues are also not of great concern to end consumers, further lowering their awareness and
instincts in protecting themselves and their data. Not to mention password management and other
basic measures are very poor. Thus there is often little need to steal data from server systems when
the end users are themselves so vulnerable.
Note that on-line banking and related systems in China do account for this and have very tough
security practices and processes, almost always involving hardware tokens, mobile phone SMS
codes, and more to ensure safety. This is especially important in a country with no checks, and thus
most payments are done electronically, usually on-line, and thus a much higher risk profile and is
common, for example in the U.S.
Hackers
Bad guys attacking servers are certainly active and often attacking back-end infrastructure, via a
variety of methods. SQL injection attacks are quite common, as are uploading malicious images,
etc. These attacks are made worse by poor coding practices, very limited testing, rapid
development cycles, and a priority on features over security. For example, large systems still often
have no hashed passwords, let along best practices salted hashes.
DDoS
The most common types of attacks are DDoS which in China can reach very large sizes, of tens of
Gbps or more. These appear to be run for fun, for ransom, and as anti-competitive methods. The
ransom versions are most dangerous as considerable resources are often in play and they are hard to
mitigate.

In particular, most IDCs have no defenses at all and the regional carriers do not cooperate on DDoS
to nearly the extent seen in the West. In addition, more sophisticated techniques like black-hole
routing are not very common. IDCs will often just terminate and not route any customer IPs and if
a common problem, terminate the hosting contract.
The best IDCs do have DDoS protection hardware, routing control, and carrier relationships to
mitigate all but the largest attacks, though some of these such as ChinaNetCenter, require the use of
their CDN services.
Recently security services such as AnQuanBao have come on line to further help manage these
attacks. These are effectively CDNs with distributed nodes and large-scale bandwidth, able to
withstand the largest DDoS attacks. They also have other benefits such as virus, XSS, and other
problem detection and prevention.
IDC Local Attacks
Some attacks are actually hijack attacks that take place inside the IDC on shared bandwidth links,
often via ARP. Hacker servers will send gratuitous ARPs to the switches so the default gateway
points to them, thus all traffic from your server goes through them first.
Their server then injects evil javascript to point to a virus or other bad code so the user will
download it and become infected. Since user PCs are so vulnerable, this is a very effective process
in China. It's also difficult to detect and nearly impossible to mitigate against without special tools.
Physical Attacks
It's theoretically possible for bad guys to physically attack systems, such as stealing whole servers
and their disks filled with data. This is extremely rare in China, so much so that it's not a real
concern. Of course, best practice is to encrypt very valuable data such as credit cards, but having
servers walk out the door is not near the top of the list of worrying issues.

Regulatory
China has a somewhat complex regulatory environment for Internet companies, though contrary to
international perceptions, it's quite clear how things work, including what is possible and what is
not allowed.
There are two general areas of concern: Website licensing and website content controls.
Licensing / ICP
All websites and indeed any system serving data over HTTP are required to register with the local
government where their company is located. This is a free and relatively simple process, and
typically takes about a month. Most IDCs and cloud providers will help process the filing, though
for foreign companies, it can only happen after the subsidiary (WOFE) is formed and has its
business license.
A basic registration, known as an ICP Beian, or Internet Content Provider Registration, is available
to both local and foreign companies, but is only good for a site showing basic information such as
sales or company / product details. In other words, a standard website, but one which does not fall
into a number of special categories, discussed below. For a typical company site that does not sell
anything, this is all you need and is easy to get.
Life becomes more complex when special categories are involved, which is a complex area, but
basically including any exchange of money such as e-commerce, any games, any news, and most
video, music, and media sites.
E-Commerce and games are the most common types of systems seen, though there are other
categories, too, but these all require an ICP license and have additional requirements, including
financial. Some also need additional special licenses or approvals, for example video sites like
YouTube need approval from SARFT, and games need approval from the Ministry of Culture
(MoC).
Note that mobile applications, APIs, and sites that feed data through other websites such as Renren
(the Facebook of China) are a gray area and require negotiation with the IDC and perhaps local
authorities.
Website Content Controls and Moderation
Part of the basic registration and data center agreement is a provision that all websites and content
providers will not have prohibited content on their systems. This is the controversial aspect of the
regulatory framework from a Western perspective, as it prohibits content that would often be
allowed elsewhere.

Broadly, for most sites, this is not a significant challenge, but it's easy to run into problems when
user-generated content is allowed, for example when users can comment on products, or tell others
about themselves and their interests. Bad-behaving users will exploit this opportunity to upload and
write all kinds of problematic content.
By far the most common prohibited content is pornography, which is illegal in China, so people will
upload porn picture or stories in as many places as they can find. The second biggest problem is
often illegal products such as guns or drugs that get listed with an anonymous phone number. Note
that political, news, and other topics which can be sensitive are actually rarely problems for typical
websites.
Most government regulators and especially IDCs have very little tolerance for any of these types of
problematic content, and will often just turn off the offending servers until the problem can be
fixed. The best IDCs will call the customer and give them until 5pm, etc. to take care of the
problem.
As a result it's very important for all sites that allow user-generated content to carefully monitor and
moderate such content. Ideally, everything is reviewed before it goes live and if this is not possible,
have methods to quickly report bad content so it can be quickly examined and removed, if needed.
Beyond this, specific industries have other requirements, including recently-issues real-name
requirements for blogging sites, and user access log retention policies on other sites, for use by law
enforcement.

Website Development Services
China has a number of good website development companies, ranging from large international
outsourcing firms to small local teams that can build things quickly and inexpensively.
Most focus on particular areas, such as e-commerce, social marketing, Drupal, advertising, and
other specialties. Others also focus on geographies, such as Shanghai or Beijing, which can be
important for the necessary in-person meetings.
Like web development companies everywhere, employee retention can be a problem, as can code
quality and overall staff experience. This is especially a problem for operations in China, as these
teams often have little knowledge of performance and security-related issues and are no longer
working on the system by the time these become problems months later. In particular, performance-
related issues in SQL queries or database design are a challenge to fix once a system becomes
popular and fails under heavy load.
Also, many developers are not overly familiar with horizontal scaling architectures that includes
such requirements such as memcache-based sessions, data caching (and related coherency), DB
read/write splitting, DB sharing, mail sending scale, and simple load testing.

About ChinaNetCloud
Founded in Shanghai in 2008, ChinaNetCloud is the world's largest Internet Managed Services
company, focused on server operations, especially Reliability, Performance, Scale, Security, and
Cost Savings.
Focused on large-scale systems, ChinaNetCloud helps customers design, build, and operate any
Internet-facing system and service, up to hundreds of millions of users and billions of page view per
day.
Global Headquarters
1238 Xietu Lu, 1-601
Xuhui District
Shanghai, China 200032
Beijing Office
9nd Floor - Sunjoy Mansion
No.6 Ritan Road
Chaoyang District
Beijing, China 100020
Hong Kong Office
9/F, Cambridge House
Taikoo Place
979 King’s Road, Quarry Bay
Hong Kong
+86 (21) 6422-1946
Sales@ChinaNetCloud.com
www.ChinaNetCloud.com

Running Internet Systems in China

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Running Internet Systems in China

Ähnlich wie Running Internet Systems in China (20)

Mehr von Casey Lau

Mehr von Casey Lau (16)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Running Internet Systems in China