1. Local Weather
Information
and
GNOME Shell Extension

2. Sammy Fung

3. Sammy Fung
● Software Developer
– to use and develop open source
sofware.
– Perl → PHP → Python.
– Interests in data mining / web
scraping.
– Consultant in web technology.

4. Sammy Fung
● 15+ years in Open Source Communities.
– Founding Chairman, Hong Kong Linux User Group.
– Founding Chairman, Open Source Hong Kong.
– Member, GNOME Asia committee.
– Mozilla Representative in Hong Kong.
– Organize, speak and participate in open source
conferences in East Asia and U.S. in recent years –
Taiwan, Korea, Japan, Malayisa, Singapore, and
Bay Area, CA.

5. Agenda
● Local Weather Information
– From Local Meteorological Observatories to Open
Data.
● GNOME Shell Extension
– Weather Widgets

6. Hong Kong Observatory

7. Weather Underground Hong Kong

8. Weather Information, China

9. Central Weather Bureau (CWB),
Taiwan.

10. Joint Typhoon Warning Centre
(JTWC)

11. Weather Data

12. Regional Weather Data, Hong Kong

13. Open Data

14. Open Data
Three Laws of Open Government Data by David Eaves.
1.If it can't be spidered or indexed, it doesn't exist.
2.If it isn't available in open and machine readable
format, it can't engage.
3.If a legal framework doesn't allow it to be
repurposed, it doesn't empower.
http://eaves.ca/2009/09/30/three-law-of-open-government-data/

15. Open Data
● Tim Berners-Lee
– the inventor of the
Web.
– 5stardata.info
● 5 star deployment
scheme of Open Data
suggested by Tim
Berners-Lee.

16. Five Star Open Data - 5stardata.info
1.make your stuff available on the Web (whatever format)
under an open license.
2.make it available as structured data (e.g., Excel instead of
image scan of a table)
3.use non-proprietary formats (e.g., CSV instead of Excel)
4.use URIs to denote things, so that people can point at your
stuff.
5.link your data to other data to provide context.
5stardata.info by Tim Berners-Lee, the inventor of the Web.

17. Open Data in Hong Kong

18. Open Data in Hong Kong
● Data.One
– http://www.gov.hk/en/theme/psi
– released on 2011/3/31.
– First App Competition on Data.One
● Call for Submission now till 2014/02/28.

19. Weather Information in Hong Kong
● Hong Kong Observatory
– Hourly Hong Kong Weather Report
– Regional Weather in Hong Kong (10 min updates)
– Weather Forecast and Weekly Weather Forecast
– Typhoon Report and Forecast

20. Hong Kong Observatory RSS

21. Hong Kong Observatory RSS

22. Weather at Data.One
● I posted a blog 'Progress of Open
Government Data in Hong Kong' on
2013/01/17.
● Weather at Data.One provides 7 dataset URLs,
returns RSS (XML) format (Eng/TChi/SChi)
– One word: Useless.
– Data.One dataset (RSS) is completely different
with HKO own paid service (XML).

23. Weather at Data.One
● Example - Current local weather report:
● Plain text report in RSS.
● Difference to quote report content:
– Website: a pair of HTML tags, eg. <PRE>....</PRE>.
– Data.One: a pair of RSS description tags,
<description>....</description>.
● Other weather data is missing, eg. Regional
temperture updates per each 12 mins.

24. Weather at Data.One
● Weather at Data.One is 'report' but not 'data'.
● Weather RSS is already released by HKO
before launch of Data.One.
● Technically, json/xml format is better
readable by computer programs.

25. Digital21 Strategy
Public Consultation Document
(G) Public Sector Information (PSI) as Default
"34. Through different channels (like press releases, publications, websites, etc.), the
Government releases a lot of information in different areas. However, most of such
information can only be read but cannot be used. In view of the immense benefits
of widening access to PSI for free and easy re-use, we propose to make all
Government information released for public consumption machine-readable by
default. Where appropriate, datasets will be released with application programming
interfaces (APIs), providing predefined functions to make their retrieval easier."
(G) 廣泛提供公共資料
"34. 政府透過不同途徑(例如新聞稿、出版物、網站等)發放大量不同範疇的資料。然
而,這些資料大都只可供閱讀而不能使用。有見開放公共資料以供免費再用可帶來巨大
效益,我們建議所有開放予公眾使用的政府資料都須以數碼格式編製。在適用情況下 ,
資料發布時會同時推出應用程式界面,以便提供預設功能, 讓公眾輕易地檢索資料。"

26. Digital21 Strategy
Public Consultation Document
"33. PSI datasets can be used and meshed together to create
innovative new applications, as demonstrated by the creative and
useful products and services developed from PSI in Hong Kong and
around the world. For example, using PSI datasets on traffic snapshot
images, a number of mobile apps have been developed to provide
real-time traffic situation for users to avoid traffic jams in planning
their traffic routes. Experience from other developed economies
shows that widening access to PSI datasets can open up lucrative
business opportunities and bring social benefits. By tapping the
creativity of the community and entrepreneurs, the use of PSI can
lead to positive social outcomes. For instance, in some cities in the
United States, application of PSI on hygiene inspections has led to a
significant drop in food poisoning incidents."

27. Digital21 Strategy
Public Consultation Document
"33. 由本港及世界各地利用公共資料所開發的實用創意
產品及服務所見 , 公共資料可個別及混合使用 , 以開發創
新的應用程式。例如 , 現時已有多個利用交通情況快拍
圖像的公共資料開發的流動應用程式 , 以提供實時交通
情況資料 , 讓使用者計劃行車路線 , 從而避開交通擠塞情
況。根據其他經濟體系的經驗 , 開放公共資料 , 供大眾廣
為使用 , 可開拓有利可圖的商機 , 並為社會帶來禆益。我
們可藉着開放公共資料 , 借助市民及企業家的創意來造
福社會。舉例來說 , 在美國一些城市 , 有關衞生檢查的公
共資料在開放使用後 , 食物中毒事故宗數大幅減少。 "

28. Digital21 Strategy
Public Consultation Document
"35. Apart from Government data, there are vast amounts of PSI
handled, collected and disseminated by public organisations,
which are equally useful for the development of innovative
services and products. Therefore, we propose to encourage
public organisations (e.g. public utilities and transport operators)
to release data owned by them in machine-readable format."
"35. 除了政府資料外 , 本港亦備有大量經公共機構處理、收集及發
放的公共資料 , 這些資料對開發創新服務及產品同樣有用。因此 , 我
們建議鼓勵公共機構 ( 例如公用事業及運輸機構 ) 發放以數碼格式編
製的資料。 "

29. Open Data in Taiwan
● Open Weather Data from CWB.
● Community – g0v.tw

30. ●Open Weather Data from CWB.

31. ●Open Weather Data from CWB.

32. ●Open Weather Data from CWB.

33. ●Open Weather Data from CWB.

34. g0v.tw
● Promote information transparency.
● Develop information platform and tools for a
society of citizen participation.
● Open Source model.
● Stackoverflow-like Q&A system for public to
asking for data which they are looking for.

35. g0v.tw
● Established after Taiwan Yahoo! Open Hack
Day in October 2012.
● Hackers, Professors, NGO/NPO, Students,
Writers, Visual Media, Legal Professionals.
● Organize 5+ bi-monthly hackathons since
December 2012.

36. Air Pollution Index

37. Air Pollution Index
● http://g0v.github.io/twgeojson/air.html
● Develop a web-based visualized map of air
pollution.
● Use Open Data provided by Environmental
Protection Administration
(opendata.epa.gov.tw)
● Air Pollution Indexes and Data from different
stations.

38. Moedict 萌典

39. Moedict 萌典
● Raw data from Ministry of Education (edu.tw)
● Community build of web-based chinese
dictionary with 160,000 Chinese items and
other items.
● Support auto-completion, searching and
offline versions.
● Source codes, other platforms, data are
available on 3du.tw (hackpad).

40. Why ?
Web Scraping

41. Programme List of Paid TVs in 2004

42. Programme List of Paid TVs in 2004
● I want to know live football match was
showing on which channel.
● Paid TV web site = M$ + IIS + ASP + Flash
● Slow....... Very Slow...... Extremely Slow!
● Couldn't connect at any peak hours!
● Wrote my first web crawler in PHP in 2004.

43. Public Transportation in 2006-2010
● Kowloon Motor Bus (KMB)
– No map view for a bus route
● Public Transportation Enquiry System (PTES)
– Exteremly Poor, Ugly (or much worse) map UI on
PTES.

44. HK Observatory and Joint Typhoon
Warning Center
● Any typhoon is coming to Hong Kong ? And
When will it come ?
● No easy data exchange format.
● No RSS nor ATOM.
● We aren't check websites everyday.

45. My Products
● WeatherHK ← ← ←
● TCTrack

46. WeatherHK
● http://twitter.com/weatherhk
● hourly current weather report
● weather forecast report
● tropical signal warning

47. WeatherHK
● Backend: Python + Scrapy + Database +
Twitter + NNTP......
● Frontend: Twitter + Newsgroup

48. WeatherHK
● http://twitter.com/weatherhk
● Interview by MetroPop in 2009.

49. My Products
● WeatherHK
● TCTrack ← ← ←

50. TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php
● Plot TC current and forecast tracks over
Google Map.
● Source:
– JTWC
– HKO

51. TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php
● Probably first tctrack map in HK using
GoogleMap
● Use of GMap: TCTrack -> Weather
Underground Hong Kong -> HKO

52. TCTrack
● http://twitter.com/tctrack
● Tweet JTWC updates for Northwest Pacific.

53. Web Scraping
● a computer software technique of extracting
information from websites. (Wikipedia)
● for business, hobbies, research purposes.

54. Web Scraping
● Look for right URLs to scrap.
● Look for right content from webpages.
● Saving data into data store.
● When to run the web scraping program ?

55. Use of
Open Source
Software in
Web Scraping

56. Use of Open Source Software in
Web Crawling
● Use Open Source Tools to collect useful and
meaningful machine-readable data.
● Doesn't need to wait provider to release data
in machine-readable format.

57. Open Source Tools
● Python programming lanugage
● with Regular Expression library
● Scrapy web crawling framework

58. Why python + scrapy ?
● python: my current favourite programming
language for few years.
● scrapy: web crawling framework written in
Python.

59. What is Scrapy ?
● An open source web scraping framework for
Python.
● Scrapy is a fast high-level screen scraping and
web crawling framework, used to crawl
websites and extract structured data from
their pages. It can be used for a wide range of
purposes, from data mining to monitoring
and automated testing.

60. Scrapy Features
● define data you want to scrapy
● write spider to extract data
● Built-in: selecting and extracting data from HTML
and XML
● Built-in: JSON, CSV, XML output
● Interactive shell console
● Built-in: web service, telnet console, logging
● Others

61. Starting new Open
Source project
hk0weather to create
Open Weather Data.

62. hk0weather
● Open Source.
● Web scraping HKO website.
● Output datas in standard machine-readable
data format – JSON, XML.
● https://github.com/sammyfung/hk0weather
● python + scrapy

63. hk0weather
● 1st version:
– from hourly weather report
– extracting temperture and humidity from 20+
weather stations, export in json format.
● 2nd version:
– From 10-minute update regional weather report.
– Including wind directions, wind speeds, max
gusts.

64. hk0weather

65. hk0weather
● Current version:
– Django web framework.
– Added rainfall data.
● Future:
– JSON APIs.
– Tropical cyclone data.
– Weather Warning Report.

66. hk0weather
● https://github.com/sammyfung/hk0weather
● $ virtualenv hk0weatherenv
● $ source hk0weatherenv/bin/activate
● $ git clone
https://github.com/sammyfung/hk0weather.git
● $ cd hk0weather
● $ pip install -r requirements.txt
● $ scrapy crawl regionalwx -t json -o /tmp/regionalwx.json

67. hk0weather
● Python
– import re
● Scrapy
– web crawling framework written in Python.
– HtmlXPathSelector.
– built-in JSON, CSV, XML output.

68. hk0weather
{"reptime": "2014-05-23 09:20:00", "temperturemax": 25.2, "ename": "The Peak", "scraptime": "2014-05-23 09:27:15", "station": "vp1", "cname": "u5c71u9802",
"temperturemin": 24.1, "temperture": 25.0},
{"reptime": "2014-05-23 09:20:00", "temperturemax": 26.3, "ename": "Ta Kwu Ling", "maxgust": 15, "scraptime": "2014-05-23 09:27:15", "humidity": 94, "windspeed": 8,
"cname": "u6253u9f13u5dba", "station": "tkl", "temperturemin": 25.0, "temperture": 25.8, "winddirection": "East"},
{"reptime": "2014-05-23 09:20:00", "temperturemax": 27.9, "ename": "Lau Fau Shan", "maxgust": 33, "scraptime": "2014-05-23 09:27:15", "humidity": 90, "windspeed":
21, "cname": "u6d41u6d6eu5c71", "station": "lfs", "temperturemin": 26.4, "temperture": 27.1, "winddirection": "South"},
{"reptime": "2014-05-23 09:20:00", "temperturemax": 27.6, "ename": "Peng Chau", "maxgust": 27, "scraptime": "2014-05-23 09:27:15", "humidity": 99, "windspeed": 13,
"cname": "u576au6d32", "station": "pen", "temperturemin": 26.8, "temperture": 27.6, "winddirection": "South"},
{"reptime": "2014-05-23 09:20:00", "ename": "Tap Mun", "maxgust": 6, "scraptime": "2014-05-23 09:27:15", "windspeed": 2, "cname": "u5854u9580", "station": "tap",
"winddirection": "North"},
{"reptime": "2014-05-23 09:20:00", "temperturemax": 28.7, "ename": "Shek Kong", "maxgust": 8, "scraptime": "2014-05-23 09:27:15", "humidity": 96, "windspeed": 1,
"cname": "u77f3u5d17", "station": "sek", "temperturemin": 26.6, "temperture": 27.4, "winddirection": "Variable"},
{"reptime": "2014-05-23 09:20:00", "temperturemax": 28.3, "ename": "Tsing Yi", "maxgust": 28, "scraptime": "2014-05-23 09:27:15", "humidity": 96, "windspeed": 18,
"cname": "u9752u8863", "station": "ty1", "temperturemin": 26.4, "temperture": 27.4, "winddirection": "N/A"},
{"reptime": "2014-05-23 09:20:00", "ename": "Green Island", "maxgust": 42, "scraptime": "2014-05-23 09:27:15", "windspeed": 35, "cname": "u9752u6d32", "station":
"gi", "winddirection": "South"},
{"ename": "Sai Wan Ho", "cname": "u897fu7063u6cb3", "station": "swh", "scraptime": "2014-05-23 09:27:15", "reptime": "2014-05-23 09:20:00"},
{"reptime": "2014-05-23 09:20:00", "temperturemax": 28.4, "ename": "Shau Kei Wan", "scraptime": "2014-05-23 09:27:15", "station": "skw", "cname":
"u7b72u7b95u7063", "temperturemin": 25.7, "temperture": 27.9},
{"ename": "Tai Mo Shan", "cname": "u5927u5e3du5c71", "station": "tms", "scraptime": "2014-05-23 09:27:15", "reptime": "2014-05-23 09:20:00"},
{"reptime": "2014-05-23 09:20:00", "ename": "Tsuen Wan Shing Mun Valley", "scraptime": "2014-05-23 09:27:15", "humidity": 98, "cname":
"u8343u7063u57ceu9580u8c37", "station": "tw"},
.
.
.
.
.

69. GNOME Shell
Extensions

70. GNOME Shell
● Core user interface functions for GNOME
– 2 Screenshots followings after this slide.
● Switching to windows
● Launching applications
● Panel at the top of the screen
● the Activities Overview
● Message Tray at the bottom of the screen.

73. GNOME Shell Extensions
● Small pieces of code
● Written by third party developers
– That's mean they could be most of you!
● Modify the way GNOME works.
● Similiar: Chrome Extensions or Firefox Addons.
● Extensions can be found and installed from
extensions.gnome.org.

74. What can GNOME Shell Extensions
do ?
● Extensions may make small changes.
– like moving your clock to the right-hand side of
the screen
● Or make big changes
– like arranging the windows in the Activities
Overview in a different way.

75. Weather related GNOME Shell
Extensions

77. Installation of GNOME Shell
Extensions
● "GNOME Shell Integration" plugin is installed
and enabled in your browser preferences
● Go to extensions.gnome.org, find and install.
– Whitelisted this website or turn off the click-to-
play feature at your browser.
● Make sure Unzip is installed.

78. Weather (by Neroth)

79. Weather (by Neroth)
● A simple extension for displaying weather
information from several cities in GNOME
Shell
● https://github.com/Neroth/gnome-shell-
extension-weather

80. Weather (by Neroth)
● Let's look at its codes.
– Data come from Yahoo Weather.

81. OpenWeather (by jens)

82. OpenWeather (by jens)
● Weather extension to display weather
information from OpenWeatherMap for many
cities in GNOME Shell.
● https://github.com/jenslody/gnome-shell-
extension-openweather

83. OpenWeatherMap

84. Developing next Weather Widget
● With data from cities/town-level weather
stations.
– Yahoo Weather ?
– OpenWeatherMap ?
– Open ''Weather'' Data ?
● From Observatories
● Web Scraping

85. Thank You!
sammy@sammy.hk
http://sammy.hk