In this ppt I will talk about how I hack freelancer.com and another travel web site to get here API so I can crawl information from them.

1. API DESIGN
BEST PRACTICES
FROM A HACKER’S VIEW

2. • Overview
• Stories
• Crawl all projects and bids from Freelancer.com
• Crawl 6 billion flight ticket price from a travel web site
• Summary

3. MONOLITHIC APP
• Hide system information inside app
• No internal sys call is exposed to outside

4. MICROSERVICE APP
• Hackers know your system better
• Service calls are exposed to user
• RESTful API as standard,easyto guess
• Need to consider security between every
service

5. TRIDITIONAL MODERN
XPATH
WEB PAGE API
Pure Data

6. STORY 1
CRAWL FREELANCER.COM

7. FREELANCER.COM
8M
Project Information
Bid Information

8. • Reputation and price, which is the most important factor for a success bid?
• How canI get most chance to be awarded when bidding for Australia employer?
• Should I put a lowest price or should I do more project to earn reputation

9. HOW CAN I GET THE INFORMATION AS FAST AS POSSIBLE?

10. https://www.freelancer.com/projects/Javascript/Web-Page-Scraper/

11. • Need a HTML parser and javascript executor
• Heavy work for both cpu and bandwidth
• Not easy to iterate through all the projects

12. TIP: MOST OF THE TIME
MOBILE SITE IS MUCH EASIER
TO GET INFORMATION

13. https://m.freelancer.com/projects/Javascript/Web-Page-Scraper/#info

14. RESTFUL APIS

15. https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page-
Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upg
rade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&us
er_status=true
https://www.freelancer.com/api/projects/0.1/projects/9844976/bids/?compact=true&limit=20&offset=0&reputatio
n=true&user_avatar=true&user_details=true
https://www.freelancer.com/api/projects/0.1/projects/9844976/?compact=true&full_description=true&job_details=
true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_
employer_reputation=true&user_reputation=true&user_status=true
https://www.freelancer.com/api/projects/0.1/projects/${id}/?compact=true&full_description=true&job_details=tr
ue&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_
employer_reputation=true&user_reputation=true&user_status=true

16. API Rate Limit
1000 / HOUR
8M / 1k = 8k HOUR = 333 DAYS

17. 172.246.149.100
216.219.130.171
95.227.99.197
17.124.253.149
137.238.189.207
34.155.214.35
• Number of threads depends on how many proxies you have
• Https proxy is hard to find
• Proxies are unstable
• Proxies will be used out quickly
• High cost if you buy proxies
WORKAROUND : USE HIGH ANONYMOUS PROXY

18. 160.124.89.71
13.193.36.236
182.3.152.44
85.72.136.122
……
• Loads of IPs, canbe changed every 10s
• High quality socks proxies across the world
• Able to use docker to start 10 tor clients in 1 minute
WORKAROUND: USE TOR NETWORK

19. USING THESE HACKS
I MANAGED TO GET ALL THE PROJECTS AND BIDS IN 10 DAYS
USE A SINGLE DIGITALOCEAN 5$ SERVER

20. WHAT DO I LEARN?

21. API Rate Limitation Mobile API

22. Easy to guess filters Predicable URL
https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page-
Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&
upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=tr
ue&user_status=true
Information leak

23. HOW CAN WE FIX THEM?

24. ONLY SUPPLY INFORMATION CLIENT NEEDS

25. MAKE SURE URL IS NOT PREDICTABLE
https://www.freelancer.com/api/projects/0.1/projects/UUID/?compact=true&full_description=true&job_details=tr
ue&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_
employer_reputation=true&user_reputation=true&user_status=true

26. REDUCE ANONYMOUS NETWORK ATTACK
• If your customers are in AU only, restrict
access when IP address is outside AU
• Set different limitation based on location
• 1k/h API usage • 100/h API usage
• Captcha to verify human

27. STORY 2
LEARN FROM CAWLING FLIGHT TICKET PRICE

28. How many days ahead do I need to get a cheapest price?
I need to crawl as many flight ticket price and analysis.

29. FIND API FROM MOBILE PAGE

30. data=%7B%22searchType%22%3A%……
useNative=true&ttid=201300@travel_h5_3.1.0&appKey=12574478
t=1426062775998&sign=3feb52aed67967a2c47aa7a2b9f2a417
If you access the same url to reproduce API calls, it will after 10 seconds
ANALYSE API
• Parameters inside data parameter:
• Fixed parameter:
• Sign

31. HOW CAN WE GENRATE A VALID API CALL?

32. FIND TRIGGER POINT
Search source code to find API endpoint

33. REFORMAT SOURCE CODE
• Reformat code to get readable
source code
• Help to set breakpoint

34. FIND API URL GENERATOR
• Trace down the code to find out howto generate the url

35. FIND OUT TOKEN GENERATION ALGORITHM
• Set breakpoint and watch variables to find out the secret

36. WHAT DO I LEARN?

37. • Use time token to generate dynamic urls
• Use parameter sign token to verify parameter
• Prevent repeat API calls
• JS obfuscated code is easy to hack

38. SUMMARY

39. • Make sure url is not predictable
• Only supply information client needs
• Reduce anonymous network attack
• By different strategy to different location
• Use time token to generate dynamic url
• Use sign to verify request is valid

40. THANK YOU
Github: derekhe