This document summarizes Go project layout and practices for a Go web application project. It discusses folder structure, configuration management using environment variables and files, embedding assets, command line interfaces, testing practices including fixtures, and packages for common functions like errors, middleware, models and more.

1. Go Project Layout
and Practice
Bo-Yi Wu
2019.08.29
ModernWeb

2. About me
• Software Engineer in Mediatek
• Member of Drone CI/CD Platform
• Member of Gitea Platform
• Member of Gin Golang Framework
• Teacher of Udemy Platform: Golang + Drone
https://blog.wu-boy.com

3. Agenda
• Go in Mediatek
• Go Project Layout
• Go Practices
• RESTful api and GraphQL
• Model testing (Postgres, SQLite, MySQL)
• Software Quality
• Data Metrics
• Go Testing

4. Tech Stack
• Initial Project using Go in 2018/01
• Golang
• Easy to Learn
• Performance
• Deployment

5. Repository folder
• api
• assets
• cmd
• configs
• docker
• pkg
├── api
├── assets
│ └── dist
├── cmd
│ └── ggz
├── configs
├── docker
│ ├── server
└── pkg
├── config
├── errors
├── fixtures
├── helper
├── middleware
│ ├── auth
│ └── header
├── model
├── module
│ ├── mailer
│ ├── metrics
│ └── storage
├── router
│ └── routes
├── schema
└── version

6. Root folder
• .drone.yml (deploy config)
• .revive.toml (golint config)
• docker-compose.yml (DB, Redis and UI)
• Makefile
• go module config (go.mod and go.sum)
• .env.example

7. Go Module
https://blog.golang.org/using-go-modules

8. Improve Deployment
Using Go Module Proxy
https://github.com/gomods/athens

9. save time
with proxy
97s -> 6s

10. Makefile
Build, Testing, Deploy

11. GOFMT ?= gofmt "-s"
GO ?= go
TARGETS ?= linux darwin windows
ARCHS ?= amd64 386
BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
GOFILES := $(shell find . -name "*.go" -type f)
TAGS ?= sqlite sqlite_unlock_notify
ifneq ($(shell uname), Darwin)
EXTLDFLAGS = -extldflags "-static" $(null)
else
EXTLDFLAGS =
endif
ifneq ($(DRONE_TAG),)
VERSION ?= $(subst v,,$(DRONE_TAG))
else
VERSION ?= $(shell git describe --tags --always')
endif

12. .env

13. GGZ_DB_DRIVER=mysql
GGZ_DB_USERNAME=root
GGZ_DB_PASSWORD=123456
GGZ_DB_NAME=ggz
GGZ_DB_HOST=127.0.0.1:3307
GGZ_SERVER_ADDR=:8080
GGZ_DEBUG=true
GGZ_SERVER_HOST=http://localhost:8080
GGZ_STORAGE_DRIVER=disk
GGZ_MINIO_ACCESS_ID=xxxxxxxx
GGZ_MINIO_SECRET_KEY=xxxxxxxx
GGZ_MINIO_ENDPOINT=s3.example.com
GGZ_MINIO_BUCKET=example
GGZ_MINIO_SSL=true
GGZ_AUTH0_DEBUG=true

14. docker-compose.yml

15. db:
image: mysql
restart: always
volumes:
- mysql-data:/var/lib/mysql
environment:
MYSQL_USER: example
MYSQL_PASSWORD: example
MYSQL_DATABASE: example
MYSQL_ROOT_PASSWORD: example
minio:
image: minio/minio
restart: always
ports:
volumes:
- minio-data:/data
environment:
MINIO_ACCESS_KEY: minio123456
MINIO_SECRET_KEY: minio123456
command: server /data
Development

16. Productionapi:
image: foo/bar
restart: always
ports:
- 8080:8080
environment:
- GGZ_METRICS_TOKEN=test-prometheus-token
- GGZ_METRICS_ENABLED=true
labels:
- "traefik.enable=true"
- "traefik.basic.frontend.rule=Host:${WEB_HOST}"
- "traefik.basic.protocol=http"

17. VersionCompile version info into Go binary

18. Version
• -X github.com/go-ggz/ggz/pkg/
version.Version=$(VERSION)
• -X github.com/go-ggz/ggz/pkg/
version.BuildDate=$(BUILD_DATE)
go build -o bin/api -ldflags

19. var (
// Version number for git tag.
Version string
// BuildDate is the ISO 8601 day drone was built.
BuildDate string
)
// PrintCLIVersion print server info
func PrintCLIVersion() string {
return fmt.Sprintf(
"version %s, built on %s, %s",
Version,
BuildDate,
runtime.Version(),
)
}

20. BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
ifneq ($(DRONE_TAG),)
VERSION ?= $(subst v,,$(DRONE_TAG))
else
VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed
's/^v//')
endif

21. AssetsEmbed files in Go
https://github.com/UnnoTed/fileb0x

22. func ReadSource(origPath string) (content []byte, err error) {
content, err = ReadFile(origPath)
if err != nil {
log.Warn().Err(err).Msgf("Failed to read builtin %s file.", origPath)
}
if config.Server.Assets != "" && file.IsDir(config.Server.Assets) {
origPath = path.Join(config.Server.Assets, origPath)
if file.IsFile(origPath) {
content, err = ioutil.ReadFile(origPath)
if err != nil {
log.Warn().Err(err).Msgf("Failed to read custom %s file", origPath)
}
}
}
return content, err
}
Debug Setting

23. // ViewHandler support dist handler from UI
func ViewHandler() gin.HandlerFunc {
fileServer := http.FileServer(dist.HTTP)
data := []byte(time.Now().String())
etag := fmt.Sprintf("%x", md5.Sum(data))
return func(c *gin.Context) {
c.Header("Cache-Control", "public, max-age=31536000")
c.Header("ETag", etag)
if match := c.GetHeader("If-None-Match"); match != "" {
if strings.Contains(match, etag) {
c.Status(http.StatusNotModified)
return
}
}
fileServer.ServeHTTP(c.Writer, c.Request)
}
}
File Server Handler

24. 圖片來來源：https://developers.google.com/web/fundamentals/performance/optimizing-
content-efficiency/http-caching?hl=zh-tw

25. // Favicon represents the favicon.
func Favicon(c *gin.Context) {
file, _ := dist.ReadFile("favicon.ico")
etag := fmt.Sprintf("%x", md5.Sum(file))
c.Header("ETag", etag)
c.Header("Cache-Control", "max-age=0")
if match := c.GetHeader("If-None-Match"); match != "" {
if strings.Contains(match, etag) {
c.Status(http.StatusNotModified)
return
}
}
c.Data(
http.StatusOK,
"image/x-icon",
file,
)
}
NO Cache

26. API

27. /healthz
• health check for load balancer
func Heartbeat(c *gin.Context) {
c.AbortWithStatus(http.StatusOK)
c.String(http.StatusOK, "ok")
}

28. CMDCommand line

29. Command line package
• Golang package: flag
• urfave/cli
• spf13/cobra

30. ├── agent
│ ├── config
│ │ └── config.go
│ └── main.go
├── notify
│ └── main.go
└── tcp-server
├── config
│ └── config.go
└── main.go

31. Config
Management
github.com/spf13/viper

32. Config management
• Load config from File
• .json
• .ini
• Load config from Environment Variables
• .env

33. var envfile string
flag.StringVar(&envfile, "env-file", ".env", "Read in a file of environment
variables")
flag.Parse()
godotenv.Load(envfile)
_ "github.com/joho/godotenv/autoload"

34. Logging struct {
Debug bool `envconfig:"GGZ_LOGS_DEBUG"`
Level string `envconfig:"GGZ_LOGS_LEVEL" default:"info"`
Color bool `envconfig:"GGZ_LOGS_COLOR"`
Pretty bool `envconfig:"GGZ_LOGS_PRETTY"`
Text bool `envconfig:"GGZ_LOGS_TEXT"`
}
// Server provides the server configuration.
Server struct {
Addr string `envconfig:"GGZ_SERVER_ADDR"`
Port string `envconfig:"GGZ_SERVER_PORT" default:"12000"`
Path string `envconfig:”GGZ_SERVER_PATH" default:"data"`
}
github.com/kelseyhightower/envconfig

35. config, err := config.Environ()
if err != nil {
log.Fatal().
Err(err).
Msg("invalid configuration")
}
initLogging(config)
// check folder exist
if !file.IsDir(config.Server.Path) {
log.Fatal().
Str("path", config.Server.Path).
Msg("log folder not found")
}
Load env from structure

36. /configs
Configuration file templates or default config

37. global:
scrape_interval: 5s
external_labels:
monitor: 'my-monitor'
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'ggz-server'
static_configs:
- targets: ['ggz-server:8080']
bearer_token: 'test-prometheus-token'

38. /docker
Docker file template

39. ├── ggz-redirect
│ ├── Dockerfile.linux.amd64
│ ├── Dockerfile.linux.arm
│ ├── Dockerfile.linux.arm64
│ ├── Dockerfile.windows.amd64
│ └── manifest.tmpl
└── ggz-server
├── Dockerfile.linux.amd64
├── Dockerfile.linux.arm
├── Dockerfile.linux.arm64
├── Dockerfile.windows.amd64
└── manifest.tmpl

40. /integrations

41. ctx := context.Background()
req := testcontainers.ContainerRequest{
Image: "goggz/ggz-server",
ExposedPorts: []string{"8080/tcp"},
WaitingFor: wait.ForLog("Starting shorten server on :8080")
}
ggzServer, err := testcontainers.GenericContainer(
ctx,
testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
})
if err != nil {
t.Fatal(err)
}
github.com/testcontainers/testcontainers-go

42. /pkg

43. ├── config
├── errors
├── fixtures
├── helper
├── middleware
│ ├── auth
│ └── header
├── model
├── module
│ ├── metrics
│ └── storage
│ ├── disk
│ └── minio
├── router
│ └── routes
├── schema
└── version

44. /pkg/errors

45. // Type defines the type of an error
type Type string
const (
// Internal error
Internal Type = "internal"
// NotFound error means that a specific item does not exis
NotFound Type = "not_found"
// BadRequest error
BadRequest Type = "bad_request"
// Validation error
Validation Type = "validation"
// AlreadyExists error
AlreadyExists Type = "already_exists"
// Unauthorized error
Unauthorized Type = "unauthorized"
)

46. // ENotExists creates an error of type NotExist
func ENotExists(msg string, err error, arg ...interface{}) error {
return New(NotFound, fmt.Sprintf(msg, arg...), err)
}
// EBadRequest creates an error of type BadRequest
func EBadRequest(msg string, err error, arg ...interface{}) error {
return New(BadRequest, fmt.Sprintf(msg, arg...), err)
}
// EAlreadyExists creates an error of type AlreadyExists
func EAlreadyExists(msg string, err error, arg ...interface{}) error {
return New(AlreadyExists, fmt.Sprintf(msg, arg...), err)
}

47. /pkg/fixtures

48. Rails-like test fixtures
Write tests against a real database
github.com/go-testfixtures/testfixtures

49. fixtures/
posts.yml
comments.yml
tags.yml
posts_tags.yml
users.yml

50. -
id: 1
email: test@gmail.com
full_name: test
avatar: http://example.com
avatar_email: test@gmail.com
-
id: 2
email: test1234@gmail.com
full_name: test1234
avatar: http://example.com
avatar_email: test1234@gmail.com

51. Unit Testing with Database

52. func TestMain(m *testing.M) {
// test program to do extra
setup or teardown before or after
testing.
os.Exit(m.Run())
}
https://golang.org/pkg/testing/#hdr-Main

53. func MainTest(m *testing.M, pathToRoot string) {
var err error
fixturesDir := filepath.Join(pathToRoot, "pkg", "fixtures")
if err = createTestEngine(fixturesDir); err != nil {
fatalTestError("Error creating test engine: %vn", err)
}
os.Exit(m.Run())
}
func createTestEngine(fixturesDir string) error {
var err error
x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared")
if err != nil {
return err
}
x.ShowSQL(config.Server.Debug)
return InitFixtures(&testfixtures.SQLite{}, fixturesDir)
}
Testing with SQLite

54. func TestIsUserExist(t *testing.T) {
assert.NoError(t, PrepareTestDatabase())
exists, err := IsUserExist(0, "test@gmail.com")
assert.NoError(t, err)
assert.True(t, exists)
exists, err = IsUserExist(0, "test123456@gmail.com")
assert.NoError(t, err)
assert.False(t, exists)
exists, err = IsUserExist(1, "test1234@gmail.com")
assert.NoError(t, err)
assert.True(t, exists)
exists, err = IsUserExist(1, "test123456@gmail.com")
assert.NoError(t, err)
assert.False(t, exists)
} go test -v -run=TestIsUserExist ./pkg/models/

55. /pkg/helper

56. Helper func
• Encrypt and Decrypt
• Regexp func
• IsEmail, IsUsername
• Zipfile

57. /pkg/middleware

58. func Secure(c *gin.Context) {
c.Header("Access-Control-Allow-Origin", "*")
c.Header("X-Frame-Options", "DENY")
c.Header("X-Content-Type-Options", "nosniff")
c.Header("X-XSS-Protection", "1; mode=block")
if c.Request.TLS != nil {
c.Header("Strict-Transport-Security", "max-age=31536000")
}
}

59. /pkg/model

60. Use gorm or xorm

61. Build in
SQLite3

62. // +build sqlite
package model
import (
_ "github.com/mattn/go-sqlite3"
)
func init() {
EnableSQLite3 = true
}
go build -v -tags 'sqlite sqlite_unlock_notify'

63. /pkg/module

64. ├── cron
├── download
├── jwt
├── ldap
├── mailer
│ ├── ses
│ └── smtp
├── queue
├── metrics
├── redis
└── storage
├── disk
└── minio

65. Integration with
Prometheus + Grafana

66. func NewCollector() Collector {
return Collector{
Users: prometheus.NewDesc(
namespace+"users",
"Number of Users",
nil, nil,
),
}
// Collect returns the metrics with values
func (c Collector) Collect(ch chan<- prometheus.Metric) {
stats := model.GetStatistic()
ch <- prometheus.MustNewConstMetric(
c.Users,
prometheus.GaugeValue,
float64(stats.Counter.User),
)
}

67. Prometheus Handler

68. func Metrics(token string) gin.HandlerFunc {
h := promhttp.Handler()
return func(c *gin.Context) {
if token == "" {
h.ServeHTTP(c.Writer, c.Request)
return
}
header := c.Request.Header.Get("Authorization")
if header == "" {
c.String(http.StatusUnauthorized, errInvalidToken.Error())
return
}
bearer := fmt.Sprintf("Bearer %s", token)
if header != bearer {
c.String(http.StatusUnauthorized, errInvalidToken.Error())
return
}
h.ServeHTTP(c.Writer, c.Request)
}
}

69. c := metrics.NewCollector()
prometheus.MustRegister(c)
if config.Metrics.Enabled {
root.GET("/metrics", router.Metrics(config.Metrics.Token))
}
Your prometheus token

70. /pkg/schema

71. RESTful vs GraphQL
See the Slide: GraphQL in Go

72. var rootQuery = graphql.NewObject(
graphql.ObjectConfig{
Name: "RootQuery",
Description: "Root Query",
Fields: graphql.Fields{
"queryShortenURL": &queryShortenURL,
"queryMe": &queryMe,
},
})
var rootMutation = graphql.NewObject(
graphql.ObjectConfig{
Name: "RootMutation",
Description: "Root Mutation",
Fields: graphql.Fields{
"createUser": &createUser,
},
})
// Schema is the GraphQL schema served by the server.
var Schema, _ = graphql.NewSchema(
graphql.SchemaConfig{
Query: rootQuery,
Mutation: rootMutation,
})

73. Write the GraphQL Testing

74. assert.NoError(t, model.PrepareTestDatabase())
t.Run("user not login", func(t *testing.T) {
test := T{
Query: `{
queryMe {
email
}
}`,
Schema: Schema,
Expected: &graphql.Result{
Data: map[string]interface{}{
"queryMe": nil,
},
Errors: []gqlerrors.FormattedError{
{
Message: errorYouAreNotLogin,
},
},
},
}
})
}

75. Best Practice
Testing your Go code

77. Testable Code
• Code Quality
• Readability
• Maintainability
• Testability

78. #1. Testing in Go
func TestFooBar(t *testing.T) {}
func ExampleFooBar(t *testing.T) {}
func BenchmarkFooBar(b *testing.B) {}
go test package_name

79. #2. Benchmark Testing
Profiling: CPU, Memory, Goroutine Block

80. func BenchmarkPlaylyfeGraphQLMaster(b *testing.B) {
for i := 0; i < b.N; i++ {
context := map[string]interface{}{}
variables := map[string]interface{}{}
playlyfeExecutor.Execute(context, "{hello}", variables, "")
}
}
func BenchmarkGophersGraphQLMaster(b *testing.B) {
for i := 0; i < b.N; i++ {
ctx := context.Background()
variables := map[string]interface{}{}
gopherSchema.Exec(ctx, "{hello}", "", variables)
}
} http://bit.ly/2L0CG3Q

82. #3. Example Testing
Examples on how to use your code

83. func ExampleFooBar() {
fmt.Println(strings.Compare("a", "b"))
fmt.Println(strings.Compare("a", "a"))
fmt.Println(strings.Compare("b", "a"))
// Output:
// -1
// 0
// 1
}

84. $ go test -v -tags=sqlite -run=ExampleFooBar ./pkg/model/...
=== RUN ExampleFooBar
--- PASS: ExampleFooBar (0.00s)
PASS
ok github.com/go-ggz/ggz/pkg/model 0.022s

85. #4. Subtests in Testing Package
func (t *T) Run(name string, f func(t *T)) bool {}
func (b *B) Run(name string, f func(b *B)) bool {}

86. tests := []struct {
name string
fields fields
args args
}{}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
c := Collector{
Shortens: tt.fields.Shortens,
Users: tt.fields.Users,
}
c.Describe(tt.args.ch)
})
}
}

87. #5. Skipping Testing
t.Skip()

88. package metrics
import (
"os"
"testing"
)
func TestSkip(t *testing.T) {
if os.Getenv("DEBUG_MODE") == "true" {
t.Skipf("test skipped")
}
}

89. #6. Running Tests in Parallel
Speedup your CI/CD Flow
t.Parallel()

90. func TestFooBar01(t *testing.T) {
t.Parallel()
time.Sleep(time.Second)
}
func TestFooBar02(t *testing.T) {
t.Parallel()
time.Sleep(time.Second * 2)
}
func TestFooBar03(t *testing.T) {
t.Parallel()
time.Sleep(time.Second * 3)
}

91. Just only use
one package
github.com/stretchr/testify

92. https://www.udemy.com/course/golang-fight/?couponCode=GOLANG2019

93. https://www.udemy.com/course/devops-oneday/?couponCode=DRONE2019

94. END