Ratchet

An example web application in Golang.

https://lets-go.alexedwards.net/sample/02.09-serving-static-files.html

Project Structure

Following https://go.dev/doc/modules/layout#server-project the implementation

Loosely inspired by the organization of WTFDial,

• Application domain types reside in the project root (User, UserService, etc)
• Implementations of the application domain reside in the subpackages sqlite, http, etc.
• Everything is tied together in the cmd subpackages

Application Domain

This is project is an Indie reader inspired by Feedi.

Packages/Tools to investigate

• ent
• chi router
• https://github.com/FiloSottile/mkcert

Implementation Subpackages

The subpackages function as an adapter between our domain and the technology used to implement the domain. For example sqlite.FeedService implements the ratchet.FeedService using SQLite.

Subpackages ideally SHOULD NOT know about one another and SHOULD communicate in terms of the application domain.

A special mock packages can be used to create simple mocks for each application domain interface. This will allow each subpackages unit tests to share a common set of mocks so layers can be tested in isolatation.

Binary Packages

With loosely coupledc subpackages the application is wired together in the cmd subpackages to produce a final binary.

The cmd packages are ultimately the interface between the application domain and the operator. Configuration of types and any CLI flags SHOULD live in these packages.

Development

You can build ratchet locally by cloning the respository, then run

make go install ./cmd/...

The ratchetd cmd binary uses Oauth so you will need to create a new Oauth App. The vlaue of the authorization callback must be the hostname and IP at which clients can access the ratchetd server.

Additional Resources

• SQLite DB backups with LiteStream
• Unit / Integration Testing
• Graceful Shutdown
• Content Range Requests
• HTTP 204 and 205 Status Codes
• How to Disable FileServer Directory Listings
• Understand Mutexs in Go
• Structured Logging in Go with log/slog
• Exit Codes with special meaning
• Organizing Database Access in Go

Go http.FileServer

Supports If-Modified-Since and Last-Modified headers

curl -i -H "If-Modified-Since: 
Thu, 04 May 2017 13:07:52 GMT" http://localhost:5001/static/img/logo.png
HTTP/1.1 304 Not Modified
Last-Modified: Thu, 04 May 2017 13:07:52 GMT
Date: Sun, 12 Jan 2025 14:26:06 GMT

Supports Range Requests and 206 Partial Content responses.

curl -i -H "Range: bytes=100-199" --output - http://localhost:5001/static/img/logo.png
HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Content-Length: 100
Content-Range: bytes 100-199/1075
Content-Type: image/png
Last-Modified: Thu, 04 May 2017 13:07:52 GMT
Date: Sun, 12 Jan 2025 14:18:32 GMT

• The Content-Type is automatically set from the file extension using the mime.TypeByExtension() function. You can add your own custom extensions and content types using the mime.AddExtensionType() function if necessary.

• Sometimes you might want to serve a single file from within a handler. For this there’s the http.ServeFile() function, which you can use like so:

func downloadHandler(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "./ui/static/file.zip")
}

Warning: http.ServeFile() does not automatically sanitize the file path. If you’re constructing a file path from untrusted user input, to avoid directory traversal attacks you must sanitize the input with filepath.Clean() before using it.

Databases

The Let's-go book calls for MySQL. We use go-sqlite3 and go-migrate tool to manage migrations instead. Use make check-system-deps to validate all tools for this repository are installed.

Packages to consider adopting include sqlx or https://github.com/blockloop/scan. Both have the potential to reduce the verbosity that comes will using the standard database/sql package.

Nulls Gotcha

Beware of Nulls when using DB.Scan() To avoid issues consider using the sql.Null* types or putting a NOT NULL constraint in your DDL. Also a sensible Default

Managing Dependencies

To upgrade to latest minor/patch version of a package in your go mod.go you can use the -u flag:

go get -u github.com/foo/bar

To update to a specific package

go get -u github.com/foo/bar@v2.0.0

To remove the package you can use go mod tidy if all references have been removed, or:

go get github.com/foo/bar@none

Templates

The official template docs can be confusing to sort out. If you are new to templates check this out first: https://www.evandemond.com/kb/go-templates

• Always use html/templates because it escapes characters, preventing XSS attacks.
• With Nested templates you must explicitly pass or pipeline your template data. This occurs in {{template}} or {{block}} and appears like {{template "main" .}} or {{block "sidebar" .}}{{end}}
• If the type that you’re yielding between {{ }} tags has methods defined against it, you can call these methods (so long as they are exported and they return only a single value — or a single value and an error). Example .Snippet.Created struct field has the underlying type time.Time so <span>{{.Snippet.Created.Weekday}}</span>. You can also pass parameters <span>{{.Snippet.Created.AddDate 0 6 0}}</span>
• The html/template package always strips out any HTML comments you include in your templates which also help avoid XSS attacks when rendering dynamic content.
• Variable names must be prefixed by a dollar sign and can contain alphanumeric characters only. Ex: {{$bar := len .Foo}}
• You can combine multiple template functions with (). Ex: {{if (gt (len .Foo) 99)}} C1 {{end}}
• Within a {{range}} action you can use the {{break}} command to end the loop early, and {{continue}} to immediately start the next loop iteration.

Template Actions

Action	Description
{{ define }}	Defines a reusable named template. The content inside {{ define "name" }} ... {{ end }} can be invoked using {{ template "name" }}. Nested define calls are not allowed.
{{ template }}	Executes a template defined by {{ define "name" }}. Can optionally pass data into the template as dot (e.g., {{ template "name" .Data }}).
{{ block }}	Defines a named template block, similar to define, but allows the content to be overridden when embedding templates. Often used in base templates to create extendable sections.
{{/* a comment */}}	A comment that is ignored by the template engine. Use it to add notes or disable parts of the template without affecting the rendered output.

Template Functions

• For all actions below the {{else}} clause is optional.
• The empty values are false, 0, any nil pointer or interface value, and any array, slice, map, or string of length zero.
• the with and range actions change the value of dot. Once you start using them, what dot represents can be different depending on where you are in the template and what you’re doing.
• -}} and {{- trims white space. "{{23 -}} < {{- 45}}" becomes "23<45"

Function	Description
{{ if .Foo }} C1 {{ else }} C2 {{ end }}	If .Foo is not empty then render the content C1, otherwise render the content C2.
{{ with .Foo }} C1 {{ else }} C2 {{ end }}	If .Foo is not empty, then set dot to the value of .Foo and render the content C1, otherwise render the content C2.
{{ range .Foo }} C1 {{ else }} C2 {{ end }}	If the length of .Foo is greater than zero then loop over each element, setting dot to the value of each element and rendering the content C1. If the length of .Foo is zero then render the content C2. The underlying type of .Foo must be an array, slice, map, or channel.
{{ define }}
{{ define }}

Action	Description
{{ eq .Foo .Bar }}	Yields true if .Foo is equal to .Bar.
{{ ne .Foo .Bar }}	Yields true if .Foo is not equal to .Bar.
{{ not .Foo }}	Yields the boolean negation of .Foo.
{{ or .Foo .Bar }}	Yields .Foo if .Foo is not empty; otherwise yields .Bar.
{{ index .Foo i }}	Yields the value of .Foo at index i. The underlying type of .Foo must be a map, slice, or array, and i must be an integer value.
{{ printf "%s-%s" .Foo .Bar }}	Yields a formatted string containing the .Foo and .Bar values. Works in the same way as fmt.Sprintf().
{{ len .Foo }}	Yields the length of .Foo as an integer.
{{$bar := len .Foo}}	Assigns the length of .Foo to the template variable $bar.

Middleware

• Function that forms a closure over the next.ServerHTTP function in a call chain
• myMiddleware → servemux → application handler applies to all requests
• servemux → myMiddleware → application handler wraps specific handlers. Example Auth middleware.
• The control flow actually looks like commonHeaders → servemux → application handler → servemux → commonHeaders
  • This means defered blocks or code after next.ServeHTTP() will execute on the way back through.
• Early returns before next.ServeHTTP() will hand controlflow back upsteam.
  • Auth middle ware is a good example. w.WriteHeader(http.StatusForbidden); return

Needs:

• Access Logs
• Rate Limiting
• CORS
• Common Headers
• Recovery from Panic

Headers

• Primer on Content Security Policy

Form Validation

• https://www.alexedwards.net/blog/validation-snippets-for-go: Covers snippets for common form validation logic.

Session Managment

• https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html
• https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.html
• https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html

Session management used here: go get github.com/alexedwards/scs/v2@v2

Session ids should be meaningless. The server side stored information can typically include:

• the client IP address
• User-Agent
• e-mail
• username
• user ID
• role
• privilege level
• access rights
• language preferences
• account ID
• current state
• last login
• session timeouts
• and other internal session details.

The preferred session ID exchange mechanism should allow defining advanced token properties, such as the token expiration date and time, or granular usage constraints. - Cookies foot the bill which is why they are used.

• the Secure cookie attribute must be used to ensure the session ID is only exchanged through an encrypted channel.

• Do not switch a given session from HTTP to HTTPS, or vice-versa, as this will disclose the session ID in the clear through the network.

  • When redirecting to HTTPS, ensure that the cookie is set or regenerated after the redirect has occurred.

• Do not mix encrypted and unencrypted contents (HTML pages, images, CSS, JavaScript files, etc) in the same page, or from the same domain.

• HTTP Strict Transport Security (also named HSTS) can be used to enforce HTTPS connections. Can also https://www.leviathansecurity.com/blog/the-double-edged-sword-of-hsts-persistence-and-privacy result in user tracking even without cookies.

• It is important to emphasize that TLS does not protect against session ID prediction, brute force, client-side tampering or fixation; however, it does provide effective protection against an attacker intercepting or stealing session IDs through a man in the middle attack.

Cookies

• Secureattribute instructs web browsers to only send the cookie through an encrypted HTTPS (SSL/TLS) connection.
  • protects against man in the middle
• The HttpOnly cookie attribute instructs web browsers not to allow scripts (e.g. JavaScript or VBscript) an ability to access the cookies via the DOM document.cookie object.
  • This session ID protection is mandatory to prevent session ID stealing through XSS attacks.
    • XSS + CSRF though could still leak the session id.
• SameSite defines a cookie attribute preventing browsers from sending a SameSite flagged cookie with cross-site requests
• The Domain cookie attribute instructs web browsers to only send the cookie to the specified domain and all subdomains.
  • THE DOMAIN ATTRIBUTE SHOULD NOT BE SET, which will restrict the cookie to the origin server.
  • You don't want to mess with cross subdomain cookies. That can just be a mess.
• The Path cookie attribute instructs web browsers to only send the cookie to the specified directory or subdirectories (or paths or resources) within the web application
  • Path attribute should be set as restrictive as possible to the web application path that makes use of the session ID.
• Session management mechanisms based on cookies can make use of two types of cookies, non-persistent (or session) cookies, and persistent cookies. If a cookie presents the Max-Age (that has preference over Expires) or Expires attributes, it will be considered a persistent cookie and will be stored on disk by the web browser based until the expiration time.
  • Typically, session management capabilities to track users after authentication make use of non-persistent cookies. This forces the session to disappear from the client if the current web browser instance is closed.
  • Therefore, it is highly recommended to use non-persistent cookies for session management purposes, so that the session ID does not remain on the web client cache for long periods of time, from where an attacker can obtain it.
    • Depends on what the application does. An RSS reader doesn't need a short lived session.

If we look at reddit cookies we can find one called token_v2.

It has:

• Domain: .reddit.com
• Path: /
• Expires/ Max-Age for roughly 3 days
• httpOnly: true
• Secure: true
• SameSite: None

Local TLS Certs

https://github.com/FiloSottile/mkcert Can be used to create local certs with a trusted CA installed on the machine.

CSRF

• http://www.gnucitizen.org/blog/csrf-demystified/
• https://stackoverflow.com/questions/6412813/do-login-forms-need-tokens-against-csrf-attacks

Mitigations

• SameSite cookies. In chapter 11 we had Lax. This means that the session cookie "won't be sent by the browser for POST, PUT, or DELETE requests. As long as you stick to only using POST, PUT, or DELETE for state changing requests (login, signup, create snippet) Lax will prevent CRSF attacks.

• Token-based mitigation The OWASP guidanceis to use some form of token check if using TLS 1.2 or lower. Like session and password management you may be best served by using a known and tested third party package. This project uses justinas/nosurf (gorilla/csrf` is a valid alternative). Both use the double submit cookie pattern. In this pattern a random CSRF token is generated and sent to the user in a CSRF cookie. This CSRF token is then added to a hidden field in each HTML form that is potentially vulnerable to CSRF. When the form is submitted, both packages use some middleware to check that the hidden field value and cookie value match.

Due to the fact that no browser exists which supports TLS 1.3 and does not support SameSite cookies. If you only allow HTTPS requests to your application and enforce TLS 1.3 as the minimum TLS version, you don’t need to make any additional mitigation against CSRF attacks (like using the justinas/nosurf package). Just make sure that you always:

Set SameSite=Lax or SameSite=Strict on the session cookie; and
Use the POST, PUT or DELETE HTTP methods for any state-changing requests.