|
|
# Ratchet
|
|
|
|
|
|
|
|
|
|
|
|
An example web application in Golang.
|
|
|
|
|
|
https://lets-go.alexedwards.net/sample/02.09-serving-static-files.html
|
|
|
|
|
|
## Project Structure
|
|
|
|
|
|
Following https://go.dev/doc/modules/layout#server-project the implementation
|
|
|
|
|
|
Loosely inspired by the organization of [WTFDial](https://github.com/benbjohnson/wtf?tab=readme-ov-file#project-structure),
|
|
|
- Application domain types reside in the project root (User, UserService, etc)
|
|
|
- Implementations of the application domain reside in the subpackages `sqlite`, `http`, etc.
|
|
|
- Everything is tied together in the `cmd` subpackages
|
|
|
|
|
|
|
|
|
### Application Domain
|
|
|
|
|
|
This is project is an Indie reader inspired by [Feedi](https://github.com/facundoolano/feedi).
|
|
|
|
|
|
### Packages/Tools to investigate
|
|
|
|
|
|
- ent
|
|
|
- chi router
|
|
|
- https://github.com/FiloSottile/mkcert
|
|
|
|
|
|
|
|
|
### Implementation Subpackages
|
|
|
|
|
|
The subpackages function as an adapter between our domain and the technology used to implement the domain. For example `sqlite.FeedService` implements the `ratchet.FeedService` using SQLite.
|
|
|
|
|
|
Subpackages ideally **SHOULD NOT** know about one another and **SHOULD** communicate in terms of the application domain.
|
|
|
|
|
|
A special `mock` packages can be used to create simple mocks for each application domain interface. This will allow each subpackages unit tests to share a common set of mocks so layers can be tested in isolatation.
|
|
|
|
|
|
### Binary Packages
|
|
|
|
|
|
With loosely coupledc subpackages the application is wired together in the `cmd` subpackages to produce a final binary.
|
|
|
|
|
|
The `cmd` packages are ultimately the interface between the application domain and the operator. Configuration of types and any CLI flags *SHOULD* live in these packages.
|
|
|
|
|
|
## Development
|
|
|
|
|
|
You can build `ratchet` locally by cloning the respository, then run
|
|
|
|
|
|
`make`
|
|
|
`go install ./cmd/...`
|
|
|
|
|
|
The `ratchetd` cmd binary uses Oauth so you will need to create a new Oauth App. The vlaue of the authorization callback must be the hostname and IP at which clients can access the `ratchetd` server.
|
|
|
|
|
|
## Additional Resources
|
|
|
|
|
|
|
|
|
- [SQLite DB backups with LiteStream](https://litestream.io/getting-started/)
|
|
|
- [Unit / Integration Testing](https://dev.to/sha254/testing-rest-apis-in-go-a-guide-to-unit-and-integration-testing-with-gos-standard-testing-library-2o9l)
|
|
|
- [Graceful Shutdown](https://dev.to/mokiat/proper-http-shutdown-in-go-3fji)
|
|
|
- [Content Range Requests](https://web.archive.org/web/20230918195519/https://benramsey.com/blog/2008/05/206-partial-content-and-range-requests/)
|
|
|
- [HTTP 204 and 205 Status Codes](https://web.archive.org/web/20230918193536/https://benramsey.com/blog/2008/05/http-status-204-no-content-and-205-reset-content/)
|
|
|
- [How to Disable FileServer Directory Listings](https://www.alexedwards.net/blog/disable-http-fileserver-directory-listings)
|
|
|
- [Understand Mutexs in Go](https://www.alexedwards.net/blog/understanding-mutexes)
|
|
|
- [Structured Logging in Go with log/slog](https://pkg.go.dev/log/slog)
|
|
|
- [Exit Codes with special meaning](https://tldp.org/LDP/abs/html/exitcodes.html)
|
|
|
- [Organizing Database Access in Go](https://www.alexedwards.net/blog/organising-database-access)
|
|
|
|
|
|
### Go http.FileServer
|
|
|
|
|
|
Supports If-Modified-Since and Last-Modified headers
|
|
|
```
|
|
|
curl -i -H "If-Modified-Since:
|
|
|
Thu, 04 May 2017 13:07:52 GMT" http://localhost:5001/static/img/logo.png
|
|
|
HTTP/1.1 304 Not Modified
|
|
|
Last-Modified: Thu, 04 May 2017 13:07:52 GMT
|
|
|
Date: Sun, 12 Jan 2025 14:26:06 GMT
|
|
|
```
|
|
|
|
|
|
Supports Range Requests and 206 Partial Content responses.
|
|
|
```
|
|
|
curl -i -H "Range: bytes=100-199" --output - http://localhost:5001/static/img/logo.png
|
|
|
HTTP/1.1 206 Partial Content
|
|
|
Accept-Ranges: bytes
|
|
|
Content-Length: 100
|
|
|
Content-Range: bytes 100-199/1075
|
|
|
Content-Type: image/png
|
|
|
Last-Modified: Thu, 04 May 2017 13:07:52 GMT
|
|
|
Date: Sun, 12 Jan 2025 14:18:32 GMT
|
|
|
```
|
|
|
|
|
|
- The `Content-Type` is automatically set from the file extension using the `mime.TypeByExtension()` function. You can add your own custom extensions and content types using the `mime.AddExtensionType()` function if necessary.
|
|
|
|
|
|
- Sometimes you might want to serve a single file from within a handler. For this there’s the `http.ServeFile()` function, which you can use like so:
|
|
|
|
|
|
```go
|
|
|
func downloadHandler(w http.ResponseWriter, r *http.Request) {
|
|
|
http.ServeFile(w, r, "./ui/static/file.zip")
|
|
|
}
|
|
|
```
|
|
|
|
|
|
Warning: http.ServeFile() does not automatically sanitize the file path. If you’re constructing a file path from untrusted user input, to avoid directory traversal attacks you must sanitize the input with filepath.Clean() before using it.
|
|
|
|
|
|
## Databases
|
|
|
|
|
|
The Let's-go book calls for MySQL. We use [go-sqlite3](https://github.com/mattn/go-sqlite3) and [go-migrate]() tool to manage migrations instead. Use `make check-system-deps` to validate all tools for this repository are installed.
|
|
|
|
|
|
Packages to consider adopting include sqlx or https://github.com/blockloop/scan. Both have the potential to reduce the verbosity that comes will using the standard database/sql package.
|
|
|
|
|
|
### Nulls Gotcha
|
|
|
|
|
|
Beware of Nulls when using `DB.Scan()` To avoid issues consider using the sql.Null* types or putting a `NOT NULL` constraint in your DDL. Also a sensible Default
|
|
|
|
|
|
## Managing Dependencies
|
|
|
|
|
|
To upgrade to latest minor/patch version of a package in your go mod.go you can use the `-u` flag:
|
|
|
|
|
|
```
|
|
|
go get -u github.com/foo/bar
|
|
|
```
|
|
|
|
|
|
To update to a specific package
|
|
|
|
|
|
```
|
|
|
go get -u github.com/foo/bar@v2.0.0
|
|
|
```
|
|
|
|
|
|
To remove the package you can use `go mod tidy` if all references have been removed, or:
|
|
|
|
|
|
```
|
|
|
go get github.com/foo/bar@none
|
|
|
```
|
|
|
|
|
|
## Templates
|
|
|
|
|
|
The [official template docs](https://pkg.go.dev/text/template#hdr-Functions) can be confusing to sort out. If you are new to templates check this out first: https://www.evandemond.com/kb/go-templates
|
|
|
|
|
|
- Always use `html/templates` because it escapes characters, preventing XSS attacks.
|
|
|
- With Nested templates you must explicitly pass or pipeline your template data. This occurs in `{{template}}` or `{{block}}` and appears like `{{template "main" .}}` or `{{block "sidebar" .}}{{end}}`
|
|
|
- If the type that you’re yielding between `{{ }}` tags has methods defined against it, you can call these methods (so long as they are exported and they return only a single value — or a single value and an error). Example `.Snippet.Created` struct field has the underlying type `time.Time` so `<span>{{.Snippet.Created.Weekday}}</span>`. You can also pass parameters `<span>{{.Snippet.Created.AddDate 0 6 0}}</span>`
|
|
|
- The `html/template` package always strips out any HTML comments you include in your templates which also help avoid XSS attacks when rendering dynamic content.
|
|
|
- Variable names must be prefixed by a dollar sign and can contain alphanumeric characters only. Ex: `{{$bar := len .Foo}}`
|
|
|
- You can combine multiple template functions with `()`. Ex: `{{if (gt (len .Foo) 99)}} C1 {{end}}`
|
|
|
- Within a `{{range}}` action you can use the `{{break}}` command to end the loop early, and `{{continue}}` to immediately start the next loop iteration.
|
|
|
|
|
|
### Template Actions
|
|
|
|
|
|
| Action | Description |
|
|
|
|-----------------------------------------------|-----------------------------------------------------------------------------------------------|
|
|
|
| `{{ define }}` | Defines a reusable named template. The content inside `{{ define "name" }} ... {{ end }}` can be invoked using `{{ template "name" }}`. Nested `define` calls are not allowed. |
|
|
|
| `{{ template }}` | Executes a template defined by `{{ define "name" }}`. Can optionally pass data into the template as `dot` (e.g., `{{ template "name" .Data }}`). |
|
|
|
| `{{ block }}` | Defines a named template block, similar to `define`, but allows the content to be overridden when embedding templates. Often used in base templates to create extendable sections. |
|
|
|
| `{{/* a comment */}}` | A comment that is ignored by the template engine. Use it to add notes or disable parts of the template without affecting the rendered output. |
|
|
|
|
|
|
### Template Functions
|
|
|
|
|
|
- For all actions below the `{{else}}` clause is optional.
|
|
|
- The _empty_ values are false, 0, any nil pointer or interface value, and any array, slice, map, or string of length zero.
|
|
|
- the `with` and `range` actions change the value of dot. Once you start using them, what dot represents can be different depending on where you are in the template and what you’re doing.
|
|
|
- `-}}` and `{{-` trims white space. `"{{23 -}} < {{- 45}}"` becomes `"23<45"`
|
|
|
|
|
|
| Function | Description |
|
|
|
|-----------------------------------------------|-------------------------------------------------------------------------------------|
|
|
|
| `{{ if .Foo }} C1 {{ else }} C2 {{ end }}` | If `.Foo` is not empty then render the content C1, otherwise render the content C2. |
|
|
|
| `{{ with .Foo }} C1 {{ else }} C2 {{ end }}` | If `.Foo` is not empty, then set `dot` to the value of `.Foo` and render the content C1, otherwise render the content C2.|
|
|
|
| `{{ range .Foo }} C1 {{ else }} C2 {{ end }}` | If the length of `.Foo` is greater than zero then loop over each element, setting dot to the value of each element and rendering the content C1. If the length of `.Foo` is zero then render the content C2. The underlying type of `.Foo` must be an `array`, `slice`, `map`, or `channel`.|
|
|
|
| `{{ define }}` | |
|
|
|
| `{{ define }}` | |
|
|
|
|
|
|
| Action | Description |
|
|
|
|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
| `{{ eq .Foo .Bar }}` | Yields `true` if `.Foo` is equal to `.Bar`. |
|
|
|
| `{{ ne .Foo .Bar }}` | Yields `true` if `.Foo` is not equal to `.Bar`. |
|
|
|
| `{{ not .Foo }}` | Yields the boolean negation of `.Foo`. |
|
|
|
| `{{ or .Foo .Bar }}` | Yields `.Foo` if `.Foo` is not empty; otherwise yields `.Bar`. |
|
|
|
| `{{ index .Foo i }}` | Yields the value of `.Foo` at index `i`. The underlying type of `.Foo` must be a map, slice, or array, and `i` must be an integer value. |
|
|
|
| `{{ printf "%s-%s" .Foo .Bar }}`| Yields a formatted string containing the `.Foo` and `.Bar` values. Works in the same way as `fmt.Sprintf()`. |
|
|
|
| `{{ len .Foo }}` | Yields the length of `.Foo` as an integer. |
|
|
|
| `{{$bar := len .Foo}}` | Assigns the length of `.Foo` to the template variable `$bar`. |
|
|
|
|
|
|
|
|
|
## Middleware
|
|
|
|
|
|
- Function that forms a closure over the `next.ServerHTTP` function in a call chain
|
|
|
- `myMiddleware → servemux → application handler` applies to all requests
|
|
|
- `servemux → myMiddleware → application handler` wraps specific handlers. Example Auth middleware.
|
|
|
- The control flow actually looks like `commonHeaders → servemux → application handler → servemux → commonHeaders`
|
|
|
- This means defered blocks or code after `next.ServeHTTP()` will execute on the way back through.
|
|
|
- Early returns before `next.ServeHTTP()` will hand controlflow back upsteam.
|
|
|
- Auth middle ware is a good example. `w.WriteHeader(http.StatusForbidden); return`
|
|
|
|
|
|
Needs:
|
|
|
- [x] Access Logs
|
|
|
- [ ] Rate Limiting
|
|
|
- [ ] CORS
|
|
|
- [X] Common Headers
|
|
|
- [X] Recovery from Panic
|
|
|
|
|
|
### Headers
|
|
|
|
|
|
- [Primer on Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
|
|
|
|
|
## Form Validation
|
|
|
|
|
|
- https://www.alexedwards.net/blog/validation-snippets-for-go: Covers snippets for common form validation logic.
|
|
|
|
|
|
## Session Managment
|
|
|
|
|
|
- https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html
|
|
|
- https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.html
|
|
|
- https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html
|
|
|
|
|
|
Session management used here: `go get github.com/alexedwards/scs/v2@v2`
|
|
|
|
|
|
Session ids should be meaningless. The server side stored information can typically include:
|
|
|
- the client IP address
|
|
|
- User-Agent
|
|
|
- e-mail
|
|
|
- username
|
|
|
- user ID
|
|
|
- role
|
|
|
- privilege level
|
|
|
- access rights
|
|
|
- language preferences
|
|
|
- account ID
|
|
|
- current state
|
|
|
- last login
|
|
|
- session timeouts
|
|
|
- and other internal session details.
|
|
|
|
|
|
The preferred session ID exchange mechanism should allow defining advanced token properties, such as the token expiration date and time, or granular usage constraints.
|
|
|
- Cookies foot the bill which is why they are used.
|
|
|
|
|
|
- the Secure cookie attribute must be used to ensure the session ID is only exchanged through an encrypted channel.
|
|
|
|
|
|
- Do not switch a given session from HTTP to HTTPS, or vice-versa, as this will disclose the session ID in the clear through the network.
|
|
|
- When redirecting to HTTPS, ensure that the cookie is set or regenerated after the redirect has occurred.
|
|
|
|
|
|
- Do not mix encrypted and unencrypted contents (HTML pages, images, CSS, JavaScript files, etc) in the same page, or from the same domain.
|
|
|
- HTTP Strict Transport Security (also named HSTS) can be used to enforce HTTPS connections. Can also https://www.leviathansecurity.com/blog/the-double-edged-sword-of-hsts-persistence-and-privacy result in user tracking even without cookies.
|
|
|
|
|
|
- It is important to emphasize that TLS does not protect against session ID prediction, brute force, client-side tampering or fixation; however, it does provide effective protection against an attacker intercepting or stealing session IDs through a man in the middle attack.
|
|
|
|
|
|
### Cookies
|
|
|
- `Secure`attribute instructs web browsers to only send the cookie through an encrypted HTTPS (SSL/TLS) connection.
|
|
|
- protects against man in the middle
|
|
|
- The `HttpOnly` cookie attribute instructs web browsers not to allow scripts (e.g. JavaScript or VBscript) an ability to access the cookies via the DOM document.cookie object.
|
|
|
- This session ID protection is mandatory to prevent session ID stealing through XSS attacks.
|
|
|
- XSS + CSRF though could still leak the session id.
|
|
|
- `SameSite` defines a cookie attribute preventing browsers from sending a SameSite flagged cookie with cross-site requests
|
|
|
- The `Domain` cookie attribute instructs web browsers to only send the cookie to the specified domain and all subdomains.
|
|
|
- THE DOMAIN ATTRIBUTE SHOULD NOT BE SET, which will restrict the cookie to the origin server.
|
|
|
- You don't want to mess with cross subdomain cookies. That can just be a mess.
|
|
|
- The `Path` cookie attribute instructs web browsers to only send the cookie to the specified directory or subdirectories (or paths or resources) within the web application
|
|
|
- `Path` attribute should be set as restrictive as possible to the web application path that makes use of the session ID.
|
|
|
- Session management mechanisms based on cookies can make use of two types of cookies, non-persistent (or session) cookies, and persistent cookies. If a cookie presents the `Max-Age` (that has preference over `Expires`) or `Expires` attributes, it will be considered a persistent cookie and will be stored on disk by the web browser based until the expiration time.
|
|
|
- Typically, session management capabilities to track users after authentication make use of non-persistent cookies. This forces the session to disappear from the client if the current web browser instance is closed.
|
|
|
- Therefore, it is highly recommended to use non-persistent cookies for session management purposes, so that the session ID does not remain on the web client cache for long periods of time, from where an attacker can obtain it.
|
|
|
- Depends on what the application does. An RSS reader doesn't need a short lived session.
|
|
|
|
|
|
If we look at reddit cookies we can find one called `token_v2`.
|
|
|
|
|
|
It has:
|
|
|
|
|
|
- Domain: .reddit.com
|
|
|
- Path: /
|
|
|
- Expires/ Max-Age for roughly 3 days
|
|
|
- httpOnly: true
|
|
|
- Secure: true
|
|
|
- SameSite: None
|
|
|
|
|
|
## Local TLS Certs
|
|
|
|
|
|
https://github.com/FiloSottile/mkcert Can be used to create local certs with a trusted CA installed on the machine.
|
|
|
|
|
|
## CSRF
|
|
|
|
|
|
- http://www.gnucitizen.org/blog/csrf-demystified/
|
|
|
- https://stackoverflow.com/questions/6412813/do-login-forms-need-tokens-against-csrf-attacks
|
|
|
|
|
|
### Mitigations
|
|
|
|
|
|
- **SameSite cookies**. In chapter 11 we had `Lax`. This means that the session cookie "won't be sent by the browser for POST, PUT, or DELETE requests. As long as you stick to only using POST, PUT, or DELETE for state changing requests (login, signup, create snippet) `Lax` will prevent CRSF attacks.
|
|
|
|
|
|
- **Token-based mitigation** The [OWASP guidance](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#token-based-mitigation)is to use some form of token check if using TLS 1.2 or lower. Like session and password management you may be best served by using a known and tested third party package. This project uses `justinas/nosurf (`gorilla/csrf` is a valid alternative). Both use the [double submit cookie](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#double-submit-cookie) pattern. In this pattern a random CSRF token is generated and sent to the user in a CSRF cookie. This CSRF token is then added to a hidden field in each HTML form that is potentially vulnerable to CSRF. When the form is submitted, both packages use some middleware to check that the hidden field value and cookie value match.
|
|
|
|
|
|
Due to the fact that no browser exists which supports TLS 1.3 and does not support SameSite cookies. If you only allow HTTPS requests to your application and enforce TLS 1.3 as the minimum TLS version, you don’t need to make any additional mitigation against CSRF attacks (like using the justinas/nosurf package). Just make sure that you always:
|
|
|
|
|
|
Set SameSite=Lax or SameSite=Strict on the session cookie; and
|
|
|
Use the POST, PUT or DELETE HTTP methods for any state-changing requests.
|