Michael McGuinness
@@ -1,6 +1,6 @@
|
||||
@@@Owners @"Michael McGuinness"
|
||||
|
||||
@@@Backend @@Owners
|
||||
@@@Infrastructure @@Owners
|
||||
|
||||
@@@Owners @"Michael McGuinness"
|
||||
|
||||
@@@Backend @@Owners
|
||||
@@@Infrastructure @@Owners
|
||||
|
||||
**/*.go @@Backend
|
||||
+32
-32
@@ -1,32 +1,32 @@
|
||||
# Include any files or directories that you don't want to be copied to your
|
||||
# container here (e.g., local build artifacts, temporary files, etc.).
|
||||
#
|
||||
# For more help, visit the .dockerignore file reference guide at
|
||||
# https://docs.docker.com/go/build-context-dockerignore/
|
||||
|
||||
**/.DS_Store
|
||||
**/.classpath
|
||||
**/.dockerignore
|
||||
**/.env
|
||||
**/.git
|
||||
**/.gitignore
|
||||
**/.project
|
||||
**/.settings
|
||||
**/.toolstarget
|
||||
**/.vs
|
||||
**/.vscode
|
||||
**/*.*proj.user
|
||||
**/*.dbmdl
|
||||
**/*.jfm
|
||||
**/bin
|
||||
**/charts
|
||||
**/docker-compose*
|
||||
**/compose.y*ml
|
||||
**/Dockerfile*
|
||||
**/node_modules
|
||||
**/npm-debug.log
|
||||
**/obj
|
||||
**/secrets.dev.yaml
|
||||
**/values.dev.yaml
|
||||
LICENSE
|
||||
README.md
|
||||
# Include any files or directories that you don't want to be copied to your
|
||||
# container here (e.g., local build artifacts, temporary files, etc.).
|
||||
#
|
||||
# For more help, visit the .dockerignore file reference guide at
|
||||
# https://docs.docker.com/go/build-context-dockerignore/
|
||||
|
||||
**/.DS_Store
|
||||
**/.classpath
|
||||
**/.dockerignore
|
||||
**/.env
|
||||
**/.git
|
||||
**/.gitignore
|
||||
**/.project
|
||||
**/.settings
|
||||
**/.toolstarget
|
||||
**/.vs
|
||||
**/.vscode
|
||||
**/*.*proj.user
|
||||
**/*.dbmdl
|
||||
**/*.jfm
|
||||
**/bin
|
||||
**/charts
|
||||
**/docker-compose*
|
||||
**/compose.y*ml
|
||||
**/Dockerfile*
|
||||
**/node_modules
|
||||
**/npm-debug.log
|
||||
**/obj
|
||||
**/secrets.dev.yaml
|
||||
**/values.dev.yaml
|
||||
LICENSE
|
||||
README.md
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
* text=auto
|
||||
* text eol=lf
|
||||
+77
-77
@@ -1,77 +1,77 @@
|
||||
# These are some examples of commonly ignored file patterns.
|
||||
# You should customize this list as applicable to your project.
|
||||
# Learn more about .gitignore:
|
||||
# https://www.atlassian.com/git/tutorials/saving-changes/gitignore
|
||||
|
||||
# Ignore the binary files generated by `go build`
|
||||
/main
|
||||
|
||||
# Dependency directories (uncomment if using vendoring)
|
||||
# vendor/
|
||||
|
||||
# Ignore Go workspace files
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Output of the go coverage tool
|
||||
*.out
|
||||
out/
|
||||
coverage/
|
||||
cover
|
||||
|
||||
# Test binary, built with `go test -c`
|
||||
*.test
|
||||
|
||||
# Node artifact files
|
||||
node_modules/
|
||||
dist/
|
||||
|
||||
# Compiled Java class files
|
||||
*.class
|
||||
|
||||
# Compiled Python bytecode
|
||||
*.py[cod]
|
||||
|
||||
# Log files
|
||||
*.log
|
||||
|
||||
# Package files
|
||||
*.jar
|
||||
|
||||
# Maven
|
||||
target/
|
||||
dist/
|
||||
|
||||
# JetBrains IDE
|
||||
.idea/
|
||||
*.iml
|
||||
|
||||
# Unit test reports
|
||||
TEST*.xml
|
||||
|
||||
# VS Code
|
||||
.vscode/
|
||||
|
||||
# Generated by MacOS
|
||||
.DS_Store
|
||||
|
||||
# Generated by Windows
|
||||
Thumbs.db
|
||||
|
||||
# Applications
|
||||
*.app
|
||||
*.exe
|
||||
*.war
|
||||
*.exe~
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
|
||||
# Large media files
|
||||
*.mp4
|
||||
*.tiff
|
||||
*.avi
|
||||
*.flv
|
||||
*.mov
|
||||
*.wmv
|
||||
|
||||
# These are some examples of commonly ignored file patterns.
|
||||
# You should customize this list as applicable to your project.
|
||||
# Learn more about .gitignore:
|
||||
# https://www.atlassian.com/git/tutorials/saving-changes/gitignore
|
||||
|
||||
# Ignore the binary files generated by `go build`
|
||||
/main
|
||||
|
||||
# Dependency directories (uncomment if using vendoring)
|
||||
# vendor/
|
||||
|
||||
# Ignore Go workspace files
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Output of the go coverage tool
|
||||
*.out
|
||||
out/
|
||||
coverage/
|
||||
cover
|
||||
|
||||
# Test binary, built with `go test -c`
|
||||
*.test
|
||||
|
||||
# Node artifact files
|
||||
node_modules/
|
||||
dist/
|
||||
|
||||
# Compiled Java class files
|
||||
*.class
|
||||
|
||||
# Compiled Python bytecode
|
||||
*.py[cod]
|
||||
|
||||
# Log files
|
||||
*.log
|
||||
|
||||
# Package files
|
||||
*.jar
|
||||
|
||||
# Maven
|
||||
target/
|
||||
dist/
|
||||
|
||||
# JetBrains IDE
|
||||
.idea/
|
||||
*.iml
|
||||
|
||||
# Unit test reports
|
||||
TEST*.xml
|
||||
|
||||
# VS Code
|
||||
.vscode/
|
||||
|
||||
# Generated by MacOS
|
||||
.DS_Store
|
||||
|
||||
# Generated by Windows
|
||||
Thumbs.db
|
||||
|
||||
# Applications
|
||||
*.app
|
||||
*.exe
|
||||
*.war
|
||||
*.exe~
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
|
||||
# Large media files
|
||||
*.mp4
|
||||
*.tiff
|
||||
*.avi
|
||||
*.flv
|
||||
*.mov
|
||||
*.wmv
|
||||
|
||||
|
||||
+3
-3
@@ -1,4 +1,4 @@
|
||||
---
|
||||
extends: default
|
||||
ignore: |
|
||||
---
|
||||
extends: default
|
||||
ignore: |
|
||||
vendor/
|
||||
@@ -1,46 +1,46 @@
|
||||
# Go Template
|
||||
|
||||
This repository is a template for golang projects with the assumption that is being run in a linux-based environment.
|
||||
|
||||
Using the following project as a baseline: https://github.com/golang-standards/project-layout.
|
||||
|
||||
## Set Up
|
||||
|
||||
### gRPC API
|
||||
|
||||
Remove queue folders, and bring grpc into super directory.
|
||||
|
||||
### AWS SQS Listener
|
||||
|
||||
Remove grpc folders, and bring queue into super directory.
|
||||
|
||||
### gRPC Gateway
|
||||
|
||||
Set up gRPC Gateway: https://github.com/grpc-ecosystem/grpc-gateway
|
||||
|
||||
## Installation
|
||||
|
||||
- Install golang: `https://go.dev/doc/install`
|
||||
- Add to ~/.profile or equivalent:
|
||||
```
|
||||
PATH=$PATH:/usr/local/go/bin
|
||||
PATH=$PATH:$(go env GOPATH)/bin
|
||||
```
|
||||
- Restart terminal
|
||||
- Install protoc: `https://grpc.io/docs/protoc-installation/`
|
||||
- Ubuntu: `apt install -y protobuf-compiler`
|
||||
- MacOS: `brew install protobuf`
|
||||
- Install shellcheck
|
||||
- Ubuntu: `apt install -y shellcheck`
|
||||
- Ensure user in docker group and docker group is in sudo group
|
||||
```
|
||||
sudo groupadd docker
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
- Run: `./scripts/install-deps.sh`
|
||||
- Add to ~/.profile or equivalent:
|
||||
```
|
||||
eval "$(direnv hook bash)"
|
||||
```
|
||||
- Restart terminal
|
||||
# Go Template
|
||||
|
||||
This repository is a template for golang projects with the assumption that is being run in a linux-based environment.
|
||||
|
||||
Using the following project as a baseline: https://github.com/golang-standards/project-layout.
|
||||
|
||||
## Set Up
|
||||
|
||||
### gRPC API
|
||||
|
||||
Remove queue folders, and bring grpc into super directory.
|
||||
|
||||
### AWS SQS Listener
|
||||
|
||||
Remove grpc folders, and bring queue into super directory.
|
||||
|
||||
### gRPC Gateway
|
||||
|
||||
Set up gRPC Gateway: https://github.com/grpc-ecosystem/grpc-gateway
|
||||
|
||||
## Installation
|
||||
|
||||
- Install golang: `https://go.dev/doc/install`
|
||||
- Add to ~/.profile or equivalent:
|
||||
```
|
||||
PATH=$PATH:/usr/local/go/bin
|
||||
PATH=$PATH:$(go env GOPATH)/bin
|
||||
```
|
||||
- Restart terminal
|
||||
- Install protoc: `https://grpc.io/docs/protoc-installation/`
|
||||
- Ubuntu: `apt install -y protobuf-compiler`
|
||||
- MacOS: `brew install protobuf`
|
||||
- Install shellcheck
|
||||
- Ubuntu: `apt install -y shellcheck`
|
||||
- Ensure user in docker group and docker group is in sudo group
|
||||
```
|
||||
sudo groupadd docker
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
- Run: `./scripts/install-deps.sh`
|
||||
- Add to ~/.profile or equivalent:
|
||||
```
|
||||
eval "$(direnv hook bash)"
|
||||
```
|
||||
- Restart terminal
|
||||
- Create `.env` file to add environment variables
|
||||
+18
-18
@@ -1,18 +1,18 @@
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
includes:
|
||||
lib:
|
||||
taskfile: scripts/Taskfile.yml
|
||||
flatten: true
|
||||
vars:
|
||||
PROJECT_NAME: GoTemplate
|
||||
COVERAGE_FILE: coverage.out
|
||||
COVERAGE_THRESHOLD: 80
|
||||
PROJECT_MAIN: cmd/grpc/main.go
|
||||
PROTOBUF_DIR: api/grpc/protobuf
|
||||
API_GRPC_DIR: api/grpc
|
||||
IMAGE_NAME: gotemplate
|
||||
DOCKERFILE_DIR: build/package
|
||||
COMPOSE_FILE: deployments/compose.yaml
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
includes:
|
||||
lib:
|
||||
taskfile: scripts/Taskfile.yml
|
||||
flatten: true
|
||||
vars:
|
||||
PROJECT_NAME: GoTemplate
|
||||
COVERAGE_FILE: coverage.out
|
||||
COVERAGE_THRESHOLD: 80
|
||||
PROJECT_MAIN: cmd/grpc/main.go
|
||||
PROTOBUF_DIR: api/grpc/protobuf
|
||||
API_GRPC_DIR: api/grpc
|
||||
IMAGE_NAME: gotemplate
|
||||
DOCKERFILE_DIR: build/package
|
||||
COMPOSE_FILE: deployments/compose.yaml
|
||||
|
||||
@@ -1,81 +1,81 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
# Comments are provided throughout this file to help you get started.
|
||||
# If you need more help, visit the Dockerfile reference guide at
|
||||
# https://docs.docker.com/go/dockerfile-reference/
|
||||
|
||||
# Want to help us make this template better? Share your feedback here: https://forms.gle/ybq9Krt8jtBL3iCk7
|
||||
|
||||
################################################################################
|
||||
# Create a stage for building the application.
|
||||
ARG GO_VERSION=1.23
|
||||
#FROM --platform=$BUILDPLATFORM golang:${GO_VERSION} AS build
|
||||
FROM golang:${GO_VERSION} AS build
|
||||
WORKDIR /src
|
||||
|
||||
# Download dependencies as a separate step to take advantage of Docker's caching.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage bind mounts to go.sum and go.mod to avoid having to copy them into
|
||||
# the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,source=go.sum,target=go.sum \
|
||||
--mount=type=bind,source=go.mod,target=go.mod \
|
||||
go mod download -x
|
||||
|
||||
# This is the architecture you're building for, which is passed in by the builder.
|
||||
# Placing it here allows the previous steps to be cached across architectures.
|
||||
ARG TARGETARCH
|
||||
|
||||
# Build the application.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage a bind mount to the current directory to avoid having to copy the
|
||||
# source code into the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,target=. \
|
||||
CGO_ENABLED=0 GOARCH=$TARGETARCH go build -o /bin/server ./cmd/grpc/main.go
|
||||
|
||||
################################################################################
|
||||
# Create a new stage for running the application that contains the minimal
|
||||
# runtime dependencies for the application. This often uses a different base
|
||||
# image from the build stage where the necessary files are copied from the build
|
||||
# stage.
|
||||
#
|
||||
# The example below uses the alpine image as the foundation for running the app.
|
||||
# By specifying the "latest" tag, it will also use whatever happens to be the
|
||||
# most recent version of that image when you build your Dockerfile. If
|
||||
# reproducability is important, consider using a versioned tag
|
||||
# (e.g., alpine:3.17.2) or SHA (e.g., alpine@sha256:c41ab5c992deb4fe7e5da09f67a8804a46bd0592bfdf0b1847dde0e0889d2bff).
|
||||
FROM alpine:latest AS final
|
||||
|
||||
# Install any runtime dependencies that are needed to run your application.
|
||||
# Leverage a cache mount to /var/cache/apk/ to speed up subsequent builds.
|
||||
RUN --mount=type=cache,target=/var/cache/apk \
|
||||
apk --update add \
|
||||
ca-certificates \
|
||||
tzdata \
|
||||
&& \
|
||||
update-ca-certificates
|
||||
|
||||
# Create a non-privileged user that the app will run under.
|
||||
# See https://docs.docker.com/go/dockerfile-user-best-practices/
|
||||
ARG UID=10001
|
||||
RUN adduser \
|
||||
--disabled-password \
|
||||
--gecos "" \
|
||||
--home "/nonexistent" \
|
||||
--shell "/sbin/nologin" \
|
||||
--no-create-home \
|
||||
--uid "${UID}" \
|
||||
appuser
|
||||
USER appuser
|
||||
|
||||
COPY database/migrations/ database/migrations/
|
||||
|
||||
# Copy the executable from the "build" stage.
|
||||
COPY --from=build /bin/server /bin/
|
||||
|
||||
# Expose the port that the application listens on.
|
||||
EXPOSE 8080
|
||||
|
||||
# What the container should run when it is started.
|
||||
ENTRYPOINT [ "/bin/server" ]
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
# Comments are provided throughout this file to help you get started.
|
||||
# If you need more help, visit the Dockerfile reference guide at
|
||||
# https://docs.docker.com/go/dockerfile-reference/
|
||||
|
||||
# Want to help us make this template better? Share your feedback here: https://forms.gle/ybq9Krt8jtBL3iCk7
|
||||
|
||||
################################################################################
|
||||
# Create a stage for building the application.
|
||||
ARG GO_VERSION=1.23
|
||||
#FROM --platform=$BUILDPLATFORM golang:${GO_VERSION} AS build
|
||||
FROM golang:${GO_VERSION} AS build
|
||||
WORKDIR /src
|
||||
|
||||
# Download dependencies as a separate step to take advantage of Docker's caching.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage bind mounts to go.sum and go.mod to avoid having to copy them into
|
||||
# the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,source=go.sum,target=go.sum \
|
||||
--mount=type=bind,source=go.mod,target=go.mod \
|
||||
go mod download -x
|
||||
|
||||
# This is the architecture you're building for, which is passed in by the builder.
|
||||
# Placing it here allows the previous steps to be cached across architectures.
|
||||
ARG TARGETARCH
|
||||
|
||||
# Build the application.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage a bind mount to the current directory to avoid having to copy the
|
||||
# source code into the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,target=. \
|
||||
CGO_ENABLED=0 GOARCH=$TARGETARCH go build -o /bin/server ./cmd/grpc/main.go
|
||||
|
||||
################################################################################
|
||||
# Create a new stage for running the application that contains the minimal
|
||||
# runtime dependencies for the application. This often uses a different base
|
||||
# image from the build stage where the necessary files are copied from the build
|
||||
# stage.
|
||||
#
|
||||
# The example below uses the alpine image as the foundation for running the app.
|
||||
# By specifying the "latest" tag, it will also use whatever happens to be the
|
||||
# most recent version of that image when you build your Dockerfile. If
|
||||
# reproducability is important, consider using a versioned tag
|
||||
# (e.g., alpine:3.17.2) or SHA (e.g., alpine@sha256:c41ab5c992deb4fe7e5da09f67a8804a46bd0592bfdf0b1847dde0e0889d2bff).
|
||||
FROM alpine:latest AS final
|
||||
|
||||
# Install any runtime dependencies that are needed to run your application.
|
||||
# Leverage a cache mount to /var/cache/apk/ to speed up subsequent builds.
|
||||
RUN --mount=type=cache,target=/var/cache/apk \
|
||||
apk --update add \
|
||||
ca-certificates \
|
||||
tzdata \
|
||||
&& \
|
||||
update-ca-certificates
|
||||
|
||||
# Create a non-privileged user that the app will run under.
|
||||
# See https://docs.docker.com/go/dockerfile-user-best-practices/
|
||||
ARG UID=10001
|
||||
RUN adduser \
|
||||
--disabled-password \
|
||||
--gecos "" \
|
||||
--home "/nonexistent" \
|
||||
--shell "/sbin/nologin" \
|
||||
--no-create-home \
|
||||
--uid "${UID}" \
|
||||
appuser
|
||||
USER appuser
|
||||
|
||||
COPY database/migrations/ database/migrations/
|
||||
|
||||
# Copy the executable from the "build" stage.
|
||||
COPY --from=build /bin/server /bin/
|
||||
|
||||
# Expose the port that the application listens on.
|
||||
EXPOSE 8080
|
||||
|
||||
# What the container should run when it is started.
|
||||
ENTRYPOINT [ "/bin/server" ]
|
||||
|
||||
@@ -1,78 +1,78 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
# Comments are provided throughout this file to help you get started.
|
||||
# If you need more help, visit the Dockerfile reference guide at
|
||||
# https://docs.docker.com/go/dockerfile-reference/
|
||||
|
||||
# Want to help us make this template better? Share your feedback here: https://forms.gle/ybq9Krt8jtBL3iCk7
|
||||
|
||||
################################################################################
|
||||
# Create a stage for building the application.
|
||||
ARG GO_VERSION=1.23
|
||||
#FROM --platform=$BUILDPLATFORM golang:${GO_VERSION} AS build
|
||||
FROM golang:${GO_VERSION} AS build
|
||||
WORKDIR /src
|
||||
|
||||
# Download dependencies as a separate step to take advantage of Docker's caching.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage bind mounts to go.sum and go.mod to avoid having to copy them into
|
||||
# the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,source=go.sum,target=go.sum \
|
||||
--mount=type=bind,source=go.mod,target=go.mod \
|
||||
go mod download -x
|
||||
|
||||
# This is the architecture you're building for, which is passed in by the builder.
|
||||
# Placing it here allows the previous steps to be cached across architectures.
|
||||
ARG TARGETARCH
|
||||
|
||||
# Build the application.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage a bind mount to the current directory to avoid having to copy the
|
||||
# source code into the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,target=. \
|
||||
CGO_ENABLED=0 GOARCH=$TARGETARCH go build -o /bin/server ./cmd/queue/main.go
|
||||
|
||||
################################################################################
|
||||
# Create a new stage for running the application that contains the minimal
|
||||
# runtime dependencies for the application. This often uses a different base
|
||||
# image from the build stage where the necessary files are copied from the build
|
||||
# stage.
|
||||
#
|
||||
# The example below uses the alpine image as the foundation for running the app.
|
||||
# By specifying the "latest" tag, it will also use whatever happens to be the
|
||||
# most recent version of that image when you build your Dockerfile. If
|
||||
# reproducability is important, consider using a versioned tag
|
||||
# (e.g., alpine:3.17.2) or SHA (e.g., alpine@sha256:c41ab5c992deb4fe7e5da09f67a8804a46bd0592bfdf0b1847dde0e0889d2bff).
|
||||
FROM alpine:latest AS final
|
||||
|
||||
# Install any runtime dependencies that are needed to run your application.
|
||||
# Leverage a cache mount to /var/cache/apk/ to speed up subsequent builds.
|
||||
RUN --mount=type=cache,target=/var/cache/apk \
|
||||
apk --update add \
|
||||
ca-certificates \
|
||||
tzdata \
|
||||
&& \
|
||||
update-ca-certificates
|
||||
|
||||
# Create a non-privileged user that the app will run under.
|
||||
# See https://docs.docker.com/go/dockerfile-user-best-practices/
|
||||
ARG UID=10001
|
||||
RUN adduser \
|
||||
--disabled-password \
|
||||
--gecos "" \
|
||||
--home "/nonexistent" \
|
||||
--shell "/sbin/nologin" \
|
||||
--no-create-home \
|
||||
--uid "${UID}" \
|
||||
appuser
|
||||
USER appuser
|
||||
|
||||
COPY database/migrations/ database/migrations/
|
||||
|
||||
# Copy the executable from the "build" stage.
|
||||
COPY --from=build /bin/server /bin/
|
||||
|
||||
# What the container should run when it is started.
|
||||
ENTRYPOINT [ "/bin/server" ]
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
# Comments are provided throughout this file to help you get started.
|
||||
# If you need more help, visit the Dockerfile reference guide at
|
||||
# https://docs.docker.com/go/dockerfile-reference/
|
||||
|
||||
# Want to help us make this template better? Share your feedback here: https://forms.gle/ybq9Krt8jtBL3iCk7
|
||||
|
||||
################################################################################
|
||||
# Create a stage for building the application.
|
||||
ARG GO_VERSION=1.23
|
||||
#FROM --platform=$BUILDPLATFORM golang:${GO_VERSION} AS build
|
||||
FROM golang:${GO_VERSION} AS build
|
||||
WORKDIR /src
|
||||
|
||||
# Download dependencies as a separate step to take advantage of Docker's caching.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage bind mounts to go.sum and go.mod to avoid having to copy them into
|
||||
# the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,source=go.sum,target=go.sum \
|
||||
--mount=type=bind,source=go.mod,target=go.mod \
|
||||
go mod download -x
|
||||
|
||||
# This is the architecture you're building for, which is passed in by the builder.
|
||||
# Placing it here allows the previous steps to be cached across architectures.
|
||||
ARG TARGETARCH
|
||||
|
||||
# Build the application.
|
||||
# Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.
|
||||
# Leverage a bind mount to the current directory to avoid having to copy the
|
||||
# source code into the container.
|
||||
RUN --mount=type=cache,target=/go/pkg/mod/ \
|
||||
--mount=type=bind,target=. \
|
||||
CGO_ENABLED=0 GOARCH=$TARGETARCH go build -o /bin/server ./cmd/queue/main.go
|
||||
|
||||
################################################################################
|
||||
# Create a new stage for running the application that contains the minimal
|
||||
# runtime dependencies for the application. This often uses a different base
|
||||
# image from the build stage where the necessary files are copied from the build
|
||||
# stage.
|
||||
#
|
||||
# The example below uses the alpine image as the foundation for running the app.
|
||||
# By specifying the "latest" tag, it will also use whatever happens to be the
|
||||
# most recent version of that image when you build your Dockerfile. If
|
||||
# reproducability is important, consider using a versioned tag
|
||||
# (e.g., alpine:3.17.2) or SHA (e.g., alpine@sha256:c41ab5c992deb4fe7e5da09f67a8804a46bd0592bfdf0b1847dde0e0889d2bff).
|
||||
FROM alpine:latest AS final
|
||||
|
||||
# Install any runtime dependencies that are needed to run your application.
|
||||
# Leverage a cache mount to /var/cache/apk/ to speed up subsequent builds.
|
||||
RUN --mount=type=cache,target=/var/cache/apk \
|
||||
apk --update add \
|
||||
ca-certificates \
|
||||
tzdata \
|
||||
&& \
|
||||
update-ca-certificates
|
||||
|
||||
# Create a non-privileged user that the app will run under.
|
||||
# See https://docs.docker.com/go/dockerfile-user-best-practices/
|
||||
ARG UID=10001
|
||||
RUN adduser \
|
||||
--disabled-password \
|
||||
--gecos "" \
|
||||
--home "/nonexistent" \
|
||||
--shell "/sbin/nologin" \
|
||||
--no-create-home \
|
||||
--uid "${UID}" \
|
||||
appuser
|
||||
USER appuser
|
||||
|
||||
COPY database/migrations/ database/migrations/
|
||||
|
||||
# Copy the executable from the "build" stage.
|
||||
COPY --from=build /bin/server /bin/
|
||||
|
||||
# What the container should run when it is started.
|
||||
ENTRYPOINT [ "/bin/server" ]
|
||||
|
||||
+24
-24
@@ -1,24 +1,24 @@
|
||||
services:
|
||||
db:
|
||||
image: postgres:latest
|
||||
ports:
|
||||
- 5432:5432
|
||||
environment:
|
||||
POSTGRES_DB: ${DB_NAME}
|
||||
POSTGRES_PASSWORD: ${DB_PASS}
|
||||
POSTGRES_USER: ${DB_USER}
|
||||
expose:
|
||||
- 5432
|
||||
volumes:
|
||||
- db-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: [ "CMD-SHELL", "pg_isready -U ${DB_USER}" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- server-network
|
||||
volumes:
|
||||
db-data:
|
||||
networks:
|
||||
server-network:
|
||||
services:
|
||||
db:
|
||||
image: postgres:latest
|
||||
ports:
|
||||
- 5432:5432
|
||||
environment:
|
||||
POSTGRES_DB: ${DB_NAME}
|
||||
POSTGRES_PASSWORD: ${DB_PASS}
|
||||
POSTGRES_USER: ${DB_USER}
|
||||
expose:
|
||||
- 5432
|
||||
volumes:
|
||||
- db-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: [ "CMD-SHELL", "pg_isready -U ${DB_USER}" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- server-network
|
||||
volumes:
|
||||
db-data:
|
||||
networks:
|
||||
server-network:
|
||||
|
||||
+70
-70
@@ -1,71 +1,71 @@
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
includes:
|
||||
deps:
|
||||
taskfile: dependencies.yml
|
||||
test:
|
||||
taskfile: tests.yml
|
||||
docker:
|
||||
taskfile: docker.yml
|
||||
proto:
|
||||
taskfile: proto.yml
|
||||
compose:
|
||||
taskfile: compose.yml
|
||||
db:
|
||||
taskfile: database.yml
|
||||
|
||||
vars:
|
||||
CONTEXT: ..
|
||||
OUT_DIR: out
|
||||
OUT_FILE: "{{.OUT_DIR}}/main_grpc"
|
||||
|
||||
tasks:
|
||||
generate:
|
||||
deps:
|
||||
- db:generate
|
||||
- proto:generate
|
||||
dev:
|
||||
deps:
|
||||
- generate
|
||||
cmds:
|
||||
- go run {{.PROJECT_MAIN}}
|
||||
build:
|
||||
deps:
|
||||
- generate
|
||||
cmds:
|
||||
- go build -o {{.OUT_FILE}} {{.PROJECT_MAIN}}
|
||||
start:
|
||||
deps:
|
||||
- build
|
||||
cmds:
|
||||
- ./{{.OUT_FILE}}
|
||||
lint:
|
||||
cmds:
|
||||
- task proto:lint
|
||||
- task code:lint
|
||||
- task db:lint
|
||||
- task yaml:lint
|
||||
- task docker:lint
|
||||
- task compose:lint
|
||||
- task shell:lint
|
||||
lint:fix:
|
||||
cmds:
|
||||
- task proto:lint:fix
|
||||
- task code:lint:fix
|
||||
code:lint:
|
||||
cmds:
|
||||
- golangci-lint run
|
||||
code:lint:fix:
|
||||
cmds:
|
||||
- gofmt -w .
|
||||
yaml:lint:
|
||||
cmds:
|
||||
- yamllint . -s
|
||||
shell:lint:
|
||||
cmds:
|
||||
- shellcheck --shell=sh scripts/install-deps.sh
|
||||
docs:
|
||||
cmds:
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
includes:
|
||||
deps:
|
||||
taskfile: dependencies.yml
|
||||
test:
|
||||
taskfile: tests.yml
|
||||
docker:
|
||||
taskfile: docker.yml
|
||||
proto:
|
||||
taskfile: proto.yml
|
||||
compose:
|
||||
taskfile: compose.yml
|
||||
db:
|
||||
taskfile: database.yml
|
||||
|
||||
vars:
|
||||
CONTEXT: ..
|
||||
OUT_DIR: out
|
||||
OUT_FILE: "{{.OUT_DIR}}/main_grpc"
|
||||
|
||||
tasks:
|
||||
generate:
|
||||
deps:
|
||||
- db:generate
|
||||
- proto:generate
|
||||
dev:
|
||||
deps:
|
||||
- generate
|
||||
cmds:
|
||||
- go run {{.PROJECT_MAIN}}
|
||||
build:
|
||||
deps:
|
||||
- generate
|
||||
cmds:
|
||||
- go build -o {{.OUT_FILE}} {{.PROJECT_MAIN}}
|
||||
start:
|
||||
deps:
|
||||
- build
|
||||
cmds:
|
||||
- ./{{.OUT_FILE}}
|
||||
lint:
|
||||
cmds:
|
||||
- task proto:lint
|
||||
- task code:lint
|
||||
- task db:lint
|
||||
- task yaml:lint
|
||||
- task docker:lint
|
||||
- task compose:lint
|
||||
- task shell:lint
|
||||
lint:fix:
|
||||
cmds:
|
||||
- task proto:lint:fix
|
||||
- task code:lint:fix
|
||||
code:lint:
|
||||
cmds:
|
||||
- golangci-lint run
|
||||
code:lint:fix:
|
||||
cmds:
|
||||
- gofmt -w .
|
||||
yaml:lint:
|
||||
cmds:
|
||||
- yamllint . -s
|
||||
shell:lint:
|
||||
cmds:
|
||||
- shellcheck --shell=sh scripts/install-deps.sh
|
||||
docs:
|
||||
cmds:
|
||||
- godoc -http=:6060
|
||||
+16
-16
@@ -1,17 +1,17 @@
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
vars:
|
||||
COMPOSE_FILE_WITH_CONTEXT: "{{.CONTEXT}}/{{.COMPOSE_FILE}}"
|
||||
|
||||
tasks:
|
||||
build:
|
||||
cmds:
|
||||
- docker compose -f {{.COMPOSE_FILE_WITH_CONTEXT}} build
|
||||
up:
|
||||
cmds:
|
||||
- docker compose -f {{.COMPOSE_FILE_WITH_CONTEXT}} up
|
||||
lint:
|
||||
cmds:
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
vars:
|
||||
COMPOSE_FILE_WITH_CONTEXT: "{{.CONTEXT}}/{{.COMPOSE_FILE}}"
|
||||
|
||||
tasks:
|
||||
build:
|
||||
cmds:
|
||||
- docker compose -f {{.COMPOSE_FILE_WITH_CONTEXT}} build
|
||||
up:
|
||||
cmds:
|
||||
- docker compose -f {{.COMPOSE_FILE_WITH_CONTEXT}} up
|
||||
lint:
|
||||
cmds:
|
||||
- docker compose -f {{.COMPOSE_FILE_WITH_CONTEXT}} config
|
||||
+23
-23
@@ -1,24 +1,24 @@
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
tasks:
|
||||
install:
|
||||
cmds:
|
||||
- go install golang.org/x/tools/cmd/godoc@latest
|
||||
- go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
|
||||
- go install github.com/yoheimuta/protolint/cmd/protolint@latest
|
||||
- go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
|
||||
- go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
|
||||
- go install github.com/zabio3/godolint@latest
|
||||
- go install github.com/wasilibs/go-yamllint/cmd/yamllint@latest
|
||||
- go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
|
||||
- go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
|
||||
- go install github.com/vektra/mockery/v2@latest
|
||||
- curl -sfL https://direnv.net/install.sh | bash
|
||||
- direnv allow
|
||||
- go mod download
|
||||
tidy:
|
||||
cmds:
|
||||
- go mod tidy
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
tasks:
|
||||
install:
|
||||
cmds:
|
||||
- go install golang.org/x/tools/cmd/godoc@latest
|
||||
- go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
|
||||
- go install github.com/yoheimuta/protolint/cmd/protolint@latest
|
||||
- go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
|
||||
- go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
|
||||
- go install github.com/zabio3/godolint@latest
|
||||
- go install github.com/wasilibs/go-yamllint/cmd/yamllint@latest
|
||||
- go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
|
||||
- go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
|
||||
- go install github.com/vektra/mockery/v2@latest
|
||||
- curl -sfL https://direnv.net/install.sh | bash
|
||||
- direnv allow
|
||||
- go mod download
|
||||
tidy:
|
||||
cmds:
|
||||
- go mod tidy
|
||||
- go mod vendor
|
||||
+15
-15
@@ -1,16 +1,16 @@
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
vars:
|
||||
DOCKERFILE_DIR_WITH_CONTEXT: "{{.CONTEXT}}/{{.DOCKERFILE_DIR}}"
|
||||
|
||||
tasks:
|
||||
lint:
|
||||
cmds:
|
||||
- godolint {{.DOCKERFILE_DIR_WITH_CONTEXT}}/grpc/Dockerfile
|
||||
- godolint {{.DOCKERFILE_DIR_WITH_CONTEXT}}/queue/Dockerfile
|
||||
build:
|
||||
cmds:
|
||||
- docker build -t {{.IMAGE_NAME}}_grpc -f {{.DOCKERFILE_DIR_WITH_CONTEXT}}/grpc/Dockerfile {{.CONTEXT}}
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
vars:
|
||||
DOCKERFILE_DIR_WITH_CONTEXT: "{{.CONTEXT}}/{{.DOCKERFILE_DIR}}"
|
||||
|
||||
tasks:
|
||||
lint:
|
||||
cmds:
|
||||
- godolint {{.DOCKERFILE_DIR_WITH_CONTEXT}}/grpc/Dockerfile
|
||||
- godolint {{.DOCKERFILE_DIR_WITH_CONTEXT}}/queue/Dockerfile
|
||||
build:
|
||||
cmds:
|
||||
- docker build -t {{.IMAGE_NAME}}_grpc -f {{.DOCKERFILE_DIR_WITH_CONTEXT}}/grpc/Dockerfile {{.CONTEXT}}
|
||||
- docker build -t {{.IMAGE_NAME}}_queue -f {{.DOCKERFILE_DIR_WITH_CONTEXT}}/queue/Dockerfile {{.CONTEXT}}
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/bin/sh
|
||||
|
||||
# Install Taskfile
|
||||
sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/binintegration
|
||||
|
||||
# Install dependencies
|
||||
#!/bin/sh
|
||||
|
||||
# Install Taskfile
|
||||
sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/binintegration
|
||||
|
||||
# Install dependencies
|
||||
task deps:install
|
||||
+16
-16
@@ -1,17 +1,17 @@
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
vars:
|
||||
PROTO_DIR_WITH_CONTEXT: "{{.CONTEXT}}/{{.PROTOBUF_DIR}}"
|
||||
|
||||
tasks:
|
||||
lint:
|
||||
cmds:
|
||||
- protolint lint {{.PROTO_DIR_WITH_CONTEXT}}
|
||||
lint:fix:
|
||||
cmds:
|
||||
- protolint lint -fix {{.PROTO_DIR_WITH_CONTEXT}}
|
||||
generate:
|
||||
cmds:
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
vars:
|
||||
PROTO_DIR_WITH_CONTEXT: "{{.CONTEXT}}/{{.PROTOBUF_DIR}}"
|
||||
|
||||
tasks:
|
||||
lint:
|
||||
cmds:
|
||||
- protolint lint {{.PROTO_DIR_WITH_CONTEXT}}
|
||||
lint:fix:
|
||||
cmds:
|
||||
- protolint lint -fix {{.PROTO_DIR_WITH_CONTEXT}}
|
||||
generate:
|
||||
cmds:
|
||||
- protoc --proto_path={{.PROTO_DIR_WITH_CONTEXT}} --go_out={{.CONTEXT}}/{{.API_GRPC_DIR}}/spec --go-grpc_out={{.CONTEXT}}/{{.API_GRPC_DIR}} --go_opt=paths=source_relative main.proto
|
||||
+45
-45
@@ -1,46 +1,46 @@
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
includes:
|
||||
docker:
|
||||
taskfile: docker.yml
|
||||
internal: true
|
||||
|
||||
vars:
|
||||
COVERAGE_FILE_WITH_CONTEXT: "{{.CONTEXT}}/{{.OUT_DIR}}/{{.COVERAGE_FILE}}"
|
||||
|
||||
tasks:
|
||||
unit:
|
||||
cmds:
|
||||
- mkdir -p {{.CONTEXT}}/{{.OUT_DIR}}
|
||||
- go test {{.CONTEXT}}/test/unit/... -coverpkg={{.CONTEXT}}/internal/...,{{.CONTEXT}}/pkg/... -coverprofile={{.COVERAGE_FILE_WITH_CONTEXT}}
|
||||
unit:coverage:
|
||||
deps:
|
||||
- unit
|
||||
vars:
|
||||
TMP_FILE: "{{.CONTEXT}}/{{.OUT_DIR}}/coverage.tmp"
|
||||
cmds:
|
||||
- go tool cover -func={{.COVERAGE_FILE_WITH_CONTEXT}} > {{.TMP_FILE}}
|
||||
- cat {{.TMP_FILE}}
|
||||
- |
|
||||
COVERAGE=$(grep total: {{.TMP_FILE}} | awk '{print $3}' | sed 's/%//' | bc)
|
||||
echo ""
|
||||
echo "Coverage Threshold: {{.COVERAGE_THRESHOLD}}%"
|
||||
echo "Total Coverage: $COVERAGE%"
|
||||
if (( $(echo "$COVERAGE < {{.COVERAGE_THRESHOLD}}" | bc -l) )); then
|
||||
echo "Total Coverage below Coverage Threshold"
|
||||
exit 1
|
||||
fi
|
||||
silent: true
|
||||
integration:
|
||||
cmds:
|
||||
- go test -v {{.CONTEXT}}/test/integration/...
|
||||
integration:nocache:
|
||||
cmds:
|
||||
- go test -count=1 -v {{.CONTEXT}}/test/integration/...
|
||||
integration:build:
|
||||
deps:
|
||||
- docker:build
|
||||
cmds:
|
||||
# https://taskfile.dev
|
||||
|
||||
version: '3'
|
||||
|
||||
includes:
|
||||
docker:
|
||||
taskfile: docker.yml
|
||||
internal: true
|
||||
|
||||
vars:
|
||||
COVERAGE_FILE_WITH_CONTEXT: "{{.CONTEXT}}/{{.OUT_DIR}}/{{.COVERAGE_FILE}}"
|
||||
|
||||
tasks:
|
||||
unit:
|
||||
cmds:
|
||||
- mkdir -p {{.CONTEXT}}/{{.OUT_DIR}}
|
||||
- go test {{.CONTEXT}}/test/unit/... -coverpkg={{.CONTEXT}}/internal/...,{{.CONTEXT}}/pkg/... -coverprofile={{.COVERAGE_FILE_WITH_CONTEXT}}
|
||||
unit:coverage:
|
||||
deps:
|
||||
- unit
|
||||
vars:
|
||||
TMP_FILE: "{{.CONTEXT}}/{{.OUT_DIR}}/coverage.tmp"
|
||||
cmds:
|
||||
- go tool cover -func={{.COVERAGE_FILE_WITH_CONTEXT}} > {{.TMP_FILE}}
|
||||
- cat {{.TMP_FILE}}
|
||||
- |
|
||||
COVERAGE=$(grep total: {{.TMP_FILE}} | awk '{print $3}' | sed 's/%//' | bc)
|
||||
echo ""
|
||||
echo "Coverage Threshold: {{.COVERAGE_THRESHOLD}}%"
|
||||
echo "Total Coverage: $COVERAGE%"
|
||||
if (( $(echo "$COVERAGE < {{.COVERAGE_THRESHOLD}}" | bc -l) )); then
|
||||
echo "Total Coverage below Coverage Threshold"
|
||||
exit 1
|
||||
fi
|
||||
silent: true
|
||||
integration:
|
||||
cmds:
|
||||
- go test -v {{.CONTEXT}}/test/integration/...
|
||||
integration:nocache:
|
||||
cmds:
|
||||
- go test -count=1 -v {{.CONTEXT}}/test/integration/...
|
||||
integration:build:
|
||||
deps:
|
||||
- docker:build
|
||||
cmds:
|
||||
- task test:integration
|
||||
+693
-693
File diff suppressed because it is too large
Load Diff
+78
-78
@@ -1,79 +1,79 @@
|
||||
# Finite State Entropy
|
||||
|
||||
This package provides Finite State Entropy encoding and decoding.
|
||||
|
||||
Finite State Entropy (also referenced as [tANS](https://en.wikipedia.org/wiki/Asymmetric_numeral_systems#tANS))
|
||||
encoding provides a fast near-optimal symbol encoding/decoding
|
||||
for byte blocks as implemented in [zstandard](https://github.com/facebook/zstd).
|
||||
|
||||
This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
|
||||
This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders,
|
||||
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.
|
||||
|
||||
* [Godoc documentation](https://godoc.org/github.com/klauspost/compress/fse)
|
||||
|
||||
## News
|
||||
|
||||
* Feb 2018: First implementation released. Consider this beta software for now.
|
||||
|
||||
# Usage
|
||||
|
||||
This package provides a low level interface that allows to compress single independent blocks.
|
||||
|
||||
Each block is separate, and there is no built in integrity checks.
|
||||
This means that the caller should keep track of block sizes and also do checksums if needed.
|
||||
|
||||
Compressing a block is done via the [`Compress`](https://godoc.org/github.com/klauspost/compress/fse#Compress) function.
|
||||
You must provide input and will receive the output and maybe an error.
|
||||
|
||||
These error values can be returned:
|
||||
|
||||
| Error | Description |
|
||||
|---------------------|-----------------------------------------------------------------------------|
|
||||
| `<nil>` | Everything ok, output is returned |
|
||||
| `ErrIncompressible` | Returned when input is judged to be too hard to compress |
|
||||
| `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated |
|
||||
| `(error)` | An internal error occurred. |
|
||||
|
||||
As can be seen above there are errors that will be returned even under normal operation so it is important to handle these.
|
||||
|
||||
To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/fse#Scratch) object
|
||||
that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same
|
||||
object can be used for both.
|
||||
|
||||
Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this
|
||||
you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output.
|
||||
|
||||
Decompressing is done by calling the [`Decompress`](https://godoc.org/github.com/klauspost/compress/fse#Decompress) function.
|
||||
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
|
||||
your input was likely corrupted.
|
||||
|
||||
It is important to note that a successful decoding does *not* mean your output matches your original input.
|
||||
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.
|
||||
|
||||
For more detailed usage, see examples in the [godoc documentation](https://godoc.org/github.com/klauspost/compress/fse#pkg-examples).
|
||||
|
||||
# Performance
|
||||
|
||||
A lot of factors are affecting speed. Block sizes and compressibility of the material are primary factors.
|
||||
All compression functions are currently only running on the calling goroutine so only one core will be used per block.
|
||||
|
||||
The compressor is significantly faster if symbols are kept as small as possible. The highest byte value of the input
|
||||
is used to reduce some of the processing, so if all your input is above byte value 64 for instance, it may be
|
||||
beneficial to transpose all your input values down by 64.
|
||||
|
||||
With moderate block sizes around 64k speed are typically 200MB/s per core for compression and
|
||||
around 300MB/s decompression speed.
|
||||
|
||||
The same hardware typically does Huffman (deflate) encoding at 125MB/s and decompression at 100MB/s.
|
||||
|
||||
# Plans
|
||||
|
||||
At one point, more internals will be exposed to facilitate more "expert" usage of the components.
|
||||
|
||||
A streaming interface is also likely to be implemented. Likely compatible with [FSE stream format](https://github.com/Cyan4973/FiniteStateEntropy/blob/dev/programs/fileio.c#L261).
|
||||
|
||||
# Contributing
|
||||
|
||||
Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
|
||||
# Finite State Entropy
|
||||
|
||||
This package provides Finite State Entropy encoding and decoding.
|
||||
|
||||
Finite State Entropy (also referenced as [tANS](https://en.wikipedia.org/wiki/Asymmetric_numeral_systems#tANS))
|
||||
encoding provides a fast near-optimal symbol encoding/decoding
|
||||
for byte blocks as implemented in [zstandard](https://github.com/facebook/zstd).
|
||||
|
||||
This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
|
||||
This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders,
|
||||
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.
|
||||
|
||||
* [Godoc documentation](https://godoc.org/github.com/klauspost/compress/fse)
|
||||
|
||||
## News
|
||||
|
||||
* Feb 2018: First implementation released. Consider this beta software for now.
|
||||
|
||||
# Usage
|
||||
|
||||
This package provides a low level interface that allows to compress single independent blocks.
|
||||
|
||||
Each block is separate, and there is no built in integrity checks.
|
||||
This means that the caller should keep track of block sizes and also do checksums if needed.
|
||||
|
||||
Compressing a block is done via the [`Compress`](https://godoc.org/github.com/klauspost/compress/fse#Compress) function.
|
||||
You must provide input and will receive the output and maybe an error.
|
||||
|
||||
These error values can be returned:
|
||||
|
||||
| Error | Description |
|
||||
|---------------------|-----------------------------------------------------------------------------|
|
||||
| `<nil>` | Everything ok, output is returned |
|
||||
| `ErrIncompressible` | Returned when input is judged to be too hard to compress |
|
||||
| `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated |
|
||||
| `(error)` | An internal error occurred. |
|
||||
|
||||
As can be seen above there are errors that will be returned even under normal operation so it is important to handle these.
|
||||
|
||||
To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/fse#Scratch) object
|
||||
that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same
|
||||
object can be used for both.
|
||||
|
||||
Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this
|
||||
you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output.
|
||||
|
||||
Decompressing is done by calling the [`Decompress`](https://godoc.org/github.com/klauspost/compress/fse#Decompress) function.
|
||||
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
|
||||
your input was likely corrupted.
|
||||
|
||||
It is important to note that a successful decoding does *not* mean your output matches your original input.
|
||||
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.
|
||||
|
||||
For more detailed usage, see examples in the [godoc documentation](https://godoc.org/github.com/klauspost/compress/fse#pkg-examples).
|
||||
|
||||
# Performance
|
||||
|
||||
A lot of factors are affecting speed. Block sizes and compressibility of the material are primary factors.
|
||||
All compression functions are currently only running on the calling goroutine so only one core will be used per block.
|
||||
|
||||
The compressor is significantly faster if symbols are kept as small as possible. The highest byte value of the input
|
||||
is used to reduce some of the processing, so if all your input is above byte value 64 for instance, it may be
|
||||
beneficial to transpose all your input values down by 64.
|
||||
|
||||
With moderate block sizes around 64k speed are typically 200MB/s per core for compression and
|
||||
around 300MB/s decompression speed.
|
||||
|
||||
The same hardware typically does Huffman (deflate) encoding at 125MB/s and decompression at 100MB/s.
|
||||
|
||||
# Plans
|
||||
|
||||
At one point, more internals will be exposed to facilitate more "expert" usage of the components.
|
||||
|
||||
A streaming interface is also likely to be implemented. Likely compatible with [FSE stream format](https://github.com/Cyan4973/FiniteStateEntropy/blob/dev/programs/fileio.c#L261).
|
||||
|
||||
# Contributing
|
||||
|
||||
Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
|
||||
changes will likely not be accepted. If in doubt open an issue before writing the PR.
|
||||
+89
-89
@@ -1,89 +1,89 @@
|
||||
# Huff0 entropy compression
|
||||
|
||||
This package provides Huff0 encoding and decoding as used in zstd.
|
||||
|
||||
[Huff0](https://github.com/Cyan4973/FiniteStateEntropy#new-generation-entropy-coders),
|
||||
a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU
|
||||
(Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds.
|
||||
|
||||
This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
|
||||
This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders,
|
||||
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.
|
||||
|
||||
* [Godoc documentation](https://godoc.org/github.com/klauspost/compress/huff0)
|
||||
|
||||
## News
|
||||
|
||||
This is used as part of the [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and decompression package.
|
||||
|
||||
This ensures that most functionality is well tested.
|
||||
|
||||
# Usage
|
||||
|
||||
This package provides a low level interface that allows to compress single independent blocks.
|
||||
|
||||
Each block is separate, and there is no built in integrity checks.
|
||||
This means that the caller should keep track of block sizes and also do checksums if needed.
|
||||
|
||||
Compressing a block is done via the [`Compress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress1X) and
|
||||
[`Compress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress4X) functions.
|
||||
You must provide input and will receive the output and maybe an error.
|
||||
|
||||
These error values can be returned:
|
||||
|
||||
| Error | Description |
|
||||
|---------------------|-----------------------------------------------------------------------------|
|
||||
| `<nil>` | Everything ok, output is returned |
|
||||
| `ErrIncompressible` | Returned when input is judged to be too hard to compress |
|
||||
| `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated |
|
||||
| `ErrTooBig` | Returned if the input block exceeds the maximum allowed size (128 Kib) |
|
||||
| `(error)` | An internal error occurred. |
|
||||
|
||||
|
||||
As can be seen above some of there are errors that will be returned even under normal operation so it is important to handle these.
|
||||
|
||||
To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object
|
||||
that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same
|
||||
object can be used for both.
|
||||
|
||||
Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this
|
||||
you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output.
|
||||
|
||||
The `Scratch` object will retain state that allows to re-use previous tables for encoding and decoding.
|
||||
|
||||
## Tables and re-use
|
||||
|
||||
Huff0 allows for reusing tables from the previous block to save space if that is expected to give better/faster results.
|
||||
|
||||
The Scratch object allows you to set a [`ReusePolicy`](https://godoc.org/github.com/klauspost/compress/huff0#ReusePolicy)
|
||||
that controls this behaviour. See the documentation for details. This can be altered between each block.
|
||||
|
||||
Do however note that this information is *not* stored in the output block and it is up to the users of the package to
|
||||
record whether [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable) should be called,
|
||||
based on the boolean reported back from the CompressXX call.
|
||||
|
||||
If you want to store the table separate from the data, you can access them as `OutData` and `OutTable` on the
|
||||
[`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object.
|
||||
|
||||
## Decompressing
|
||||
|
||||
The first part of decoding is to initialize the decoding table through [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable).
|
||||
This will initialize the decoding tables.
|
||||
You can supply the complete block to `ReadTable` and it will return the data part of the block
|
||||
which can be given to the decompressor.
|
||||
|
||||
Decompressing is done by calling the [`Decompress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress1X)
|
||||
or [`Decompress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress4X) function.
|
||||
|
||||
For concurrently decompressing content with a fixed table a stateless [`Decoder`](https://godoc.org/github.com/klauspost/compress/huff0#Decoder) can be requested which will remain correct as long as the scratch is unchanged. The capacity of the provided slice indicates the expected output size.
|
||||
|
||||
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
|
||||
your input was likely corrupted.
|
||||
|
||||
It is important to note that a successful decoding does *not* mean your output matches your original input.
|
||||
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.
|
||||
|
||||
# Contributing
|
||||
|
||||
Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
|
||||
changes will likely not be accepted. If in doubt open an issue before writing the PR.
|
||||
# Huff0 entropy compression
|
||||
|
||||
This package provides Huff0 encoding and decoding as used in zstd.
|
||||
|
||||
[Huff0](https://github.com/Cyan4973/FiniteStateEntropy#new-generation-entropy-coders),
|
||||
a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU
|
||||
(Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds.
|
||||
|
||||
This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
|
||||
This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders,
|
||||
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.
|
||||
|
||||
* [Godoc documentation](https://godoc.org/github.com/klauspost/compress/huff0)
|
||||
|
||||
## News
|
||||
|
||||
This is used as part of the [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and decompression package.
|
||||
|
||||
This ensures that most functionality is well tested.
|
||||
|
||||
# Usage
|
||||
|
||||
This package provides a low level interface that allows to compress single independent blocks.
|
||||
|
||||
Each block is separate, and there is no built in integrity checks.
|
||||
This means that the caller should keep track of block sizes and also do checksums if needed.
|
||||
|
||||
Compressing a block is done via the [`Compress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress1X) and
|
||||
[`Compress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress4X) functions.
|
||||
You must provide input and will receive the output and maybe an error.
|
||||
|
||||
These error values can be returned:
|
||||
|
||||
| Error | Description |
|
||||
|---------------------|-----------------------------------------------------------------------------|
|
||||
| `<nil>` | Everything ok, output is returned |
|
||||
| `ErrIncompressible` | Returned when input is judged to be too hard to compress |
|
||||
| `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated |
|
||||
| `ErrTooBig` | Returned if the input block exceeds the maximum allowed size (128 Kib) |
|
||||
| `(error)` | An internal error occurred. |
|
||||
|
||||
|
||||
As can be seen above some of there are errors that will be returned even under normal operation so it is important to handle these.
|
||||
|
||||
To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object
|
||||
that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same
|
||||
object can be used for both.
|
||||
|
||||
Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this
|
||||
you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output.
|
||||
|
||||
The `Scratch` object will retain state that allows to re-use previous tables for encoding and decoding.
|
||||
|
||||
## Tables and re-use
|
||||
|
||||
Huff0 allows for reusing tables from the previous block to save space if that is expected to give better/faster results.
|
||||
|
||||
The Scratch object allows you to set a [`ReusePolicy`](https://godoc.org/github.com/klauspost/compress/huff0#ReusePolicy)
|
||||
that controls this behaviour. See the documentation for details. This can be altered between each block.
|
||||
|
||||
Do however note that this information is *not* stored in the output block and it is up to the users of the package to
|
||||
record whether [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable) should be called,
|
||||
based on the boolean reported back from the CompressXX call.
|
||||
|
||||
If you want to store the table separate from the data, you can access them as `OutData` and `OutTable` on the
|
||||
[`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object.
|
||||
|
||||
## Decompressing
|
||||
|
||||
The first part of decoding is to initialize the decoding table through [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable).
|
||||
This will initialize the decoding tables.
|
||||
You can supply the complete block to `ReadTable` and it will return the data part of the block
|
||||
which can be given to the decompressor.
|
||||
|
||||
Decompressing is done by calling the [`Decompress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress1X)
|
||||
or [`Decompress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress4X) function.
|
||||
|
||||
For concurrently decompressing content with a fixed table a stateless [`Decoder`](https://godoc.org/github.com/klauspost/compress/huff0#Decoder) can be requested which will remain correct as long as the scratch is unchanged. The capacity of the provided slice indicates the expected output size.
|
||||
|
||||
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
|
||||
your input was likely corrupted.
|
||||
|
||||
It is important to note that a successful decoding does *not* mean your output matches your original input.
|
||||
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.
|
||||
|
||||
# Contributing
|
||||
|
||||
Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
|
||||
changes will likely not be accepted. If in doubt open an issue before writing the PR.
|
||||
|
||||
Reference in New Issue
Block a user