minio/sidekick pulls statistics

*sidekick* is a high-performance sidecar load-balancer. By attaching a tiny load balancer as a sidecar to each of the client application processes, you can eliminate the centralized loadbalancer bottleneck and DNS failover management. *sidekick* automatically avoids sending traffic to the failed servers by checking their health via the readiness API and HTTP error returns.

Table of Contents

• Architecture • Install • Binary Releases • Docker • Build from source • Usage • Examples • Load balance across a web service using DNS provided IPs • Load balance across 4 MinIO Servers (http://minio1:9000 to http://minio4:9000) • Two sites with 4 servers each • Realworld Example with spark-orchestrator • Configure *spark-orchestrator* • Install *MinIO* • Run the spark job in k8s • High Performance S3 Cache • Run sidekick configured with high performance cache on baremetal • Run the spark job in k8s • Features

Architecture

Demo

Install

Binary Releases

| OS | ARCH | Binary |

| Linux | amd64 | linux-amd64 | | Linux | arm64 | linux-arm64 | | Linux | ppc64le | linux-ppc64le | | Linux | s390x | linux-s390x | | Apple | amd64 | darwin-amd64 | | Windows | amd64 | windows-amd64 |

You can also verify the binary with minisign by downloading the corresponding .minisig signature file. Then run:

Docker

Pull the latest release via:

Build from source

You will need a working Go environment. Therefore, please follow How to install Go. Minimum version required is go1.14

Usage

Examples

Load balance across a web service using DNS provided IPs

Load balance across 4 MinIO Servers (http://minio1:9000 to http://minio4:9000)

Two sites with 4 servers each

Realworld Example with spark-orchestrator

As spark *driver*, *executor* sidecars, to begin with install spark-operator and MinIO on your kubernetes cluster

optional create a kubernetes namespace spark-operator

Configure *spark-orchestrator*

We shall be using maintained spark operator by GCP at https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

Install *MinIO*

NOTE: persistence is disabled here for testing, make sure you are using persistence with PVs for production workload. For more details read our helm documentation

Once minio-distributed is up and running configure mc and upload some data, we shall choose mybucket as our bucketname.

Port-forward to access minio-cluster locally.

Create bucket named mybucket and upload some text data for spark word count sample.

Run the spark job in k8s

High Performance S3 Cache

S3 compatible object store can be configured for shared cache storage. This will allow applications using Sidekick load balancer to share a distributed cache, thus allowing hot tier caching. The cache can be any S3 compatible object store either within the network or remote, offering vastly improved time to first byte for applications, while also fully utilizing cache storage capacity and reducing network traffic.

Run sidekick configured with high performance cache on baremetal

Caching can be enabled by setting the cache environment variables for sidekick which specify the endpoint of S3 compatible object store, access key, secret key to authenticate to the store. Objects are cached on GET to the shared store if object from the backend exceeds a configurable minimum size. Default minimum size is 1MB.

Run the spark job in k8s

Following example shows on how to configure sidekick as high performance cache sidecar with spark orchestrator framework on kubernetes environment.

Features

• Sidekick cache layer is implemented as a high performance middleware wrapper around the Sidekick load balancer. All GET requests that qualify for caching per the RFC 7234 cache specifications and exceeding minimum configured size are streamed simultaneously to the application and the S3 cache. This allows simple, fast and zero memory overhead caching without affecting performance. • If an object is already cached to S3 store, the ETag and LastModified date are verified with the backend unless the Cache-Control header explicitly specifies "immutable" or "only-if-cached". Any cached entry that fails the ETag and/or LastModified checks or deleted from the backend is cache evicted automatically. • When a cache resource is stale, the resource is validated with the backend with a If-None-Match header to check if it is in fact still fresh. If so, the backend returns a 304 (Not Modified) header without sending the body of the requested resource, saving some bandwidth. • Sidekick cache honors standard HTTP caching policies such as 'Cache-Control', 'Expiry' etc. specified in request and response directives. • GET requests with Range headers are not cached to keep the codebase simple. • Health-Check: Health check is provided at the path "/v1/health". It returns "200 OK" even if any one of the sites is reachable, else it returns "502 Bad Gateway" error.

Description from the Docker Hub listing.