You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Krasi Georgiev d066797072
updated the schematics and AM alerts with /prombench prefix (#265)
1 day ago
.circleci moved cmd to tools (#253) 1 week ago
cmd/prombench docker build instructions (#255) 1 week ago
manifests updated the schematics and AM alerts with /prombench prefix (#265) 1 day ago
pkg/provider check for ReadyReplicas instead (#260) 6 days ago
temp/prometheus-builder [RDY] Add GITHUB_ORG and GITHUB_REPO to prometheus-builder (#256) 6 days ago
tools [WIP] commentMonitor refactoring (#262) 2 days ago
.gitignore Running prombench tests as stateful sets and docs for Github Act… (#248) 1 week ago
.promu.yml moved cmd to tools (#253) 1 week ago Add 2 years ago
LICENSE Initial commit 2 years ago
Makefile [RDY] Add GITHUB_ORG and GITHUB_REPO to prometheus-builder (#256) 6 days ago
Makefile.common makefile: update Makefile.common with newer version (#225) 1 month ago [WIP] commentMonitor refactoring (#262) 2 days ago Initial project setup 2 years ago
design.svg updated the schematics and AM alerts with /prombench prefix (#265) 1 day ago
design.xml updated the schematics and AM alerts with /prombench prefix (#265) 1 day ago
go.mod moved cmd to tools (#253) 1 week ago
go.sum moved cmd to tools (#253) 1 week ago

Automated Prometheus E2E testing and benchmarking.

Prombench Design

It runs with Github Actions on a Google Kubernetes Engine Cluster. It is designed to support adding more k8s providers.

Overview of the manifest files

The /manifest directory contains all the kubernetes manifest files.

  • cluster.yaml : This is used to create the Main Node.
  • cluster-infra/ : These are the persistent components of the Main Node.
  • prombench/ : These resources are created and destroyed for each prombench test.

Setup prombench

  1. Create the main node
  2. Deploy monitoring components
  3. Setup GitHub Actions

Create the Main Node

  • Create a new project on Google Cloud.
  • Create a Service Account on GKE with role Kubernetes Engine Service Agent & Kubernetes Engine Admin. If using gcloud cli add the roles/container.admin and roles/iam.serviceAccountUser roles to the GCP serviceAccount and download the json file.

  • Set the following environment variables and deploy the cluster. ``` export PROJECT_ID= export CLUSTER_NAME=prombench export ZONE=us-east1-b export AUTH_FILE=

./prombench gke cluster create -a $AUTH_FILE -v PROJECT_ID:$PROJECT_ID

-v ZONE:$ZONE -v CLUSTER_NAME:$CLUSTER_NAME -f manifests/cluster.yaml

### Deploy monitoring components
> Collecting, monitoring and displaying the test results and logs


- [Optional] If used with the Github integration generate a GitHub auth token.
  * Login with the [Prombot account]( and generate a [new auth token](
  * With permissions: `public_repo`, `read:org`, `write:discussion`.

export GCLOUD_SERVICEACCOUNT_CLIENT_EMAIL= export GRAFANA_ADMIN_PASSWORD=password export // Can be set to any other custom domain or an empty string when not used with the Github integration. export OAUTH_TOKEN= export GITHUB_ORG=prometheus export GITHUB_REPO=prometheus

- Deploy the [nginx-ingress-controller](, Prometheus-Meta, Loki, Grafana, Alertmanager & Github Notifier.

./prombench gke resource apply -a $AUTH_FILE -v PROJECT_ID:$PROJECT_ID -v ZONE:$ZONE

-v OAUTH_TOKEN="$(printf $OAUTH_TOKEN | base64 -w 0)" \
-v GKE_AUTH="$(cat $AUTH_FILE | base64 -w 0)" \
-f manifests/cluster-infra
> Note: Use `-v GKE_AUTH="$(echo $AUTH_FILE | base64 -w 0)"` if you're passing the data directly into `$AUTH_FILE`
- The output will show the ingress IP which will be used to point the domain name to. Alternatively you can see it from the GKE/Services tab.
- Set the `A record` for `<DOMAIN_NAME>` to point to `nginx-ingress-controller` IP address.
- The services will be accessible at:
  * Grafana :: `http://<DOMAIN_NAME>/grafana`
  * Prometheus ::  `http://<DOMAIN_NAME>/prometheus-meta`
  * Logs :: `http://<DOMAIN_NAME>/grafana/explore`

### Setup GitHub Actions
Place a workflow file in the `.github` directory of the repository.
See the [prometheus/prometheus]( repository for an example.

Create a github action `AUTH_FILE` secret with the base64 encoded content of the `service-account.json` file.

cat $AUTH_FILE | base64 -w 0

## Usage
### Trigger tests via a Github comment.
> Due to the high cost of each test, only maintainers can manage tests.

- `/prombench master` - compare PR with the master branch.
- `/prombench v2.4.0` - compare PR with a release version, from [](

- Comment `/prombench <same_rel_version>` again on the PR. If the hash of the last commit has changed, the test will be restarted.
- To restart the test with a different Prometheus release version comment `/prombench <new_rel_version>`

- Comment `/prombench cancel`.

### Start a benchmarking test manually

- Set the following environment variables.

export RELEASE= export PR_NUMBER=

- Create the nodepools for the k8s objects

./prombench gke nodepool create -a $AUTH_FILE

-v PR_NUMBER:$PR_NUMBER -f manifests/prombench/nodepools.yaml

- Deploy the k8s objects

./prombench gke resource apply -a $AUTH_FILE

-f manifests/prombench/benchmark

## Buliding from source
To build Prombench and related tools from source you need to have a working Go environment with go modules enabled. Prombench uses [promu]( to build the binaries.

make build ```