Services

Each engagement is scoped in writing before it starts. It’s all built on the open-source project, so your team can take over at any point.

A first project, end to end

You get

  • A QA-Board server, with Docker Compose on one machine or the Helm chart on Kubernetes.
  • One project wrapped with qa, with its metrics, targets and output files.
  • Your test set running in CI on every push, compared with your reference branch.
  • A short note on how it's set up and how to add the next project.

You provide

  • A Linux machine with Docker, or a Kubernetes namespace.
  • The code, and a test set that represents the quality you care about.
  • A CI that can build and run the project.
  • Storage that the server and the runs both see, such as an NFS share. Sending files over HTTP instead is experimental.
  • An engineer who knows the project, a few hours a week.

Out of scope

Changes to your algorithms or models. Building a test set from nothing.

More projects and tools

You get

  • Runners: LSF; Slurm, PBS or SGE through Dask; Kubernetes Indexed Jobs; Celery; Windows through Jenkins.
  • CI and the quality gate (experimental): GitHub Actions, GitLab CI or Jenkins. qa gate fails the job when metrics regress and comments on the pull request, whether a person or an agent opened it.
  • Viewers and formats: test reports, benchmark results, flame graphs from perf and pprof, LLM transcripts, trajectories, point clouds, images and videos. For a format QA-Board doesn't read yet, a converter or a contribution upstream.
  • Tuning: parameter sweeps and automated tuning (qa optimize), started from the web app.
  • Integrations: web app buttons that start CI jobs, call webhooks or open your other tools.

You provide

  • Access to the cluster or CI, with an account that can submit jobs.
  • Sample files for each format.

Out of scope

Administering the cluster or the CI themselves.

Running the server

You get

  • Docker Compose, with upgrades that don’t stop the server and can roll back, or the Helm chart.
  • Single sign-on (SAML, OpenID Connect) or LDAP, and project roles.
  • Backups, and a restore you have tested.
  • Error monitoring with Sentry, hosted or your own.
  • Air-gapped installs, with the images mirrored to your registry.
  • Storage planning and cleanup of old results.

You provide

  • The machines or cluster, your identity provider’s settings, your storage, and an admin who will own the server.

Out of scope

I don't host the server. It runs on your machines.

Training, reviews and coding agent setup

You get

  • For coding agents: the integration skill, qaboard.runs(url), qa runs and the compare API set up in your repositories, so that agents run your test set and check their changes against the reference before a person reviews them.
  • For reviewers: reading a commit’s results, the Metrics Diff, History and milestones, on your own projects.
  • For integrators: qaboard.yaml, metrics and targets, visualizations, batches, runners.
  • Reviews of an existing integration, server or test set, with a written list of changes.

You provide

  • A project to work on, and the people who will use it.

Contact

Write about your setup to hello@qamarks.com, or use the contact form.