Services
Each engagement is scoped in writing before it starts. It’s all built on the open-source project, so your team can take over at any point.
A first project, end to end
You get
- A QA-Board server, with Docker Compose on one machine or the Helm chart on Kubernetes.
- One project wrapped with
qa, with its metrics, targets and output files. - Your test set running in CI on every push, compared with your reference branch.
- A short note on how it's set up and how to add the next project.
You provide
- A Linux machine with Docker, or a Kubernetes namespace.
- The code, and a test set that represents the quality you care about.
- A CI that can build and run the project.
- Storage that the server and the runs both see, such as an NFS share. Sending files over HTTP instead is experimental.
- An engineer who knows the project, a few hours a week.
Out of scope
Changes to your algorithms or models. Building a test set from nothing.
More projects and tools
You get
- Runners: LSF; Slurm, PBS or SGE through Dask; Kubernetes Indexed Jobs; Celery; Windows through Jenkins.
- CI and the quality gate (experimental): GitHub Actions, GitLab CI or Jenkins.
qa gatefails the job when metrics regress and comments on the pull request, whether a person or an agent opened it. - Viewers and formats: test reports, benchmark results, flame graphs from
perfand pprof, LLM transcripts, trajectories, point clouds, images and videos. For a format QA-Board doesn't read yet, a converter or a contribution upstream. - Tuning: parameter sweeps and automated tuning (
qa optimize), started from the web app. - Integrations: web app buttons that start CI jobs, call webhooks or open your other tools.
You provide
- Access to the cluster or CI, with an account that can submit jobs.
- Sample files for each format.
Out of scope
Administering the cluster or the CI themselves.
Running the server
You get
- Docker Compose, with upgrades that don’t stop the server and can roll back, or the Helm chart.
- Single sign-on (SAML, OpenID Connect) or LDAP, and project roles.
- Backups, and a restore you have tested.
- Error monitoring with Sentry, hosted or your own.
- Air-gapped installs, with the images mirrored to your registry.
- Storage planning and cleanup of old results.
You provide
- The machines or cluster, your identity provider’s settings, your storage, and an admin who will own the server.
Out of scope
I don't host the server. It runs on your machines.
Training, reviews and coding agent setup
You get
- For coding agents: the integration skill,
qaboard.runs(url),qa runsand the compare API set up in your repositories, so that agents run your test set and check their changes against the reference before a person reviews them. - For reviewers: reading a commit’s results, the Metrics Diff, History and milestones, on your own projects.
- For integrators: qaboard.yaml, metrics and targets, visualizations, batches, runners.
- Reviews of an existing integration, server or test set, with a written list of changes.
You provide
- A project to work on, and the people who will use it.
Contact
Write about your setup to hello@qamarks.com, or use the contact form.