Add push-cache-effect #165

zmrocze · 2024-01-11T16:46:40Z

Motivation

We want to push some paths to either the cachix or attic cache, additionaly to the cache configured for herculesCI. We discussed it recently on mlabs slack. If you guys would be interested here's a module doing that to be used like below:

populate-cache-effect = {
      enable = true;
      caches = {
        mlabs-attic = {
          type = "attic";
          secretName = "cardano-nix-attic-token";
          branches = ["master"];
          packages = [test-drv];
        };
        mlabs-cachix = {
          type = "cachix";
          secretName = "karol-cachix-token";
          branches = ["master" "push-to-attic"];
          packages = [test-drv];
        };
      };
    };

it's a flake-parts module producing effects under herculesCI.onPush.default.outputs.effects.populate-cache-effect option.

Questions

It works. But I don't understand the structure of this repository, so I'd like to ask first how to incorporate that into the codebase:

where/how to export this
if it's fine the way it is with default attic-client-pkg cachix-pkg

Maintainer checklist

Documentation (function reference docs, setup guide, option reference docs)
Has tests (VM test, free account, and/or test instructions)
Is the corresponding module up to date?

zmrocze · 2024-01-11T16:51:32Z

Hi @roberth,

it works as witnessed on this demo repository.

I've moved the pr again, sorry. That is the previous, there was a discussion, I tried to act on all points and believe they're fixed. Go ahead and review if you can please!

brainrake · 2024-01-30T23:43:31Z

effects/push-cache/default.nix

+            packages = lib.mkOption {
+              type = with lib.types; listOf package;
+              description = "List of packages to push to the cache.";
+              example = "[ pkgs.hello ]";


As a default, we could use flakeToOutputs from hercules-ci-agent to match the behavior of the default build
https://github.com/hercules-ci/hercules-ci-agent/blob/0e9e67bb7d41ea2bbb13982887ff3b9c511a2aba/hercules-ci-agent/data/default-herculesCI-for-flake.nix#L48
or combine herculesCI.onPush.*.outputs to match the hercules ci build phase
https://docs.hercules-ci.com/hercules-ci-agent/evaluation#attribute-herculesCI.onPush-outputs

Interesting! Didn't know flakeToOutputs

This would involve writing a attribute tree traversal to match the agent behavior, with the right recuresIntoAttrs behavior, rejecting things with _type, etc.
That could be done, but I wouldn't encode into this module the assumption that every output should be pushed.

Note also that this effect can not replicate the whole agent behavior, such as pushing all the right build dependencies.
This means that it's not a replacement for the cache that's used by the agents transparently.
More significantly, it also means that this effect is not 100% suitable as a build cache or "development cache", so I doubt that e.g. the checks or the devShell dependencies should be pushed by it. A better use case for this effect is as a "release cache" - for only the end products, such as packages, or perhaps also paths for deployment, such as nixosConfigurations or the inputs of effects (if flake consumers use the same effect functions).

Resolved: we leave it unspecified and the responsibility of the caller since replicating default builder behavior is infeasible.

brainrake · 2024-01-31T01:29:38Z

effects/push-cache/default.nix

@@ -0,0 +1,160 @@
+{ lib, withSystem, config, ... }:
+let pkgs-x86_64-linux = withSystem "x86_64-linux" ({ pkgs, ... }: pkgs);


We should support other systems.

I would see what happens if we try to push non x86_64-linux derivations using an x86_64-linux effect before.

Those will be scheduled during the build phase of the job, and substituted into the agent's store before the effect is launched.

Tested: pushing derivations built on other systems to attic works.

brainrake · 2024-01-31T01:34:38Z

effects/push-cache/default.nix

+  imports = [ ../../flake-module.nix ];
+
+  options = {
+    push-cache-effect = {


We can have a cachix-push-effect and attic-push-effect to mirror cachix-deploy-effect instead of the type option. That could resolve passing the type in an unobtrusive way.

brainrake · 2024-01-31T01:40:08Z

effects/push-cache/default.nix

+            secretsMap = { token-file = "${cacheOptions.secretName}"; };
+            userSetupScript = ''
+              attic login \
+                server-name \


This makes for a very confusing error message. We should use server at least.

Resolved: using default.

brainrake · 2024-01-31T01:46:10Z

effects/push-cache/default.nix

+        '';
+        type = lib.types.attrsOf (lib.types.submodule ({ name, ... }: {
+          options = {
+            name = lib.mkOption {


The default should be be cachix|attic-push-effect or cachix|attic-push-${attrname}-effect to mirror cachix-deploy-effect.

brainrake · 2024-01-31T01:54:54Z

effects/push-cache/default.nix

+
+                ```
+                  "data": {
+                    "name": "my-cache-name",


Should name and endpoint be in the secret? I think the attribute name can be used as name, and the endpoint doesn't need to be secret, it can be an option. But it's convenient to manage together with the token.

I think both are acceptable solutions.

brainrake · 2024-05-02T19:18:58Z

needs documentation

brainrake · 2024-05-02T19:20:58Z

i would separate the two effects since the abstraction is too complicated, there are significant differences, and there is still repetition.

brainrake · 2024-05-02T19:21:02Z

I'd also add mkCachixPushEffect and mkAtticPushEffect functions.

zmrocze · 2024-05-06T16:21:56Z

@brainrake what if instead type would take a list, i.e.: types = ["attic" "cachix"]?
How it is currently I agree that the effect should be split into two effects.

zmrocze requested a review from roberth as a code owner January 11, 2024 16:46

zmrocze mentioned this pull request Jan 11, 2024

Add populate-cache-effect #163

Closed

3 tasks

brainrake suggested changes Jan 30, 2024

View reviewed changes

This was referenced Jan 30, 2024

Push packages to attic mlabs-haskell/cardano.nix#12

Merged

Hercules CI's effect to push to attic mlabs-haskell/spo-anywhere#10

Merged

brainrake reviewed Jan 31, 2024

View reviewed changes

brainrake mentioned this pull request Mar 14, 2024

Add populate-cache-effect mlabs-haskell/hercules-ci-effects#1

Closed

3 tasks

zmrocze and others added 11 commits March 14, 2024 06:08

Add populate-cache-effect

69feb35

Remove attic flake input & Rename to push & small fixes

88b2d2e

Fix branches option

4ac73c0

Rename

bb67abf

Export push-cache effect

79681bd

Move export

f3d0426

Format

b225c97

Fix

ee2cdc1

Fix path

275eaa5

Rename and unformat

a4888df

change attic server name to "default"

f5ed263

brainrake force-pushed the push-cache-effect branch from 832a4a3 to f5ed263 Compare March 14, 2024 06:08

fix push-cache-effect when type=cachix branches=null

5ad8f96

zmrocze mentioned this pull request May 5, 2024

Cardano-db-sync module mlabs-haskell/cardano.nix#59

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add push-cache-effect #165

Add push-cache-effect #165

zmrocze commented Jan 11, 2024

zmrocze commented Jan 11, 2024

brainrake Jan 30, 2024 •

edited

Loading

aciceri Jan 31, 2024

roberth Jan 31, 2024

brainrake Feb 6, 2024

brainrake Jan 31, 2024 •

edited

Loading

aciceri Jan 31, 2024

roberth Jan 31, 2024

brainrake Feb 15, 2024

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading

brainrake Feb 6, 2024

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024

roberth Jan 31, 2024

brainrake commented May 2, 2024

brainrake commented May 2, 2024

brainrake commented May 2, 2024

zmrocze commented May 6, 2024

		@@ -0,0 +1,160 @@
		{ lib, withSystem, config, ... }:
		let pkgs-x86_64-linux = withSystem "x86_64-linux" ({ pkgs, ... }: pkgs);

Add push-cache-effect #165

Are you sure you want to change the base?

Add push-cache-effect #165

Conversation

zmrocze commented Jan 11, 2024

Motivation

Questions

Maintainer checklist

zmrocze commented Jan 11, 2024

brainrake Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brainrake Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brainrake Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

brainrake Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brainrake Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brainrake commented May 2, 2024

brainrake commented May 2, 2024

brainrake commented May 2, 2024

zmrocze commented May 6, 2024

brainrake Jan 30, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading