Add ability to include other justfiles

[cledoux]

Would it be possible to add file inclusion to Justfiles?

A semi-common pattern I have with makefiles is to use separate makefiles for different environments with a common Makefile included between them.

A trimmed down example I'm currently using is below. There are three files:

• Makefile.include: Contains common configuration and targets.
• Makefile.vagrant: Configuration and targets specific to the development install.
• Makefile.staging: Configuration and targets specific to the cloud staging environment.

# Makefile.include  Common configuration and targets

ENVIRONMENT_TYPE ?= Default
INVENTORY ?= ./inventory
ANSIBLE_ARGS ?= --verbose
INSTALL_PLAYBOOK ?= playbooks/install.yml
DEPLOY_PLAYBOOK ?= playbooks/deploy.yml
UPGRADE_PLAYBOOK ?= playbooks/upgrade.yml
ANSIBLE = ansible-playbook $(ANSIBLE_ARGS) --inventory=$(INVENTORY)

all: provision install deploy

install:
	$(ANSIBLE) $(INSTALL_PLAYBOOK)

deploy:
	$(ANSIBLE) $(DEPLOY_PLAYBOOK)

upgrade:
	$(ANSIBLE) $(UPGRADE_PLAYBOOK)

# Makefile.vagrant   Local installs using vagrant.
INVENTORY ?= .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory
ENVIRONMENT_TYPE = Vagrant

include Makefile.include

provision:
	vagrant up

nuke:
	vagrant destroy -f
	rm -rf .vagrant

# Makefile.staging  Build staging environment in the cloud
INVENTORY ?= ./inventory/gce.py

PROVISION_PLAYBOOK ?= gcloud/provision.yml
ENVIRONMENT_TYPE = Staging

include Makefile.include

provision:
	$(ANSIBLE) $(PROVISION_PLAYBOOK)
	./inventory/gce.py --refresh-cache > /dev/null

nuke:
	$(ANSIBLE) gcloud/teardown.yml

If you have a better suggestion for solving my use case that doesn't include multiple files, I'm all ears. I'm always happy to learn. 😃

[casey]

Would it be possible to just have everything in one file, and rename the recipes? For example, provision-local, nuke-local.

I know that's a little bit annoying, but this feature would add a bit of complexity to the parser, lexer, and error messages, since they would have to be aware of multiple files.

If a lot of users wanted it though, I wouldn't be opposed to adding it.

[cledoux]

I've considered several alternatives so far. All are workable, but not as clean as including files.

Single file with namespaced targets. -- e.g. provision-local, nuke-local

I think this would result in a large amount of boilerplate for my use case. The main difference between the different files is the definition of a configuration variable. Thus, almost all the rules would have the basic form of the below example:

VAGRANT_INVENTORY=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory
STAGING_INVENTORY=./inventory/gce.py
install {{INVENTORY}}:
	ansible-playbook {{ANSIBLE_ARGS}} --inventory={{INVENTORY}} {{INSTALL_PLAYBOOK}}

install-vagrant:
	just install {{VAGRANT_INVENTORY}}

install-staging:
	just install {{STAGING_INVENTORY}}

Multiple files without inclusion.

In this option, I would just maintain independent Justfiles for each environment and duplicate the common code.

This requires manually ensure the common code areas stay properly synced, but this shouldn't be any more annoying than the boilerplate required when using a single file. I never have more than three separate files.

This is the method I would probably use, but that's just a personal opinion. We (my team and I) are already used to selecting the environment to operate on by switching makefiles. I would want to preserve this workflow and allow switching environments by selecting the appropriate justfile.

[cledoux]

this feature would add a bit of complexity

I fully understand not wanting to complicate the code to solve what may well be a rare corner case.

For now, I'll either use one of the previously mentioned options, or just fall back to Makefiles if I really need file inclusion.

I appreciate the work you have and continue to put into this tool. It's an amazing utility that I've wanted for a while.

How complicated is it to get started in rust? Working on a parser/lexer sounds like fun, but I don't know rust nor know how hard it is to get started using it.

[casey]

Rust is a bit hard to get started with. I definitely spent a week or two trying to figure out owned vs. borrowed, and then probably another week or two on lifetimes and appeasing the borrow checker. Also, there are some idioms to learn around error handling.

Overall though, I think it's really nice! Also, if you do decide to pick up rust, definitely check out the rust gitter and IRC channels. They are chock full of nice, smart, helpful, beginner-friendly people.

I think for now I'd like to keep this issue open, and also encourage others to comment if it's something they want or think they would use.

Also, thank you very much for the kind words!

[Fleshgrinder]

Definitely an important feature, especially within monorepos where each microservice has its dedicated Just/Makefile but some things are shared. Another thing that directly comes to my mind is splitting of config and recipes.

[runeimp]

I would like to see this feature added. Regarding includes complexity with the lexer, etc. I'm wondering if it would be easier to simply build a temporary concatenated version of the main file and it's imports as a 1st phase prior to parsing, lexing, etc.? I've no idea if that would or could be done easily. I've not reviewed the code and played very little with Rust so far. Just throwing it out there.

BTW, finally tried implementing Just and love it! Seems to be exactly what I've been looking for over the last year or so.

[Fleshgrinder]

The problem with concatenating is that one cannot report errors with precise location anymore. This would be very frustrating for the user.

[casey]

@runeimp So glad you like it!

Also, I think that I would want to make it clear where things were coming from, something like:

import foo

bar: foo.baz
  echo {{foo.SOME_VARIABLE}}

Instead of putting them all in the same scope/namespace.

[Fleshgrinder]

Should foo resolve to ./foo/Justfile? I think that this is actually a great idea, as it ensures that targets will not collide.

Variables that were exported should not have the prefix otherwise it would be weird, right?

[cledoux]

Personally, I think having include act as a copy-paste as suggested by @runeimp is good enough. This is my understanding of how includes work in Makefiles already.

As for debugging, I feel having the --dump command print the justfile with all includes resolved would be sufficient.

To put it another way, I'd rather have a copy-paste style include than no include at all.

[cledoux]

@casey Wouldn't having to namespace imported variables break my original use case? The purpose of imports is so multiple top level justfiles can change configuration variables as needed and the imported justfile can remain unchanged. The collision of variables is intentional.

# Justfile.include
foo:
    bar {{OPTIONS}}

# Justfile.staging
OPTIONS = <staging options>
import Justfile.include

# Justfile.production
OPTIONS = <production options>
import Justfile.include

The foo target is then invoked with either just -f Justfile.production foo or just -f Justfile.staging foo.

[Fleshgrinder]

I personally would never solve the issue as you describe it in that way. I would use an environment variable that defines the, well, environment. This is currently not possible with just but could easily be supported. Make has it via ifdef.

That being said, I see how it could break some things if you have a common bootstrap Justfile were you define reusable options and targets for others to be included.

[casey]

@cledoux Ah, yeah, it wouldn't work in that case. This feature is going to have to be pretty carefully designed up front. I think I'll need to be able to enumerate all possible use-cases, and then figure out how the feature can support all of them.

It seems like there are two use-cases being discussed in this thread, one being sharing code between different justfiles in the same directory, doing essentially configured versions of the same thing, and another being sharing configuration between different subdirectories of a mono repo.

I think for the former, I would prefer to have some way to mark options as not coming from the current file:

# Justfile.include
foo:
    bar {{super::OPTIONS}}

# Justfile.staging
OPTIONS = <staging options>
import Justfile.include

# Justfile.production
OPTIONS = <production options>
import Justfile.include

[cledoux]

@Fleshgrinder I may not have been clear. In my case, both commands are run from the same computer, so environment variables aren't really an option because I need the variable to hold different things at different times all on the same computer.

If you have a better way to solve my use case, I'm all ears. I'm always eager to learn. 😃

[Fleshgrinder]

Another way would be to allow different ways of inclusion, e.g. import imports and use works as you envisioned it. I personally see value in the prefixing of variables in some situations.

@cledoux working with environment variables is actually very powerful, especially if you need to ad-hoc change things. You would basically define them while calling just:

$ just foo # this uses the defaults
$ ENV=stage just foo
$ ENV=prod just foo

You could also define aliases for these in your .bashrc/.zshrc/…:

just-stage() { ENV=stage just "$@"; }
just-prod() { ENV=prod just "$@"; }

See also #245

[casey]

@Fleshgrinder If you have to define shortcuts in another file, to use with just, a program that lets you put shortcuts into a file, then just has kind of failed ;)

Environment variables are always there, of course, but I think that variables defined within just have important advantages, for example being cross platform (windows environment variables are case-insensitive), getting compile-time errors when they aren't defined, being utf8, and not polluting the downstream environment needlessly.

[Fleshgrinder]

I disagree, adding or removing functionality based on the environment is typical in task runners and build systems. Environment variables are the typical choice for that, especially because of CI systems. Make also has special functionality to solve the previously mentioned things:

build: export ENV ?= dev
build:
    # do something

build-staging: export ENV := stage
build-staging: build

build-prod: export ENV := prod
build-prod: build

This is one of the reasons why I ask for conditional assignment in #245. 😉

[cledoux]

I have no idea how complicated it would be to implement, but what about something like the way python handles imports? An import foo namespaces everything under foo while from foo import * puts everything into the global namespace.

Then again, that might be overcomplicating everything.

@Fleshgrinder There are multiple configurations that need changing when targeting different environments and the contents of these options are too long to type out each time I run a command. In order to use environment variables in the manner you suggest (or least how I understand you to have suggested them), I would need to do the following:

if ENV == stage:
    TARGET = <a long path name>
    OPTIONS1 = <a list of options>
    OPTIONS2 = <another long list of options>
else if ENV == dev
    TARGET = < a different path>
    OPTIONS1 = < a different list of options>
    # OPTIONS2 is not set in this case
# Everything else

[Fleshgrinder]

I like the from foo import * idea, should not be too hard to implement.

What you would need in such cases would be a conditional include, too keep things nice and tidy. Something make also does not offer. This is actually one of the reasons why tools like autoconf were created.

foo: export ENV ?= dev
foo:
import {{ENV}}.just
    # command

foo-stage: export ENV = stage
foo-stage: foo

foo-prod: export ENV = prod
foo-prod: foo

This does not work in make, but would be the best thing.

[gregwebs]

For the conditional environment example, I do one of 2 approaches:

1. explicitly pass the environment as the first argument
2. set it as a just variable

The second works better if env=dev is a good default. This is because you cannot pass arguments to dependent tasks, but they do have access to top-level variables (there is a different issue for this).

I do have a use-case in mind where I want to be in different directories and usually do the same thing and be able to share those tasks but customize some tasks. I think for now I can accomplish this by having a shared justfile invoke a second custom just file:

infrastructure:
	#!/bin/bash
	set -euo pipefail
	if test -f custom.just && (just -l -d . -f custom.just | grep infrastructure); then
	  just -f custom.just -d . env={{env}} infrastructure
	fi

I think this is a decent solution. However, there are two problems:

1. boilerplate
2. forgetting to pass variables

The second is where things can easily go awry. If I default to env=dev then it is easy to not notice (no errors are produced) that I didn't pass the variable along to the next justfile. I cannot do this automatically with just -d . -f custom.just --evaluate | awk '{print "--set " $1 "=" "?"}' because I don't have access to the current environment (as an abstraction where I can lookup a string).

This can be solved by source inclusion, i.e. just -d . -f custom.just --dump | ogrep 'infrastructure:', but that has to happen on every justfile parse rather than lazily when the command is invoked (like the above delegation).

To summarize, I think this problem can be solved without adding import support to just, but instead with delegation, but we run into safety issues. Having a function similar to env_var that can be called from a template could solve that issue, and then we would be left with a fair amount of boilerplate that just could add special support to reduce. This would leave the connection unsafe in the sense that runtime failure will occur, but failure should occur and be noticed, which may be an acceptable trade-off if the implementation is much simpler.

[dionjwa]

Another use case is sharing common justfile commands between repositories. There's a common set of commands for ci steps (build/test/deploy) that is great to live in the root of a repository, but then each repository also has commands specific to that repo. Without the ability to import justfiles, you cannot easily share sets of common tasks. If you have a separate justfile, then it needs a different name, which breaks some other functionality (just doesn't know the name of the running justfile, it assumes it is justfile when it might be ci.justfile which breaks e.g. --list. If this use case is unclear, I'm happy to elaborate or simplify. just is great, keep up the good work!

[runeimp]

In the case of environment variables Just now automatically loads .env files. This was not a feature when this discussion started but has been for about a year now I believe. In any case this does not directly affect some of the use cases but I wonder if it changes the discussion at all?

[dionjwa]

It doesn't really change the use case I'm interested in: sharing modular justfiles that can be integrated into the root justfile.

[casey]

I just posted a little update on where things stand with modules, which I think is the feature which will eventually enable the use cases mentioned in this thread.

[fgblomqvist]

What's the 2021 take on this? It seems like the discussions have stranded for both this and #271 😕 .
I guess a basic workaround that perhaps solves a part of the problem is something like:

foo:
  just -f ../common.justfile -d . bar

But it ain't that pretty and also more prone to breaking than an official solution. Naturally no variable sharing either.

If anyone knows an alternative to just that isn't primarily a build tool, please share!

[casey]

The 2021 take is, sadly, that things are still where they were in 2017.

I started working on inline modules, thinking it might be easier than doing import statements, but it wasn't, so I abandoned the attempt.

I want Just to support this, but I don't personally have any justfiles that are large enough to want to break up into modules, so I don't have a personal use-case that would motivate me to do this. So for now, it's in limbo until I somehow find the motivation or someone else picks it up.

[fgblomqvist]

Alright, thanks for the update at least! 🙂

[runeimp]

I'm still wondering if concatenation of files is a simple first option. I know @Fleshgrinder said it would be imprecise for errors but isn't that exactly why source map files are generated by JavaScript code minifiers such as Uglify? I'm not saying it would easy. But I think it probably wouldn't be too bad to implement compared to other options. Here are some resources to consider.

• Source Map Revision 3 Proposal - Google Docs
• Use a source map - Firefox Developer Tools | MDN
• mishoo/UglifyJS: JavaScript parser / mangler / compressor / beautifier toolkit
  • UglifyJS/sourcemap.js at master · mishoo/UglifyJS

[MadBomber]

I created an experiment justprep before reading through all of the just issues. It is a pre-processor that does the include thing. Its in Ruby and Crystal. I've using the crystal created executable in my environment.

https://github.com/MadBomber/experiments/tree/master/just_playing

Dewayne
o-*

[mihaigalos]

As an intermediate solution, why not:

execute_myreceipe_from_other_justfile:
  just --justfile /my/path/justfile myrecipe myarguments

This doesn't expose the recipes to just --list, but still make the justfiles "composable".

[ceejbot]

Concatenation of files before full parsing, perhaps via an include statement at the top, would satisfy our include needs completely. We have boilerplate that we'd like to repeat in each subproject in a monorepo, exactly like the make file use case in the original post. Namespacing variables would break this particular use case, but that would maybe be something we could work around if an include feature that worked that way arrived.

[casey]

@ceejbot You should check out justprep: https://github.com/MadBomber/justprep

It's a justfile preprocessor that does what you want. I'm hesitant to include include, because although it's expedient, the semantics of literal text inclusion are sub-optimal.

[casey]

I'm going to close this issue, since there are problems with literal text inclusion. See #383 and #929 for modules and subcommands. Maybe someone should create a discussion for workarounds, like justprep and other preprocessing steps?

[jpbochi]

I can't see it directly mentioned here, but there's an alternative to including other files: https://github.com/casey/just#invoking-justfiles-in-other-directories

Yes, this alternative may not be perfect for every case, but may be an acceptable solution for most people looking for the include feature.

[mrchantey]

Using the new [no-cd] asttribute we can use a workflow like this:

#./justfile

submodule *args:
  just -f ./config/submodule {{args}}

#./config/submodule.justfile

[no-cd]
do-thing:
  pwd

# usage:
just submodule do-thing

PS I'd love love to have a no-cd setting for the whole file, keep up the good work team!