Booleans specified with -e / --extra-settings do not evaluate to False

[r4victor]

• I have read the Filing Issues and subsequent “How to Get Help” sections of the documentation.
• I have searched the issues (including closed ones) and believe that this is not a duplicate.

• OS version and name: macOS Mojave
• Python version: 3.9.7
• Pelican version: 4.7.1
• Link to theme: default produced by pelican-quickstart
• Links to plugins: default produced by pelican-quickstart
• Link to your site: –
• Link to your source: https://github.com/r4victor/pelican/tree/boolean_overrides_demo/website
• Link to a Gist with the contents of your settings file: https://github.com/r4victor/pelican/blob/boolean_overrides_demo/website/pelicanconf.py

Issue

1. Pick any boolean settings parameter that is set to True by default or in pelicanconf.py.
2. Try to override it to False with -e / --extra-settings.
3. Observe that parameter is still True.

For example, I set DELETE_OUTPUT_DIRECTORY = True in pelicanconf.py and then run:

pelican --debug -e DELETE_OUTPUT_DIRECTORY=false

I see how the old files in the output directory are deleted. Here's the debug log: https://github.com/r4victor/pelican/blob/boolean_overrides_demo/website/debug.txt

Setting -e param=False or -e param=0 doesn't work as well. The only way is to set the parameter to an empty string like -e param=. This is definitely not what users are supposed to do according to the docs (https://docs.getpelican.com/en/latest/settings.html#settings).

It seems the problem was introduced when fixing issue #2789 and originates in the coerce_overrides() function in settings.py:

def coerce_overrides(overrides):
    if overrides is None:
        return {}
    coerced = {}
    types_to_cast = {int, str, bool}
    for k, v in overrides.items():
        if k not in DEFAULT_CONFIG:
            logger.warning('Override for unknown setting %s, ignoring', k)
            continue
        setting_type = type(DEFAULT_CONFIG[k])
        if setting_type not in types_to_cast:
            coerced[k] = json.loads(v)
        else:
            try:
                coerced[k] = setting_type(v)
            except ValueError:
                logger.debug('ValueError for %s override with %s, try to '
                             'load as json', k, v)
                coerced[k] = json.loads(v)
    return coerced

So, when types_to_cast is bool, setting -e k=v results in coerced[k] = bool(v), but in Python bool(v) is True for any string except for the empty string. bool("false") is True.

The solution would be to evaluate "false", "False", "0", "" (something else?) to False, and everything else to True.

[justinmayer]

Hi Victor. Many thanks for reporting this problem and posting such a detailed analysis. Much appreciated! 👏

Since you seem to have a good handle on a potential solution, would you consider submitting a PR that addresses the problem, preferably including first a commit with a (failing) test and then another that fixes it? That would really help us out!

cc: @jwodder @sabaini

[r4victor]

Hi, Justin. Sure, I'll work on it.

[r4victor]

I now realized that coerce_overrides() and thus -e / --extra-settings are completely broken. It's not just booleans.

To parse a value of some parameter, coerce_overrides() does coerced[k] = json.loads(v). But to avoid parsing string parameters, it also looks at the type of the parameter's default value and so does coerced[k] = str(v) if it's str. (It does the same for int, and for bool before my PR for no good reason). The problem with this is that in Pelican a settings parameter can take values of different Python types. For instance, PAGE_TRANSLATION_ID is 'slug' by default but can be None or False. Many other parameters are nullable strings. DEFAULT_PAGINATION can be False or int. All those parameters are parsed incorrectly.

With default PAGINATED_TEMPLATES run pelican -e DEFAULT_PAGINATION=123. You'll get one blog entry per page because Pelican sees that DEFAULT_PAGINATION is False in defaults and parses 123 as True. Then Paginator(... per_page=True) works as Paginator(... per_page=1), but that's another story. The same for strings and so on. My PR doesn't affect this behaviour.

Another observation is that though we allowed non-json booleans like False and True for boolean types, it's still not possible to do pelican -e JINJA_ENVIRONMENT='{"trim_blocks": True}'. You'll get JSONDecodeError.

I see two possible solutions:

1. Associate each parameter with a type (maybe a compound type). Then parse each type accordingly.
2. Give up on custom values like True and False and rely completely on json.loads(). Show users a useful error message on JSONDecodeError.

I'm 100% for the second solution since it's much simpler and predictable. I wouldn't be afraid of breaking code that relies on True and False since it was so broken anyway.

[r4victor]

I should note that relying on json.loads() will force users to quote string parameters like so: pelican -e k='"some string"'. We'll parse 'null' as None and '"some string"' as 'some string', but parsing 'some string' will throw an error.

If you want to accept 'some string' literally, you need to handle 'null', and this again means you need to introduce some kind of type system for parameters.

I'd go with json.loads() + informative error messages + overview in the docs.