Meltano Version

2.11.0

Python Version

3.10

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

Linux

Description

version: 1
default_environment: dev
project_id: f71334e3-2231-4661-baa5-ddc3aeb9f852
send_anonymous_usage_stats: false
env:
  LEVEL: '2'
  STACK: ${STACK}-2
  CHAIN2: ${CHAIN1}-2
plugins:
  utilities:
  - name: pyjokes
    namespace: pyjokes
    pip_url: pyjokes
    commands:
      joke:
        executable: pyjoke
        args: STACK=${STACK}, CHAIN1=${CHAIN1}, CHAIN2=${CHAIN2}, CHAIN3=${CHAIN3}, CHAIN4=${CHAIN4}, CHAIN5=${CHAIN5}, LEVEL=${LEVEL}
    env:
      LEVEL: '4'
      STACK: ${STACK}-4
      CHAIN4: ${CHAIN3}-4
environments:
- name: dev
  env:
    LEVEL: '3'
    STACK: ${STACK}-3
    CHAIN3: ${CHAIN2}-3
  config:
    plugins:
      utilities:
      - name: pyjokes
        env:
          LEVEL: '5'
          STACK: ${STACK}-5
          CHAIN5: ${CHAIN4}-5
- name: staging
- name: prod

Using the above meltano.yml, we expect the following output from meltano invoke pyjokes:joke:

2022-12-08T20:19:15.638460Z [info     ] Environment 'dev' is active
usage: pyjoke [-h] [-c {neutral,chuck,all,twister}] [-l {en,de,es,gl,eu,it}]
pyjoke: error: unrecognized arguments: STACK=1-2-3-4-5, CHAIN1=1, CHAIN2=1-2, CHAIN3=1-2-3, CHAIN4=1-2-3-4, CHAIN5=1-2-3-4-5, LEVEL=5

Instead we get the following:

2022-12-08T20:19:15.638460Z [info     ] Environment 'dev' is active
usage: pyjoke [-h] [-c {neutral,chuck,all,twister}] [-l {en,de,es,gl,eu,it}]
pyjoke: error: unrecognized arguments: STACK=1-3-4-5, CHAIN1=1, CHAIN2=${CHAIN1}-2, CHAIN3=-2-3, CHAIN4=-2-3-4, CHAIN5=-2-3-4-5, LEVEL=5

Env var expansion in the top-level env field in meltano.yml normally works, as it is used within the env precedence integration test.

I have confirmed that this is not a new issue as of Meltano 2.11.0 - it also affects Meltano 2.10.0, and Meltano 2.9.0. I did not test further.

I am not sure exactly what fields are affected, aside from the args field of plugin commands.

Running with --no-environment does not resolve the issue. We still see CHAIN2=${CHAIN1}-2 in the output.

As with most other env var related issues, I would recommend against prioritizing a fix for this until the Meltano manifest feature has been implemented:

• Feature proposal: meltano-manifest.json #6876

Code

No response