Custom pipeline variables do not populate on initial load of new pipeline page

Summary

When custom pipeline variables are configured for a project on GitLab.com, navigating to the /-/pipelines/new page does not consistently fetch the variables. A page refresh is required after loading the page to load the custom variables, and if the page is left idle for some time, a subsequent refresh will also return empty results until another page refresh is performed.

Issue reported by a customer in this support ticket: https://gitlab.zendesk.com/agent/tickets/354900

Steps to reproduce

1. Create a project on GitLab.com
2. Add a .gitlab-ci.yml file containing some variables:

variables:
  ENVIRONMENT:
    value: all
    description: This directs which environment to run against
  STAGE_OF_PIPELINE:
    value: test
    description: This variable defines the pipeline step to run
  OTHER_CUSTOM_VARIABLE:
    value: value1,value2
    description: This variable sets something

3. Navigate to the /-/pipelines/new page for the project, and observe that variables do not populate until the page is refreshed

Example Project

https://gitlab.com/gitlab-silver/dannyjb-test/354900/-/pipelines/new

What is the current bug behavior?

The graphql query response when /-/pipelines/new page is loaded initially returns empty results:

[{"data":{"project":{"id":"gid://gitlab/Project/41966387","ciConfigVariables":null,"__typename":"Project"}}}]

When the page is refreshed, the variables are returned as expected:

[{"data":{"project":{"id":"gid://gitlab/Project/41966387","ciConfigVariables":[{"description":"This directs which environment to run against","key":"ENVIRONMENT","value":"all","valueOptions":null,"__typename":"CiConfigVariable"},{"description":"This variable defines the pipeline step to run","key":"STAGE_OF_PIPELINE","value":"test","valueOptions":null,"__typename":"CiConfigVariable"},{"description":"This variable sets something","key":"OTHER_CUSTOM_VARIABLE","value":"value1,value2","valueOptions":null,"__typename":"CiConfigVariable"}],"__typename":"Project"}}}]

If the page is left idle for some time after refreshing the page, a subsequent refresh will return another empty result and the variables will disappear again. A refresh of the page for the third time will bring the variables back.

What is the expected correct behaviour?

The query should load the variables the first time, every time, and not require a page refresh to fetch them correctly.

Relevant logs and/or screenshots

Screen capture:

https://www.loom.com/share/6ee198f90e87435289cb392bf080765a

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes