On Thu, Nov 26, 2020 at 05:57, Sorin Sbarnea ssbarnea@redhat.com wrote:

Thanks! i works!

tMaybe now you can propose a CR to update the old script that no longer worked? https://opendev.org/x/coats/src/branch/master/coats/openstack_gerrit_zuul_st...

Sorry, I have no intention to do a full update on the original script. As far as I remember the old gerrit CI table did a lot more than what I hacked together and I have not enough time and knowledge to cover that. Also I'm not sure how the above script relates the solution in the old gerrit that was injected server side instead of injected from the client side.

There is one example where it does not play very nice due to lots of changes sharing the same topic: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/734217

What I see here is that there is a lot of related changes and the table is added at the end of the long list. But overall it works for me on this patch too. What is your specific problem?

Cheers, gibi

Probably you can just replace the script, but I would bother to increase the versioning because once we merge it, existing users may be able to detect an update.

PS. Add me as reviewer.

/sorin

On 26 Nov 2020 at 13:00:01, Lucio Seki lucioseki@gmail.com wrote:

...

Thanks a lot Balázs!

I minified [0] the script and put it in a bookmarklet [1]. It works like a charm [2] :-)

Lucio

[0] https://javascript-minifier.com/ [1] https://superuser.com/questions/279107/add-a-bookmarklet-in-google-chrome/13... [2] https://imgur.com/a/iy5jPIn

On Thu, 26 Nov 2020 at 09:42, Balázs Gibizer balazs.gibizer@est.tech wrote:

...

Hi,

I understand that adapting the old CI test result table to the new gerrit review UI is not a simple task. As a temporary measure I hacked together a tampermonkey user script[1] that does a limited version of the CI table injection from the browser side. I tested it with recent chrome and firefox. Let me know if you have problems with it and I'll try to help. Also feel free to adapt it to your own needs.

Cheers, gibi

[1] https://gist.github.com/gibizer/717e0cb5b0da0d60d5ecf1e18c0c171a

On Thu, Nov 26, 2020 at 14:37, Jeremy Stanley fungi@yuggoth.org wrote:

On 2020-11-26 05:57:03 -0800 (-0800), Sorin Sbarnea wrote:

...

Maybe now you can propose a CR to update the old script that no longer worked?

[...]

As discussed over months leading up to the upgrade, modern Gerrit gives us the ability to have properly supported WebUI plugins using a real extension API, so that we can stop limping along with hacks to modify the interface internals in unstable and unsupported ways. I really don't want us to be stuck continuing to maintain the old mechanism.

I totally agree. The user script I created is a hack that is hard to maintain. I intended it as a temporary solution until a proper server side solution can be created.

Cheers, gibi

-- Jeremy Stanley

On Thu, Nov 26, 2020 at 14:37 Jeremy Stanley wrote:

On 2020-11-26 05:57:03 -0800 (-0800), Sorin Sbarnea wrote:

...

Maybe now you can propose a CR to update the old script that no longer worked?

[...]

As discussed over months leading up to the upgrade, modern Gerrit gives us the ability to have properly supported WebUI plugins using a real extension API, so that we can stop limping along with hacks to modify the interface internals in unstable and unsupported ways. I really don't want us to be stuck continuing to maintain the old mechanism. -- Jeremy Stanley

Here is a CR to add the test result table back in the new gerrit UI:

https://review.opendev.org/c/opendev/system-config/+/763891

This new implementation uses the javascript plugin extension API.

Regards, -Tristan

On Thu, 2020-11-26 at 15:19 +0000, Jeremy Stanley wrote:

On 2020-11-26 15:10:09 +0000 (+0000), Tristan Cacqueray wrote: [...]

...

Here is a CR to add the test result table back in the new gerrit UI:

https://review.opendev.org/c/opendev/system-config/+/763891

This new implementation uses the javascript plugin extension API.

Thanks--that was quick work!

is that going to be moved to the zuul org in opendev as a part of the zuul projects deliverables.

i kind of think it makes sne to develop it there rather then on the rdo gerrit instance.

looking at the impleation it does not look like this owrks teh ssame way by parsing the comment https://softwarefactory-project.io/cgit/software-factory/zuul-results-gerrit...

will this just show the results form zuul. it need to pars all the comments form the third party cis too. basicaly you need to parse all coment form a author that end isn CI and comments form zuul

On 2020-11-26 17:30:12 +0000 (+0000), Tristan Cacqueray wrote: [...]

It should do that already, though I haven't tested it with real data. I guess the next step would be to enable such plugin on review-dev.

Well, review-dev hasn't been kept consistent with production and we were planning to decommission it entirely in the coming days along with the temporary review-test we used to run through various upgrade scenarios to get timing data.

Probably the easiest solution will be for one of OpenDev's Zuul admins to arrange an autohold for a build which includes your new plugin, and then we can demo it there before releasing it again (three cheers for "testing like production!").

On Thu, Nov 26, 2020 at 06:10:34PM +0000, Tristan Cacqueray wrote:

...

Probably the easiest solution will be for one of OpenDev's Zuul admins to arrange an autohold for a build which includes your new plugin, and then we can demo it there before releasing it again (three cheers for "testing like production!").

Though I'd be happy to get a test instance and feed it with real ci comments to demonstrate the results.

We probably want to document this. As a start, I put a hold on a job.

After I set /etc/hosts for review.opendev.org to the held node locally, I could log into the test gerrit (otherwise i'm redirected to the real site).

I then created a openstack/diskimage-builder project via the UI, just for testing. I started getting some errors which [2] resolved after a container restart.

I copied in the raw git tree from the real openstack/diskimage-builder git repo, and ran a re-index by sshing in as the 'Gerrit Code Review' user using suexec

ssh -i ~gerrit2/review_site/etc/ssh_host_rsa_key -p 29418 -l 'Gerrit Code Review' localhost 'suexec --as "Ian Wienand" gerrit index changes-in-project openstack/diskimage-builder'

I don't know if that's right, but it seemed to work. Fiddling with this was where I found [2]

Then I could see the dib changes; e.g.

https://104.130.172.52/c/openstack/diskimage-builder/+/751610

Very willing to take suggestions on what we could do better here. It would probably be awesome if we could figure out automated import of a live tree in the system-config test, so we have a bunch of changes with zuul comments pre-populated. Perhaps something like keep a zip of a smaller repo that has a bunch of reviews and comments, and populate that into the testing gerrit?

I can see the plugin is installed, because in the web console you get the log from [3,4]. However, I can't actually see the table.

Tristan -- have sent you a note separately on how to log in for live debugging if you like.

I can't say too much, because Tristan has written something and I haven't, but the choice of Reason or BuckleScript or OCaml or whatever is going on [5] really lowers the "bus factor" of this. I can read it and get the gist of what's going on; but it compiles down to [6] which I have no idea how to debug. AFAICT, Reason 3.0.4 was released 3 years ago, so it's either finished or ... not that active. I use emacs and write email in mutt -- I'm the first to admit I'm behind on the latest web framework trends. But after investigating TBH I remain unconvinced mostly that we can mantain this indefinitely, and secondly that the tooling will remain maintained.

Anyway, getting a better flow for testing plugins will serve us all well at any rate.

-i

[1] https://review.opendev.org/c/opendev/system-config/+/764396 gerrit: fix db/ mount for gate testing [2] https://review.opendev.org/c/opendev/system-config/+/764395 gerrit: set ownership on ~gerrit2/.ssh directory [3] https://softwarefactory-project.io/cgit/software-factory/re-gerrit/tree/src/... [4] https://softwarefactory-project.io/cgit/software-factory/zuul-results-gerrit... [5] https://reasonml.github.io/ [6] https://104.130.172.52/plugins/zuul-results/static/zuul-results.js [7] https://github.com/facebook/reason/releases/tag/3.0.4

On Fri, Nov 27, 2020 at 14:20 Ian Wienand wrote:

On Thu, Nov 26, 2020 at 06:10:34PM +0000, Tristan Cacqueray wrote:

...

...

Probably the easiest solution will be for one of OpenDev's Zuul admins to arrange an autohold for a build which includes your new plugin, and then we can demo it there before releasing it again (three cheers for "testing like production!").

...

Though I'd be happy to get a test instance and feed it with real ci comments to demonstrate the results.

We probably want to document this. As a start, I put a hold on a job.

After I set /etc/hosts for review.opendev.org to the held node locally, I could log into the test gerrit (otherwise i'm redirected to the real site).

I then created a openstack/diskimage-builder project via the UI, just for testing. I started getting some errors which [2] resolved after a container restart.

I copied in the raw git tree from the real openstack/diskimage-builder git repo, and ran a re-index by sshing in as the 'Gerrit Code Review' user using suexec

ssh -i ~gerrit2/review_site/etc/ssh_host_rsa_key -p 29418 -l 'Gerrit Code Review' localhost 'suexec --as "Ian Wienand" gerrit index changes-in-project openstack/diskimage-builder'

I don't know if that's right, but it seemed to work. Fiddling with this was where I found [2]

Then I could see the dib changes; e.g.

https://104.130.172.52/c/openstack/diskimage-builder/+/751610

Very willing to take suggestions on what we could do better here. It would probably be awesome if we could figure out automated import of a live tree in the system-config test, so we have a bunch of changes with zuul comments pre-populated. Perhaps something like keep a zip of a smaller repo that has a bunch of reviews and comments, and populate that into the testing gerrit?

I can see the plugin is installed, because in the web console you get the log from [3,4]. However, I can't actually see the table.

Tristan -- have sent you a note separately on how to log in for live debugging if you like.

Thanks, I have switched the auth to DEVELOPMENT_BECOME_ANY_ACCOUNT and created a fake Zuul CI comment in https://104.130.172.52/c/openstack/diskimage-builder/+/554002

I can't say too much, because Tristan has written something and I haven't, but the choice of Reason or BuckleScript or OCaml or whatever is going on [5] really lowers the "bus factor" of this. I can read it and get the gist of what's going on; but it compiles down to [6] which I have no idea how to debug. AFAICT, Reason 3.0.4 was released 3 years ago, so it's either finished or ... not that active. I use emacs and write email in mutt -- I'm the first to admit I'm behind on the latest web framework trends. But after investigating TBH I remain unconvinced mostly that we can mantain this indefinitely, and secondly that the tooling will remain maintained.

From what I understand, we can either use java and the polymer template

system, or the javascript api to implement the zuul results table.

I used a `compile to js` language named ReasonML because this let me write applications where most of the runtime errors are simply not possible. Moreover, the code can be compiled to other targets such as Erlang or native assembly. The language was created by the author of ReactJS and it is designed for Javascript developper, see the difference here: https://rescript-lang.org/docs/manual/v8.0.0/overview

To debug [6], you would use the same tools as for any javascript applications. For example, using a source map so that the debugger can hook into the non-minified source.

I'd be happy to use another implementation, I wrote this one rather quickly to restore the zuul-results build table in the new gerrit. Though I think ReasonML is the right tool for this job.

Regards, -Tristan

Anyway, getting a better flow for testing plugins will serve us all well at any rate.

-i

[1] https://review.opendev.org/c/opendev/system-config/+/764396 gerrit: fix db/ mount for gate testing [2] https://review.opendev.org/c/opendev/system-config/+/764395 gerrit: set ownership on ~gerrit2/.ssh directory [3] https://softwarefactory-project.io/cgit/software-factory/re-gerrit/tree/src/... [4] https://softwarefactory-project.io/cgit/software-factory/zuul-results-gerrit... [5] https://reasonml.github.io/ [6] https://104.130.172.52/plugins/zuul-results/static/zuul-results.js [7] https://github.com/facebook/reason/releases/tag/3.0.4

On Wed, Dec 02, 2020 at 22:14 Ian Wienand wrote:

On Fri, Nov 27, 2020 at 02:11:21PM +0000, Tristan Cacqueray wrote:

...

From what I understand, we can either use java and the polymer template system, or the javascript api to implement the zuul results table.

This was discussed today in the infra meeting @ [1]

There was a couple of broad conclusions. There's no dispute that it works, for now. The maintenace of this in perpituity is the major concern.

Some points:

There are already two Zuul plugins in the upstream repos

zuul : https://gerrit.googlesource.com/plugins/zuul -- This is rather badly namespaced, and is focused on showing the status of Depends-On: changes a cyclic dependencies

zuul-status : https://gerrit.googlesource.com/plugins/zuul-status/ -- This is also using a broad namespace, and appears to be related to showing the ongoing job status in the UI directly, but does not show final results. An example I found at [2] from an old wikimedia thread

Note that my proposal goal is just to restore the missing table from the new review.opendev.org user interface. I agree that it would be better to follow how upstream does plugins.

Upstream have shown to take some interest in these plugins when undertaking upgrades to polymer, etc. and thus integrating there is seen as the best practice. Unless we have a strong reason not to, we would like to consume any plugins from upstream for this reason.

It would be in our best interest to work with these existing plugins to clarify what they do more directly and come up with some better namespacing to make it more obvious as potential number of plugins grows.

The current proposed implementation does not look like any of the upstream plugins. Admittedly, [3] shows this is not a strong eco-system; the documentation is mostly TODO and examples are thin. Implementing this with Reason [4] adds one more significant hurdle; we have no track record of maintaing this javascript dialect and it is not used upstream in any way. It also doesn't use the same build and test frameworks; this may not be a blocker but again creates hurdles by being different to upstream. One more concrete comment is that I'm pretty sure walking the shadow DOM to update things is [5] not quite as intended and we should be using endpoints.

In trying to find the best place to start, I've pulled out bits of the checks plugin, image-diff plugin and codeeditor plugin into the simplest thing I could get to a tab with a table in it @ [6]. You can see this for now @ [7]. I think we're going to get more potential eyes using polymer components and making things look like the existing plugins. It's a matter of taste but I think the blank canvas of a separate tab serves the purpose well. I think with the Bazel build bits, we can ultimately write an upstream Zuul job to build this for us in gerrit's Zuul (corvus might know if that's right?)

Tristan -- maybe as step one, we could try integrating what you have into that framework? Maybe we rewrite the ReasonML ... but having it something is step 1. Overall I think this is the broad strokes of what we would look at sending upstream. If they think it's too specific, Zuul tenant seems a logical home.

This is looking great, thank you for looking into that. I guess we can directly re-use the showResult function in-place of the gr-zuul-summary-status-view_html.js content.

If using polymer template is required, then we can still re-use the the `CI.fromMessages` function exported by the re-gerrit library to count the rechecks and extract build results from the message array.

We would also like to expand the gate testing to better handle testing plugins. This will involve us automatically adding a repo, sample change, and Zuul user comments during the gate test. A logical extension of this would be to take samples using a headless browser for reporting as artifacts; held nodes can also be used for advanced debugging. This will give us better confidence as we keep our Gerrit up-to-date.

I'd like to solict some feedback on the checks plugin/API, which Zuul added support for with [8]. My understanding is that this has never really consolidated upstream and is under redevelopment [9]. I don't think there's much there for us at this point; even still seeing as we are so Zuul centric we might be able to do things with a specific plugin this API will never do.

It seems like the checks API enables a similar user interface where the CI build details are integrated in the tablist along with a chip-view under the commit message. It's unclear what are the requirements but that looks like a good long term solution since we wouldn't need a custom plugin for Zuul.

-Tristan

[1] http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-12-01-19.01.lo... [2] https://imgur.com/a/uBk2oxQ [3] https://gerrit-review.googlesource.com/Documentation/pg-plugin-dev.html [4] https://reasonml.github.io/ [5] https://github.com/softwarefactory-project/zuul-results-gerrit-plugin/blob/m... [6] https://github.com/ianw/gerrit-zuul-summary-status [7] https://104.130.172.52/c/openstack/diskimage-builder/+/554002 [8] https://opendev.org/zuul/zuul/commit/e78e948284392477d385d493fc9ec194d544483... [9] https://www.gerritcodereview.com/design-docs/ci-reboot.html

On Tue, Jan 19, 2021 at 8:52 AM Andrii Ostapenko anost1986@gmail.com wrote:

One click more than before, but looks great! Thank you!

+1 it really does look great very clear and easy to see the failure (and bonus it's even clearer than it was in the 'old' zuul)

thanks very much Balazs for bringing this up and big thanks to Ian and anyone else who worked on that

On Tue, Jan 19, 2021 at 12:00 AM Ian Wienand iwienand@redhat.com wrote:

...

On Thu, Nov 26, 2020 at 01:39:13PM +0100, Balázs Gibizer wrote:

...

I understand that adapting the old CI test result table to the new

gerrit

...

...

review UI is not a simple task.

We got there in the end :) Change [1] enabled the zuul-summary-results plugin, which is available from [2]. I just restarted opendev gerrit with it, and it seems to be working. Look for the new "Zuul Summary" tab next to "Files". I would consider it a 0.1 release and welcome any contributions to make it better.

If you want to make changes, you should be able to submit a change to system-config with a Depends-On: and trigger the system-config-run-review test; in the results returned there are screenshot artifacts that will show the results (expanding this testing also welcome!). We can also a put a node on hold for you to work on the plugin if you have interest. It's also fairly easy to run the container locally, so there's plenty of options.

Thanks,

-i

[1] https://review.opendev.org/c/opendev/system-config/+/767079 [2]

https://gerrit-review.googlesource.com/admin/repos/plugins/zuul-results-summ...

...

On Tue, Jan 19, 2021 at 7:01 AM Ian Wienand iwienand@redhat.com wrote:

On Thu, Nov 26, 2020 at 01:39:13PM +0100, Balázs Gibizer wrote:

...

I understand that adapting the old CI test result table to the new gerrit review UI is not a simple task.

We got there in the end :) Change [1] enabled the zuul-summary-results plugin, which is available from [2]. I just restarted opendev gerrit with it, and it seems to be working. Look for the new "Zuul Summary" tab next to "Files". I would consider it a 0.1 release and welcome any contributions to make it better.

Thank you, this is amazing! I wonder if we could also run the plugin that shows the live progress (it was mentioned somewhere in the thread).

Dmitry

If you want to make changes, you should be able to submit a change to system-config with a Depends-On: and trigger the system-config-run-review test; in the results returned there are screenshot artifacts that will show the results (expanding this testing also welcome!). We can also a put a node on hold for you to work on the plugin if you have interest. It's also fairly easy to run the container locally, so there's plenty of options.

Thanks,

-i

[1] https://review.opendev.org/c/opendev/system-config/+/767079 [2] https://gerrit-review.googlesource.com/admin/repos/plugins/zuul-results-summ...

On 2021-01-19 12:56:41 +0000 (+0000), Sean Mooney wrote:

On Tue, 2021-01-19 at 12:37 +0100, Dmitry Tantsur wrote:

[...]

...

I wonder if we could also run the plugin that shows the live progress (it was mentioned somewhere in the thread).

i belive showing the live progress of the jobs is effectivly a ddos vector. infra have ask that we not use javascript to pool the the live status of the jobs in our browser in the past.

[...]

i know that we previously tried enbeding the zuul job status directly into gerrit a few years ago and that had to be qickly reverted as it does not take many developers leave review open in a tab to quickly make that unworkable. i know i for one often leave review open over night if im pinged to review something shortly before i finish for the day so that its open on my screen when i log in the next day.

[...]

I think it's probably worth trying again. The previous attempts hit a wall because of several challenges:

1. The available Zuul status API returned data on all enqueued refs (a *very* large JSON blob when the system is under heavy use)

2. Calls to the API were handled by a thread of the scheduler daemon, so often blocked or were blocked by other things going on, especially when Zuul was already under significant load

3. Browsers at the time continued running Javascript payloads in "background" tabs so the volume of queries was multiplied not just by the number of users but also by the average number of review tabs they had open

Over time we added a ref-scoped status method so callers could request the status of a specific change. The Zuul REST API is now served by a separate zuul-web daemon, which we can move to a different server entirely if load demands that (and can even scale horizontally with more than one instance of it, I think?). Browser tech has also improved, and popular ones these days suspend Javascript stacks when tabs are not exposed. We may also be caching status API responses more aggressively than we used to do. All of these factors combined could make live status info in a Gerrit plug-in entirely tractable, we'll just need someone with time to try it and see... and be prepared for multiple Gerrit service restarts to enable/disable it, so probably not when utilization is as high as it has been the past couple of weeks.

On Wed, Jan 20, 2021 at 9:56 AM Mark Goddard mark@stackhpc.com wrote:

On Tue, 19 Jan 2021 at 15:18, Jeremy Stanley fungi@yuggoth.org wrote:

...

On 2021-01-19 12:56:41 +0000 (+0000), Sean Mooney wrote:

...

On Tue, 2021-01-19 at 12:37 +0100, Dmitry Tantsur wrote:

[...]

...

...

I wonder if we could also run the plugin that shows the live progress (it was mentioned somewhere in the thread).

i belive showing the live progress of the jobs is effectivly a ddos vector. infra have ask that we not use javascript to pool the the live status of the jobs in our browser in the past.

[...]

...

i know that we previously tried enbeding the zuul job status directly into gerrit a few years ago and that had to be qickly reverted as it does not take many developers leave review open in a tab to quickly make that unworkable. i know i for one often leave review open over night if im pinged to review something shortly before i finish for the day so that its open on my screen when i log in the next day.

[...]

I think it's probably worth trying again. The previous attempts hit a wall because of several challenges:

1. The available Zuul status API returned data on all enqueued refs

(a *very* large JSON blob when the system is under heavy use)

2. Calls to the API were handled by a thread of the scheduler

daemon, so often blocked or were blocked by other things going on, especially when Zuul was already under significant load

3. Browsers at the time continued running Javascript payloads in

"background" tabs so the volume of queries was multiplied not just by the number of users but also by the average number of review tabs they had open

Over time we added a ref-scoped status method so callers could request the status of a specific change. The Zuul REST API is now served by a separate zuul-web daemon, which we can move to a different server entirely if load demands that (and can even scale horizontally with more than one instance of it, I think?). Browser tech has also improved, and popular ones these days suspend Javascript stacks when tabs are not exposed. We may also be caching status API responses more aggressively than we used to do. All of these factors combined could make live status info in a Gerrit plug-in entirely tractable, we'll just need someone with time to try it and see... and be prepared for multiple Gerrit service restarts to enable/disable it, so probably not when utilization is as high as it has been the past couple of weeks.

A refresh button to update live results on demand could be a good compromise between UX and unnecessary polling.

I'd be totally fine with a refresh button, very infrequent (once a minute?) automatic refreshes and aggressive caching.

Dmitry

...

-- Jeremy Stanley

On 2021-01-21 17:27:12 +0000 (+0000), Sean Mooney wrote: [...]

if you agressivly cache there is no point in frequent pooling i would consider anything less then 5 mints to be still relitivly frequent by the way so 1 minute is not very infrequent.

[...]

When I said "aggressively" I meant we're actually caching what would normally be considered dynamic data. The Zuul API sets...

Cache-Control: public, max-age=1

...so that browsers will try to avoid requesting the file more than once a second, and we front it with Apache mod_proxy so the server will also refresh its cache roughly that often.

I think frequent requests are fine, maybe not once a second, but once every five or ten would likely be okay (our Zuul dashboard refreshes its status every 5 seconds in your browser, and that's pulling the full status blob for the tenant, not just change-specific data like the Gerrit plugin would presumably do). We have capacity and a scalable architecture now, so I'd err on the side of responsiveness and user convenience unless we see that it's actually still a problem.

Thanks Ian! Very useful.

On Tue, Jan 19, 2021 at 12:48 PM Balazs Gibizer balazs.gibizer@est.tech wrote:

On Tue, Jan 19, 2021 at 16:58, Ian Wienand iwienand@redhat.com wrote:

...

On Thu, Nov 26, 2020 at 01:39:13PM +0100, Balázs Gibizer wrote:

...

I understand that adapting the old CI test result table to the new gerrit review UI is not a simple task.

We got there in the end :) Change [1] enabled the zuul-summary-results plugin, which is available from [2]. I just restarted opendev gerrit with it, and it seems to be working. Look for the new "Zuul Summary" tab next to "Files". I would consider it a 0.1 release and welcome any contributions to make it better.

If you want to make changes, you should be able to submit a change to system-config with a Depends-On: and trigger the system-config-run-review test; in the results returned there are screenshot artifacts that will show the results (expanding this testing also welcome!). We can also a put a node on hold for you to work on the plugin if you have interest. It's also fairly easy to run the container locally, so there's plenty of options.

Awesome. Thank you Ian!

Cheers, gibi

...

Thanks,

-i

[1] https://review.opendev.org/c/opendev/system-config/+/767079 [2]

https://gerrit-review.googlesource.com/admin/repos/plugins/zuul-results-summ...

...

On Thu, Jan 21, 2021 at 10:58:05AM +0000, Stephen Finucane wrote:

Thanks for this. One issues I've noted is that it doesn't update as I click through a chain of patches. The results from patch N will be shown for any other patch I navigate to. I don't know if this is an issue with my browser, though it sounds like something is being cached and that cache should be invalidated?

Thanks, you're right. I think it's not observing when the "change" objects ... change. I'll look into it.

-i