Open Stack

 > Data migration from where to where? We access the current state by 
retrieving the data from leader db, or am I missing something here?

In case there are changes in how data is stored in one version of a 
charm vs the other.

Another problem is application versioning: we do have version-specific 
templates but this data may be versioned too. Old entries may not be 
simple strings, e.g. they can be small objects which can change 
following version changes (new data added or removed in a pre-defined way).

I can also see potential scenarios where you would need to gracefully 
retire old data as features get deprecated and, eventually, removed. So, 
during an upgrade you would have two copies of stateful data and a charm 
would react differently depending on the current application version 
set. After an upgrade the old copy could be automatically discarded.

 > Perhaps I'm being naive but I don't see these developing into data 
sets large enough to cause performance problems.

I don't think it's going to be used for large data sets either but you 
never know.

 > Each time the action is run the context associated with the action is 
deleted and recreated. If an action argument is unset I guess we could 
interpret that as leave-unchanged.

Leave unchanged - yes. Still need to be able to delete completely though.

What I like about actions is that you can clearly express imperative 
steps with arguments that you have to perform after a deployment and 
they have a very specific type of data in mind which is fetched from 
stateful applications out of band by an operator.

On 19.02.2018 18:11, Liam Young wrote:
> On Mon, Feb 19, 2018 at 9:05 AM, Dmitrii Shcherbakov
> <dmitrii.shcherbakov at canonical.com> wrote:
>> Hi Liam,
>>
>>> I was recently looking at how to support custom configuration that relies
>>> on post deployment setup.
>> I would describe the problem in general as follows:
>>
>> 1) charms can get context not only from Juju (config options, relation data,
>> leader data), environment (operating system release, OpenStack release,
>> services running etc.) but also from a stateful data store (e.g. a Keystone
>> database);
>> 2) it's not easy to track application state from a charm because:
>> authentication is needed to fetch persistent state, notifications from a
>> data store cannot be reliably set up because charm code is ran periodically
>> and it is not always present in memory (polling is neither timely nor
>> efficient). Another problem is that software that holds the state needs to
>> support data change notifications which raises version compatibility
>> questions.
>>
>> By using actions we move the responsibility for data retrieval and change
>> notifications to an operator but a more generic scenario would be modeling a
>> feedback loop from an application to Juju as a modeling system where changes
>> can be either automatic or gated by an operator (an orchestrator). Making it
>> automatic would mean that a service would get notifications/poll data from a
>> state store and would be authorized to use Juju client to make certain
>> changes.
> This is an interesting idea, but there is no such mechanism within
> Juju that I know of.
>
>> Another problem to solve is maintenance of that state: if we start
>> maintaining a key-value DB in leader settings we need to think about data
>> migration over time and how to access the current state.
> Data migration from where to where? We access the current state by retrieving
> the data from leader db, or am I missing something here?
>
>> In other words, in
>> CRUD, the "C" part is relatively straightforward, "R" is more complicated
>> with large data sets (if I have a lot of leader data, how do I interpret it
>> efficiently?),
> Perhaps I'm being naive but I don't see these developing into data
> sets large enough
> to cause performance problems.
>
>> "UD" is less clear - seems like there will have to be 3 or 4
>> actions per feature for C, [R], U and D or one action that can multiplex
>> commands.
> Each time the action is run the context associated with the action is deleted
> and recreated. If an action argument is unset I guess we could interpret that as
> leave-unchanged.
>
>> This brings me to the question of how is it different from state-specific
>> config values with a complex structure.
> To my mind the difference is complexity for the end user. An action has clearly
> defined arguments and the charm action code looks after forming this into
> the correct context.
>
>> Instead of leader data, a per-charm
>> config option could hold state data in some format namespaced by a feature
>> name or config file name to render. A data model would be needed to make
>> sure we can create versioned application-specific state buckets (e.g. for
>> upgrades, hold both states, then remove the old one).
>>
>> Application version-specific config values is something not modeled in Juju
>> although custom application versions are present
>> (https://jujucharms.com/docs/2.3/reference-hook-tools#application-version-set).
>> Version information has to be set via a hook tool which means that it has to
>> come from a custom config option anyway. Each charm has its own method to
>> specify an application version and config dependencies are not modeled
>> explicitly - one has to implement that logic in a charm without any Juju API
>> for charms present the way I see it.
>>
>> config('key', 'app-version') - would be something to aim for.
>>
>> Do you have any thoughts about leader data vs a special complex config
>> option per charm and versioning?
>>
>> Thanks!
> Thanks for the feedback Dmitrii