January 2019 - openstack-discuss - lists.openstack.org

[goals][upgrade-checkers] Week R-14 Update
by Matt Riedemann 04 Jan '19

04 Jan '19

There has not been much progress since the R-16 update [1] let's assume because of the holiday break. There is a new trove patch which replaces the placeholder check with a real upgrade check [2]. I have left comments on that review. I am not sure if that is due to some new changes in trove which require the upgrade check, or if this is just something that has always been needed when upgrading trove. [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/001266… [2] https://review.openstack.org/#/c/627555/ -- Thanks, Matt

1 0

[placement] update 19-00
by Chris Dent 04 Jan '19

04 Jan '19

HTML: https://anticdent.org/placement-update-19-00.html Welcome to the first placement update of 2019. May all your placements have sufficient resources this year. # Most Important A few different people have mentioned that we're approaching crunch time on pulling the trigger on deleting the placement code from nova. The week of the 14th there will be a meeting to iron out the details of what needs to be done prior to that. If this is important to you, watch out for an announcement of when it will be. This is a separate issue from putting placement under its own governance, but some of the requirements [declared](http://lists.openstack.org/pipermail/openstack-dev/2018-September… for that, notably a deployment tool demonstrating an upgrade from placement-in-nova to placement-alone, are relevant. Therefore, reviewing and tracking the deployment tool related work remains critical. Those are listed below. Also, it is spec freeze next week. There are quite a lot of specs that are relevant to placement and scheduling that are close, but not quite. Mel has sent out [an email](http://lists.openstack.org/pipermail/openstack-discuss/2019-January/… about which specs most need attention. # What's Changed * There's an [os-resource-classes](https://pypi.org/p/os-resource-classes) now, already merged in placement, with a change posted [for nova](https://review.openstack.org/#/c/628278/). It's effectively the same as os-traits, but for resource classes. * There's a release of a 0.1.0 of placement [pending](https://review.openstack.org/628400). This won't have complete documentation, but will mean that there's an actually usable openstack-placement on PyPI, with what we expect to be the final python module requirements. * This has been true for a while, but it seems worth mentioning, via coreycb: "you can install placement-api on bionic with the stein cloud archive enabled". * A `db stamp` command has been added to `placement-manage` tool which makes it possible for someone who has migrated their data from nova to say "I'm at version X". * placement functional tests have been removed from nova. * Matt did a mess of work to make initializing the scheduler report client in nova less expensive and redundant. * Improving the handling of [allocation ratios](https://docs.openstack.org/nova/latest/admin/configuration/schedule… has merged, allowing for "initial allocation ratios". # Bugs * Placement related [bugs not yet in progress](https://goo.gl/TgiPXb): 15. -2. * [In progress placement bugs](https://goo.gl/vzGGDQ) 15. +2 # Specs Spec freeze next week! Only one of the previously listed specs has merged since early December and a new one has been added (at the end). * <https://review.openstack.org/#/c/544683/> Account for host agg allocation ratio in placement (Still in rocky/) * <https://review.openstack.org/#/c/595236/> Add subtree filter for GET /resource_providers * <https://review.openstack.org/#/c/597601/> Resource provider - request group mapping in allocation candidate * <https://review.openstack.org/#/c/549067/> VMware: place instances on resource pool (still in rocky/) * <https://review.openstack.org/#/c/555081/> Standardize CPU resource tracking * <https://review.openstack.org/#/c/599957/> Allow overcommit of dedicated CPU (Has an alternative which changes allocations to a float) * <https://review.openstack.org/#/c/591037/> Modelling passthrough devices for report to placement * <https://review.openstack.org/#/c/603955/> Nova Cyborg interaction specification. * <https://review.openstack.org/#/c/601596/> supporting virtual NVDIMM devices * <https://review.openstack.org/#/c/552924/> Proposes NUMA topology with RPs * <https://review.openstack.org/#/c/569011/> Count quota based on resource class * <https://review.openstack.org/#/c/141219/> Adds spec for instance live resize * <https://review.openstack.org/#/c/612497/> Provider config YAML file * <https://review.openstack.org/#/c/509042/> Propose counting quota usage from placement and API database * <https://review.openstack.org/603545> Resource modeling in cyborg. * <https://review.openstack.org/#/c/609960/> Support filtering of allocation_candidates by forbidden aggregates * <https://review.openstack.org/601596> support virtual persistent memory # Main Themes ## Making Nested Useful Progress continues on gpu-reshaping for libvirt and xen: * <https://review.openstack.org/#/q/topic:bp/reshape-provider-tree+status:open> Also making use of nested is bandwidth-resource-provider: * <https://review.openstack.org/#/q/topic:bp/bandwidth-resource-provider> There's a [review guide](http://lists.openstack.org/pipermail/openstack-discuss/2018-December… for those patches. Eric's in the process of doing lots of cleanups to how often the ProviderTree in the resource tracker is checked against placement, and a variety of other "let's make this more right" changes in the same neighborhood: * Stack at: <https://review.openstack.org/#/c/615677> ## Extraction The [extraction etherpad](https://etherpad.openstack.org/p/placement-extract-stein-4) is starting to contain more strikethrough text than not. Progress is being made. I'll refactor that soon so it is more readable, before the week of the 14th meeting. The main tasks are the reshaper work mentioned above and the work to get deployment tools operating with an extracted placement: * [TripleO](https://review.openstack.org/#/q/topic:tripleo-placement-extractio… * [OpenStack Ansible](https://review.openstack.org/#/q/project:openstack/openstack-ansib… * [Kolla and Kolla Ansible](https://review.openstack.org/#/q/topic:split-placement) Loci's change to have an extracted placement has merged. Kolla has a patch to [include the upgrade script](https://review.openstack.org/#/q/topic:upgrade-placement). It raises the question of how or if the `mysql-migrate-db.sh` should be distributed. Should it maybe end up in the pypi distribution? Documentation tuneups: * Release-notes: <https://review.openstack.org/#/c/618708/> This is blocked until we refactor the release notes to reflect _now_ better. * The main remaining task here is participating in [openstack-manuals](https://docs.openstack.org/doc-contrib-guide/doc-index.h…, to that end: * <https://review.openstack.org/#/c/628183/> A stack of changes to nova to remove placement from the install docs. * <https://review.openstack.org/#/c/628220/> Install docs in placement. I wrote to the [mailing list](http://lists.openstack.org/pipermail/openstack-discuss/2019-January/0… asking for input on making sure these things are close to correct, especially with regard to distro-specific things like package names. * <https://review.openstack.org/#/c/628324> Change to openstack-manuals to assert that placement is publishing install docs. Depends on the above. * There is a patch to [delete placement](https://review.openstack.org/#/c/618215/) from nova that we've put an administrative -2 on while we determine where things are (see about the meeting above). * There's a pending patch to support [online data migrations](https://review.openstack.org/#/c/624942/). This is important to make sure that fixup commands like `create_incomplete_consumers` can be safely removed from nova and implemented in placement. # Other There are currently 13 [open changes](https://review.openstack.org/#/q/project:openstack/placement+statu… in placement itself. Most of the time critical work is happening elsewhere (notably the deployment tool changes listed above). Of those placement changes, the [database-related](https://review.openstack.org/#/q/owner:nakamura.tetsuro%2… ones from Tetsuro are the most important. Outside of placement: * <https://review.openstack.org/#/q/topic:minimum-bandwidth-allocation-placeme…> Neutron minimum bandwidth implementation * <https://review.openstack.org/#/c/602160/> Add OWNERSHIP $SERVICE traits * <https://review.openstack.org/#/c/586960/> zun: Use placement for unified resource management * <https://review.openstack.org/#/c/621645/> WIP: add Placement aggregates tests (in tempest) * <https://review.openstack.org/627326> blazar: Consider the number of reservation inventory * <https://review.openstack.org/622316> Add placement client for basic GET operations (to tempest) # End Lot's of good work in progress. Our main task is making sure it all gets review and merged. The sooner we do, the sooner people get to use it and find all the bugs we're sure to have left lying around. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

1 0

[oslo] Problem when use library "oslo.messaging" for HA Openstack
by Thành Nguyễn Bá 04 Jan '19

04 Jan '19

Dear all, I have a problem when use 'notification listener' oslo-message for HA Openstack. It raise 'oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 172.16.4.125:5672 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'false' but current is 'true''. How can i fix this?. I think settings default in my program set 'durable' is False so it can't listen RabbitMQ Openstack? This is my nova.conf > http://paste.openstack.org/show/738813/ And section [oslo_messaging_rabbit] [oslo_messaging_rabbit] > rabbit_ha_queues = true > rabbit_retry_interval = 1 > rabbit_retry_backoff = 2 > amqp_durable_queues= true *Nguyễn Bá Thành* *Mobile*: 0128 748 0391 *Email*: bathanhtlu(a)gmail.com

3 4

Openstack CLI using a non-zero return code for successful command.
by Rafael Weingärtner 04 Jan '19

04 Jan '19

Hello OpenStackers, I have been using "openstack" CLI for a while now, and for most of the commands (the ones I used so far), when it is successful, I receive a return code (RC) 0 (ZERO). However, when using the command "openstack federation protocol set --identity-provider <idP> --mapping <mappingName> <protocol>", I am getting an RC 1 (ONE) as the exit code for successful executions as well. Is this a known bug? -- Rafael Weingärtner

1 0

[nova][dev] tracking spec reviews ahead of spec freeze
by melanie witt 04 Jan '19

04 Jan '19

Hey all, Spec freeze is coming up shortly after the holidays on January 10, 2019. Since we don't have much time after returning to work next year before spec freeze, let's keep track of specs that are close to approval or otherwise candidates to focus on in the last stretch before freeze. Here's an etherpad where I've collected a list of possible candidates for focus the first week of January. Feel free to add notes and specs I might have missed: https://etherpad.openstack.org/p/nova-stein-blueprint-spec-freeze Cheers, -melanie

2 3

[nova][ops] Trying to get per-instance live migration timeout action spec unstuck
by Matt Riedemann 04 Jan '19

04 Jan '19

I'm looking at this (previously approved [1]) spec again [2] and trying to sort out what needs to happen to reach agreement on this feature. Note the dependent blueprint is now complete in Stein [3]. The idea is pretty simple: provide new parameters to the live migration API to (1) override [libvirt]/live_migration_completion_timeout [4] and/or (2) provide a timeout action in case the provided (or configured) timeout is reached which would override [libvirt]/live_migration_timeout_action [5]. The use case is also pretty simple: you can have a default timeout and action (abort) configured but there could be cases where you need to override that on a per-instance basis to move a set of VMs off a host for maintenance, so you want to tell nova to force complete (post-copy or pause) in case of a timeout. The abort and force-complete actions are the same as in the API ([6] and [7] respectively). There are two main sticking points against this in the review: 1. This can already be done using existing APIs (as noted) client-side if monitoring the live migration and it times out for whatever you consider a reasonable timeout at the time. 2. The libvirt driver is the only one that currently supports abort and force-complete. For #1, while valid as a workaround, is less than ideal since it would mean having to orchestrate that into any tooling that needs that kind of workaround, be that OSC, openstacksdk, python-novaclient, gophercloud, etc. I think it would be relatively simple to pass those parameters through with the live migration request down to nova-compute and have the parameters override the config options and then it's natively supported in the API. For #2, while also true, I think is not a great reason *not* to support per-instance timeouts/actions in the API when we already have existing APIs that do the same thing and have the same backend compute driver limitations. To ease this, I think we can sort out two things: a) Can other virt drivers that support live migration (xenapi, hyperv, vmware in tree, and powervm out of tree) also support abort and force-complete actions? John Garbutt at least thought it should be possible for xenapi at the Stein PTG. I don't know about the others - driver maintainers please speak up here. The next challenge would be getting driver maintainers to actually add that feature parity, but that need not be a priority for Stein as long as we know it's possible to add the support eventually. b) There are pre-live migration checks that happen on the source compute before we initiate the actual guest transfer. If a user (admin) specified these new parameters and the driver does not support them, we could fail the live migration early. This wouldn't change the instance status but the migration would fail and an instance action event would be recorded to explain why it didn't work, and then the admin can retry without those parameters. This would shield us from exposing something in the API that could give a false sense of functionality when the backend doesn't support it. Given all of this, are these reasonable compromises to continue trying to drive this feature forward, and more importantly, are other operators looking to see this functionality added to nova? Huawei public cloud operators want it because they routinely are doing live migrations as part of maintenance activities and want to be able to control these values per-instance. I assume there are other deployments that would like the same. If this is something you'd like to see move forward, please speak up soon since the nova spec freeze for Stein is January 10. [1] https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-m… [2] https://review.openstack.org/#/c/600613/ [3] https://blueprints.launchpad.net/nova/+spec/live-migration-force-after-time… [4] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.li… [5] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.li… [6] https://developer.openstack.org/api-ref/compute/?expanded=#delete-abort-mig… [7] https://developer.openstack.org/api-ref/compute/?expanded=#force-migration-… -- Thanks, Matt

5 9

[openstack-dev] [neutron] Cancelling Neutron drivers meeting on January 4th
by Miguel Lavalle 04 Jan '19

04 Jan '19

Hi Neutrinos, Today I spent time triaging RFEs. I posted comments and questions in several of them in order to get them in good shape to be discussed by the Drivers team. None are ready to be discussed, though. As a consequence. I am cancelling the meeting on January 4th. Best regards Miguel

1 0

[loci] Loci Meeting for January 3, 2019
by Chris Hoge 04 Jan '19

04 Jan '19

On Friday, January 3 we will resume our Loci team meetings at 7 AM PT/ 15 UTC after an extended end-of-year break. We will revisit our plan for building stable branches (in light of several failures to build stable branches on a few distributions because of requirements failures), coming up with a more robust testing strategy, and planning new development priorities for the remainder of the cycle. Thanks, Chris https://etherpad.openstack.org/p/loci-meeting

1 1

[tripleo] OVB stable/1.0 branch created
by Ben Nemec 03 Jan '19

03 Jan '19

Just a quick update on the status of OVB. There's some discussion on [1] about how to handle importing it to Gerrit, and it looks like we first want to wrap up the 2.0-dev feature branch so we don't have to mess with that after import. As a result, I've cut the stable/1.0 branch from current OVB master. Anyone who's not ready to try out 2.0 should start pointing at stable/1.0 instead of master as the 2.0-dev branch will soon be merged back to master. This will leave us with a branch structure that better matches most other OpenStack projects. We should probably give consumers some time to get switched over, but it sounds like TripleO CI already pins to a specific commit in OVB so it may not affect that. Once a little time has passed we can do the 2.0-dev merge and proceed with the import. I also made a wordier blog post about this[2]. Let me know if you have any feedback. Thanks. -Ben 1: https://review.openstack.org/#/c/620613/ 2: http://blog.nemebean.com/content/openstack-virtual-baremetal-import-plans

2 1

[tripleo] Scenario Standalone ci jobs update - voting and promotion pipeline
by Marios Andreou 03 Jan '19

03 Jan '19

o/ TripleO's & Happy New Year \o/ if you are tracking the ci squad you may know that one area of focus recently is moving the scenario-multinode jobs to harder/better/faster/stronger (most importantly *smaller* but not as cool) scenario-standalone equivalents. Scenarios 1-4 are now merged [1]. For the current sprint [2] ci squad is doing cleanup on those. This includes making sure the new jobs are used in all the places the multinode jobs were e.g.[3][4] (& scens 2/3 will follow) and fixing any missing services or any other nits we find. Once done we can move on to the rest - scenarios 5/6 etc. We are looking for any feedback about the jobs in general or any one in particular if you have some special interest in a particular service (see [5] for reminder about services and scenarios). Most importantly those jobs are now being set as voting (e.g. already done for 1/4 at [1]) and the next natural step once voting is to add them into the master promotion pipeline. Please let us know if you think this is a bad idea or with any other feedback or suggestion. regards &thanks for reading! marios [1] https://github.com/openstack-infra/tripleo-ci/blob/3d634dc2874f95a9d4fd97a1… [2] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-3 [3] https://review.openstack.org/#/q/topic:replace-scen1 [4] https://review.openstack.org/#/q/topic:replace-scen4 [5] https://github.com/openstack/tripleo-heat-templates#service-testing-matrix

3 2

[neutron] Switching Ocata branch to Extended maintenance
by Bernard Cafarelli 03 Jan '19

03 Jan '19

Hello and happy new year, as discussed in our final 2018 meeting [0], we plan to switch the Ocata branch of Neutron to Extended maintenance [1]. There will be a final Ocata release for Neutron itself. Stadium projects backports only included test fixes so do not really need a final release. We will then mark the branch as ocata-em. Any comments or objections? Especially from driver projects maintainers, as these have independent releases. [0] http://eavesdrop.openstack.org/meetings/networking/2018/networking.2018-12-… [1] https://docs.openstack.org/project-team-guide/stable-branches.html#extended… -- Bernard Cafarelli

1 0

[release] Release countdown for week R-13, Jan 7-11
by Sean McGinnis 03 Jan '19

03 Jan '19

Welcome back from the holiday lull! Development Focus ----------------- Teams should be focused on completing milestone 2 activities by January 10 and checking overall progress for the development cycle. General Information ------------------- The Stein membership freeze coincides with the Stein-2 milestone on the 10th. While doing an audit of all officially governed team repos, there are quite a few that have not had an official release done yet. If your team has added any repos for deliverables you would like to have included in the Stein coordinated release, please add at least an empty template deliverable file for now so we can help track those. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. Another reminder about the changes this cycle with library deliverables that follow the cycle-with-milestones release model. As announced, we will be automatically proposing releases for these libraries at milestone 2 if there have been any functional changes since the last release to help ensure those changes are picked up by consumers with plenty of time to identify and correct issues. More detail can be found in the original mailing list post describing the changes: http://lists.openstack.org/pipermail/openstack-dev/2018-October/135689.html Any other cycle-with-intermediary deliverables that have not requested a release by January 10 will be switched to the cycle-with-rc model as discussed previously: http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000465… Upcoming Deadlines & Dates -------------------------- Stein-2 Milestone: January 10 -- Sean McGinnis (smcginnis)

1 0

[placement] [packaging] [docs] WIP placement install docs
by Chris Dent 03 Jan '19

03 Jan '19

I've started a very wippy work in progress [1] for placement installation docs within the extracted placement repository, using the existing docs from nova as a base. I'd like to draw the attention of rdo, ubuntu, and obs packagers as the docs follow the existing pattern of documenting install on those three distros. However, since placement is not fully packaged yet, I'm making some guesses in the docs about how things will be set up and how packages will be named. So if people who are interested in such things could have a look that would be helpful. I've stubbed out, but not yet completed, a from-pypi page as well, as with placement, installation can become as simple as pip install + run a single command line with the right args, if that's what you want. I think people should be able to see that too. Thanks. [1] https://review.openstack.org/628220 -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

1 0

[tc] agenda for Technical Committee Meeting 3 Jan 2019 @ 1400 UTC
by Doug Hellmann 03 Jan '19

03 Jan '19

TC Members, Our next meeting will be this Thursday, 3 Jan at 1400 UTC in #openstack-tc. This email contains the agenda for the meeting, based on the content of the wiki [0]. If you will not be able to attend, please include your name in the "Apologies for Absence" section of the wiki page [0]. [0] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee We have several items of old business to wrap up: * technical vision for openstack We have approved the first draft of the vision [1], and have one update proposed by gmann [2]. What are the next steps for this change to the vision? Are we ready to remove this topic from the tracker now, and treat further updates as individual work items? [1] https://governance.openstack.org/tc/reference/technical-vision.html [2] https://review.openstack.org/#/c/621516/ * next step in TC vision/defining the role of the TC We also approved a document explaining the role of the TC [3]. Is there more work to do here, or are we ready to remove this from the tracker now? [3] https://governance.openstack.org/tc/reference/role-of-the-tc.html * keeping up with python 3 releases We have approved all of the patches for documenting the policy and for selecting the versions to be covered in Stein. What are the next steps for ensuring that any implementation work is handled? * Reviewing TC Office Hour Times and Locations The most recent mailing list thread [4] was resolved with no changes to the number of office hours. Does anyone want to propose different times, or are we happy with the current schedule? [4] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000542… and we also have a couple of items of new business to take up: * Train cycle goals selection update Checking with lbragstad and evrardjp for any updates to share with us this month. * health check status for stein How is it going contacting the PTLs for the health check for stein? Does anyone have anything to raise based on what they have learned in their conversations? -- Doug

1 1

[glance] (Silly) question about container_format and disk_format
by Andre Florath 03 Jan '19

03 Jan '19

Hello! I do not completely understand the parameters 'container_format' and 'disk_format' as described in [1]. The documentation always uses 'the format' but IMHO there might be two formats involved. Are those formats either (A) the formats of the image file that is passed in. Like (from the official documentation [2]) $ openstack image create --disk-format qcow2 --container-format bare \ --public --file ./centos63.qcow2 centos63-image qcow2 / bare are the formats of the passed in image. or (B) the formats that are used internally to store the image Like $ openstack image create --disk-format vmdk --container-format ova \ --public --file ./centos63.qcow2 centos63-image vmdk / ova are formats that are used internally in OpenStack glance to store the image. In this case there must be an auto-detection of the image file format that is passed in and an automatic conversion into the new format. Kind regards Andre [1] https://developer.openstack.org/api-ref/image/v2/index.html?expanded=create… [2] https://docs.openstack.org/glance/pike/admin/manage-images.html

2 2

Re: [Infra][Cinder] Request for voting permission from LINBIT LINSTOR CI (linbit_ci@linbit.com) for Cinder projects
by Woojay Poynter 03 Jan '19

03 Jan '19

Hello, I would like to request a voting permission on LINBIT LINSTOR CI for Cinder project. We are looking to satisfy third-party testing requirement for Cinder volume driver for LINSTOR (https://review.openstack.org/#/c/624233/) The CI's test result of the lastest patch is at http://us.linbit.com:8080/CI-LINSTOR/33/456/ While the driver itself is still being tweaked on the remaining two failed tests, I hope I can have the CI portion of the requirement ready w/ this request. The testbeds are jenkins slave running fresh installation of devstack w/ target patch triggered from gerrit with lvmthin storage backend managed by LINSTOR driver for Cinder. LINSTOR is storage provisioner using DRBD for block replication. Some of the test results of the LINBIT LINSTOR CI running on openstack-dev/ci-sandbox are here; https://review.openstack.org/#/c/627599/ https://review.openstack.org/#/c/627596/ https://review.openstack.org/#/c/626625/ Thank you for your time during the holidays. Sincerely, Woojay Poynter -- 541-435-1001 woojay(a)linbit.com LINBIT | Keeping the Digital World Running DRBD – Corosync – Pacemaker DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. On Fri, Dec 28, 2018 at 1:51 PM Woojay Poynter <woojay(a)linbit.com> wrote: > Hello, > > I would like to request a voting permission on LINBIT LINSTOR CI for > Cinder project. We are looking to satisfy third-party testing requirement > for Cinder volume driver for LINSTOR ( > https://review.openstack.org/#/c/624233/) The CI's test result of the > lastest patch is at http://us.linbit.com:8080/CI-LINSTOR/33/456/ > > While the driver itself is still being tweaked on the remaining two failed > tests, I hope I can have the CI portion of the requirement ready w/ this > request. > > The testbeds are jenkins slave running fresh installation of devstack w/ > target patch triggered from gerrit with lvmthin storage backend managed by > LINSTOR driver for Cinder. LINSTOR is storage provisioner using DRBD for > block replication. > > Some of the test results of the LINBIT LINSTOR CI running on > openstack-dev/ci-sandbox are here; > https://review.openstack.org/#/c/627599/ > https://review.openstack.org/#/c/627596/ > https://review.openstack.org/#/c/626625/ > > Thank you for your time during the holidays. > > Sincerely, > > Woojay Poynter > -- > 541-435-1001 > woojay(a)linbit.com > > LINBIT | Keeping the Digital World Running > DRBD – Corosync – Pacemaker > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > -- Woojay Poynter – Software Developer +1 503 573-1262 x211 woojay(a)linbit.com LINBIT | Keeping the Digital World Running DRBD – Corosync – Pacemaker DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

2 1

[cinder] Is the cinder Active-Active feature OK？
by Jaze Lee 03 Jan '19

03 Jan '19

Hello, In my opinion, all rest api will get to manager of cinder volume. If the manager is using DLM, we can say cinder volume can support active-active. So, can we rewrite the comments and option's help in page https://github.com/openstack/cinder/blob/master/cinder/cmd/volume.py#L78 ? Thanks a lot. -- 谦谦君子

2 2

[neutron] Bug deputy report 17th~23rd Dec 2018
by Paweł Suder 02 Jan '19

02 Jan '19

Hello Neutrons, Following bugs/RFE/issues have been raised during last week. Some of them were already recognized, triaged, checked: >From oldest: https://bugs.launchpad.net/neutron/+bug/1808731 [RFE] Needs to restart metadata proxy with the start/restart of l3/dhcp agent - IMO need to discus how to do that. https://bugs.launchpad.net/neutron/+bug/1808916 Update mailinglist from dev to discuss - it is done. https://bugs.launchpad.net/neutron/+bug/1808917 RetryRequest shouldn't log stack trace by default, or it should be configurable by the exception - confirmed by Sławek ;) https://bugs.launchpad.net/neutron/+bug/1809037 [RFE] Add anti_affinity_group to binding:profile - RFE connected with another RFE from Nova related to NFV and (anti)affinity of resources like PFs. https://bugs.launchpad.net/neutron/+bug/1809080 reload_cfg doesn't work correctly - change in review, I tried to review, but no comments left - new contributor https://bugs.launchpad.net/neutron/+bug/1809134 - TypeError in QoS gateway_ip code in l3-agent logs - review in progress https://bugs.launchpad.net/neutron/+bug/1809238 - [l3] `port_forwarding` cannot be set before l3 `router` in service_plugins - review in progress https://bugs.launchpad.net/neutron/+bug/1809447 - performance regression from mitaka to ocata - not triaged, I am not sure how to handle it - it is a wide thing.. https://bugs.launchpad.net/neutron/+bug/1809497 - bug noticed on gates related to another bug opened last week: https://bugs.launchpad.net/neu tron/+bug/1809134 Wish you a good time this week and for the next year! :) Cheers, Paweł

2 2

[all][ptl][heat][senlin][magnum]New SIG for Autoscaling? plus Session Summary: Autoscaling Integration, improvement, and feedback
by Rico Lin 02 Jan '19

02 Jan '19

Dear all Tl;dr; I gonna use this ML to give a summary of the forum [1] and asking for feedback for the idea of new SIG. We have a plan to create a new SIG for autoscaling which to cover the common library, docs, and tests for cross-project services (Senlin, Heat, Monasca, etc.) and cross-community (OpenStack, Kubernetes, etc). And the goal is not just to have a place to keep those resources that make sure we can guarantee the availability for use cases, also to have a force to keep push the effort to integrate across services or at least make sure we don't go to a point where everyone just do their own service and don't care about any duplication. So if you have any thoughts for the new SIG (good or bad) please share it here. Here I summarize our discussion in the forum session `Autoscaling Integration, improvement, and feedback`. If you like to learn more or input your thoughts, feel free to put it in etherpad [1] or simply reply to this email. In the forum, we have been discussed the scope and possibility to integrate effort from Heat, Senlin, and also autoscaling across OpenStack to K8s. There are some long-term goals that we can start on like create a document for general Autoscaling on OpenStack, Common library for cross-project usage, or create real scenario cases to test on our CI. And the most important part is how can we help users and satisfied use cases without confuse them or making too much duplication effort across communities/projects. So here's an action we agree on, is to trigger discussion for either we need to create a new SIG for autoscaling. We need to define the scope and the goal for this new SIG before we go ahead and create one. The new SIG will cover the common library, docs, and tests for cross-project services (Senlin, Heat, Monasca, etc.) and cross-community (OpenStack, Kubernetes, etc). And the goal is not just to have a place to keep those resources that make sure we can guarantee the availability for use cases, also to have a force to keep push the effort to integrate across services or at least make sure we don't go to a point where everyone just do their own service and don't care about any duplication. For example, we can have a document about do autoscaling in OpenStack, but we need a place to put it and keep maintain it. And we can even have a place to set up CI to test all scenario for autoscaling. I think it's possible to extend the definition of this SIG, but we have to clear our goal and make sure we actually doing a good thing and make everyone's life easier. On the other hand we also need to make sure we do not duplicate the effort of other SIGs/WGs. Also The reason I add `ptl` tag for this ML is that this SIG or the concept of `autoscaling` might be very deferent to different projects. So I really wish to hear from anyone and any projects who are interested in this topic. [1] https://etherpad.openstack.org/p/autoscaling-integration-and-feedback -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

4 3

[nova] Granular locks in the API
by Matt Riedemann 02 Jan '19

02 Jan '19

I wanted to float something that we talked about in the public cloud SIG meeting today [1] which is the concept of making the lock API more granular to lock on a list of actions rather than globally locking all actions that can be performed on a server. The primary use case we discussed was around a pre-paid pricing model for servers. A user can pre-pay resources at a discount if let's say they are going to use them for a month at a fixed rate. However, once they do, they can't resize those servers without going through some kind of approval (billing) process to resize up. With this, the provider could lock the user from performing the resize action on the server but the user could do other things like stop/start/reboot/snapshot/etc. The pricing model sounds similar to pre-emptible instances for getting a discount but the scenario is different in that these servers couldn't be pre-empted (they are definitely more non-cloudy pets than cattle). An alternative solution for that locked resize issue is using granular policy rules such that pre-paid servers have some other kind of role attached to them so by policy you could restrict users from performing actions on those servers (but the admin could override). In reality I'm not sure how feasible that is in a public cloud with several thousand projects. The issue I see with policy controlling this is the role is attached to the project, not the resource (the server), so if you did this would users have to have separate projects for on-demand vs pre-paid resources? I believe that's what CERN and StackHPC are doing with pre-emptible instances (you have different projects with different quota models for pre-emptible resources). I believe there are probably other use cases for granular locks on servers for things like service VMs (trove creates some service VMs to run a database cluster and puts locks on those servers). Again, definitely a pet scenario but it's one I've heard before. Would people be generally in favor of this or opposed, or just meh? [1] https://etherpad.openstack.org/p/publiccloud-wg -- Thanks, Matt

4 5

[neutron] bug deputy report (Dec 24 - Dec 30)
by Hongbin Lu 02 Jan '19

02 Jan '19

Hi all, Below is the bug deputy report for last week. Since it is on the holiday, there are not too much reporting bugs. REFs: * https://bugs.launchpad.net/neutron/+bug/1809628 [RFE] Enable driver field for the api of service_providers * https://bugs.launchpad.net/neutron/+bug/1809878 [RFE] Move sanity-checks to neutron-status CLI tool Incomplete: * https://bugs.launchpad.net/neutron/+bug/1809907 Unstable ping during zuul testing Best regards, Hongbin

1 0

Re: [Openstack] [Nova][Glance] Nova imports flat images from base file despite ceph backend
by Eugen Block 02 Jan '19

02 Jan '19

Hello and a happy new year! I need to reopen this thread because there are still things going on that I don't fully understand. I changed the disk_format of all the images that were affected by my mistake a couple of weeks ago. I deleted the base files in /var/lib/nova/instances/_base and launched new instances, leading to expected cow clones: ---cut here--- control:~ # openstack image show 5f486361-5468-42a0-9993-9cdda3450b0e | grep disk_format | disk_format | raw| control:~ # rbd children images/5f486361-5468-42a0-9993-9cdda3450b0e@snap images/029bfb90-cbab-4a7c-a51c-27807ab41ce7_disk images/ccd4498f-c0a0-480a-ab4b-6224d63e78fa_disk images/d5379918-40a0-4119-9e47-773c0ab8c0f3_disk ---cut here--- These are only three clones, but I have 16 instances based on the same image: ---cut here--- +--------------------------------------+ | uuid | +--------------------------------------+ | 87dfbb4f-784d-4390-a0c9-d3162a56ea7e | | bb56995d-d3ea-464d-94fe-382880bf2a92 | | d4fde2fe-140e-4904-90e8-996cc418302d | | 7bd4fdc0-d6b7-47cb-adb3-324abff6a0e5 | | bef8e4fe-b2f4-44f5-a144-36ced37007ac | | bcc005fc-fa12-4dd9-b531-cbccfb7c426a | ###| ccd4498f-c0a0-480a-ab4b-6224d63e78fa | ###| 029bfb90-cbab-4a7c-a51c-27807ab41ce7 | ###| d5379918-40a0-4119-9e47-773c0ab8c0f3 | | abd2c0bc-2e66-4e2a-9f4c-f006937c7b27 | | e07c0f5e-403e-4ad3-b04f-fb306e99869e | | edec4aa0-9459-4444-bbb8-03097bccba4c | | ca4449f5-921d-4ce5-8559-fc719cfbc845 | | 8bff1799-391e-4620-9b11-134962084b84 | | 9d927ac4-d7d3-45cb-8628-b267a6d0e668 | | 54cfeede-44f9-49bd-9f49-ef4dcacff953 | +--------------------------------------+ ---cut here--- Those three cow clones were created after adjusting the disk_format, some of the rest were created before the changes, the others have been created after the changes. Can anyone explain why a new base image is created? I downloaded the glance image and double-checked the file format, I also exported the snapshot which should be used to create clones, there's only one discrepancy (disk size), but I don't think this could be relevant: ---cut here--- control:~ # qemu-img info /var/lib/glance/images/image-snap.img image: /var/lib/glance/images/image-snap.img file format: raw virtual size: 5.0G (5368709120 bytes) disk size: 2.5G control:~ # qemu-img info /var/lib/glance/images/image.img image: /var/lib/glance/images/image.img file format: raw virtual size: 5.0G (5368709120 bytes) disk size: 5.0G control:~ # md5sum /var/lib/glance/images/image.img b9b28dd300a6fbb1de2081f1cb8a07d0 /var/lib/glance/images/image.img control:~ # md5sum /var/lib/glance/images/image-snap.img b9b28dd300a6fbb1de2081f1cb8a07d0 /var/lib/glance/images/image-snap.img ---cut here--- I also tried to find something with the glance-cache-manage cli, but I can't even connect to that service, I guess it has never been used. control:~ # glance-cache-manage list-cached Failed to show cached images. Got error: Connect error/bad request to Auth service at URL http://control1.cloud.hh.nde.ag:5000/v3/tokens. I don't know how to fix that yet, but could this be a way to resolve that issue? Regards, Eugen Zitat von melanie witt <melwittt(a)gmail.com>: > On Fri, 12 Oct 2018 20:06:04 -0700, Remo Mattei wrote: >> I do not have it handy now but you can verify that the image is >> indeed raw or qcow2 >> >> As soon as I get home I will dig the command and pass it on. I have >> seen where images have extensions thinking it is raw and it is not. > > You could try 'qemu-img info <image>' and get output like this, > notice "file format": > > $ qemu-img info test.vmdk > (VMDK) image open: flags=0x2 filename=test.vmdk > image: test.vmdk > file format: vmdk > virtual size: 20M (20971520 bytes) > disk size: 17M > > [1] https://en.wikibooks.org/wiki/QEMU/Images#Getting_information > > -melanie > > > > > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack(a)lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

1 0

queens heat db deadlock
by Ignazio Cassano 02 Jan '19

02 Jan '19

Hi All, I installed queens on centos 7. Heat seems to work fine with templates I wrote but when I create magnum cluster I often face with db deadlock in heat-engine log: 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource Traceback (most recent call last): 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 918, in _action_recorder 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource yield 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2035, in delete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource *action_args) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource step = next(subtask) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 977, in action_handler_task 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource done = check(handler_data) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 587, in check_delete_complete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource return self._check_status_complete(self.DELETE) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 454, in _check_status_complete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource action=action) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource ResourceFailure: resources[0]: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/2j85) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource 2018-12-13 11:48:46.030 89597 INFO heat.engine.stack [req-9a43184f-77d8-4fad-ab9e-2b9826c10b70 - admin - default default] Stack DELETE FAILED (swarm-clustergp27-ebhsalhb4bop-swarm_primary_master-tzgkh3ncmymw): Resource DELETE failed: resources[0]: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/2j85) 2018-12-13 11:48:46.844 89595 INFO heat.engine.resource [req-9a43184f-77d8-4fad-ab9e-2b9826c10b70 - admin - default default] DELETE: ResourceGroup "swarm_primary_master" [6984db3c-3ac1-4afc-901f-21b3e7f230a7] Stack "swarm-clustergp27-ebhsalhb4bop" [b43256fa-52e2-4613-ac15-63fd9340a8be] I read this issue was solved in heat version 10 but seems not. Regards Ignazio

5 14

Re: [heat] Bug : Heat cannot create Octavia Load Balancer
by Jeff Yang 02 Jan '19

02 Jan '19

Please confirm whether "auth_strategy" is set as ''keystone" in configure file. I remember that the value of "auth_strategy" is set as "noauth" in "/usr/share/octavia/octavia-dist.conf" default if you install octavia by rpm. If the value was set as "noauth", you must manually specify "project_id" for octavia. Jeff Yang <yjf1970231893(a)gmail.com> 于2019年1月2日周三下午12:26写道： > Please confirm whether "auth_strategy" is set as ''keystone" in > configure file. I remember that the value of "auth_strategy" is set as > "noauth" in "/usr/share/octavia/octavia-dist.conf" default if you install > octavia by rpm. If the value was set as "noauth", you must manually specify > "project_id" for octavia. > > Zane Bitter <zbitter(a)redhat.com> 于2019年1月2日周三上午11:23写道： > >> On 2/01/19 1:34 AM, Adam Harwell wrote: >> > I'm just on my phone over the holidays, but it kinda looks like the >> code >> > for this was just updated 12 days ago: >> > https://review.openstack.org/#/c/619577/ >> > >> > If you're using that new code, I imagine it's possible there could be a >> > bug that wasn't yet caught... If you're NOT using that code, maybe try >> > it and see if it helps? I'm guessing it's related one way or another. >> If >> > you come to the #openstack-lbaas channel once more people are around >> > (later this week?), we can probably take a look. >> >> That's only the example template (previously it was an example for >> LBaaSv2; now it's an example for Octavia); there's been no recent change >> to the code. >> >> > --Adam Harwell (rm_work) >> > >> > On Sun, Dec 30, 2018, 03:37 Zufar Dhiyaulhaq <zufardhiyaulhaq(a)gmail.com >> > <mailto:zufardhiyaulhaq@gmail.com>> wrote: >> > >> > I have try creating load balancer with Heat. but always get this >> error : >> > >> > Resource CREATE failed: OctaviaClientException: >> > resources.loadbalancer: Validation failure: Missing project ID in >> > request where one is required. (HTTP 400) (Request-ID: >> > req-b45208e1-a200-47f9-8aad-b130c4c12272) >> > >> > OctaviaClientException: resources.loadbalancer: Validation failure: >> > Missing project ID in request where one is required. (HTTP 400) >> > (Request-ID: req-b45208e1-a200-47f9-8aad-b130c4c12272) >> >> What version of OpenStack are you using? >> >> The issue is that Heat is sending a "tenant_id" but Octavia wants a >> "project_id", which is the new name for the same thing. (I think you >> likely modified that template after trying it but before uploading it, >> because there is no "project_id" property in Heat's >> OS::Octavia::LoadBalancer resource type.) >> >> This bug has been reported and there is a patch up for review in Heat: >> https://storyboard.openstack.org/#!/story/2004650 >> >> There was a change to Octavia in Pike >> (https://review.openstack.org/455442) to add backwards compatibility, >> but it was either incomplete or the problem reoccurred and was fixed >> again in Rocky (https://review.openstack.org/569881) My guess is that >> it's likely broken in Pike and Queens. >> >> I'd certainly have expected Heat's gate tests to pick up the problem, >> and it's a bit of a mystery why they didn't. Perhaps we're not >> exercising the case where a project_id is required (using it at all is >> an admin-only feature, so that's not too surprising I guess; it's >> actually more surprising that there's a case where it's _required_). >> >> cheers, >> Zane. >> >> > I create 2 openstack environment : >> > >> > * Heat with Octavia (Octavia Heat Template : >> > http://paste.opensuse.org/view//33592182 ) >> > * Heat with Neutron Lbaasv2 (Neutron LBaaSv2 Heat Template : >> > http://paste.opensuse.org/view//71741503) >> > >> > But always error when creating with octavia : >> > >> > * Octavia Log (https://imgur.com/a/EsuWvla) >> > * LBaaS v2 (https://imgur.com/a/BqNGRPH) >> > >> > Are Heat code is broken to create Octavia Load Balancer? >> > >> > Best Regards, >> > Zufar Dhiyaulhaq >> > >> >> >>

2 1

[heat] Bug : Heat cannot create Octavia Load Balancer
by Zufar Dhiyaulhaq 02 Jan '19

02 Jan '19

I have try creating load balancer with Heat. but always get this error : Resource CREATE failed: OctaviaClientException: resources.loadbalancer: Validation failure: Missing project ID in request where one is required. (HTTP 400) (Request-ID: req-b45208e1-a200-47f9-8aad-b130c4c12272) OctaviaClientException: resources.loadbalancer: Validation failure: Missing project ID in request where one is required. (HTTP 400) (Request-ID: req-b45208e1-a200-47f9-8aad-b130c4c12272) I create 2 openstack environment : - Heat with Octavia (Octavia Heat Template : http://paste.opensuse.org/view//33592182 ) - Heat with Neutron Lbaasv2 (Neutron LBaaSv2 Heat Template : http://paste.opensuse.org/view//71741503) But always error when creating with octavia : - Octavia Log (https://imgur.com/a/EsuWvla) - LBaaS v2 (https://imgur.com/a/BqNGRPH) Are Heat code is broken to create Octavia Load Balancer? Best Regards, Zufar Dhiyaulhaq

3 2

[Heat][Octavia] Is autoscaling feature missing?
by Viktor Shulhin 02 Jan '19

02 Jan '19

Hi all, I am trying to create Heat template with autoscaling and loadbalancing. I didn't find any similar Heat template examples. Individually, loadbalancing and autoscaling work well, but loadbalancing OS::Octavia::PoolMember can be added only manually. Is there any way to use OS::Heat::AutoScalingGroup as server pool for loadbalancing? Regards, Viktor.

3 2

[Cyborg] no irc meeting this week
by Li Liu 01 Jan '19

01 Jan '19

Hi Team, Since it's public holiday for folks in US and Canada, we will not have the irc meeting this week. Enjoy the new year guys :) Thank you Regards Li Liu

1 0

[dev][karbor]No meeting today
by 焦鹏举 01 Jan '19

01 Jan '19

Hi Karbor Team, We will skip karbor weekly meeting today due to holidays. Next meeting will come on 15 January. Thanks. Pengju Jiao

1 0

[chef] State of the Kitchen: 8th Edition
by Samuel Cassiba 01 Jan '19

01 Jan '19

It has been some time since the last State of the Kitchen. Since the release of Chef OpenStack 17 (Queens), we have been operating in a minimal churn mode to give people time to test/upgrade deployments and handle any regressions that emerge. There are three main areas that are still in progress upstream from Chef OpenStack, but can affect its cadence regardless. As a result, this update focuses more on those areas. Consider this more of a year-end review. ### Important Happenings * *fog-openstack*[^1] Beginning in August, we started receiving reports of breakage due to changes in fog-core. As a reactionary measure, we implemented upper-level constraints in the client resource cookbook to maintain a consistent outcome. The fog-openstack library has continued to receive changes to further align with fog-core, and we are following its progress to find a good time to move ChefDK and Chef OpenStack to a post-1.0 release of fog-openstack. We are targeting the 18th release of Chef OpenStack, due to Keystone endpoint changes that need to happen. * *Sous Chefs*[^2] One of the biggest strengths of Chef and OpenStack is the collective outcome of their unique communities. Within the Chef ecosystem, the Sous Chefs group was formed in response to a need for the continued existence of Chef components, libraries, and utilities that need a long-term home. Across the globe, Sous Chefs work to keep some of the most heavily used cookbooks in existence, such as [apache2](https://supermarket.chef.io/cookbooks/apache2), [mysql](https://supermarket.chef.io/cookbooks/mysql) (and [mariadb](https://supermarket.chef.io/cookbooks/mariadb)!), [postgresql](https://supermarket.chef.io/cookbooks/postgresql), as well as [redisio](https://supermarket.chef.io/cookbooks/redisio), and many more. Chef OpenStack depends on MariaDB, Apache, and their related cookbooks, for compatibility without operators needing to plumb those resources internally. * *poise-python*[^3] In early October, pip 18.1 was released, which made some additional waves in the ecosystem. Workarounds were devised and implemented to limit the fallout. Currently, the fix has been merged to poise-python's master, but cannot be released safely due to CI changes in the current workflow. There are limitations on what the Sous Chefs can reasonably maintain. The maintenance of poise is rather beyond that boundary, not to discount or disparage anyone involved. Anyone interested with spare cycles over the holiday season might consider joining the conversation. ### Upcoming Changes * In Chef OpenStack 18... - The MariaDB version will default to 10.3, consistent with the default in the 2.0 version of the cookbook. Please plan accordingly. - Keystone's endpoint will be changing to drop the hardcoded API version - the cloud primitives (client) cookbook is in the process of migrating from cookbook-openstackclient to cookbook-openstack-client (named openstack_client, to conform with current best practices in the Chef community) - Ubuntu will be upgraded from 16.04 to 18.04, and as such we will be gating against Bionic at that time. Plainly put, previous Chef OpenStack releases will not be moving to Bionic jobs, and will continue to work at best effort until they succumb to the detritus of time. ### Meetings Since the Summit, a few people have reached out through various means about Chef OpenStack and how to work together to improve the outcome. As a result, I would like to propose holding regular meetings for Chef OpenStack once more set aside a dedicated period where we can come together and talk about food, or other things. We have the IRC channel, but IRC has proven less effective for a small group to dedicate time consistently, so I would something more high bandwidth for technical conversations, such as video with a publicized method for joining and viewing. I will follow up with a more expanded proposal outside this update. ### On The Menu This would not be a State of the Kitchen without something to eat. My partner and I try to cook with recipes that are not overly complicated, but can be infinitely complex with just the right nudge. Sometimes we incorporate our own opinions into someone else's recipe to make it our own thing, and sometimes they're great just as they come. *Dat Dough, Doe* * 170g / 6oz grated mozzarella or Edam, or another mild cheese with similar melting consistency * 85g / 3oz almond meal/flour * 28g / 2 tbsp cream cheese or Neufchatel * 1 egg * pinch of salt to taste 1. Mix the shredded/grated cheese and the almond meal in a microwaveable bowl, then add the cream cheese. Microwave on high for 1 minute. Stir the mixture, then microwave on high for another 30 seconds. 2. Add egg, salt, additional spices or flavorings, and mix or fold gently. 3. Shape using parchment paper into the desired outcome, be it flat like a disc or rounded, like a boule. 4. Create vents to ensure that the finished product cooks evenly. 5. Fry, bake, broil or grill as desired. Lipids can be friends here. More commonly known as the "Fat Head" dough, out of these few ingredients, one can make food that can taste every bit like pizza, pasta, bread, even pão de queijo. Or, perhaps, cinnamon rolls, or danishes, as one might consider making. With these basic suggestions, one can apply their own opinions and set of requirements to create complex pieces of work, which can taste every bit like an artform and a science. See you in 2019! Your humble pastry chef, -scas [^1]: https://github.com/fog/fog-openstack/issues/434 [^2]: https://sous-chefs.org/ [^3]: https://github.com/poise/poise-python/issues/133

1 0