Hi everyone,
here is a summary of the discussions. The full agenda can be found here [1].
== Current pain points and challenges of interop testing ==
The current guidelines don't contain complex enough tests - apart from api tests we should also have more functional tests which mock users' interactions with the cloud - we should test as a real user would use the cloud.
People could fake the refstack results - after a discussion we came to an agreement that those people would always find a way, there is no need for coming up with sophisticated ways to prevent it. The point of Interop is to provide a check for vendors/users to see where they stand with their product based on the agreed standards.
Use of Interop's tooling is complicated - there have been reports that users struggle with refstack-client and tempest. There is an ongoing effort to create a refstack container which would contain all necessary tools and could be run with just one command.
Passing the Interop's guidelines means interoperability on which level? Passing the guidelines should mean being in compliance with the beforehand community agreed upon standards.
Getting vendors running the interop tests more often - we only came up with the strategies which are already being used, such as expirations and guideline versioning.
== Future strategy ==
The future strategy could be summarized in 3 points:
* discuss the possibility for using sdk-based and tobiko testing
* better describe to users the expectations they should have for clouds which assert interoperability
* look into expiration of older standards and require that compliant providers indicate to users which version of the standard they last tested/passed
Regards,
-- Martin Kopec
Principal Software Quality Engineer
Red Hat EMEA
IM: kopecmartin