Modularity is Dead, Long Live Modularity!

-----BEGIN PGP SIGNED MESSAGE-----  Hash: SHA256    Modularity is Dead, Long Live Modularity!  =========================================    See this post in glorious technicolor at:  https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/      Summary  - -------    Fedora's Modularity initiative aims to make it easy for packagers to  create alternative versions of software and for users to consume those  streams simply. We've been working on this for several years, resulting  in the "Boltron" prototype this summer and the recent Fedora Modular  Server beta. Feedback shows that these test releases didn't meet the  goal, and we're incorporating that in a modified design which we think  _will_. We plan to demo the new approach by DevConf.cz and FOSDEM.      Retrospective  - -------------    As you have probably read, the members of the Modularity Working Group  and the Server Working Group decided that the current implementation of  Modularity would not meet Fedora's standard of excellence in time for a  release in Fedora 27 — even with the extra time that the Fedora Council  granted to the project. In particular, feedback received from the  Fedora 27 Modular Server Beta release alerted us to several serious  problems in the design.    The most common feedback from users was: "How do I install package  `foo` in Modular Server?". In this case, `foo` ranged across a wide  variety of software, including everything from the screen package to  complex third-party applications. In that version of Modularity, a  system was either all Modular or none of it was. To make software  available in the fully-modular system, its packages needed to be part  of a module — and unfortunately, we didn't succeed in making many of  those.    The lack of available modules was a symptom of a larger problem with  packager involvement. Some of this problem was due to disagreements  with the implementation or design, and some of it was because creating  modules turned out to be more time-consuming and had a steeper learning  curve than we wanted. Additionally, the whole process was not obvious  or well-socialized. The team produced a lot of documentation, but  packagers needed to understand a whole lot of it before even getting  started. As a result, all of the modules slated for delivery in F27  Modular Server were being developed by the Modularity team, a group  that had limited expertise around many of the packages we desired to  ship.    There were other problems, of course. We didn't really have a strong  plan for how to handle upgrades from traditional deployments. We lacked  a story for how third-party software should be loaded onto the system.  And, we never got away from needing a "bootstrap" module (as explained  below). So, we decided not to ship Modular Server in Fedora 27, and we  activated our fallback plan of releasing Fedora 27 Server Edition in  its classic incarnation at https://getfedora.org/server/.      Moving Forward  - --------------    With all of this in mind, we took Modularity back to the drawing board  (or perhaps "chopping board"). We analyzed the various functionality  that the design offered us and made some trade-offs.    First and foremost, we gave up on the idea of a strictly-maintained,  stable buildroot. Traditional Fedora builds are performed by building  an RPM in a buildroot containing the latest packages available in the  stable updates repository for a release (plus the overrides repo when  needed to allow building against pre-release packages). With  modularity, we hoped that we could define a small and specific  buildroot which would be stable and maintained for the life of a  release. We would then build every module against it. It turned out  when we tried to implement this that it was basically impossible. It  required us to find a point at which the entire buildroot could be used  to build _itself_, a feat which has not been accomplished in Fedora for  many years (if ever). Fedora packages are really only guaranteed to  build successfully against a buildroot for which they have _already_  been built. If it builds against any other buildroot, this is a happy  accident.    There are many reasons for this; something in the dependency chain for  one of the build dependencies may have broken, a bug may have been  introduced in the compiler or linker resulting in an inability to  continue, new default compiler flags may have been added that result in  warnings becoming errors and so on. This was the easiest requirement to  drop as it had never worked. We used a kludgy "bootstrap" module to  work around this which provided a set of packages copied from  non-Modular Fedora to be able to build the platform. In its place, **we  will now build modules against the standard Fedora buildroot**.    Since our buildroot now has access to everything that has been built  for Fedora, we reconsidered another of our original goals that of  providing a module to be the base platform upon which all of the other  modules would depend. Originally, this was intended to define the  provided API of the Base and indicate which portions of it should be  considered stable. However, Fedora already has a stable updates policy  for how things must act within a release. This policy has only a few  outliers (Kernel, KDE, etc.), but for the most part, once we release  the Beta, the platform API is stable for the life of the Fedora  release. By relaxing this requirement, we realized that we could  eliminate (or at least postpone) the requirement for having a defined  platform module.    What we decided instead is to treat Fedora's "Everything" repository  (essentially, the complete set of software available within a Fedora  release) as the "platform module", though the tooling will not report  this content as a module. In practical terms, this means that  **creators of modules will no longer need to go through the very  painful process of tracking down which modules provide a dependency  that they need**. Instead, they will be able to depend on the system  version available in the Everything repo. If that version does not meet  their needs, they will have several options described later.    This provides us with several advantages, including **a straightforward  upgrade path from a traditional deployment**, because we will retain  the traditional repositories as well as a set of default modules. This  means it will be possible to upgrade from a current Fedora 27 system to  a modular Fedora 28 system without any special steps. In fact, this  approach will also mean that modularity need not be limited to the  Server Edition.    Another practical advantage to this change is that **module-creation  will become _significantly_ simpler**. Instead of a complex  collection of a package and all of its dependencies, modules will now  only need to describe the parts that differ from the base repository.  For example, Fedora 28 will ship with the Node.js 8.x LTS release in  the standard repository, and a module could be built to provide the 9.x  experimental release as an option. We could also easily provide the  older 6.x LTS release to support older applications. In these cases, we  can ship very simple module definitions which just lists the dist-git  branches matching the desired upstream releases.    In fact, this will now be so simple that **we plan to provide tools to  automate the creation of these modules**. Since most modules will now  only require a single source package be in the components list, we plan  to enable support for automatically building a single branch in  dist-git for all active Fedora (and EPEL) releases. Even for more  complex multi-package modules, the automatically-created module  definitions provide an easy and obvious starting point.      What Will It Look Like?  - -----------------------    From an end-user's perspective, **Fedora will ship with two sets of  repositories.** One will be the traditional Fedora repositories  (fedora, updates, and updates-testing) and the other will be a new set  of repositories providing alternative and supplementary modules. We  haven't decided on a final name for these yet, so we will use the  placeholder terms modular, modular-updates, and  modular-updates-testing.    With this design, **anyon who does not wish to access the additional  versions of software provided by modules can disable the modular and  modular-updates repositories and their system will function exactly as  it does today**. Packages built with Fedora's traditional process will  be installed and managed from the regular fedora repository, as will  default versions of packages which use the new process behind the  scenes.    **For anyone who wants access to additional versions of packages, these  new module repositories will make them available**. Users will be able  to interact with these new repositories by taking advantage of some new  syntax in DNF, the same as was used in the Modular Server Beta. If a  user wants to install a particular module stream, they can do `dnf  install module[:stream[/profile]]` (e.g. `dnf install @nodejs:6`, which  will install the default profile containing the nodejs and npm  packages). If the user just wants to install whatever version is the  default for this system, a `dnf install package` will continue to work  as it always has.      Conclusion  - ----------    This refined plan offers an understandable, approachable, and  deliverable future for Modularity. Packagers who don't want to produce  modules will be able to continue packaging exactly as they always have  with **no modification** to their workflows. Those who want to provide  alternative versions of software in a single release or to easily  provide the same version across multiple releases will have new tools  to simplify this.    As the number of available modules grows, users of Fedora will have a  much easier access to the exact version of software they want to  accomplish their tasks. People doing rapid-prototyping can more easily  access newer versions of packages and at the same time people running  older applications can continue to access the older streams that they  need.    -----BEGIN PGP SIGNATURE-----  Version: Mailvelope v2.1.0  Comment: https://www.mailvelope.com    wkYEAREIABAFAlo5f/YJEHolVWI2uqOjAAD5RgCfeRL8DcETF34kmCOT5pXJ  yzL0LLYAmwVmmd2DPHf1gzKZOjUfxiHlbuML  =ZnhF  -----END PGP SIGNATURE-----