Content Package Type Validation Forces Rethink of Standard Project Structure for Adobe Experience Manager

June 17, 2021
Principal Architect

With the launch of Adobe Experience Manager (AEM) as a Cloud Service, Adobe has redefined the way content, code, and configurations are stored in AEM. The changes create a more explicitly structured project with strict enforcement, which translates to how AEM projects are set up and deployed. Let’s dive into the before and after pictures of that change, as well as providing reasoning for the changes in order to effectively deploy AEM as a Cloud Service.

Prehistoric FileVault

For about a decade, 2009 to the beginning of 2019, we built and deployed CQ5/AEM content packages like this, where question marks, dashed lines, and cloudy boundaries indicate areas of ambiguity, non-determinism, and opportunities to become more proficient with the Felix Admin Console:

Diagram showing how CQ5/AEM content packages were built and deployed from 2009-2019

 

In the above diagram, we build two top-level content packages, herein labeled mixed, which is the new package type reserved for what is now the legacy approach to content package composition.

P1 could represent a base, or common content package created by a project team, and P2 might represent a content package that contains site- or brand-specific overlays and initial content. P1 also contains a subpackage, SUB1, which might represent the acs-aem-commons-content package of old, containing the project's OSGi bundles, UI templates and components, and other content dependencies.

The cloudy blobs labeled NODES LOL1, NODES LOLSUB1, and NODES LOL2 represent the context-agnostic DocView XML files included in each content package, masked by a hand-written filter.xml that might overwrite any repository path, including authorable content, OSGi bundles, and configurations, user nodes, group nodes, index definitions.

To better visualize the likely JCR layout of the abstract resources in the diagram, here's what might be contained if P1 was a package named myapp-content.zip:

example of what might be contained if P1 was a package named myapp-content.zip

 

Because the role and nature of the content package artifact were undifferentiated, the packaging conventions followed by the majority of AEM project teams evolved to favor artifact simplicity, doubling down on using a single deployable content package artifact, whenever possible, to encode all direct JCR repository changes to all environments, and which indirectly affected all OSGi classpath and configuration changes in all environments, which required a heavy investment in runtime smoke testing in multiple shared pre-production environments.

And because we have traditionally included the core OSGi bundle and configurations inside of the same package containing the Sling scripts that depend on the service APIs delivered by the bundle, there has always been a brief (in the best case) transition period during package installation after the Sling scripts have been imported (and therefore resolvable by the Servlet Resolver) but before the bundle had been activated. Where the custom scripts extended or overrode platform features, those platform features would be broken in this interim state.

In an ideal world, those Sling scripts should only be resolvable when their runtime classpath dependencies had been satisfied, but the FileVault format and packaging conventions did not allow building artifacts with that kind of deterministic relationship between OSGi bundles and Sling script deployments.

Throughout this bygone era, we often had lackluster and ad-hoc options for validation of content packages, but build and deployment pipelines for on-premise customers were free to evolve together with the AEM environments they were tightly coupled to, eventually converging on a point of mutual stability with or without package validation tooling, at the cost of less frequent and more painful product upgrades and migrations.

Arrival of the Composite Node Store, Package Types, and FileVault Validation

The dawn of the modern era, heralding the coming of "AEM as a Cloud Service," was marked when Adobe communicated a new hard requirement for a package to contain either all immutable content (/apps/…) or all mutable content (not /apps/…). This was a hint to how AEM Cloud Service would isolate code from content in order to allow JCR to achieve cloud-scale, using a CompositeNodeStore to mount /apps and /libs as a read-only partition at runtime. We made a big push in projects like acs-aem-commons to implement that split, so that a ui.apps package would only contain everything under /apps/myapp, including components, templates, OSGi bundles, and configurations. Everything else would go into a ui.content package.

The new age was upon us, but headaches would still remain to get upstream projects to play nicely with bleeding-edge conventions implemented by Adobe Cloud Manager Pipelines. To help get us across the finish line, a significant amount of work has been invested in the tooling for building FileVault content packages more reliably and uniformly.

Recently, with the release of FileVault Package Maven Plugin Validation (>= 1.1.0) we are presented with three new package types, with explicit rules about what each can contain:

  • Container: include sub-packages/embedded packages, OSGi bundles, and OSGi configs.
  • Application: any immutable content (/apps), but not including sub-packages/embedded packages, OSGi bundles, or OSGi configs.
  • Content: Everything else.

One detail of this package type scheme was largely unexpected, and AEM old-timers hoping to just go with the flow will probably find it to be the most troublesome aspect of this transition because it kind of changes everything.

Your OSGi bundles and configurations do not belong in your ui.apps package alongside your templates and component HTL files.

How This Changes Everything

I don't mean it in the "This. changes. EVERYTHING." sense. I mean it in the "this will likely affect all of your day-to-day AEM development activities in some small way," sense. And if you don't grasp the importance of the change upfront, you will likely end up spending a lot of time subconsciously fighting against it in little ways throughout your development workflow.

  • You will likely need to refactor your maven module structure a bit. Look to recent versions of the AEM project archetype (>=24) for examples for all , ui.structure, and ui.config packages.
  • If you contribute to a third-party project like acs-aem-commons or integrate it into your own project, you should expect to have a bunch of new content-package modules/dependencies to be aware of that will exist to target different levels of platform support for these newer conventions between AEM Classic and AEM as a Cloud Service.
  • Your own project code JCR paths will likely be distributed across at least two or more content-package modules.
  • You will likely need to relearn your go-to mvn install commands to remain efficient in local development.

When you use the new package type model in your maven build with filevault-package-maven-plugin validation enabled, you can ensure that your artifacts conform to a directed, acyclic dependency graph, which will allow Cloud Manager and other CI/CD pipeline tools to enforce more opinionated and specific validation rules to help your development team improve and maintain code quality over time, and to help protect authored content from being corrupted.

If we revisit the abstract package structure diagram presented before, we can see an encouraging concreteness to the relationships between packages and bundles.

after diagram of how CQ5/AEM content packages are now built and deployed

 

In this diagram, we have two top-level container packages. We can think of C1 as the core, or base deployable package. C2 might be another deployable package that contains site- or brand-specific content and trivial component overrides. Within C1, we have an embedded container package EC1, which might represent a third-party artifact containing our main project dependencies.

Within each container package are several Sling installable resources:

C1:

  • B1 (bundle)
  • B2 (bundle)
  • EC1 (container package)
  • A1 (application package)
  • A2 (application package)

EC1:

  • EB1 (bundle)
  • EA1 (application package)

C2:

  • A3 (application package)
  • N1 (content package)

Before diving into the inner elements of the container packages, let's consider the relationships between them and what an install sequence might look like:

  1. C1 contains EC1, and so C1 must be installed before EC1 can be installed.
  2. After C1 is installed, EC1 will not have any explicit dependencies to satisfy, so it can be installed immediately after installing C1.
  3. The embedded package, EC1, is only registered for Sling installation upon completion of C1 installation. It is not necessarily chosen to be installed immediately after C1, so if C2 had also already been registered by this point, but not yet installed, Sling would be free to choose either EC1 or C2 as the next installable resource, or it might choose to look at any of the other installable objects extracted by C1, like bundle B1 or B2, or application package A1 or A2.
  4. Thus, the container packages would be installed in sequence C1, C2, EC1 or C1, EC1, C2 or C2, C1, EC1.

Again, if we were to translate this abstraction into a practical example similar to the one presented earlier for myapp-content.zip, it might look like this myapp.all.zip package after an initial refactoring:

myapp.all.zip package after an initial refactoring

 

Of course, it's difficult to convince an experienced AEM architect to proceed with a large refactoring without providing a set of assertions and calling them "Best Practices." So here we go.

The New Best Practices for AEM Content Packages

Best Practice One

Use explicit <dependencies> in your filevault-package-maven-plugin configurations throughout your project, where appropriate.

Keep the following in mind:

A container package is not allowed to list any package dependencies. External/third-party container packages should always be embedded in one of your own container packages.

An application package should only explicitly depend on other application packages (and OSGi bundles as maven dependencies). These can be external/third-party dependencies.

A content package should only be explicitly dependent on application or content packages. These can be external/third-party dependencies.

Best Practice Two

The workspace filter of a container package should never overlap with the workspace filter of an application package.

This might seem like an obvious thing to state, but definitely don't allow it.

Application packages should not have explicit dependencies on container packages, so a package of one type should not be relying on a package of the other type to establish any ancestor paths in order to satisfy validation by the filevault-package-maven-plugin.

For organizations with multiple AEM development teams, a JCR path convention delineating container-owned roots versus application-owned roots should be established up-front and be rigidly enforced for any AEM environment where multiple CI pipelines deploy from different branches. For example, container packages might only be allowed to import paths matching a regular expression, ^/apps/[^/]+-packages/.*, and application packages would NOT be allowed to import paths matching that same expression.

Best Practice Three

Make sure your filevault:analyze-classes goal is actually collecting the imports of HTL scripts in your application packages.

This requires that the htl-maven-plugin be configured to generate Java sources from your src/main/content/jcr_root directory.

This enables the Cloud Manager pipeline and OSGi installer to check for missing java dependencies before your customers get a chance to.

Best Practice Four

Don't design your OSGi resources to be dependent on JCR resources that cannot be defined purely by other OSGi resources.

For example, don't create an OSGi Service that tries to read a specific repository path on service activation that is only defined by a DocView XML file in an application type package.

Instead, establish the path in code if it doesn't exist or use a RepositoryInitializer factory config ("repoinit script") to create the path.

Best Practice Five

Only container type packages should contain resources activated under different run mode combinations.

In other words, an application or content type package should only install content that is:

  • Appropriate for ALL run mode combinations (behaving exactly the same on author or publish, dev or prod).
  • Or, it should install content that is Appropriate for ANY SINGLE run mode combination. In this case, it should only be embedded in a container package at an /apps/*/install(.*) path that reflects that run mode combination.
  • More narrowly scoped by run mode than all of its dependencies, and more specifically, all of its dependencies must be safely installable under the same set of run modes.

An application or content package that is only scoped to the prod run mode should not depend on a package that is only scoped to the author run mode, because the dependency could not be safely installed on prod.publish. On the other hand, a prod.author-scoped package could be made dependent on an author-scoped package.

Best Practice Six

Don't use content packages to initialize anything your code might depend on.

Use repoinit instead of content packages whenever possible. Repoinit operations are never reverted and are designed to be declarative and idempotent.

Content packages will be installed after every other type of OSGi bundle, repoinit configuration, and application package has been installed, assuming you followed the rules above regarding explicit package dependencies.

Content package repository scope should only include paths that are safe to revert upon uninstallation of the package. For example, content package filters should not include mutable paths that OSGi services might be dependent on, such as roots under /home or /var. Instead, repoinit scripts deployed in container packages should be responsible for creating such resources, because RepositoryInitialization OSGi configurations will not be reverted if they are uninstalled.

Another example, don't create a system user in repoinit, and then only define a critical property on that user’s profile node in a content package. It will appear to work on installation, but if someone or some event triggers an un-installation of the package in the future, you might break the associated service functionality in unexpected ways if a snapshot package is installed.

Content package install permissions in the Cloud Service publish tier are explicitly defined for the sling-distribution-importer service user. If it doesn't have permission to read/write to the paths the content package is installing to, it can block the replication queues.

Content packages should be treated with deeper scrutiny by acceptance testing and code quality checks when built and deployed by a CI pipeline, because their installation may have a destructive effect on resources that on one hand are not easily identified by a simple application startup or post-installation integration test, and on the other hand will not be automatically corrected with a bugfix deployment.

Best Practice Seven

Cloud Manager Targets other than none should only be set on container packages.

In other words, Cloud Manager deployment pipelines should never handle application or content packages directly. Cloud Manager should only deploy container packages.

Any use of pipeline variables or maven profiles to customize build artifacts should be limited in effect to content-package modules of the container type.

Application and content package artifacts and bundle artifacts should always be assembled the same way in any execution environment.

Best Practice Eight

Use container packages to deliver all other application artifacts to your AEM servers.

Container packages should be nothing more than a transparent wrapper around a group of common-source OSGi-installable JCR resources that defines a mapping for each embedded resource to an immutable repository path.

Container packages should be the ONLY package type in a deployment to extract JCR nodes that might affect the state of the OSGi framework.

The act of installing/uninstalling a container package should have exactly the same effect as requesting installation/uninstallation of the OSGi resources that it contains.

Some wisdom for local development: If you want to most closely mimic Cloud Manager deployments, container artifacts should be the only thing you ever upload to AEM using the content-package-maven-plugin, and embedded artifacts (bundles and application/content packages) should be uploaded to a stable JCR path without using the Felix Web Console or CRX Package Manager servlets.

Practicality will make it more advantageous in almost all cases to use the content-package-maven-plugin for embedded packages, because the JCR Installer does not provide direct, synchronous feedback to the maven build.

In the small fraction of maven projects where an embedded package target path is subject to any conditions like run mode specificity (like a package that should only be installed in publish or only in prod) or if it might be shadowed by a higher priority path according to the JCR Installer, individual deployment of the artifact using the content-package-maven-plugin should be avoided, or should rely on a WebDAV(MKCOL + PUT) or SlingPostServlet deployment method, like what the sling-maven-plugin supports for OSGi bundle install.