To our dear friend BitBucket, whom we have reliably heard through the office grapevine will be finally retiring near the end of October. We wish her well on her journeys.
So long and thanks for all the commits
[Please have all your Jenkins jobs and other deployment configurations reviewed and updated as soon as possible]
If you're here because you're having issues with your pipelines failing see our blog post
If anyone is still experiencing issues with failing ECS cluster upgrade jobs, please ensure that the corresponding Jenkinsfile (you can view the job configuration for the failing job to determine the appropriate repository and Jenkinsfile) is set to use the 1.0.0 or 2.0.0 version of the devops library to avoid the "unable to cleanup in PREPROD or PROD" issue that may be occurring on other branches.
devops@<1.0.0|2.0.0>
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Friday, September 19th 2025 14:00 until Tuesday, September 23rd 2025 03:59 UTC. See what time the maintenance will occur in your region
We have identified a minor issue with container instance caching and are testing the fix in preproduction now. If anyone notices an issue with scaling, please report through the standard channels
As you may have noticed, we are experiencing significant loss of functionality at times in both AWS accounts due to an increase in the number of list tags for resource calls being made to the ECS API. We traced this back to the 1.96.0 release of the ECS agent which is deployed to all of our EC2 based ECS compute nodes. AWS in turn traced this to the Datadog agents running on most of those nodes. Datadog is calling the agent to describe the tasks that are running in each node to marry metadata to metrics being pulled from the Docker daemon and other places.
This is crashing head on into a function deep in Deployoryah's ECS library where service location is wrapped in the ability to fallback to tag based discovery, as a leftover from the days when services could only be tombstoned and not deleted and configuration mistakes resulted in -vN+1 suffixes on service names. Tags were used to assign "authoritative" (e.g content-aca-solr-preprod or apm-preprod-2022.1) names and use those instead of having to track which version (e.g. content-aca-solr-preprod-v3 or apm-preprod-2022.1-v4).
While we have made many efforts to reduce the load on this contested API through caching for use in Apirah which shares the library with Deployoryah, the same strategy doesn't work for Deployoryah as it runs from a cold start for each call. Even with caching in place, Apirah's performance degradation within throttling storms is significant enough that we're retiring its use as an ECS management interface and are working with the users of that function on temporary workarounds and long term replacements that bypass the old library entirely, allowing us to skip the issue entirely.
For users of Peregrine who are struggling in the validate stage because of this issue, one workaround that can be employed is to allow validate to run and if it is taking long or fails unexpectedly, to then manually confirm successful completion of the deployment in ECS followed by a rebuild of the pipeline using the START_FROM_STAGE parameter to skip ahead to publish to ensure the necessary state data is published for the deployment. For the adventurous or those that don't care to manually gate and rerun, you can just comment out the validate stage from the YAML that is configuring the deployment. The archetype only runs stages it finds a concrete configuration for.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Wednesday, September 17th 2025 16:00 until Thursday, September 18th 2025 16:00 UTC. See what time the maintenance will occur in your region
We are rolling out an eFix to migrate the ECS Cluster deployment archetype onto nimbus controlled demolition as its cleanup mechanism in the release stage. This replaces the nimbus cleanup command which is now prohibited in preprod and prod pipelines. This has caused some upgrade pipelines to fail, leaving the clusters in an unmanaged state.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, September 9th 2025 10:00 until Tuesday, September 9th 2025 12:00 UTC. See what time the maintenance will occur in your region
Test message
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Monday, September 8th 2025 18:00 until Wednesday, September 10th 2025 23:59 UTC. See what time the maintenance will occur in your region
The peregrine-next fix version encompasses 8 critical DVOPS tickets addressing core infrastructure stability and deployment reliability. The primary focus areas include Ruby compatibility modernization, pipeline resilience improvements, and infrastructure automation enhancements. Key resolved issues include fixing the Deployoryah::Aws::Deployment class loading problem that was blocking deployments, resolving CloudFormation template validation failures, and addressing deprecated Ruby API usage that threatened long-term maintainability.
The release delivers significant improvements to deployment pipeline reliability, with fixes for ECR enhanced scanning compatibility, Jenkins build error resolution, and ECS deployment rollback detection. Infrastructure automation capabilities have been enhanced through nimbus controlled demolition improvements and Cloudflare command stabilization. All critical blockers have been resolved, with the remaining tickets in Ready for Verification status representing final validation steps rather than development work, ensuring a stable foundation for production deployments.
Change Details: CHG-7840
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, September 2nd 2025 13:30 until Tuesday, September 2nd 2025 15:30 UTC. See what time the maintenance will occur in your region
We will be performing the following:
Peregrine 2025.09.1 Release Release Details {}Release Name{}: Peregrine 2025.09.1 {}Scheduled Date{}: September 2, 2025 at 9:00 AM {}Fix Version{}: peregrine-next Included Tickets Summary apirah Performance & Stability Fixes {}DVOPS-23152{} - apirah error on getting ECS services with tags {}DVOPS-23119{} - apirah error on service overview forbid and sanction {}DVOPS-22656{} - apirah scheduled tasks overview inconsistent docker image Deployoryah Safety & Automation Enhancements {}DVOPS-23171{} - The Deployoryah::Aws::Deployment class is not defined or not properly loaded in the deployoryah gem version 2025.08.24 {}DVOPS-23149{} - Deployoryah: Implement validation of prod stacks before deletion {}DVOPS-23148{} - Deployoryah: Modifications to prevent unintended automatic prod stack 'cleanups' Release Impact This release focuses on critical stability improvements for apirah service performance and enhanced safety mechanisms for deployoryah automated cleanup processes. All tickets have been verified and are ready for production deployment as well as general performance and security fixes in the underlying OS images
Verification Status All included tickets are in 'Ready for Verification' status and tagged with peregrine-next fix-version.
New Blog Post
The most recent versions of the AWS SDK gems released to https://rubygems.org have dropped support for Ruby 2.5 which is bundled in Chef 14. If you're still using Chef 14, please get in touch for help on upgrading, but in the meantime there's a new version of pq_aws released to GHE. This version pins the necessary SDK gems at their last 2.5 compatible releases.
cookbook "pq_aws",
git: "https://${GITHUB_USER}:${GITHUB_TOKEN}@github.com/clarivate-prod/pq-common-chef.git",
rel: "pq_aws",
branch: "chef-14"
For those of you that joined our ProDevDay session, we mentioned that we were aiming to publish more content about what we're doing with Peregrine, so here's a start:
Evolving - An Agentic Engineering Blog describes some of what we're working on as an evolution of both the platform and how we develop it
As part of our ongoing infrastructure scheduling improvements, we are updating the execution frequency for maintenance and deployment jobs related to sidecar images—including Filebeat, LogCleaner, and the Datadog Agent—across all environments:
These changes are designed to align better with environment usage patterns and enhance the stability and efficiency of our deployment pipelines.
The 2.0.0 version of the pipeline library now supports GHE credentials for resolving Berksfile dependencies and git operations. If you're using 2.0.0-beta-1 or 2.0.0-rc-1, update your Jenkinsfile to use the v2 release branch (2.0.0).
If you're still using v1 AMI/EC2 pipelines, this would be a good time to upgrade to v2.
Please review the updated guide here
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, April 29th 2025 15:00 until Tuesday, April 29th 2025 17:00 UTC. See what time the maintenance will occur in your region
We will be performing an eFix for Peregrine today including:
In order to implement the resource property injector, we need to be able to parse the templates and manipulate the data. This does mean that our standards may be a bit stricter than Cloudformation's when it comes to syntax, especially when the !Tag form of a function is used. As the YAML processors strips those tags when it loads the file, we internally convert them to their Fn::Tag form before submitting the template and that normally works fine.
There may be some cases though, where the input syntax may be valid for Cloudformation but not necessarily for our YAML processor as the following:
SecurityGroupIds: [Fn::ImportValue: !If [IsProd, books-books-infrastructure-security-prod-main-securitygroup:BookServiceSecurityGroup, books-books-infrastructure-security-preprod-main-securitygroup:BookServiceSecurityGroup], Fn::ImportValue: !If [IsProd, books-books-infrastructure-security-prod-main-securitygroup:SGBase, books-books-infrastructure-security-preprod-main-securitygroup:SGBase]]
That line will result in an error about an unexpected : in a scaler value. The solution is unfortunately to convert it to work entirely in the Fn::Tag form:
SecurityGroupIds:
- Fn::ImportValue:
Fn::If:
- IsProd
- books-books-infrastructure-security-prod-main-securitygroup:BookServiceSecurityGroup
- books-books-infrastructure-security-preprod-main-securitygroup:BookServiceSecurityGroup
- Fn::ImportValue:
Fn::If:
- IsProd
- books-books-infrastructure-security-prod-main-securitygroup:SGBase
- books-books-infrastructure-security-preprod-main-securitygroup:SGBase
This issue is being caused by the ":" in the middle of the stack output reference required by !ImportValue. It can also be worked around by forcing YAML to treat it as a proper string by enclosing the entire string(s) in quotes:
SecurityGroupIds: [Fn::ImportValue: !If [IsProd, "books-books-infrastructure-security-prod-main-securitygroup:BookServiceSecurityGroup", "books-books-infrastructure-security-preprod-main-securitygroup:BookServiceSecurityGroup"], Fn::ImportValue: !If [IsProd, "books-books-infrastructure-security-prod-main-securitygroup:SGBase", "books-books-infrastructure-security-preprod-main-securitygroup:SGBase"]]
You may also find an error similar to:
E, [2025-04-16T17:58:46.022040 #42045] ERROR -- Commander::Runner: Tried to load unspecified class: Date (Psych::DisallowedClass)
This is likely caused by a similar issue, where the template format version attribute of the template has not been enclosed in quotes and is being treated as a date. In that case, a similar correction is required:
-AWSTemplateFormatVersion: 2010-09-09
+AWSTemplateFormatVersion: "2010-09-09"
One more fun one...
If you're passing a list of values in your YAML, such as a list of valid response codes to a health check, they need to be quoted as well (at least if there are two values, as the , will be interpreted as a thousands separator):
Matcher:
HttpCode: 200,301
should be
Matcher:
HttpCode: "200,301"
Please contact us as necessary to assist. These should be rare though
We've found a bug in the new template resource attribute injector that's causing !GetAtt functions to be improperly preprocessed if the attribute to be located contains dots in its name. We will push out an eFix later today. For now, you can manually convert the template to use the Fn::GetAtt form if necessary.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, April 15th 2025 13:00 until Tuesday, April 15th 2025 15:00 UTC. See what time the maintenance will occur in your region
The next Peregrine release has been scheduled for 15.04.24. The release notes have been updated with the changes. The JDK 21 change called out separately here will be the biggest change and future proof us for a few more years of Jenkins releases, but should have minimal real impact on your builds as the default JDK should never be in use.
As always, contact DevOps with any issues.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, April 1st 2025 04:00 until Tuesday, April 15th 2025 04:00 UTC. See what time the maintenance will occur in your region
With the first release of new Jenkins Node AMIs in April 2025 for Peregrine (i.e a 2025.04.X release), we will be upgrading to Java 21 as the default JDK on the Jenkins nodes that are automatically provisioned to run your build executors. This will not change the JDK used to build your artifacts if you are using one of our supported pipeline branches (1.0.0, 2.0.0-beta-1 or 2.0.0-rc-1) and a standard build stage actor (e.g. pqisGradle, gradle, maven etc) as we will default a corretto-11 tool installation if one is not specified in your pipeline definition. This will allow us to emulate the current setup for any pipeline that does not currently override the JDK.
Please contact ISDevOps with any questions prior to this date.
Our friends in CloudEng have been attempting to turn on additional controls in our accounts (pqp-pqis-[dev|prod] specifically in this case) that will prevent EC2 instances from being launched if particular tagging criteria have not been met. This has been challenging because of a particular peculiarity in the way that we apply tags on our deployments and how they trickle down to EC2 instances through Launch Template derived AutoScaling Groups and to EBS Volumes attached to those instances.
To support their effort, we will be rolling out a change to nimbus in the March Peregrine release that will cause it to inject the appropriate set of tags, as inline properties in your CloudFormation templates, when the input template is deemed deficient. This will cause the deployed resources to be tagged correctly and allow the new controls to be enabled. Without this in place, instances created from this type of configuration will be blocked from launching, when otherwise properly tagged.
As this process is being introduced in a location in Peregrine that is executed by nearly every pipeline that has been setup, we're bound to uncover edge cases that we've not considered as we roll it out. To that end, if you do have any pipelines currently running on devl AMIs, they will be executing this new code. Please reach out directly to @graeme if anything looks suspicious in a nimbus create_stack_v2 call during a deploy stage, as it's something I'll likely need to look into . There is some translation required as we have to parse the template, inject our own configuration and then try to write something back out that maintains any intrinsic functions etc that might be swallowed during the import process, so it's a bit... tricky
If you happen to attempt to run any of our deployment tools locally (the deployoryah suite, cask, wooster etc), you may notice they don't work if you have migrated to identity center. Don't worry, this is by design as identity center didn't exist when we designed them ;)
We need to upgrade some SDK components in all of the tools to support the new credentials provider, this will be undertaken in the next couple of weeks as we've just been able to test this ourselves. In the meantime, you can get a temporary ~/.aws/credentials file through the web console login page that can be used in an emergency
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Wednesday, January 1st 2025 05:00 until Saturday, February 1st 2025 04:59 UTC. See what time the maintenance will occur in your region
This is just a quick notice to inform that the next release of Peregrine's Jenkins AMIs will begin the transition from rtx to mise as the version manager of choice for our Jenkins nodes. This just represents the renaming of a tool because the original (rtx) has become slightly overloaded in today's GPU rich environment. The developer of the tool transitioned a year or so ago, we've remained on the old version as it performed all of its required tasks admirably, but now the developer has let the old rtx named domain that once hosted the installation mechanism so it has now forced our hand.
As as user of Peregrine, you will not be expected to make any configuration changes to support this transition, this is purely informational in case things don't go as smoothly as planned. Step one began to be rolled out today, you'll notice the supported versions of our pipeline libraries (1.0.0, 2.0.0-beta-1 and 2.0.0-rc-1) will now attempt to use both mise and rtx when previously only rtx would be attempted. This is to support the transition as the newly built devl AMIs have the new tool and the current prod AMIs have the old. The next step will be to promote the new AMIs, after which we will swap the priority and attempt mise first, and then fall back to rtx (in case any one has old AMIs pinned in their cloud configs). Finally, we will remove all references to rtx in the library.
If you are knowingly using an unsupported branch, please be in touch and we'll extend transitional support to any other branches we're made aware of. If we miss something, you will encounter job failures, but the fix is relatively simple so a ticket in the DVOPS Jira queue should be actioned quickly, but will still be required post-release.
The timing of this is TBD, but will be completed by the end of this month.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Friday, November 22nd 2024 19:00 until Friday, November 22nd 2024 19:30 UTC. See what time the maintenance will occur in your region
Per DVOPS-22108, people have been experiencing delays at times with tasks launching promptly when submitted through Firkin's CloudWatch Task Initiator module.
We believe we have traced the cause of this to a function added when testing the impact of SOCI indexes on Fargate start times and this change will make this an optional feature.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, November 12th 2024 14:30 until Tuesday, November 12th 2024 15:30 UTC. See what time the maintenance will occur in your region
We will be performing the next scheduled release of our delivery platform on Tuesday, November 12th at 9:30am ET.
Please view the changelog for more details and contact ISDevOps if you have any questions or concerns
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Friday, November 1st 2024 04:00 until Saturday, November 30th 2024 16:59 UTC. See what time the maintenance will occur in your region
We have had reports that Jenkins updates are now installing a version that has deprecated support for Java 11 as shown on the table on this page - https://www.jenkins.io/doc/book/platform-information/support-policy-java/. This only applies to the JVM used to run the Jenkins agent process on the remote nodes, not to the JDK that's used to build your software, so Java 11 support isn't being dropped entirely, just as the default JVM on the build nodes.
We have been preparing for this for a while, so we have a replacement set of AMIs prepared under the beta environment tag that can be used. So far, we have not experienced any issues swapping these in for their Java 11 equivalents and do not expect anything to come up other than a small piece of housekeeping. In order to build applications using Java 11, you will need to use one of the supported methods to install the JDK into your workspace. If you are using one of our pipeline archetypes, there's likely already a javaVersion argument configured in your build stage. If you have been relying on Java 11 being the default JVM on the build nodes and have no argument value configured, we will be making a small change to target the "corretto-11" named tool installer that should be present on everyone's instances. As long as this installer has a matching architecture to the instance class assigned to your build nodes, you won't have to do anything. If any issues do crop up (i.e. you have a mix of ARM64 and x64 nodes and the default arch is incorrect), a simple change to assign the correct named tool (or use our rtx/mise based tool management processs) will fix you up.
We can provide details on how to do so if need be.
We will be forcing the matter before the end of the month, exact date TBD. This post is mainly informational in case you decide to do a quick yum update jenkins before that time.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Monday, October 21st 2024 16:00 until Friday, November 15th 2024 04:59 UTC. See what time the maintenance will occur in your region
We have taken the step to disable smoke tests as part of the ECS Service Validation actor (triggered in the validate stage of an ECS deployment pipeline).
This step uses cucumber to validate the deployment and often pulls in a library of shared test steps, based on the same code as our deployment tools. Occasionally, this pulls in a more recent version of our deployment tools code which then overlays the tools on the node that performed the test. This can result in build instability as this code can be untested and would normally live in "devl" AMIs and not participate in stable pipelines.
Until we can better isolate these tests to protect from build failures, we are going to eliminate this step as it's mostly noise and not adding much. We will revisit as part of the Peregrine stability enhancement project.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, October 1st 2024 04:00 until Monday, October 7th 2024 15:59 UTC. See what time the maintenance will occur in your region
This is a feature of the platform with little current use, but in case anyone is out there that's experiencing issues and we're unaware, an update to the soci-snapshotter tool is necessary. soci-snapshotter lets us build lazy loading images for Fargate which can decrease container launch time enough to achieve dramatic cost savings when spawning thousands of tasks on an annual basis.
If you are attempting to build soci indexes for your images and are currently experiencing an error with unknown layer or media types, please add this line to the init script for your docker node:
sudo curl https://pq-devops.s3.amazonaws.com/tools/soci/soci-0.7.0-x86_64 -o /usr/local/bin/soci && sudo chmod +x /usr/local/bin/soci
This will overlay a more recent version of soci-snapshotter into your node (replace the architecture with aarch64 for Graviton build nodes)
This change is also available in devl dockerslave AMIs if you're willing to drop a stability level for the interim and will promoted with the next release
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, September 24th 2024 12:00 until Wednesday, September 25th 2024 15:59 UTC. See what time the maintenance will occur in your region
Release notes:
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Thursday, September 12th 2024 04:00 until Saturday, November 30th 2024 16:59 UTC. See what time the maintenance will occur in your region
ISDevOps, in cooperation with CloudEng, have historically produced a set of "GIO AMIs" as the basic building blocks for EC2 based deployments. These AMIs take an upstream AWS provided AmazonLinux 2 or AmazonLinux 2023 OS image, layer on security updates and a set of common configurations, such as the SAD agents (Crowdstrike, Qualys et al), and are made available to be consumed directly for EC2 instance launches or as the input to an AMI producing pipeline.
Clarivate separately had developed its own common AMI, known as the "Clarivate Golden AMI". The original implementation was a bit to purpose specific for our use, but the CloudEng team have been hard at work to produce a more general purpose AMI and we are now ready to start integrating. We have started building a set of beta AMIs using the new Clarivate Golden AMI in place of the AWS provided OS images as the upstream to the existing pipelines. The outputs are a drop in replacement for what we were previously producing and should not require any changes downstream for compatibility.
We are aiming to cutover our AMI pipelines in November of this year as we are in a change freeze until the end of October. This change likely roll out with a change to the system tags used to identify these AMIs, more on that change to follow.
We will communicate specific details about the change as the time gets closer. This is intended as a heads up to ensure everyone is aware that the change is coming.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Wednesday, September 11th 2024 04:00 until Monday, September 30th 2024 15:59 UTC. See what time the maintenance will occur in your region
A gap exists with the !PriorityExact tag processor that can result in incorrect rule priority assignments during template rendering when configuring a shared ALB for path based routing. The "exact" flag, when provided to the rule matcher and applied to a path based route, creates a potentially overly restrictive filter causing the process to not find a previously configured rule. In some cases, this can cause pipeline failures as the reservation process may subsequently trigger a race condition during rule priority reassignment.
A change to the exact matcher has been made and is available in both the beta (JDK 21) and devl Jenkins Agent AMIs. This change means that the matcher can accept the condition in any of the following patterns:
These should cover most of our common use cases and will work with the most frequently used methods of providing the matcher to the !PriorityMacro processor. If your use case falls outside of one of these patterns, you can either open a ticket with ISDevOps to update the pattern list or update your deployment configurations to provide a more exact representation of the path in the YAML.
Contact ISDevOps if you need assistance.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Friday, August 30th 2024 12:00 until Friday, August 30th 2024 14:00 UTC. See what time the maintenance will occur in your region
Jenkins servers OS updates are required to complete mitigation of RegreSSHion critical vulnerability CVE-2024-6387 This notice applies to all Jenkins servers marked in this notice. OS reboot required to complete the updates. See change management issue for more details: https://jira.clarivate.io/browse/CHG-6945
Duration of work: about 2 hour (duration updated due to an extended update workflow)
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, July 23rd 2024 17:00 until Wednesday, July 24th 2024 04:00 UTC. See what time the maintenance will occur in your region
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, June 11th 2024 18:00 until Tuesday, June 11th 2024 19:00 UTC. See what time the maintenance will occur in your region
This is a regularly scheduled platform upgrade. Please review the Changelog for more details
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Monday, June 3rd 2024 15:40 until Monday, June 3rd 2024 16:40 UTC. See what time the maintenance will occur in your region
We will be consolidating the EC2Box database with the RDS instance used to store data for the status page(s) as a cost savings effort. The existing data has been migrated and there will be a brief connectivity blip as we roll out the new configuration
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, May 14th 2024 13:00 until Tuesday, May 14th 2024 14:00 UTC. See what time the maintenance will occur in your region
We will be performing a regular release of our delivery platform this afternoon including a refresh of all Jenkins Agent AMIs.
Please see the associated release notes here
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, May 14th 2024 04:00 until Friday, May 31st 2024 15:59 UTC. See what time the maintenance will occur in your region
Today's delivery platform release has added support for adjustable compression settings for your ECR images. To take advantage of this, you'll need to use the ecrDockerMultiArch builder for your image stage and your pipeline must be using the latest, 2.0.0-rc-1, release of our pipeline library (migration from beta-1 to rc-1 is just a change to the library declaration in your Jenkinsfile, migration from a 1.0.0 branch may required DevOps support).
We are also releasing initial support for building SOCI indexes, specifically for images that will be deployed as Fargate tasks. SOCI indexes allow the image to be lazy loaded at start time, where the files required to bootstrap the containerized process are first downloaded, with the remainder pulled as time allows.
Notes for enabling both of these options can be viewed here
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Wednesday, May 1st 2024 04:00 until Saturday, June 1st 2024 03:59 UTC. See what time the maintenance will occur in your region
The recent updates to the YAML configuration engine have necessitated a change to the priority mixin.
Now that the !Macros directly update the rendered configuration document without an intermediate ERB representation, the mixin (e.g. <%= priority... %>) form of the same function call through to the YAML tag processor. As the tag version of the !Priority mixin takes the resource label and match pattern values directly, we need to remove the secondary call to <%= value() %> when using the mixin priority form (such as when needing to use the assume role functionality exposed through the :account argument).
The old version:
HTTPSPriority: <%= priority(value('my-loadbalancer-pipeline::RESOURCES.HTTPListenerARN'), '/my-service-path*', :exact => true, :account => 'pqisprod') %>
The updated syntax:
HTTPSPriority: <%= priority('my-loadbalancer-pipeline::RESOURCES.HTTPListenerARN', '/my-service-path*', :exact => true, :account => 'pqisprod') %>
Note that if you're using the "single config" model, it now supports automatic role assumption based on the value of the PIPELINE_ENVIRONMENT variable, so the macro forms !Priority and !PriorityExact can be used in place of the mixin form. If you're on a v2 library branch (2.0.0-beta-1 or 2.0.0-rc-1) but not using single config, you can define a PIPELINE_ENVIRONMENT variable to simulate the effect.
Upcoming work to prep for a new multi-account model should eliminate the need even for that, though may introduce additional configuration elements. More to come on that topic at a later date
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, April 30th 2024 18:00 until Tuesday, April 30th 2024 19:00 UTC. See what time the maintenance will occur in your region
We will be performing a regular release of our delivery platform this afternoon including a refresh of all Jenkins Agent AMIs.
Please see the associated release notes here
The find_image mixin has been deprecated with the 30.04.2024 release and should be replaced in your configuration files.
Existing:
filebeat_image: <%= find_image('system=devops.logstash.filebeat', 'devl').image_name %>
Replace WIth:
filebeat_image: <%= firkin('system=devops.logstash.filebeat', 'devl') %>
If you have a pipeline that includes a script activity that is suddenly showing signs of missing single quotes, then you may need to read this.
The latest release of our delivery platform seems to contain a bug where certain script configurations may improperly render causing this:
s3 ls s3://bucket/app/artifacts | grep MyArtifact.zip | awk '{print $NF}') MyArtifact.zip
to execute as
s3 ls s3://bucket/app/artifacts | grep MyArtifact.zip | awk {print } MyArtifact.zip
and fail.
Instead, update the script to use two single quotes.
s3 ls s3://bucket/app/artifacts | grep MyArtifact.zip | awk ''{print $NF}'') MyArtifact.zip
Which will render back to the original form before executing.
We are looking into the cause of this problem, but this should function as a workaround until we have a solution in place. Our apologies
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Wednesday, April 17th 2024 19:00 until Wednesday, April 17th 2024 21:00 UTC. See what time the maintenance will occur in your region
This is a scheduled maintenance for https://ebooksjenkins.proquest.com/. The service will not be available for the duration of this maintenance for a back-end server upgrade to Oracle Enterprise Linux 9
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, April 16th 2024 15:00 until Tuesday, April 16th 2024 17:00 UTC. See what time the maintenance will occur in your region
This is a scheduled maintenance for https://ecmsjenkins.proquest.com/. The service will not be available for the duration of this maintenance for a back-end server upgrade to Oracle Enterprise Linux 9.
We'll be performing brief maintenance from Tuesday, April 16th 2024 09:00 until Tuesday, April 16th 2024 10:00 EST. See what time the maintenance will occur in your region
We will be performing a release of some products of our Delivery Platform to add extra needed features. The following products will be impacted: deployoryah apirah firkin
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, April 9th 2024 04:00 until Saturday, April 13th 2024 03:59 UTC. See what time the maintenance will occur in your region
In order to simplify the configuration of multi-phase releases, we are now mandating the use of the Hidden Parameter plugin as it will be used to automatically create the job parameters necessary for release integration, without polluting the parameters submission form.
We have reviewed the configuration of the current fleet of Jenkins instances and added the plugin where we found it missing, for instances that we know are using our delivery platform. If you are using the 2.0.0-rc-1 branch of the DevOps pipeline library and start experiencing issues submitting pipeline jobs, please ensure this plugin has been installed on your instance before contacting DevOps
This is a scheduled maintenance for https://tdmjenkins.proquest.com/. The service will not be available for the duration of this maintenance for a back-end server upgrade to Oracle Enterprise Linux 9.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Friday, April 5th 2024 18:00 until Friday, April 5th 2024 20:00 UTC. See what time the maintenance will occur in your region
This is a scheduled maintenance for https://pgsjenkins.proquest.com/. The service will not be available for the duration of this maintenance for a back-end server upgrade to Oracle Enterprise Linux 9.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Wednesday, April 3rd 2024 19:00 until Wednesday, April 3rd 2024 21:00 UTC. See what time the maintenance will occur in your region
This is a scheduled maintenance for https://optimusjenkins.proquest.com/. The service will not be available for the duration of this maintenance for a back-end server upgrade to Oracle Enterprise Linux 9.
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing brief maintenance from Tuesday, April 2nd 2024 23:00 until Wednesday, April 3rd 2024 01:00 UTC. See what time the maintenance will occur in your region
We will be performing an eFix of our delivery pipeline this evening, necessitated by an error in the !ComponentResource tag processor.
A few additional enhancements and bug fixes are rolled into this release, the notes can be found here
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, April 2nd 2024 18:00 until Tuesday, April 2nd 2024 19:00 UTC. See what time the maintenance will occur in your region
This is a scheduled maintenance for https://dgjenkins.proquest.com/. The service will not be available for the duration of this maintenance for a back-end server update for Oracle Enterprise Linux 9
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Tuesday, April 2nd 2024 18:00 until Tuesday, April 2nd 2024 19:00 UTC. See what time the maintenance will occur in your region
This is a scheduled maintenance for https://agaijenkins.proquest.com/. The service will not be available for the duration of this maintenance for a back-end server update for Oracle Enterprise Linux 9
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Thursday, March 28th 2024 19:00 until Thursday, March 28th 2024 21:00 UTC. See what time the maintenance will occur in your region
This is an OS upgrade process for pqqajenkins.proquest.com. Intermittent availability to complete outage. Please do not attempt to use this service during this maintenance.
The change to eliminate the token processor in Deployoryah's templating engine has uncovered a few areas where deprecated syntax is still in use in some pipeline configuration YAML files.
If you encounter an error messages such as error: undefined method '-' for true:TrueClass. Use --trace to view backtrace, this means that you're using an old, now deprecated, method of injecting pipeline variables into your configuration.
For example:
ALBDNSName: <%= !pgs-etdadmin-loadbalancer-devl-team-pipeline::RESOURCES.ALBDNSName %>
With the token based parser, this statement would lookup the ALBDNSName resource from the named etaadmin load balancer pipeline and to set the value for the key ALBDNSName. With the change to use a custom YAML tag preprocessor this now is treated as a Ruby statement which results in a attempt to negate the subtraction of the variables pgs, etdadmin, loadbalancer, devl, team and the constant pipeline::RESOURCES.ALBDNSName. As these variables derive to the value false, the above error is thrown.
Instead, this should be represented using the !Value macro (or it's corresponding <%= value() %> function such as in the following example:
ALBDNSName: !Value pgs-etdadmin-loadbalancer-devl-team-pipeline::RESOURCES.ALBDNSName
or
ALBDNSName: <%= value('pgs-etdadmin-loadbalancer-devl-team-pipeline::RESOURCES.ALBDNSName') %>
Another change is that the double "moustache" syntax no longer works to denote placeholder values. Previously, \{\{ a_parameter \}\} would compile down to <%= a_parameter %> in the preprocessor, which would cause the templating engine to insert the value found in the pipeline state for A_PARAMETER. Which the token based preprocessor deprecated, this is no longer supported and these must be changed to use the post-processed form.
The 26.03.24 Delivery Platform release is underway. Today's release involves the promotion of a new set of Jenkins Agent AMIs which bring bug fixes and new functionality to our platform tools. We also have an update to the 2.0.0-rc-1 branch of our Jenkins pipeline library that introduces new error reporting functionality to more clearly indicate the cause of pipeline failures.
Full release notes can be found here
For those of you who are receiving build notifications for your ECS deployments that contain Image Scan Results, we have added a link to the header for each image in the email that links to ECR's page for the image. This will allow you to view the details for any Medium or Low severity vulnerabilities that have been discovered for the image. For brevity's sake, these are not currently included in the body of the notification as most images have a rather long list of vulnerabilities at these severity levels.
This link can be found with the title Full Results beside the image digest. Note that you may need a previously authenticated session with AWS for the link to work properly as otherwise you may be redirected to the standard AWS login page which is non-functional with our SAML based auth setup.
Note that this only applies to the 2.0.0-rc-1 branch of the pipeline library. If you have not done so yet, please update your pipelines to this version at your earliest convenience.
A regular Jenkins AMI release is scheduled for 05.03.24 at 8am EST. The release notes contain the details for what's included in this release. Additionally, the normal slate of security updates will be included.
Please contact PQ DevOps if you experience any difficulties after the release
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Monday, February 26th 2024 13:00 until Monday, February 26th 2024 15:00 UTC. See what time the maintenance will occur in your region
Performing maintenance for booksjenkins101 and booksjenkins201
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Thursday, February 22nd 2024 17:00 until Thursday, February 22nd 2024 22:00 UTC. See what time the maintenance will occur in your region
We are upgrading the OS to Oracle Enterprise Linux (OEL) 9.
An upcoming release for Firkin has been scheduled for 21/02/2024. This release contains a minor bugfix for the target tracking variant of the ECS Autoscaling Monitor.
In the current version, if the networking infrastructure for an existing service is updated so that the service now belongs to a new target group, if that same service is configured to track the request rate metric for its target group, scaling for that service would become disabled and would need manual intervention to resume once the networking change completed. In the new version, firkin will automatically detect the updated configuration and re-enable scaling automatically once the new target group is in place.
This change will be promoted to preprod on 20/02/2024 and production on 21/02/2024 assuming no new issues are encountered
At ProQuest, we're always working to enhance the functionality and security of our products.
We'll be performing maintenance from Friday, February 16th 2024 16:00 until Friday, February 16th 2024 19:00 UTC. See what time the maintenance will occur in your region
Starting around 11:00 EST today, devopsjenkins-pre.proquest.com will be unavailable. This will impact some automation in our delivery platform (scheduled Jenkins jobs, some automated cleanups etc), but will not impact the delivery of any applications that takes place on development team specific Jenkins instances.
At ProQuest, we're always working to enhance the functionality and security of our products. Please note that the following AMIs are scheduled for deprecation on October 1st, 2024. We strongly recommend migrating to the latest, supported AMIs listed in the table that describes the latest available AMIs from https://devops-docs.dev.int.proquest.com/user-guide/how-to/locate-a-base-ami/
Please note that the import format for Service Registry tag imports has changed to support a new, parent product taxonomy within the registry. A new product attribute will be required to be provided in the import file. Please review the updated Documentation for details.