Amazon Managed Workflows for Apache Airflow (Amazon MWAA) has grow to be a cornerstone for organizations embracing data-driven decision-making. As a scalable answer for managing complicated information pipelines, Amazon MWAA allows seamless orchestration throughout AWS companies and on-premises methods. Though AWS manages the underlying infrastructure, you should fastidiously plan and execute your Amazon MWAA atmosphere updates based on the shared duty mannequin. Upgrading to the newest Amazon MWAA model can present vital benefits, together with enhanced safety by way of crucial safety patches and potential enhancements in efficiency with sooner DAG parsing and decreased database load. You need to use superior options whereas sustaining ecosystem compatibility and receiving prioritized AWS assist. The important thing to profitable upgrades lies in selecting the best answer and following a methodical implementation method.
On this publish, we discover greatest practices for upgrading your Amazon MWAA atmosphere and supply a step-by-step information to seamlessly transition to the newest model.
Resolution overview
Amazon MWAA gives two major improve options:
In-place improve – This technique works greatest when you possibly can accommodate deliberate downtime. You deploy the brand new model straight in your present infrastructure. In-place model upgrades on Amazon MWAA are supported for environments operating Apache Airflow model 2.x and later. Nevertheless, should you’re operating model 1.10.z or older variations, you should create a brand new atmosphere and migrate your assets, as a result of these variations don’t assist in-place upgrades.
Cutover improve – This technique helps reduce disruption to manufacturing environments. You create a brand new Amazon MWAA atmosphere with the goal model after which transition out of your outdated atmosphere to the brand new one.
Every answer affords a distinct method that will help you improve whereas working to keep up information integrity and system reliability.
In-place improve
In-place upgrades work nicely for environments the place you possibly can schedule a upkeep window for the improve course of. Throughout this window, Amazon MWAA preserves your workflow historical past. This technique works greatest when you possibly can accommodate deliberate downtime. It helps preserve historic information, gives a simple improve course of, and consists of rollback capabilities if points happen throughout provisioning. You additionally use fewer assets since you don’t must create a brand new atmosphere.
You’ll be able to carry out in-place upgrades by way of the AWS Administration Console with a single operation. This course of helps cut back operational overhead by managing many improve steps for you.
In the course of the improve course of, your atmosphere can’t schedule or run new duties. Amazon MWAA helps handle the improve course of and implements security measures—if points happen through the provisioning part, the service makes an attempt to revert to the earlier steady model.
Earlier than you start an in-place improve, we suggest testing your DAGs for compatibility with the goal model, as a result of DAG compatibility points can have an effect on the improve course of. You need to use the Amazon MWAA native runner to check DAG compatibility earlier than you begin the improve. You can begin the improve utilizing both the console and specifying the brand new model or the AWS Command Line Interface (AWS CLI). The next is an instance Amazon MWAA improve command utilizing the AWS CLI:
aws mwaa update-environment –name –airflow-version
The next diagram reveals the Amazon MWAA in-place improve workflow and states.
Consult with Introducing in-place model upgrades with Amazon MWAA for extra particulars.
Cutover improve
A cutover improve gives another answer when it is advisable reduce downtime, although it requires extra handbook steps and operational planning. With this method, you create a brand new Amazon MWAA atmosphere, migrate your metadata, and handle the transition between environments. Though this technique affords extra management over the improve course of, it requires further planning and execution effort in comparison with an in-place improve.
This technique can work nicely for environments with complicated workflows, significantly while you plan to make vital modifications alongside the model improve. The method affords a number of advantages: you possibly can reduce manufacturing downtime, carry out complete testing earlier than switching environments, and preserve the power to return to your unique atmosphere if wanted. You may as well evaluation and replace your configurations through the transition.
Contemplate the next facets of the cutover method. Once you run two environments concurrently, you pay for each environments. The pricing for every Amazon MWAA atmosphere relies on:
Period of atmosphere uptime (billed hourly with per-second decision)
Atmosphere dimension configuration
Computerized scaling capability for staff
Scheduler capability
AWS calculates the price of further computerized scaled staff individually. You’ll be able to estimate prices in your particular configuration utilizing the AWS Pricing Calculator.
To assist stop information duplication or corruption throughout parallel operation, we suggest implementing idempotent DAGs. The Airflow scheduler robotically populates some metadata tables (dag, dag_tag, and dag_code) in your new atmosphere. Nevertheless, it is advisable plan the migration of the next further metadata parts:
DAG historical past
Variables
Slot pool configurations
SLA miss information
XCom information
Job information
Log tables
You’ll be able to select this method when your necessities prioritize minimal downtime and you’ll handle the extra operational complexity.
The cutover improve course of entails three important steps: creating a brand new atmosphere, restoring it with the prevailing information, and performing the improve. The next diagram illustrates the total workflow.
Within the following sections, we stroll by way of the important thing steps to carry out a cutover improve.
Stipulations
Earlier than you start the improve course of, full the next steps:
Create a brand new atmosphere
Full the next steps to create a brand new atmosphere:
Generate a template in your new atmosphere configuration utilizing the AWS CLI:
aws mwaa create-environment –generate-cli-skeleton > .json
Modify the generated JSON file:
Copy configurations out of your backup file .json to .json.
Replace the atmosphere title.
Hold the AirflowVersion parameter worth out of your present atmosphere.
Overview and replace different configuration parameters as wanted.
Create your new atmosphere:
aws mwaa create-environment –cli-input-json
Restore the brand new atmosphere
Full the next steps to revive the brand new atmosphere:
Use the DR PyPI package deal to create and run the restore DAG.
This course of copies metadata out of your S3 backup bucket to the brand new atmosphere.
Confirm that your new atmosphere incorporates the anticipated metadata out of your unique atmosphere.
Carry out the model improve
Full the next steps to carry out the model improve:
Improve your atmosphere:
aws mwaa update-environment –name –airflow-version
Monitor the improve:
Observe the atmosphere standing on the console.
Look ahead to error messages or warnings.
Confirm the atmosphere reaches the AVAILABLE
Plan your transition timing fastidiously. When your unique atmosphere continues to course of workflows throughout this improve, the metadata between environments can change.
Clear up
After you confirm the steadiness of your upgraded atmosphere by way of monitoring, you possibly can start the cleanup course of:
Take away your unique Amazon MWAA atmosphere utilizing the AWS CLI command:
aws mwaa delete-environment –name
Clear up your related assets by eradicating unused backup information from S3 buckets, deleting momentary AWS Id and Entry Administration (IAM) roles and insurance policies created for the improve, and updating your DNS or routing configurations.
Earlier than eradicating any assets, be sure you comply with your group’s backup retention insurance policies, preserve essential backup information in your compliance necessities, and doc configuration modifications made through the improve.
This method helps you carry out a managed improve with alternatives for testing and the power to return to your unique atmosphere if wanted.
Monitoring and validation
You’ll be able to observe your improve progress utilizing Amazon CloudWatch metrics, with a concentrate on DAG processing metrics and scheduler heartbeat. Your atmosphere transitions by way of a number of states through the improve course of, together with UPDATING and CREATING. When your atmosphere reveals the AVAILABLE state, you possibly can start validation testing. We suggest checking system accessibility, testing crucial workflow operations, and verifying exterior connections. For detailed monitoring steering, see Monitoring and metrics for Amazon Managed Workflows for Apache Airflow.
Key issues
Think about using infrastructure as code (IaC) practices to assist preserve constant atmosphere administration and assist repeatable deployments. Schedule metadata backups utilizing DR in periods of low exercise to assist defend your information. When designing your workflows, implement idempotent pipelines to assist handle potential interruptions, and preserve documentation of your configurations and dependencies.
Conclusion
A profitable Amazon MWAA improve begins with deciding on an method that aligns along with your operational necessities. Whether or not you select an in-place or cutover improve, thorough preparation and testing assist assist a managed transition. Utilizing obtainable instruments, monitoring capabilities, and advisable practices may also help you improve to the newest Amazon MWAA options whereas working to keep up your workflow operations.
For added particulars and code examples on Amazon MWAA, seek advice from the Amazon MWAA Consumer Information and Amazon MWAA examples GitHub repo.
Apache, Apache Airflow, and Airflow are both registered logos or logos of the Apache Software program Basis in america and/or different nations.
In regards to the Authors
Anurag Srivastava works as a Senior Massive Knowledge Cloud Engineer at Amazon Internet Companies (AWS), specializing in Amazon MWAA. He’s keen about serving to clients construct scalable information pipelines and workflow automation options on AWS.
Sriharsh Adari is a Senior Options Architect at Amazon Internet Companies (AWS), the place he helps clients work backwards from enterprise outcomes to develop progressive options on AWS. Over time, he has helped a number of clients on information platform transformations throughout business verticals. His core space of experience embody Know-how Technique, Knowledge Analytics, and Knowledge Science. In his spare time, he enjoys taking part in sports activities, binge-watching TV reveals, and taking part in Tabla.
Venu Thangalapally is a Senior Options Architect at AWS, based mostly in Chicago, with deep experience in cloud structure, information and analytics, containers, and software modernization. He companions with Monetary Companies business clients to translate enterprise objectives into safe, scalable, and compliant cloud options that ship measurable worth. Venu is keen about leveraging know-how to drive innovation and operational excellence. Exterior of labor, he enjoys spending time together with his household, studying, and taking lengthy walks.
Chandan Rupakheti is a Senior Options Architect at AWS. His important focus at AWS lies within the intersection of analytics, serverless, and AdTech companies. He’s a passionate technical chief, researcher, and mentor with a knack for constructing progressive options within the cloud. Exterior of his skilled life, he loves spending time together with his household and buddies, and listening to and taking part in music.
