New at AWS re:Invent: Werner Vogels Keynote

Dr. Werner Vogels’ keynote is usually the most anticipated at AWS re:Invent conventions. It’s no surprise he delivers what can be considered the closing keynote. He took a break from Now Go Build – Season 3 production to visit with customers in Las Vegas at the AWS re:Invent 2022 conference. Dr. Vogels and his team have perfected the art of keynote delivery with a cinematic intro, some background education, guest speakers to support the talk’s theme, and always an inspiring set of new service introductions.

This year’s theme was “Managing Complexity by observing and imitating natural patterns in our systems”. This was a very entertaining and educational as well as inspirational keynote. There was something for everyone. Worth watching when you have an hour and a half to invest. 

The closing statement can bring the entire keynote into focus: 

“I hope that you can all agree that we can learn from something like looking around us and observing the greatest system in existence, the universe itself. The universe itself is extremely agile; It is extremely fault tolerant as well and resilient and robust. 

We should learn from the principles that we see in nature and the world around us when we start building our computer systems. I hope this talk has inspired you to build bigger, better bolder systems much faster.”

Let’s break this down as it happened. 

The cinematic intro: This was a “The Matrix Spoof” where a world that is completely ordered and synchronized one to one exists.   You realize very quickly that the concepts of throughput and latency become essential to be able and do anything.  High throughput, low latency, loosely coupled and events processing  are to be pursued in our systems if they are to continue delivering results despite a failure or malfunction.  He says, “Synchronous is a simplification, an abstraction, a convenience to write our programs, and finally an illusion” to be avoided in our systems design.  The world is asynchronous and event driven. We should imitate that as much as possible in our systems. 

The background education: 

“Synchrony implies tightly coupled systems.” and the initial Amazon S3 Design principles used in 2006 still apply to this day in order to avoid such a situation. 

Amazon S3 Design Principles: 

  1. Decentralization 
  2. Asynchrony 
  3. Local responsibility 
  4. Decompose into small well understood building blocks. 
  5. Autonomy 
  6. Controlled concurrency 
  7. Failure Tolerance 
  8. Controlled Parallelism 
  9. Symmetry 
  10. Simplicity 

Embracing asynchrony leads to loosely coupled systems and embracing workflows enable us to build applications from loosely coupled systems. 

The patterns for workflows have a general guiding set of principles which can be summarized as: 

  1. Sequence, 
  2. Retry, 
  3. Error Handling, 
  4. Parallel, 
  5. Based on data, 
  6. Concurrent iterative  

Loosely coupled systems implement: 

  1. Fewer dependencies 
  2. Failure isolation 
  3. Evolvable architectures

The natural sequence to this best practice is the journey from a monolithic application to evolve to service oriented architecture to a micro-service architecture that uses shared services via (IaaS). 

According to Gall’s Law: “All complex Systems that work evolved from simpler systems that worked.”  

“It’s evolve or perish when it comes to systems design and the best way to build systems that can evolve is to focus on event driven architectures.  Event driven architectures enable global scale. Event-driven architectures help development teams move faster. The world is built in patterns.”

Patterns in Event Driven Architectures include: 

  1. Change Data Capture 
  2. Asynchronous coupling 
  3. Self-healing replicators. 

As a way to ramp up in more detail on these ideas he offers the Amazon Distributed Computing Manifesto. You can also visit the Amazon Builders Library.

Finally, you can also take a look at the following Cloud Academy courses and have a hands-on experience with: 

The Service Introductions: 

1) Step Functions Distributed Map (Generally available)

A Serverless Solution for Large-Scale Parallel Data Processing

“Step Function’s map state executes the same processing steps for multiple entries in a dataset. The existing map state is limited to 40 parallel iterations at a time. This limit makes it challenging to scale data processing workloads to process thousands of items (or even more) in parallel. In order to achieve higher parallel processing prior to today, you had to implement complex workarounds to the existing map state component.”

“The new distributed map state allows you to write Step Functions to coordinate large-scale parallel workloads within your serverless applications. You can now iterate over millions of objects such as logs, images, or CSV files stored in Amazon Simple Storage Service (Amazon S3). The new distributed map state can launch up to ten thousand parallel workflows to process data. You can process data by composing any service API supported by Step Functions, but typically, you will invoke Lambda functions to process the data with code written in your favorite programming language.”

“This new capability is optimized to work with S3. I can configure the bucket and prefix where my data is stored directly from the distributed map configuration. The distributed map stops reading after 100 million items and supports JSON or CSV files of up to 10GB.”

“When processing large files, think about downstream service capabilities. Let’s take Lambda again as an example. Each input—a file on S3, for example—must fit within the Lambda function execution environment in terms of temporary storage and memory. To make it easier to handle large files, Lambda Powertools for Python introduced a new streaming feature to fetch, transform, and process S3 objects with minimal memory footprint. This allows Lambda functions to handle files larger than the size of their execution environment. To learn more about this new capability, check the Lambda Powertools documentation.” (per the AWS launch blog.)

See: https://aws.amazon.com/blogs/aws/step-functions-distributed-map-a-serverless-solution-for-large-scale-parallel-data-processing for details.

 2) AWS Application Composer (Preview)

Visually design and build serverless applications quickly. Visual canvas makes composing serverless applications easier. Rapidly generate ready to deploy infrastructure as code (IaC) that follows best practices. Maintain a model of your architecture that’s easy to share and build with team members.

The call to action by Dr. Vogels: “I urge you to start with AWS application Composer to make it easy for you to get started with building these applications.”

“AWS Application Composer is a visual builder that makes it easier for developers to design an application architecture by dragging, grouping, and connecting AWS services in a visual canvas. Developers can start a new architecture from scratch, or they can import an existing AWS CloudFormation or AWS Serverless Application Model (SAM) template, including those generated by AWS Application Composer. The AWS Application Composer experience is focused around common serverless services like AWS Lambda, AWS Step Functions, and Amazon EventBridge, but it can be used to compose any AWS service supported by AWS CloudFormation resources. Developers can export infrastructure as code (IaC) to incorporate into their existing processes, such as local testing with AWS SAM Command Line Interface (CLI), peer review through version control, or deployment through CloudFormation and continuous integration and delivery (CI/CD) pipelines.” 

Image: 

Image: SRC AWS Documentation

See the following URLs for details: 

3) Amazon EventBridge Pipes (Generally available)

Connects event producers and consumers in seconds. Build advanced integrations in minutes with enhanced security, reliability and scalability out of the box. 

“So basically this is pipes on steroids because it’s not just easily composed, It also has the ability to manipulate the events that are flowing through your pipe.” Dr. Werner Vogels

Write less code – Build fully managed integrations quickly with an easy-to-use interface that connects event producers with consumers.

Save costs with filtering and built-in integrations – Lower compute and usage costs by using event filters and built-in integrations – only process and pay for events that you need.

Source events in real-time – Deliver events to over 14 AWS services with EventBridge Pipes seamlessly polling for new events.

Reduce operational load – Reliably connect services without worrying about scalability, security, patching, and provisioning your infrastructure.

Image: SRC AWS Documentation

See the following URLs for details: 

4) Amazon CodeCatalyst (Preview)

Spark innovation. Accelerate delivery. 

Unified software development service to quickly build and deliver applications on AWS.

Create new projects with everything you need in minutes

CodeCatalyst project blueprints automatically setup everything you need to start a new software development project, including CI/CD, deployable code, issue tracking, and AWS services configured according to best practices.

Spend more time coding and less time managing local development environments

CodeCatalyst Dev Environments are available on-demand in the cloud and are automatically created with branch code and consistent project settings, providing faster setup, development, and testing.

Collaborate with your team using built-in features such as issue tracking

Start creating and assigning issues. Use priorities, estimates, labels, and custom fields to help your team prioritize what’s next. Exchange feedback on issues and pull requests with your team.

Quickly customize your pipelines and fully-managed build environments

Simplify the creation and customization of automated workflows by configuring pre-defined actions using the visual editor or directly editing the underlying YAML configuration. Along with a collection of CodeCatalyst actions, you can also use any GitHub Action in your workflows.

Level up your testing strategy with easily added quality gates

CodeCatalyst has built-in support for code coverage, software composition analysis, and unit tests. You can view your test results at a glance with automated visual test reporting.

The easiest way to deploy to AWS

CodeCatalyst makes it easy to deploy your application to AWS services such as AWS Lambda or Amazon ECS. You can deploy application stacks across accounts or AWS Regions by simply listing them as targets in your pipeline

Track progress across your team

CodeCatalyst’s notifications and personalized activity feed help you stay updated on relevant project activity such as successful deployments or accepted pull requests.

(Per the product page)

Visit the following URLs for details:

The keynote included a significant amount of additional material including guest speakers.  

Angela Timofte – Director, Engineering, Trustpilot and AWS Hero, spoke about the TrustPilot event driven architecture and how it has allowed them to scale to  “Scan 100% of reviews coming through while using amazon step functions orchestrate workflows to take action on abusive behavior or unusual patterns in reviews.”. 

Nathan Thomas, VP, Unreal Engine, Epic Games introduced their app “RealityScan for IOS” and demonstrated the highly sophisticated tools available from Unreal Engine to build. Some of the items that were discussed are the Matrix awakens, an Unreal Engine Five experience, meta human creator is an online web service which allows you to generate high fidelity digital humans in minutes and those are free for use in Unreal engine.   Great tools for game designers and these tools are being used in order industries as well. 

The above are only a few of the many subjects discussed on the keynote. It’s certainly worth watching and thank you for reading this post.  

Cloud Academy