nunogrl.com - devops

Automating Server Startups with CloudFormation & Sceptre

2024-03-21T00:00:00+00:00

🚀 Problem & Solution

📌 Context / Backstory

We had a requirement to start a test EC2 instance automatically at 08:00 each morning. This server, called Tests01, was only needed during working hours, and we wanted to avoid manual intervention or running it 24/7.

⚠️ The Problem

Manual starts were unreliable and easy to forget.
CloudFormation alone doesn't handle timed triggers.
Lambda + CloudWatch Events provide scheduling, but managing them manually is messy.
We needed this fully automated and reproducible, ideally within infrastructure-as-code.

💡 The Solution

We used Sceptre to deploy a CloudFormation template that provisions: - A CloudWatch rule triggered at 08:00 - A Lambda function that starts the Tests01 instance - All IAM roles and permissions necessary

👥 Who This Helps

DevOps engineers using AWS automation
Teams migrating from cron jobs to infrastructure-as-code
Anyone needing scheduled resource management in the cloud

⚙️ Technical Implementation

Let's visualize the automation flow:

flowchart LR CW[CloudWatch Event] L[Lambda Function] EC2[EC2 Instance] CW -->|"Trigger (08:00)"| L L -->|"StartInstances API"| EC2 subgraph "AWS Infrastructure" CW L EC2 end

1️⃣ Sceptre Project Setup

Project layout:

config/
  config.yaml
  dev/
    start-tests.yaml
templates/
  start-tests.yaml

Global config/config.yaml:

project_code: startup-project
region: eu-west-1

Stack config config/dev/start-tests.yaml:

template_path: start-tests.yaml
stack_name: start-tests-scheduler
parameters:
  InstanceId: i-0123456789abcdef0

2️⃣ CloudFormation Template

File: templates/start-tests.yaml

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  InstanceId:
    Type: String
Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: StartInstance
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: ec2:StartInstances
                Resource: "*"

  StartInstanceLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Runtime: python3.9
      Timeout: 30
      Code:
        ZipFile: |
          import boto3
          import os
          def handler(event, context):
              ec2 = boto3.client('ec2')
              ec2.start_instances(InstanceIds=[os.environ['INSTANCE_ID']])
      Environment:
        Variables:
          INSTANCE_ID: !Ref InstanceId

  StartEventRule:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: cron(0 8 * * ? *)
      Targets:
        - Arn: !GetAtt StartInstanceLambda.Arn
          Id: TargetFunctionV1

  PermissionForEventsToInvokeLambda:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref StartInstanceLambda
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt StartEventRule.Arn

🛠️ Troubleshooting & Debugging

Ensure the Lambda role has permissions to ec2:StartInstances.
Use aws lambda invoke to manually test the function.
Logs appear in CloudWatch Logs; confirm the instance ID is correct.
Validate your cron expression with AWS docs: cron(0 8 * * ? *) runs at 08:00 UTC.

🔁 Optimizations & Alternatives

You can also add a second Lambda + CloudWatch rule to stop the instance at 18:00.
Consider replacing this with AWS Systems Manager Automation documents if you're not using CloudFormation.
If you're managing multiple instances, parameterize the instance list or use tags.

✅ Conclusion & Takeaways

This setup gives you a reproducible, versioned, and automated method for managing instance scheduling using CloudFormation and Sceptre. It's a clean way to handle what would otherwise be a manual or error-prone task—and it fits directly into a GitOps-style workflow.

💬 Comments & Next Steps

Have you implemented similar automation patterns with CloudFormation? Share your experience or ask questions below!

Change Sets with Sceptre: Controlled AWS Changes in ITIL Environments

2024-03-21T00:00:00+00:00

🚀 Problem & Solution

📌 Context / Backstory

We needed to add a stop-instance Lambda to our existing AWS stack — but in a production environment governed by ITIL change control. That meant:

No untracked infrastructure changes
No direct "deploy and hope" workflows
Every change needed visibility, approval, and a rollback path

⚠️ The Problem

CloudFormation makes changes declarative — but deployments are immediate by default. In an ITIL environment, we needed:

A way to preview the change
A record of the proposed change
A controlled execution, ideally during a change window

💡 The Solution

We used Sceptre's built-in support for CloudFormation Change Sets to decouple proposing changes from executing them. This gave us:

Change Set visibility before applying
A file-based workflow for review and audit
Controlled deployments aligned with ITIL practices

👥 Who This Helps

Engineers working in regulated environments (finance, enterprise IT, healthcare)
DevOps teams needing pre-deployment approvals
Anyone trying to bridge automation with change control

⚙️ Technical Implementation

Let's visualize the change management workflow:

1️⃣ Add a Stop-Lambda to Your CloudFormation Template

For example, we extended our template with:

StopInstanceLambda:
  Type: AWS::Lambda::Function
  Properties:
    Handler: index.handler
    Role: !GetAtt LambdaExecutionRole.Arn
    Runtime: python3.9
    Timeout: 30
    Code:
      ZipFile: |
        import boto3
        import os
        def handler(event, context):
            ec2 = boto3.client('ec2')
            ec2.stop_instances(InstanceIds=[os.environ['INSTANCE_ID']])
    Environment:
      Variables:
        INSTANCE_ID: !Ref InstanceId

We committed this change to Git but did not deploy yet.

2️⃣ Preview with Sceptre Change Set

In your Sceptre stack config (stop-tests.yaml):

template_path: start-stop-template.yaml
stack_name: start-tests-scheduler
parameters:
  InstanceId: i-0123456789abcdef0

Then run:

sceptre create-change-set dev/stop-tests.yaml

This creates a named Change Set in CloudFormation.

3️⃣ Review the Change Set

You can now inspect the proposed changes via:

sceptre describe-change-set dev/stop-tests.yaml

This outputs a diff-like summary of added/removed/modified resources.

4️⃣ Execute the Change Set in a Controlled Window

Once approved:

sceptre execute-change-set dev/stop-tests.yaml

This applies only what was reviewed and approved, nothing more.

🛠️ Troubleshooting & Debugging

Change Sets fail if resources are renamed instead of replaced — use Retain policies or snapshots carefully.
If nothing appears in the Change Set, verify your stack is actually different from the current state.
Include --no-execute-changeset in manual aws cloudformation calls if testing outside Sceptre.

🔁 ITIL Alignment & Best Practices

Why this works for change management:

✅ Pre-approved changes: Reviewable before execution
✅ Audit trail: Change Set IDs + Git commits form a traceable chain
✅ Rollback-ready: No impact until applied; easy to cancel
✅ Automatable: Integrates with GitOps, CI/CD, and approval gates

Compare to a traditional ITIL CAB process: - Sceptre's Change Set becomes the RFC payload - Execution timing maps to change windows - Logs & ChangeSet name tie into CMDB or ticketing systems

✅ Conclusion & Takeaways

By using Sceptre's Change Sets, we introduced governed change control without sacrificing automation. It's a clean way to blend DevOps practices with ITIL compliance — reducing risk while maintaining velocity.

💬 Comments & Next Steps

Have you implemented similar change control processes in your AWS infrastructure? Share your experience or ask questions below!

Enforcing GPG-Signed Commits in Git

2024-03-21T00:00:00+00:00

🚀 Problem & Solution

📌 Context / Backstory

In our collaborative development environment, we discovered that Git's flexibility allows commits under any name and email. This became a security concern when we realized that GitHub identifies users by email, not SSH keys.

sequenceDiagram participant Alice participant Bob participant GitHub Alice->>GitHub: Commit signed with alice@example.com (GPG signed) GitHub->>GitHub: Shows "Verified" commit from Alice Bob->>GitHub: Commit using alice@example.com (no signature) GitHub->>GitHub: Shows commit as from "Alice" (Unverified) Note over GitHub: GitHub matches commits by email, not by SSH or true identity.

⚠️ The Problem

Anyone with repository access can spoof another contributor's identity
Commit history could be manipulated without detection
No cryptographic proof of commit authenticity
GitHub's user identification relies solely on email addresses

💡 The Solution

We implemented mandatory GPG-signed commits, which:

Cryptographically verify commit authenticity
Prevent identity spoofing
Create traceable commit history
Integrate with GitHub's verification system

👥 Who This Helps

Security-conscious development teams
Open-source project maintainers
Regulated environments requiring audit trails
Organizations needing verified commit history

⚙️ Technical Implementation

Let's visualize the GPG signing workflow:

flowchart LR C[Commit] -->|Sign| G[GPG Key] G -->|Verify| GH[GitHub] subgraph "Local System" C G end subgraph "Remote" GH end style GH fill:#f96,stroke:#333 style G fill:#9f6,stroke:#333

1️⃣ Setting Up GPG Keys

# Generate a new GPG key
gpg --full-generate-key

# List your keys
gpg --list-secret-keys --keyid-format=long

2️⃣ Configuring Git

# Configure Git to use your GPG key
$ git config --global user.signingkey <YOUR_KEY_ID>
$ git config --global commit.gpgsign true
$ git config --global gpg.program gpg

3️⃣ GitHub Integration

Export your public GPG key:

$ gpg --armor --export <YOUR_KEY_ID>

Add the key to your GitHub account settings

4️⃣ Enforcing Signed Commits

In GitHub repository settings:

Navigate to Settings > Branches
Add branch protection rule
Enable "Require signed commits"

🛠️ Troubleshooting & Debugging

GPG signing fails: Check gpg-agent configuration
GitHub doesn't show "Verified": Ensure GPG key is added to GitHub
CI/CD issues: Set up proper GNUPGHOME environment
Smart card/YubiKey: Verify proper card reader access

🔁 Optimizations & Best Practices

Use GPG subkeys instead of master keys
Implement regular key rotation
Set up separate signing keys for different contexts
Use environment isolation in CI/CD pipelines
Consider hardware security keys (YubiKey) for key storage

✅ Conclusion & Takeaways

GPG-signed commits provide a robust security layer for Git workflows, ensuring: - Verified commit authenticity - Protected repository history - Clear accountability - Compliance with security best practices

💬 Comments & Next Steps

How do you handle commit verification in your organization? Share your experience or ask questions below!

Password Store with GPG and Git

2024-03-21T00:00:00+00:00

🚀 Problem & Solution

📌 Context / Backstory

We needed a secure way to manage passwords and secrets across multiple servers and team members. Commercial password managers were either too complex, costly, or required external services we wanted to avoid.

⚠️ The Problem

Managing secrets across systems and teams presents several challenges:

Keeping passwords secure yet accessible
Tracking changes and maintaining history
Sharing secrets securely between team members
Avoiding dependency on external services

💡 The Solution

We implemented password-store with GPG encryption and Git integration, providing:

Secure GPG encryption for all secrets
Git-based version control and distribution
Fully local operation with no external dependencies
Command-line interface for automation

👥 Who This Helps

System administrators managing multiple servers
DevOps teams handling shared credentials
Security-conscious users wanting local password management
Teams needing version-controlled secrets

⚙️ Technical Implementation

Let's visualize the password-store workflow:

1️⃣ Generating GPG Keys in Batch Mode

For automated environments:

cat > gpg-server-key.conf <<EOF
%no-protection
Key-Type: default
Subkey-Type: default
Name-Real: Server Automation Key
Name-Email: server@example.com
Expire-Date: 0
%commit
EOF

gpg --batch --generate-key gpg-server-key.conf

2️⃣ Setting Up the Environment

export PASSWORD_STORE_GPG_OPTS="--armor"
export GNUPGHOME=/etc/password-store/.gnupg
export PASSWORD_STORE_DIR=/etc/password-store/store

3️⃣ Initializing the Password Store

mkdir -p "$GNUPGHOME" "$PASSWORD_STORE_DIR"
chmod 700 "$GNUPGHOME" "$PASSWORD_STORE_DIR"
pass init server@example.com

4️⃣ Git Integration

cd "$PASSWORD_STORE_DIR"
git init
git add .
git commit -m "Initial password store"
git remote add origin git@example.com:secrets.git
git push -u origin main

🛠️ Troubleshooting & Debugging

Ensure proper GPG key permissions (700 for directories, 600 for files)
Verify GPG recipient when encryption fails
Check Git remote access rights for sync issues
Monitor Git conflicts when multiple users update simultaneously

🔁 Optimizations & Alternatives

Consider using GPG agent for improved key handling
Implement Git hooks for pre-commit validation
Use Git branches for testing password updates
Consider pass extensions for additional features

✅ Conclusion & Takeaways

Using GPG with password-store provides a flexible, secure, and lightweight method for managing secrets across machines. With Git integration, you get version history, team sharing, and distributed backup—without compromising security.

💬 Comments & Next Steps

How do you manage shared secrets in your infrastructure? Share your experience or ask questions below!