XHD.io | Yet another devops blog
Posts Tags Categories About
XHD.io | Yet another devops blog

Contents

Cloud Nuke

TheProtos included in clouds
   1283 words   7 minutes 
/site/cloud-nuke/featured-image.png

Managing resources and objects in your cloud account is an essential aspect of maintaining an efficient and cost-effective infrastructure. Occasionally, you may need to delete specific types of resources and objects to streamline your operations, optimize costs, or meet compliance requirements.

In this article, we will guide you through the process of deleting various types of resources and objects in your cloud account.

AWS-nuke

AWS only. See project repository and documentation

Install aws-nuke

brew install ekristen/tap/aws-nuke

(OPTIONAL) Ensure your account has alias before running aws-nuke or baypass it

# Get alias name
aws iam list-account-aliases

# Create one of missing
#aws iam create-account-alias --account-alias my-account-alias

Create config to list all resources

cat <<"EOF" > aws-nuke-all.yml
regions:
  #- global
  - us-west-2

settings:
  EC2Image:
    IncludeDisabled: true
    IncludeDeprecated: true
    DisableDeregistrationProtection: true
  EC2Instance:
    DisableStopProtection: true
    DisableDeletionProtection: true
  RDSInstance:
    DisableDeletionProtection: true
  CloudFormationStack:
    DisableDeletionProtection: true
  DynamoDBTable:
    DisableDeletionProtection: true

resource-types:
  excludes:
    - IAM*
    - ServiceCatalogTagOption # Excluded due to https://github.com/rebuy-de/aws-nuke/issues/515
    - ServiceCatalogTagOptionPortfolioAttachment # Excluded due to https://github.com/rebuy-de/aws-nuke/issues/515
    - FMSNotificationChannel # Excluded because it's not available
    - FMSPolicy # Excluded because it's not available
    - MachineLearningMLModel # Excluded due to ML being unavailable
    - MachineLearningDataSource # Excluded due to ML being unavailable
    - MachineLearningBranchPrediction # Excluded due to ML being unavailable
    - MachineLearningEvaluation # Excluded due to ML being unavailable
    - RoboMakerDeploymentJob # Deprecated Service
    - RoboMakerFleet # Deprecated Service
    - RoboMakerRobot # Deprecated Service
    - RoboMakerSimulationJob
    - RoboMakerRobotApplication
    - RoboMakerSimulationApplication
    - OpsWorksApp # Deprecated service
    - OpsWorksInstance # Deprecated service
    - OpsWorksLayer # Deprecated service
    - OpsWorksUserProfile # Deprecated service
    - OpsWorksCMBackup # Deprecated service
    - OpsWorksCMServer # Deprecated service
    - OpsWorksCMServerState # Deprecated service
    - CodeStarProject # Deprecated service
    - CodeStarConnection # Deprecated service
    - CodeStarNotification # Deprecated service
    - Cloud9Environment # Deprecated service
    - CloudSearchDomain # Deprecated service
    - RedshiftServerlessSnapshot # Deprecated service
    - RedshiftServerlessNamespace # Deprecated service
    - RedshiftServerlessWorkgroup # Deprecated service


blocklist:
  - "999999999999" # production

bypass-alias-check-accounts:
  - 1234567890

accounts:
  1234567890:

List all resources in the account, no actual deletion

aws-nuke nuke -c aws-nuke-all.yml --no-alias-check --profile client-sandbox

Use case 1 - delete EKS cluster

There are clusters created by eksctl, but no application configs exist. Therefore, it is not possible to delete the clusters using eksctl. Some of the resources were deleted already, but few left. Delete all resources created by eksctl

In order to get available resources types run below

aws-nuke resource-types

Create config

cat <<"EOF" > aws-nuke-eks.yml
regions:
  - global
  - eu-west-2
  
settings:
  EC2Image:
    IncludeDisabled: true
    IncludeDeprecated: true
    DisableDeregistrationProtection: true
  EC2Instance:
    DisableStopProtection: true
    DisableDeletionProtection: true
  RDSInstance:
    DisableDeletionProtection: true
  CloudFormationStack:
    DisableDeletionProtection: true
  DynamoDBTable:
    DisableDeletionProtection: true
  
resource-types:
  includes:
    - IAMInstanceProfileRole
    - IAMInstanceProfile
    - IAMRole
    - IAMRolePolicy
    - IAMRolePolicyAttachment
  excludes:
    - S3Object # Excluded because S3 bucket removal handles removing all S3Objects
    - ServiceCatalogTagOption # Excluded due to https://github.com/rebuy-de/aws-nuke/issues/515
    - ServiceCatalogTagOptionPortfolioAttachment # Excluded due to https://github.com/rebuy-de/aws-nuke/issues/515
    - FMSNotificationChannel # Excluded because it's not available
    - FMSPolicy # Excluded because it's not available
    - MachineLearningMLModel # Excluded due to ML being unavailable
    - MachineLearningDataSource # Excluded due to ML being unavailable
    - MachineLearningBranchPrediction # Excluded due to ML being unavailable
    - MachineLearningEvaluation # Excluded due to ML being unavailable
    - RoboMakerDeploymentJob # Deprecated Service
    - RoboMakerFleet # Deprecated Service
    - RoboMakerRobot # Deprecated Service
    - RoboMakerSimulationJob
    - RoboMakerRobotApplication
    - RoboMakerSimulationApplication
    - OpsWorksApp # Deprecated service
    - OpsWorksInstance # Deprecated service
    - OpsWorksLayer # Deprecated service
    - OpsWorksUserProfile # Deprecated service
    - OpsWorksCMBackup # Deprecated service
    - OpsWorksCMServer # Deprecated service
    - OpsWorksCMServerState # Deprecated service
    - CodeStarProject # Deprecated service
    - CodeStarConnection # Deprecated service
    - CodeStarNotification # Deprecated service
    - Cloud9Environment # Deprecated service
    - CloudSearchDomain # Deprecated service
    - RedshiftServerlessSnapshot # Deprecated service
    - RedshiftServerlessNamespace # Deprecated service
    - RedshiftServerlessWorkgroup # Deprecated service

account-blocklist:
  - "999999999999" # production

bypass-alias-check-accounts:
  - 899999999999

accounts:
  "899999999999":
    filters:
      IAMInstanceProfileRole:
      - type: regex
        value: "^eksctl-sandbox-west2-"
        invert: true
      IAMInstanceProfile:
      - type: regex
        value: "^eksctl-sandbox-west2-"
        invert: true
      IAMRole:
      - type: regex
        value: "^eksctl-sandbox-west2-"
        invert: true
      IAMRolePolicy:
      - type: regex
        value: "^eksctl-sandbox-west2-"
        invert: true
      IAMRolePolicyAttachment:
      - type: regex
        value: "^eksctl-sandbox-west2-"
        invert: true
EOF

Run removal

it will ask for approval twice

aws-nuke -c aws-nuke.yml --no-alias-check --profile client-sandbox --no-dry-run

Cloud-nuke (AWS)

See project repository and documentation

Telemetry to Gruntwork

As of version v0.29.0 cloud-nuke sends telemetry back to Gruntwork to help us better prioritize bug fixes and feature improvements. The following metrics are included:

  • Command and Arguments
  • Version Number
  • Timestamps
  • Resource Types
  • Resource Counts
  • A randomly generated Run ID
  • AWS Account ID

We never collect:

  • IP Addresses
  • Resource Names

To disable it:

export DISABLE_TELEMETRY=TRUE

Getting started with cloud-nuke

Install

brew install cloud-nuke

Disable telemetry and set AWS_PROFILE

export DISABLE_TELEMETRY=TRUE 
export AWS_PROFILE=client-sandbox

Get all resources available in AWS account

cloud-nuke inspect-aws --region global --region us-west-2

# Alternative way
cloud-nuke aws --dry-run

Delete resources

Delete resources using --resource-type, run cloud-nuke aws --list-resource-types before below:

cloud-nuke aws \
--region global \
--region us-west-2 \
--resource-type ekscluster \
--resource-type asg \
--resource-type lt \
--resource-type managed-prometheus \
--resource-type opensearchdomain \
--resource-type cloudwatch-alarm \
--resource-type cloudwatch-dashboard \
--resource-type cloudwatch-loggroup \
--log-level debug

Delete resources using config file

Create config file with list of resources (see full example):

cat <<"EOF" > cloud-nuke.yml
---
iam:
  exclude:
    names_regex:
      - .*

iam-group:
  exclude:
    names_regex:
      - .*

iam-policy:
  exclude:
    names_regex:
      - .*

iam-role:
  exclude:
    names_regex:
      - .*

managed-prometheus:
  include:
    names_regex:
      - ^abc-.*
      - abc-.*
      - ^*.-efg-.*

opensearchdomain:
  include:
    names_regex:
      - ^abc-.*
      - abc-.*
      - ^*.-efg-.*

lt:
  include:
    names_regex:
      - ^abc-.*
      - abc-.*
      - ^*.-efg-.*

#--resource-type s3
s3:
  include:
    names_regex:
      - ^abc-.*
      - abc-.*
      - ^*.-efg-.*

#--resource-type lambda
LambdaFunction:
  include:
    names_regex:
      - ^abc-.*
      - abc-.*
      - ^*.-efg-.*

#lambda_layer

cloudwatch-alarm:
  include:
    names_regex:
      - ^abc-.*
      - abc-.*
      - ^*.-efg-.*

NatGateway:
  include:
    names_regex:
      - eksctl-.*
      - abc-.*
      - ^*.-efg-.*

#--resource-type eip
ElasticIP:
  include:
    names_regex:
      - eksctl-.*
      - abc-.*
      - ^*.-efg-.*

#--resource-type event-bridge-rule
EventBridgeRule:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type cloudwatch-loggroup
CloudWatchLogGroup:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type dynamodb
DynamoDB:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type sqs
sqs:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type grafana
grafana:
  include:
    names_regex:
      - .*

#--resource-type secretsmanager
secretsmanager:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type kinesis-stream
KinesisStream:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type elasticache
elasticache:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type rds-subnet-group
rds-subnet-group:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type rds-parameter-group
rds-parameter-group:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*

#--resource-type data-sync-task
data-sync-task:
  include:
    names_regex:
      - .*

#--resource-type data-sync-location
data-sync-location:
  include:
    names_regex:
      - .*

#--resource-type elasticacheParameterGroups
elasticacheParameterGroups:
  include:
    names_regex:
      - abc-.*
      - hij-.*

#--resource-type elasticacheSubnetGroups
elasticacheSubnetGroups:
  include:
    names_regex:
      - abc-.*
      - hij-.*

#--resource-type apigateway
apigateway:
  include:
    names_regex:
      - .*

#--resource-type nat-gateway
nat-gateway:
  include:
    names_regex:
      - abc-.*

#--resource-type security-group
security-group:
  include:
    names_regex:
      - abc-.*
      - ^*.-efg-.*
EOF

The resources in config file and in cli param --resource-type have different naming.

cloud-nuke aws \
--region global \
--region us-west-2 \
--resource-type nat-gateway \
--resource-type eip \
--config cloud-nuke.yml \
--log-level debug

Azure

Azure-cloud-nuke

TBD. Promising project https://gitlab.com/lmartz/azure-cloud-nuke

Powershell

Install Azure cli for your OS: https://learn.microsoft.com/en-us/powershell/azure/install-azure-powershell?view=azps-10.0.0

Install azure module

Install-Module -Name Az -Repository PSGallery -Force

Create ps1

cat <<"EOF" > azure.ps1
# Login
Connect-AzAccount 

# Get a list of all Azure subscript that the user can access
$allSubs = Get-azSubscription 
$allSubs | Sort-Object SubscriptionName | Format-Table -Property SubscriptionName, SubscriptionId, State
$theSub = Read-Host "Enter the subscriptionId you want to clean"

Write-Host "You select the following subscription. (it will be display 15 sec.)" -ForegroundColor Cyan
Get-azSubscription -SubscriptionId $theSub | Select-azSubscription 

#Get all the resources groups
$allRG = Get-azResourceGroup

foreach ($g in $allRG){

    Write-Host $g.ResourceGroupName -ForegroundColor Yellow 
    Write-Host "------------------------------------------------------`n" -ForegroundColor Yellow 
    $allResources = Get-azResource -ResourceGroupName $g.ResourceGroupName | FT

    if($allResources){
        $allResources | Format-Table -Property Name, ResourceName
    }
    else{
         Write-Host "-- empty--`n"
    } 
    Write-Host "`n`n------------------------------------------------------" -ForegroundColor Yellow 
}

$lastValidation = Read-Host "Do you want to delete ALL the resources previously listed? (YES/ NO)"

if($lastValidation.ToLower().Equals("yes")){

    foreach ($g in $allRG){

        Write-Host "Deleting " $g.ResourceGroupName 
        #Get-AzResourceGroup -Name $g.ResourceGroupName | Remove-AzResourceGroup -Verbose -Force

    }
}
else{
     Write-Host "Aborded. Nothing was deleted." -ForegroundColor Cyan
}

EOF
Updated on 2023-03-20
 Aws, Azure, Gcp, Nuke
Back | Home