DevOps Bash tools 다운로드 DevOps Bash tools 소스 코드 다운로드

Hari Sekhon -Devops Bash 도구

git.io/bash-tools

1000+ DevOps 쉘 스크립트 및 고급 배쉬 환경.

빠른 고급 시스템 엔지니어링, 자동화, API, 짧은 CLI 등

많은 Github Repos에 많이 사용되는 수십 개의 DockerHub 빌드 (DockerFiles) 및 600 개 이상의 CI 빌드가 많이 사용됩니다.

요약

많은 인기있는 DevOps 기술에 대한 스크립트, 자세한 내용은 아래 색인을 참조하십시오.
git, vim, screen, tmux, postgresql psql 등과 같은 일반적인 도구에 대한 고급 구성 ...
대부분의 주요 연속 통합 제품에 대한 CI 구성 (CI 빌드 페이지 참조)
모든 CI 빌드에서 실행되는 표준 검사의 드롭 인 프레임 워크, CI 감지, CI 환경에 걸친 설치 차이, 루트 대 사용자, VirtualEnvs 등의 설치 차이를위한 CI 스크립트.
API 스크립트 자동 처리 인증, 토큰 및 기타 세부 사항은 /path/endpoint 제공하는 몇 가지 키 스트로크로 인기있는 API를 신속하게 쿼리하기 위해 기타 세부 사항
Advanced Bash Environment- .bashrc + .bash.d/*.sh 별칭, 기능, 채색, 동적 git 및 쉘 행동 향상, 설치 용 자동 경로 및 Python, Perl, Ruby, Nodejs, Linux 배포판 및 Mac과 같은 주요 언어. .bash.d/readme.md를 참조하십시오
AWS CLI, Azure CLI, Gcloud SDK, Digital Ocean Cli, Terraform, Terragrunt, Github CLI, Kubernetes Kubectl & Kustomize, Helm, EKSCTL, Docker -Compose, JQ 및 기타 많은 패키지를위한 광범위한 패키지 목록을위한 광범위한 패키지 목록 및 MAC의 최상의 시스템 패키지를 설치합니다.
- install/ - 인기있는 오픈 소스 소프트웨어를위한 많은 설치 스크립트가 포함되어 있으며 Github 릴리스에서 직접 이진 다운로드
- configs/ - vim, 상단, 화면, tmux, mysql, postgresql 등과 같은 일반적인 기술에 대한 많은 DOT 구성이 포함되어 있습니다.
- setup/ - 설정 스크립트, 패키지 목록, 추가 구성, Mac OS X 설정 등이 포함됩니다.
수백 개의 스크립트가 사용하는 유틸리티 라이브러리 및 저장소를 가로 질러 빌드합니다.
- .bash.d/ - 대화식 라이브러리
- lib/ - 스크립팅 및 CI 라이브러리
SQL 스크립트 - PostgreSQL, MySQL, AWS Athena + CloudTrail, Google BigQuery 용 100 개 이상의 스크립트
템플릿 - 일반적인 프로그래밍 언어 및 빌드 구성 템플릿
Kubernetes 구성 -Kubernetes Yaml 프로덕션 모범 사례, 팁 및 트릭을 포함한 대부분의 일반적인 시나리오에 대한 구성

다른 언어로 유사한 DevOps 리포지어도 참조하십시오

Hari Sekhon

클라우드 및 빅 데이터 계약자, 영국

(전 Hortonworks 컨설턴트 전 Cloudera)

(LinkedIn에서 저와 연결되어 있습니다)

빠른 설정

부트 스트랩을 위해 패키지를 설치하고 쉘 프로파일에 링크하여 모든 구성을 상속 받으려면 다음을 수행하십시오.

curl -L https://git.io/bash-bootstrap | sh

.bashrc / .bash_profile 에 소싱을 추가하여 모든 기술에 대한 모든 .bash.d/*.sh 환경 향상을 자동으로 상속합니다 (아래의 인벤토리 참조).
Symlinks .* GIT, VIM, 상단, HTOP, 화면, TMUX, TMUX, Ansible, PostgreSQL .psqlrc 등을위한 $HOME 에 대한 구성 요소를 구성하십시오 (아직 존재하지 않으므로 자체 구성과 충돌이 없음).
모든 스크립트에 대한 OS 패키지 종속성 설치 (OS 감지 및 올바른 RPMS, DEBS, APK 또는 MAC Homebrew 패키지 설치)
파이썬 패키지를 설치합니다
AWS CLI를 설치합니다

스크립트를 실행하기 위해 패키지 종속성 만 설치하려면 Git Clone 디렉토리로 cd 간단히 작성하고 실행 make .

git clone https://github.com/HariSekhon/DevOps-Bash-tools bash-tools
cd bash-tools
make

make install 쉘 프로파일을 설정 하여이 리포트를 소스하십시오. 더 자세한 설치/제거 옵션은 아래의 개별 설정 부품을 참조하십시오.

색인

dot configs- .gitconfig , .vimrc , .screenrc , .tmux.conf , .toprc , .gitignore ...
배쉬 환경 및 라이브러리 - .bashrc , .bash.d/ 대화식 라이브러리, lib/ 스크립팅 라이브러리
많은 인기있는 오픈 소스 기술에 대한 설치 스크립트
Linux & Mac -Curl Oauth/JWT, LDAP, 중복 파일 찾기, SSL 인증서 GET/Validate, URL 인코딩/디코딩, Vagrant
Mac & AppleScript -Mac 설정 및 UI 자동화 스크립트, 키 스트로크 보내기, 마우스 클릭, 전경 앱 감지, 앱 스위치, 잠긴 화면 또는 스크린 세이버 감지, Screensaver 활성화
모니터링 - Grafana, Prometheus, Node Expler, Common Linux 및 Mac CLI 모니터링 통계 및 로그 위치의 스크립트 컬렉션을위한 빠른 생성 벤더 지원 Tarball 번들 로컬 및 SSH 이상
AWS -Amazon Web Services -AWS 계정 요약, 많은 IAM 보고서, CIS 벤치 마크 구성 강화, EC2, ECR, EKS, 스팟 종료, S3 액세스 로깅, KMS 키 회전 정보, SSM, CloudTrail, SNS 알림 주제와 이메일 알림 구독이있는 CloudWatch 청구 알람
GCP- Google Cloud 플랫폼 - 대규모 GCP 자동 인벤토리, GCE, GKE, GCR, Secret Manager, BigQuery, Cloud SQL, Cloud Scheduler, Terraform Service 계정 제작 스크립트
Kubernetes -Massive Kubernetes Auto -Inventory, 클러스터 관리 스크립트 및 트릭
Docker -Docker API, DockerHub API, Quay.io API 스크립트
데이터베이스 - 빠른 CLI 래퍼, 인스턴트 Docker 샌드 박스 (PostgreSQL, MySQL, MariadB, SQLite), SQL 스크립트, 모든 버전의 DB에 대한 SQL 스크립트 테스터, 고급 .psqlrc 대한 SQL 스크립트 테스터.
데이터 - AVRO, Parquet, CSV, JSON, INI / Properties Files (Java), LDAP LDIF, XML, YAML 용 데이터 도구, 변환기 및 형식 유효성 검사기
BIG DATA & NOSQL -Kafka, Hadoop, HDFS, Hive, Impala, Zookeeper, Cloudera Manager API & Cloudera Navigator API 스크립트
git -github, gitlab, bitbucket, azure devops- git local & mirror management, github, gitlab & bitbucket apis를위한 스크립트
CI / CD- 지속적인 통합 / 연속 전달 - 대부분의 주요 CI 시스템에 대한 API 스크립트 및 구축 파이프 라인 구성 :
- Jenkins, Concourse, Gocd, TeamCity- One -Touch Boot & Build
- Azure Devops 파이프 라인, Github 액션 워크 플로우, Gitlab CI, Bitbucket 파이프 라인, 앱베리오르, BuildKite, Travis CI, Circle CI, Codefresh, Codeship, Drone.io, Semapore CI, Shippable ...
- Terraform Cloud, Octopus 배포
- Checkov / Bridgecrew Cloud
AI & IPAAS- OpenAi (ChatGpt), make.com
인터넷 서비스 - CloudFlare, Datadog, Digital Ocean, Kong API Gateway, Gitguardian, Jira, Ngrok, Traefik, Pingdom, WordPress 및 다양한 페이스트 빈 및 파일 업로드 사이트
Java -Java Utilies는 자바 프로그램을 실행하는 디버그 또는 더 깊은 디버깅을위한 Java Jar 코드를 디 컴파일하는 Java Utilies
파이썬 - 파이썬 유틸리티 및 라이브러리 관리
PERL -PERL 유틸리티 및 라이브러리 관리
Golang -Golang 유틸리티
미디어 - MP3 메타 데이터 편집, 앨범 및 오디오 북의 그룹화 및 주문, mkv/avi to mp4 컨버터, YouTube 채널 다운로드
Spotify -40+ 백업 용 Spotify API 스크립트, 재생 목록 관리, 중간선 추적, URI 변환, 검색, 추가/삭제, 트랙을 따라가는 아티스트, 최고의 아티스트, 최고 트랙 등
더 많은 Linux & Mac- 더 많은 시스템 관리 스크립트, 패키지 설치 자동화
빌드, 언어 및 라인팅 - 프로그래밍 언어, 빌드 시스템 및 CI Linting
템플릿 - AWS, GCP, Terraform, Docker, Jenkins, 클라우드 빌드, Vagrant, Puthpet, Python, Bash, Go, Go, Go, Github Actions, Circleci, Jenkinsfile, Makefile, Docompope 등
Kubernetes 구성 -Kubernetes Yaml 프로덕션 모범 사례, 팁 및 트릭을 포함한 대부분의 일반적인 시나리오에 대한 구성

도트 구성

최상위 수준의 도트 파일 및 configs/ 디렉토리 :

.* - 많은 공통 소프트웨어를위한 DOT Conf 파일 예를 들어. 고급 .vimrc , .gitconfig , massive .gitignore , .editorconfig , .screenrc , .tmux.conf 등
- .vimrc 많은 멋진 VIM 조정과 Python, Perl, Bash / Shell, Dockerfiles, JSON, YAML, XML, CSV, INI / Properties 파일, LDAP LDIF 등을 포함하여 다양한 파일 유형을 줄이기위한 핫키가 포함되어 있습니다.
- .screenrc 고급 컬러 바, 대형 기록, 핫키 재 장전, 자동 블랭킹 등을 포함한 멋진 화면 구성
- .tmux.conf 멋진 Tmux 구성이 고급 컬러 바 및 플러그인, 설정, 핫키 재 장전 등이 포함됩니다.
- git :
  - .gitconfig 고급 git 구성
  - .gitignore 광범위한 git는 당신이 저 지르지 말아야 할 사소한 파일을 무시합니다.
  - 향상된 git diff
  - AWS Secret Keys 커밋에 대한 보호 또는 충돌 처리되지 않은 파일 병합

배쉬 환경 및 라이브러리

최상위 .bashrc 및 .bash.d/ directory :

.bashrc .bash.d/*.sh 의 쉘 튜닝 및 소싱
.bash.d/*.sh 수천 개의 고급 Bashrc 코드, 별명, 기능 및 환경 변수 :
- Linux & Mac
- SCM -GIT, Mercurial, Svn
- AWS
- GCP
- 도커
- Kubernetes
- 카프카
- 방랑자
- 암호를 다시 입력하지 않고 암호화 된 개인 키를 처리하기위한 자동 GPG 및 SSH 에이전트 처리 및 게으른 평가가 처음으로 Key Load를 신속하게 평가하는 게으른 평가를 처음으로 불러옵니다.
- 더 자세한 내용 -보다 자세한 목록은 .bash.d/readme를 참조하십시오.
- run make bash to link .bashrc / .bash_profile 및 .* dot config 파일을 $HOME 디렉토리로 링크하여 모든 것을 자동 상속하기 위해.
lib/*.sh -bash 유틸리티 라이브러리 Docker, Environment, CI Detection (Travis CI, Jenkins 등), 포트 및 HTTP URL 가용성 컨텐츠 확인 등의 기능으로 가득 찬 다른 GitHub 리포지토리에서 공급되는 테스트를보다 쉽게 설정합니다.

설치 스크립트

install/install_*.sh 일반적인 기술에 사용하기 쉬운 간단한 설치 스크립트 :
- AWS CLI
- Azure Cli
- Gcloud SDK
- Github Cli
- Terraform
- Terragrunt
- Direnv
- ansible
- K3S
- Minikube (Kubernetes)
- Minishift (Redhat Openshift / OKD DEV VMS)
- Maven
- Gradle
- SBT
- Epel
- rpmforge
- 홈브류
- 트래비스 CI
- Circle CI
- 앱베이어
- BuildKite
- AVRO 도구
- 파크 도구
- 프로 메테우스
- 다양한 JDK 및 RDBMS JDBC 커넥터 항아리
- 그리고 더 많은 ...

Linux & Mac

bin/ 디렉토리 :

login.sh 주요 클라우드 플랫폼에 로그인 aws, gcp, azure, github와 같은 환경에서 자격 증명이 발견되는 경우 Dockerhub, GHCR, ECR, GCR, GAR, ACR, Gitlab, Quay ...
clean_caches.sh OS 패키지 및 프로그래밍 언어 캐시 정리 - 공간을 절약하거나 도커 이미지 크기를 줄이는 데 유용합니다.
delete_duplicate_files.sh 주어진 또는 현재 디렉토리에서 웹 브라우저 다운로드로 인해 일반적으로 (N) 접미사를 사용하여 파일을 삭제합니다. 확인 안전을 위해 정확히 동일한 체크섬이있는 (N) 접미사가없는 매칭베이스 이름 파일의 정확한 복제를 확인합니다. 파일 당 삭제하라는 프롬프트. 삭제를 자동으로 승인하려면 yes | delete_duplicate_files.sh . 이것은 ~/Downloads 디렉토리를 정리하는 빠른 방법이며 사용자 crontab을 넣을 수 있습니다.
download_url_file.sh 클로버가없고 지원을 계속하는 wget을 사용하여 URL에서 파일을 다운로드하거나, 경주 조건을 피하기 위해 원자 교체로 말입니다. github/github_download_release_file.sh , github_download_release_jar.sh 에서 사용하고 install/download_*_jar.sh
curl_auth.sh -OAuth2 / JWT API 토큰 또는 사용자 이름 및 비밀번호를 환경 변수 또는 대화식 출연 암호 프롬프트에서 자동로드하여 curl 명령을 단축하여 RAM 파일 디스크립터를 통해 대화식 출연 암호 프롬프트를 통해 명령 줄에 배치하지 않도록합니다 (프로세스 목록 또는 OS 감사 파일). 다른 인접한 API 쿼리 스크립트에서 사용합니다
find_duplicate_files*.sh 주어진 디렉토리 트리에서 크기 및/또는 체크섬별로 중복 파일을 찾습니다. 체크섬은 효율성을 위해 이미 일치하는 바이트 카운트가있는 파일에서만 수행됩니다.
find_broken_links.sh 트립 방어를 피하기 위해 지연이있는 깨진 링크 찾기
find_broken_symlinks.sh 존재하지 않는 파일/디렉토리를 가리키는 부러진 상징 링크를 찾으십시오
find_lock.sh 응용 프로그램을 열/닫아야하는 프롬프트 전후에 파일 목록의 스냅 샷을 가져 와서 주어진 또는 현재 작업 디렉토리에 잠금 장치가 사용되는지 찾으려고합니다.
http_duplicate_urls.sh 주어진 웹 페이지에서 중복 URL을 찾으십시오.
ldapsearch.sh 환경 변수에서 스위치를 추론하여 ldapsearch 명령을 단축
ldap_user_recurse.sh / ldap_group_recurse.sh 모든 부모 그룹 또는 그룹을 찾기 위해 Active Directory LDAP 사용자를 다시 찾아 모든 중첩 사용자를 찾을 수 있도록 (LDAP 통합 및 그룹 기반 허가를 디버깅하는 데 유용함)
linux_distro_versions.sh 주어진 Linux 배포판에 대한 주요 버전 목록을 빠르게 반환합니다.
diff_line_threshold.sh 두 파일과 라인 카운트 diff 임계 값을 비교하여 근본적으로 다른지 여부를 결정합니다. 단순한 업데이트가 아니라 완전히 다른 파일을 덮어 쓰는 것을 피하는 데 사용됩니다.
organize_downloads.sh $HOME/Downloads 디렉토리에서 잘 알려진 확장의 파일을 1 주일보다 오래된 파일을 유형의 자본화 된 하위 디렉터로 이동하여 $HOME/Downloads/ directory 깔끔한 상태를 유지합니다.
copy_to_clipboard.sh Linux 또는 Mac의 System Clipboard에 stdin 또는 String arg를 복사합니다.
paste_from_clipboard.sh 시스템 클립 보드에서 Linux 또는 Mac의 Stdout까지의 페이스트
paste_diff_settings.sh 클립 보드 변경 전후의 스냅 샷을 가져 와서 구성 변경을 표시하도록합니다.
processes_ram_sum.sh 주어진 regex와 일치하는 모든 프로세스의 RAM 사용량을 GB의 RAM 사용을 1 십자 자리로 합산합니다.
pldd.sh -Linux의 Parses /proc 런타임을 표시합니다 .so 클래식 정적 ldd 명령과 동일하며 System pldd 명령이 프로세스에 첨부되지 않기 때문에 런타임
random_select.sh 주어진 Args 중 하나를 무작위로 선택합니다. 대형 테스트 스위트의 샘플링, 무작위 하위 집합 실행에 유용합니다.
random_number.sh 두 정수 인수 사이의 임의 정수를 인쇄합니다 (포함)
random_string.sh 주어진 길이의 임의의 영숫자 문자열을 인쇄합니다.
shields_embed_logo.sh -base64 주어진 아이콘 파일 또는 URL을 인코딩하고 logo=... URL 매개 변수를 추가해야합니다.
shred_file.sh 민감한 정보의 복구를 방지하기 위해 삭제하기 전에 파일을 DoD 표준으로 7 번 덮어 씁니다.
shred_free_space.sh 이미 삭제 된 파일에 대한 민감한 정보의 복구를 방지하기 위해 여유 공간을 덮어 쓰기
split.sh 큰 파일을 n 부품으로 분할하여 (CPU 코어 수로 기본값) 작업을 병렬화합니다.
ssl_get_cert.sh 원격 host:port Server의 SSL Cert는 예를 들어 Java TrustStores와 같은 로컬로 파이프, 저장 및 사용할 수있는 형식으로 제공됩니다.
ssl_verify_cert.sh 원격 SSL 인증서를 확인합니다 (전투 테스트 더 많은 기능이 풍부한 버전 check_ssl_cert.pl 이 고급 Nagios 플러그인 repo에 존재합니다)
ssl_verify_cert_by_ip.sh 특정 IP 주소에서 SSL 인증서를 확인하고, End-to-End의 SSL Full-Strict 모드를 활성화하기 전에 CloudFlare 프록시 소스와 같은 CDN의 SSL 소스 주소를 테스트하는 데 유용합니다 ( curl_k8s_ingress.sh 참조).
ttygif.sh ttyrec 및 ttygif 사용하여 터미널 명령을 실행하여 GIF를 생성 한 다음 결과 GIF를 엽니 다.
asciinema.sh asciinema 및 agg 사용하여 터미널 명령을 실행하여 GIF를 생성 한 다음 결과 GIF를 엽니 다.
terminalizer.sh Terminalizer를 사용하여 터미널 명령을 실행하여 GIF를 생성 한 다음 결과 GIF를 엽니 다.
urlencode.sh / urldecode.sh -URL은 명령 줄, 파이프 등의 Encode / Decode를 빠르게 encode / decode합니다.
urlextract.sh 주어진 문자열 arg, 파일 또는 표준 입력에서 URL을 추출합니다.
urlopen.sh -arg로 주어진 URL 또는 stdin 또는 주어진 파일에서 발견 된 첫 번째 URL을 엽니 다. 시스템의 기본 브라우저를 사용합니다
vagrant_hosts.sh Vagrantfile 에서 /etc/hosts 출력을 생성합니다
vagrant_total_mb.sh Vagrantfile 에서 VM에 커밋 된 RAM 계산

Linux 및 Mac의 지식 기반 노트도 참조하십시오.

Mac & Applescript

Mac Automation Scripts를 자동화하여 Mac UI 및 설정을 자동화합니다.

bin/ 디렉토리 :

mac_diff_settings.sh UI 설정의 스냅 샷 전후에 가져 와서 defaults 을 쉽게 찾을 수 있도록 setup/mac_settings.sh 설정을 저장하기 위해 쉽게 찾을 수 있습니다.
mac_iso_to_usb.sh 주어진 ISO 파일을 USB 부팅 가능한 이미지로 변환하여 주어진 삽입 된 삽입 된 USB 드라이브로 굽습니다.
copy_to_clipboard.sh Linux 또는 Mac의 System Clipboard에 stdin 또는 String arg를 복사합니다.
paste_from_clipboard.sh 시스템 클립 보드에서 Linux 또는 Mac의 Stdout까지의 페이스트
paste_diff_settings.sh 클립 보드 변경 전후의 스냅 샷을 가져 와서 구성 변경을 표시하도록합니다.

applescript/ 디렉토리 :

keystrokes.sh n 키 스트로크 조합을 보내십시오
mouse_clicks.sh n 마우스 보내기 스크린 좌표 시퀀스로 조합을 클릭하십시오.
- get_mouse_coordinates.sh 현재 마우스 좌표 인쇄 - 위의 스크립트로 전달 해야하는 내용을 알아 보려면
mouse_clicks_remote_desktop.sh Microsoft 원격 데스크탑으로 전환하고 10 초를 기다린 다음 1 분에 한 번 마우스를 클릭하여 스크린 세이버가 켜지지 않도록합니다. 스크린 세이버를 비활성화 할 수없는 Active Directory Group 정책에 대한 해결 방법. 마우스를 마우스 클릭 효과가없는 영역으로 가리키고 CMD-TAB에서 터미널을 실행하고이를 실행하십시오.
get_frontmost_process_title.scpt 가장 앞 창을 감지하십시오
- 키 스트로크 / 마우스 클릭을 보내야하는지 감지하려면)
set_frontmost_process.scpt 주어진 앱을 전경으로 가져 오려면 키 스트로크 / 마우스 클릭을 보내도록 전환합니다.
- browser_get_default.scpt 기본 구성된 브라우저를 AppleScript (위의 스크립트)로 전달할 수있는 형식으로 기본 구성 브라우저를 가져옵니다.
is_screen_locked.py 키 스트로크 또는 마우스 클릭 전송을 중지하기 위해 화면이 잠겨 있는지 감지합니다.
is_screensaver_running.scpt 스크린 세이버가 실행 중인지 키 스트로크 또는 마우스 클릭을 중지하는지 감지합니다.
screensaver_activate.scpt 스크린 세이버를 활성화합니다

Harisekhon/Knowledge-Base의 Mac Page도 참조하십시오.

모니터링

monitoring/ 디렉토리 :

dump_stats.sh 로컬 타르 볼의 텍스트 파일에 공통 명령 출력을 덤프합니다. 공급 업체 지원 사례에 대한 지원 정보를 수집하는 데 유용합니다
grafana_api.sh 인증을 통해 Grafana API를 쿼리합니다
log_timestamp_large_intervals.sh 타임 스탬프 간격이 주어진 초를 초과하는 로그 라인을 찾아 마지막 타임 스탬프와 현재 타임 스탬프의 차이로 해당 로그 라인을 출력합니다. CI/CD 로그와 같은 로그 파일에서 오랜 시간이 걸리는 작업을 찾는 데 유용합니다.
prometheus.sh 현지에서 Prometheus를 시작하여 $PATH 에서 찾을 수없는 경우 다운로드
prometheus_docker.sh docker-compose 사용하여 Docker에서 Prometheus를 시작합니다
prometheus_node_exporter.sh 현지에서 Prometheus node_exporter 시작하여 $PATH 에서 찾을 수없는 경우 다운로드
ssh_dump_stats.sh ssh 및 dump_stats.sh 사용하여 원격 서버에서 로컬 Tarball로 공통 명령 출력을 덤프합니다. 공급 업체 지원 사례에 유용합니다
ssh_dump_logs.sh SSH를 사용하여 공급 업체 지원 케이스에 업로드하기 위해 서버에서 로컬 텍스트 파일로 로그를 덤프합니다.

Grafana, Prometheus, OpentsDB, InfluxDB 등의 Harisekhon/Knowledge-Base의 DOC 페이지를 참조하십시오.

데이터베이스

mysql/ , postgres/ , sql/ 및 bin/ directories :

PostgreSQL, MySQL, Google BigQuery 및 AWS Athena CloudTrail Logs 통합 용 SQL/ -100+ SQL 스크립트
sqlite.sh -One -Touch sqlite, SMLITE3 쉘을 샘플 'Chinook'데이터베이스로드하여 시작합니다.
mysql*.sh -mysql 스크립트 :
- mysql.sh - $MYSQL_TCP_PORT mysql $DBI_USER , $MYSQL_PWD (doc 참조) 및 $ mysql_host / $ host, $MYSQL_HOST mysql_user / $ mysql_paser, $HOST , $MYSQL_USER / $USER 다른 일반적인 환경 변수와 같은 표준 환경 변수의 자동 구성 스위치를 통해 mysql 명령을 $MYSQL_PASSWORD $PASSWORD . $MYSQL_DATABASE / $DATABASE
- mysql_foreach_table.sh 각 반복에서 {db} 및 {table} 대체하여 모든 테이블에 대해 SQL 쿼리를 실행합니다. select count(*) from {table}
- mysql_*.sh 행 카운트에 mysql.sh 사용하는 다양한 스크립트, 각 테이블 반복 또는 빠른 스크립팅을위한 깨끗한 데이터베이스 및 테이블을 출력합니다.
- mysqld.sh -one -touch mysql, boots docker container + mysql 쉘에 떨어지고, /sql 스크립트가 컨테이너에 장착되어 쉽게 소싱 할 수 있습니다. source /sql/<name>.sql .sql. 선택적으로 샘플 '치누크'데이터베이스를로드합니다
- 더 많은 직접 MySQL SQL 스크립트에 대한 SQL 스크립트 Repo 참조
mariadb.sh -one -touch mariadb, 부츠 Docker 컨테이너 + mysql 쉘에 떨어지고 /sql 스크립트가 컨테이너에 장착되어 쉽게 소싱 할 수 있습니다. source /sql/<name>.sql .sql. 선택적으로 샘플 '치누크'데이터베이스를로드합니다
postgres*.sh / psql.sh PostgreSQL 스크립트 :
- postgres.sh -One -Touch Postgresql, 부츠 Docker 컨테이너 + psql 쉘에 떨어지고 /sql 스크립트가 컨테이너에 장착되어 쉽게 소싱 할 수 있습니다. i /sql/<name>.sql . 선택적으로 샘플 '치누크'데이터베이스를로드합니다
- psql.sh psql 명령은 $PG* (문서 참조)뿐만 아니라 $ $POSTGRESQL_HOST / $ postgres_host / $ host, $ postgresql_user / $ postgreser / $ postgreser / $POSTGRES_HOST / $ postgreser / $ postgreser / $ postgreser / $ postgreser / $HOST postgreser / $POSTGRESQL_USER / $POSTGRES_USER / $USER / $POSTGRESQL_PASSWORD $POSTGRES_PASSWORD / $PASSWORD , $POSTGRESQL_DATABASE / $POSTGRES_DATABASE / $DATABASE
- postgres_foreach_table.sh 각 반복에서 {db} , {schema} 및 {table} 대체하여 모든 테이블에 대해 SQL 쿼리를 실행합니다. select count(*) from {table}
- postgres_*.sh 행 카운트에 psql.sh 사용하는 다양한 스크립트, 각 테이블 반복 또는 Quick Scripting을위한 깨끗한 데이터베이스 목록, 스키마 및 테이블을 출력합니다.
- checks/check_sqlfluff.sh 주어진 또는 현재 디렉토리에있는 모든 SQL 코드 파일을 재귀 적으로 반복하고 각 경로/파일 이름/확장자의 다른 SQL 방언을 추론하여 SQLFluff Linter를 실행합니다.

AWS- 아마존 웹 서비스

aws/ 디렉토리 :

AWS 스크립트 -AWS aws_*.sh :
- aws_profile.sh ARG로 제공된 AWS 프로파일로 전환하거나 선택할 수있는 편리한 대화식 메뉴 목록으로 사용자에게 프롬프트 - AWS 작업 프로파일이 많을 때 유용합니다.
- aws_cli_create_credential.sh 관리자 권한 (또는 기타 그룹 또는 정책)을 사용하여 CI/CD 또는 CLI 용 AWS 서비스 계정 사용자를 생성하고 AWS 액세스 키를 생성하고 자격 증명 CSV를 저장하고 심지어 쉘 내보내기 명령 및 AWS 자격 증명 파일 구성을 인쇄하여 환경을 사용하도록 구성합니다. CLI Reauth에 aws sso login 매일 피하는 유용한 트릭.
- aws_terraform_create_credential.sh CI/CD 시스템이 AWS SSO 워크 플로우에서 작동 할 수 없기 때문에 TerraForm Cloud 또는 기타 CI/CD 시스템에 대한 관리자 권한이있는 AWS TerraForm 서비스 계정을 만듭니다. Access 키를 CSV 및 인쇄 쉘 내보내기 명령 및 자격 증명 파일 구성으로 저장합니다.
- .envrc-aws AWS 프로파일, 컴퓨팅 영역, EKS 클러스터 kubectl 컨텍스트 등과 같은 AWS 구성 설정을 자동 으로하기 위해 Direnv에 대한 .envrc 로 복사하십시오.
  - 전화 .envrc-kubernetes 전류 쉘로 분리 된 kubectl 컨텍스트를 설정하여 전 세계를 순진하게 변경하여 쉘과 스크립트 사이의 경주 조건을 방지하기 위해 전류 쉘로 분리 된 ~/.kube/config 컨텍스트를 설정합니다.
- aws_sso_ssh.sh 로컬 AWS SSO 인증 팝업 (아직 인증되지 않은 경우)을 시작한 다음 SCP의 최신 결과 ~/.aws/sso/cache/ 파일에 대한 원격 서버 및 SSH가있는 SSH가 서버에서 쉽게 사용하여 SSH를 쉽게 사용할 수 있습니다.
- aws_terraform_create_s3_bucket.sh 백엔드 상태를 저장하고 공개 액세스를 잠그고, 공개 액세스를 잠그고, 버전 작성, 암호화를 활성화하고, 안전을 위해 주어진 사용자/그룹/역할 ARN을 잠그기위한 Terraform S3 버킷을 생성합니다.
- aws_terraform_create_dynamodb_table.sh s3 백엔드와 함께 사용할 수 있도록 dynamodb에서 Terraform 잠금 테이블을 생성하고 권한이없는 계정에 적용 할 수있는 사용자 정의 정책을 만듭니다.
- aws_terraform_create_all.sh 위의 모든 내용을 실행하고, 사용자 정의 DynamoDB IAM 정책을 사용자에게 적용하여 계정이 덜 특권이 있는지 확인하는 경우에도 여전히 테라 양식 잠금을 얻을 수 있습니다 (GitHub 조치 환경에 유용하여 독서 한 사용자가 승인을 필요로하지 않고 풀 요청에서 TerraForf 계획을 생성 할 수 있도록하십시오.
- aws_terraform_iam_grant_s3_dynamodb.sh 이름에 terraform-state 또는 tf-state 가있는 S3 버킷 및 DynamoDB 테이블에 액세스하여 주어진 사용자에게 첨부하는 IAM 정책을 생성합니다. 제한된 권한에 유용하여 Terraform Plan Eg를 실행하는 CI/CD 계정. GitHub 작업에서 요청을 가져옵니다
- aws_account_summary.sh key = value 쌍의 AWS AccountAccessKeysPresent check_aws_root_account.py 은 AccountMFAEnabled
- aws_billing_alarm.sh 주어진 임계 값 이상의 요금을 부과 할 때 이메일을 구독하여 CloudWatch 청구 경보 및 SNS 주제를 만듭니다. 이것은 종종 계정에서 가장 먼저하고 싶은 일입니다.
- aws_budget_alarm.sh 예산의 80% 이상의 예측 요금이 90%, 실제 사용량이 90%로 발생할 때 이메일을 보내기 위해 AWS 예산 청구 경보 및 SNS 주제를 만듭니다. 이것은 종종 계정에서 가장 먼저하고 싶은 일입니다.
- aws_batch_stale_jobs.sh 주어진 대기열에서 N 시간보다 오래된 AWS 배치 작업을 나열합니다.
- aws_batch_kill_stale_jobs.sh 주어진 대기열에서 N 시간보다 오래된 AWS 배치 작업을 찾아 죽여
- aws_cloudfront_distribution_for_origin.sh 주어진 기판을 포함하는 기원을 제공하는 분포의 AWS CloudFront ARN을 반환합니다. Cloudfront를 통해 노출 된 개인 S3 버킷에 권한을 부여하는 데 필요한 Cloudfront ARN을 빠르게 찾는 데 유용합니다.
- aws_cloudtrails_cloudwatch.sh 클라우드 트레일과 CloudWatch 로그로의 마지막 전달을 나열합니다 (최근이어야 함)
- aws_cloudtrails_event_selectors.sh 클라우드 트레일 및 이벤트 선택기가 각각 하나 이상의 이벤트 선택기를 확인하도록 목록에 나열
- aws_cloudtrails_s3_accesslogging.sh 클라우드 트레일 버킷과 액세스 로깅 접두사 및 대상 버킷을 나열합니다. 확인 S3 액세스 로깅이 활성화되어 있습니다
- aws_cloudtrails_s3_kms.sh 클라우드 트레일을 나열하고 S3 버킷이 KMS 보안인지 여부
- aws_cloudtrails_status.sh 클라우드 트레일 상태를 나열합니다 - 로깅, 다중 지역 및 로그 파일 유효성 검증 활성화
- aws_config_all_types.sh AWS 구성 레코더를 나열하고 모든 리소스 유형을 확인하는 것이 지원되고 (사실이어야 함) 글로벌 리소스를 포함합니다 (사실이어야 함)
- aws_config_recording.sh AWS 구성 레코더, 레코딩 상태 (사실이어야 함) 및 마지막 상태 (성공이어야 함)를 나열합니다.
- aws_csv_creds.sh CSV 파일에서 쉘 내보내기 명령문으로 AWS 자격 증명을 인쇄합니다. API (예 : jenkins_cred_add*.sh , github_actions_repo*_set_secret.sh , gitlab_*_set_env_vars.sh , circleci, circleci_*_set_env_vars.sh . bitbucket_*_set_env_vars.sh , terraform_cloud_*_set_vars.sh , kubectl_kv_to_secret.sh ). 새 사용자 및 새로운 액세스 키 CSV 파일 형식을 지원합니다.
- aws_codecommit_csv_creds.sh CSV 파일의 쉘 내보내기 명령문으로 AWS CodeCommit git 자격 증명을 인쇄합니다. 위와 같은 유사한 사용 사례 및 체인
- aws_ec2_instance_name_to_id.sh 추가 안전 점검이있는 인스턴스 ID 만 반환하고 해당 인스턴스 ID에서 리버스 조회가 이름과 일치하는 추가 안전 확인이있는 인스턴스 이름에서 EC2 인스턴스 ID를 찾습니다. 인스턴스 ID가 전달되면 편의를 위해 그대로 반환합니다. 인접한 스크립트에서 사용합니다
- aws_ec2_instances.sh AWS EC2 인스턴스, DNS 이름 및 상태를 읽기 쉬운 테이블 출력에 나열합니다.
- aws_ec2_terminate_instance_by_name.sh 이름으로 AWS EC2 인스턴스를 종료합니다
- aws_ec2_create_ami_from_instance.sh EC2 인스턴스에서 AWS EC2 AMI를 생성하고 사용할 수 있기를 기다립니다.
- aws_ec2_clone_instance.sh 원본에서 AMI를 작성한 다음 원본 인스턴스와 동일한 설정으로 AMI에서 새 인스턴스를 부팅하여 AWS EC2 인스턴스를 복제합니다. Tableau의 서버 관리자 복구와 같은 별도의 EC2 인스턴스에서 위험한 것을 테스트하는 데 유용합니다.
- aws_ec2_amis.sh 읽기 쉬운 테이블 출력에서 귀하의 계정에 속하는 AWS EC2 AMI 목록
- aws_ec2_ami_ids.sh 매핑 테이블을 작성하고 AMI IDS를 재고 스크립트의 이름으로 변환하는 인접한 스크립트에서 사용될 AWS EC2 AMI IDS 만 라인 당 하나만 나열합니다. aws_info_ec2*.sh
- aws_ec2_ebs_*.sh -AWS EC2 EBS 스크립트 :
  - aws_ec2_ebs_volumes.sh 현재 영역의 EC2 인스턴스 및 EBS 볼륨을 나열하십시오.
  - `aws_ec2_ebs_create_snapshot_and_wait.sh- 주어진 EBS 볼륨 ID의 스냅 샷을 생성하고 지수 백 오프로 완료되기를 기다립니다.
  - `aws_ec2_ebs_resize_and_wait.sh- EBS 볼륨을 크기를 조정하고 지수 백 오프로 수정하고 선택적으로 최적화하기를 기다립니다.
  - aws_ec2_ebs_volumes_unattached.sh 테이블 형식으로 부착되지 않은 EBS 볼륨을 나열하십시오.
- aws_ecr_*.sh -AWS ECR DOCKER 이미지 관리 스크립트 :
  - aws_ecr_docker_login.sh Docker를 AWS ECR로 인증하여 현재 AWS 계정 ID 및 지역에서 ECR 레지스트리를 추론합니다.
  - aws_ecr_docker_build_push.sh Docker 이미지를 작성하고 latest Docker 태그뿐만 아니라 현재 Git Hashref 및 Git 태그로 ECR로 푸시합니다.
  - aws_ecr_list_repos.sh ECR Repos를 나열하고 Docker Image Mutability 및 이미지 스캔이 활성화되어 있는지 여부
  - aws_ecr_list_tags.sh 주어진 ECR Docker 이미지에 대한 모든 태그를 나열합니다.
  - aws_ecr_newest_image_tags.sh 최신 생성 날짜가있는 주어진 ECR Docker Image의 태그를 나열합니다 (이를 사용하여 latest 태그로 표시 할 이미지 버전을 결정할 수 있음)
  - aws_ecr_alternate_tags.sh 주어진 ECR Docker 이미지 <image>:latest 대한 모든 태그를 latest 합니다 image:tag
  - aws_ecr_tag_image.sh 그것을 잡아 당기지 않고 다른 태그로 ECR 이미지를 태그
  - aws_ecr_tag_image_by_digest.sh 위와 동일하지만 Digest를 통해 찾은 ECR 이미지를 태그 (기존 태그의 참조로 더 정확한 태그가 이동 대상이 될 수 있음). 타협되지 않은 이미지를 복구하는 데 유용합니다
  - aws_ecr_tag_latest.sh 주어진 ECR Docker image:tag Docker 이미지를 당기거나 밀지 않고 latest 으로 태그
  - aws_ecr_tag_branch.sh 주어진 ECR image:tag
  - aws_ecr_tag_datetime.sh Docker Image를 당기거나 밀지 않고 생성 날짜 및 UTC 타임 스탬프 (ECR에 업로드 된 경우)가있는 주어진 ECR Docker 이미지를 태그
  - aws_ecr_tag_newest_image_as_latest.sh Docker Image를 당기거나 밀지 않고 latest ECR Docker Image의 최신 빌드를 찾아서 태그합니다.
  - aws_ecr_tags_timestamps.sh 주어진 ECR Docker Image의 모든 태그와 타임 스탬프를 나열합니다.
  - aws_ecr_tags_old.sh 주어진 ECR Docker Image의 N Days보다 오래된 태그를 나열합니다.
  - aws_ecr_delete_old_tags.sh 주어진 ECR Docker 이미지에 대해 N 일보다 오래된 태그를 삭제합니다. 이미지 나열 : 삭제할 태그 및 확인 안전을위한 프롬프트
- aws_foreach_profile.sh AWS CLIV2로 구성된 모든 AWS에서 템플릿 명령을 실행하여 각 반복에서 {profile} 대체합니다. 강력한 기능, 감사, 설정 등을 위해 다른 스크립트와 결합하십시오. aws_kube_creds.sh 모든 환경에서 모든 EKS 클러스터에 대한 kubectl 구성 구성
- aws_foreach_region.sh 각 반복에서 {region} 대체하여 현재 계정에 활성화 된 각 AWS 영역에 대해 템플릿 명령을 실행합니다. AWS CLI 또는 스크립트와 결합하여 지역에서 자원을 찾으십시오.
- aws_iam_*.sh -AWS IAM 스크립트 :
  - aws_iam_password_policy.sh key = value 쌍 ( aws_harden_password_policy.sh 에 의해 사용).
  - aws_iam_harden_password_policy.sh CIS 기초에 따라 AWS 비밀번호 정책 강화 벤치 마크 권장 사항
  - aws_iam_replace_access_key.sh - replaces the non-current IAM access key (Inactive, Not Used, longer time since used, or an explicitly given key), outputting the new key as shell export statements (useful for piping to the same tools listed for aws_csv_creds.sh above)
  - aws_iam_policies_attached_to_users.sh - finds AWS IAM policies directly attached to users (anti-best practice) instead of groups
  - aws_iam_policies_granting_full_access.sh - finds AWS IAM policies granting full access (anti-best practice)
  - aws_iam_policies_unattached.sh - lists unattached AWS IAM policies
  - aws_iam_policy_attachments.sh - finds all users, groups and roles where a given IAM policy is attached, so that you can remove all these references in your Terraform code and avoid this error Error: error deleting IAM policy arn:aws:iam::***:policy/mypolicy: DeleteConflict: Cannot delete a policy attached to entities.
  - aws_iam_policy_delete.sh - deletes an IAM policy, by first handling all prerequisite steps of deleting all prior versions and all detaching all users, groups and roles
  - aws_iam_generate_credentials_report_wait.sh - generates an AWS IAM credentials report
  - aws_iam_users.sh - list your IAM users
  - aws_iam_users_access_key_age.sh - prints AWS users access key status and age (see also aws_users_access_key_age.py in DevOps Python tools which can filter by age and status)
  - aws_iam_users_access_key_age_report.sh - prints AWS users access key status and age using a bulk credentials report (faster for many users)
  - aws_iam_users_access_key_last_used.sh - prints AWS users access keys last used date
  - aws_iam_users_access_key_last_used_report.sh - same as above using bulk credentials report (faster for many users)
  - aws_iam_users_last_used_report.sh - lists AWS users password/access keys last used dates
  - aws_iam_users_mfa_active_report.sh - lists AWS users password enabled and MFA enabled status
  - aws_iam_users_without_mfa.sh - lists AWS users with password enabled but no MFA
  - aws_iam_users_mfa_serials.sh - lists AWS users MFA serial numbers (differentiates Virtual vs Hardware MFAs)
  - aws_iam_users_pw_last_used.sh - lists AWS users and their password last used date
- aws_ip_ranges.sh - get all AWS IP ranges for a given Region and/or Service using the IP range API
- aws_info*.sh :
  - aws_info_all_profiles.sh - calls aws_info.sh for all AWS profiles using aws_foreach_profile.sh
    - aws_info.sh - lists AWS deployed resources in the current or specified AWS account profile
      - aws_info_ec2.sh - lists AWS EC2 Instances resources deployed in the current AWS account
  - aws_info_ec2_csv.sh - lists AWS EC2 Instances in quoted CSV format in the current AWS account
  - aws_info_ec2_all_profiles_csv.sh - lists AWS EC2 Instances in quoted CSV format across all configured AWS profiles for their configured region
- aws_eks_cloudwatch_logs.sh - enables and fetches AWS EKS Master logs via CloudWatch
- aws_eks_ssh_dump_logs.sh - fetch system logs from EKS Worker Nodes EC2 VMs (eg. for support debug requests by vendors)
- aws_kms_key_rotation_enabled.sh - lists AWS KMS keys and whether they have key rotation enabled
- aws_kube_creds.sh - auto-loads all AWS EKS clusters credentials in the current --profile and --region so your kubectl is ready to rock on AWS
- aws_kubectl.sh - runs kubectl commands safely fixed to a given AWS EKS cluster using config isolation to avoid concurrency race conditions
- aws_logs_*.sh - some useful log queries in last N hours (24 hours by default):
  - aws_logs_batch_jobs.sh - lists AWS Batch job submission requests and their callers
  - aws_logs_ec2_spot.sh - lists AWS EC2 Spot fleet creation requests, their caller and first tag value for origin hint
  - aws_logs_ecs_tasks.sh - lists AWS ECS task run requests, their callers and job definitions
- aws_meta.sh - AWS EC2 Metadata API query shortcut. See also the official ec2-metadata shell script with more features
- aws_nat_gateways_public_ips.sh - lists the public IPs of all NAT gateways. Useful to give to clients to permit through firewalls for webhooks or similar calls
- aws_rds_list.sh - list RDS instances with select fields - Name, Status, Engine, AZ, Instance Type, Storage
- aws_rds_open_port_to_my_ip.sh - adds a security group to an RDS DB instance to open its native database SQL port to your public IP address
- aws_rds_get_version.sh - quickly retrieve the version of an RDS database to know which JDBC jar version to download using install/download_*_jdbc.sh when setting up connections
- aws_route53_check_ns_records.sh - checks AWS Route 53 public hosted zones NS servers are delegated in the public DNS hierarchy and that there are no rogue NS servers delegated not matching the Route 53 zone configuration
- aws_sso_accounts.sh - lists all AWS SSO accounts the current SSO user has access to
- aws_sso_configs.sh - generates AWS SSO configs for all AWS SSO accounts the currently logged in SSO user has access to
- aws_sso_configs_save.sh - saves AWS SSO configs generated by aws_sso_configs.sh to ~/.aws/config if they're not already found
- aws_sso_config_duplicate_sections.sh - lists duplicate AWS SSO config sections that are using the same sso_account_id. Useful to deduplicate configs containing a mix of hand crafted and automatically generated aws_sso_configs.sh
- aws_sso_config_duplicate_profile_names.sh - lists duplicate AWS SSO config profile names that are using the same sso_account_id
- aws_sso_env_creds.sh - retrieves AWS SSO session credentials in the format of environment export commands for copying to other systems like Terraform Cloud
- aws_sso_role_arn.sh - prints the currently authenticated AWS SSO user's role ARN in IAM policy usable format
- aws_sso_role_arns.sh - prints all AWS SSO role ARNs in IAM policy usable format
- aws_profile_config_add_if_missing.sh - reads AWS profile config blocks from stdin and appends them to the ~/.aws/config file if the profile section is not found
- aws_profile_generate_direnvs.sh - generates subdirectories containing the config.ini and .envrc for every AWS profile found in the given file or $AWS_CONFIG_FILE or ~/.aws/config . Useful to take a large generated AWS config.ini from aws_sso_configs.sh and then split it into subdirectories for direnvs
- aws_s3_bucket.sh - creates an S3 bucket, blocks public access, enables versioning, encryption, and optionally locks out any given user/group/role ARNs via a bucket policy for safety (eg. to stop Power Users accessing a sensitive bucket like Terraform state)
- aws_s3_buckets_block_public_access.sh - blocks public access to one or more given S3 buckets or files containing bucket names, one per line
- aws_s3_account_block_public_access.sh - blocks S3 public access at the AWS account level
- aws_s3_check_buckets_public_blocked.sh - iterates each S3 bucket and checks it has public access fully blocked via policy. Parallelized for speedup
- aws_s3_check_account_public_blocked.sh - checks S3 public access is blocked at the AWS account level
- aws_s3_sync.sh - syncs multiple AWS S3 URLs from file lists. Validates S3 URLs, source and destination list lengths matches, and optionally that path suffixes match, to prevent off-by-one human errors spraying data all over the wrong destination paths
- aws_s3_access_logging.sh - lists AWS S3 buckets and their access logging status
- aws_s3_delete_bucket_with_versions.sh - deletes a bucket including all versions. Use with caution!
- aws_spot_when_terminated.sh - executes commands when the AWS EC2 instance running this script is notified of Spot Termination, acts as a latch mechanism that can be set any time after boot
- aws_sqs_check.sh - sends a test message to an AWS SQS queue, retrieves it to check and then deletes it via the receipt handle id
- aws_sqs_delete_message.sh - deletes 1-10 messages from a given AWS SQS queue (to help clear out test messages)
- aws_ssm_put_param.sh - reads a value from a command line argument or non-echo prompt and saves it to AWS Systems Manager Parameter Store. Useful for uploading a password without exposing it on your screen
- aws_secret*.sh - AWS Secrets Manager scripts:
  - aws_secret_list.sh - returns the list of secrets, one per line
  - aws_secret_add.sh - reads a value from a command line argument or non-echo prompt and saves it to Secrets Manager. Useful for uploading a password without exposing it on your screen
  - aws_secret_add_binary.sh - base64 encodes a given file's contents and saves it to Secrets Manager as a binary secret. Useful for uploading things like QR code screenshots for sharing MFA to recovery admin accounts
  - aws_secret_update.sh - reads a value from a command line argument or non-echo prompt and updates a given Secrets Manager secret. Useful for updating a password without exposing it on your screen
  - aws_secret_update_binary.sh - base64 encodes a given file's contents and updates a given Secrets Manager secret. Useful for updating a QR code screenshot for a root account
  - aws_secret_get.sh - gets a secret value for a given secret from Secrets Manager, retrieving either a secure string or secure binary depending on which is available
- eksctl_cluster.sh - downloads eksctl and creates an AWS EKS Kubernetes cluster

GCP - Google Cloud Platform

gcp/ directory:

Google Cloud scripts - gcp_*.sh / gce_*.sh / gke_*.sh / gcr_*.sh / bigquery_*.sh :
- .envrc-gcp - copy to .envrc for direnv to auto-load GCP configuration settings such as Project, Region, Zone, GKE cluster kubectl context or any other GCloud SDK settings to shorten gcloud commands. Applies to the local shell environment only to avoid race conditions caused by naively changing the global gcloud config at ~/.config/gcloud/active_config
  - calls .envrc-kubernetes to set the kubectl context isolated to current shell to prevent race conditions between shells and scripts caused by otherwise naively changing the global ~/.kube/config context
- gcp_terraform_create_credential.sh - creates a service account for Terraform with full permissions, creates and downloads a credential key json and even prints the export GOOGLE_CREDENTIALS command to configure your environment to start using Terraform immediately. Run once for each project and combine with direnv for fast easy management of multiple GCP projects
- gcp_ansible_create_credential.sh - creates an Ansible service account with permissions on the current project, creates and downloads a credential key json and prints the environment variable to immediately use it
- gcp_cli_create_credential.sh - creates a GCloud SDK CLI service account with full owner permissions to all projects, creates and downloads a credential key json and even prints the export GOOGLE_CREDENTIALS command to configure your environment to start using it. Avoids having to reauth to gcloud auth login every day.
- gcp_spinnaker_create_credential.sh - creates a Spinnaker service account with permissions on the current project, creates and downloads a credential key json and even prints the Halyard CLI configuration commands to use it
- gcp_info.sh - huge Google Cloud inventory of deployed resources within the current project - Cloud SDK info plus all of the following (detects which services are enabled to query):
  - gcp_info_compute.sh - GCE Virtual Machine instances, App Engine instances, Cloud Functions, GKE clusters, all Kubernetes objects across all GKE clusters (see kubernetes_info.sh below for more details)
  - gcp_info_storage.sh - Cloud SQL info below, plus: Cloud Storage Buckets, Cloud Filestore, Cloud Memorystore Redis, BigTable clusters and instances, Datastore indexes
  - gcp_info_cloud_sql.sh - Cloud SQL instances, whether their backups are enabled, and all databases on each instance
    - gcp_info_cloud_sql_databases.sh - lists databases inside each Cloud SQL instance. Included in gcp_info_cloud_sql.sh
    - gcp_info_cloud_sql_backups.sh - lists backups for each Cloud SQL instance with their dates and status. Not included in gcp_info_cloud_sql.sh for brevity. See also gcp_sql_export.sh further down for more durable backups to GCS
    - gcp_info_cloud_sql_users.sh - lists users for each running Cloud SQL instance. Not included in gcp_info_cloud_sql.sh for brevity but useful to audit users
  - gcp_info_networking.sh - VPC Networks, Addresses, Proxies, Subnets, Routers, Routes, VPN Gateways, VPN Tunnels, Reservations, Firewall rules, Forwarding rules, Cloud DNS managed zones and verified domains
  - gcp_info_bigdata.sh - Dataproc clusters and jobs in all regions, Dataflow jobs in all regions, PubSub messaging topics, Cloud IOT registries in all regions
  - gcp_info_tools.sh - Cloud Source Repositories, Cloud Builds, Container Registry images across all major repos ( gcr.io , us.gcr.io , eu.gcr.io , asia.gcr.io ), Deployment Manager deployments
  - gcp_info_auth_config.sh - Auth Configurations, Organizations & Current Config
  - gcp_info_projects.sh - Projects names and IDs
  - gcp_info_services.sh - Services & APIs enabled
    - gcp_service_apis.sh - lists all available GCP Services, APIs and their states (enabled/disabled), and provides is_service_enabled() function used throughout the adjacent scripts to avoid errors and only show relevant enabled services
  - gcp_info_accounts_secrets.sh - IAM Service Accounts, Secret Manager secrets
- gcp_info_all_projects.sh - same as above but for all detected projects
- gcp_foreach_project.sh - executes a templated command across all GCP projects, replacing {project_id} and {project_name} in each iteration (used by gcp_info_all_projects.sh to call gcp_info.sh )
- gcp_find_orphaned_disks.sh - lists orphaned disks across one or more GCP projects (not attached to any compute instance)
- gcp_secret*.sh - Google Secret Manager scripts:
  - gcp_secret_add.sh - reads a value from a command line argument or non-echo prompt and saves it to GCP Secrets Manager. Useful for uploading a password without exposing it on your screen
  - gcp_secret_add_binary.sh - uploads a binary file to GCP Secrets Manager by base64 encoding it first. Useful for uploading QR code screenshots. Useful for uploading things like QR code screenshots for sharing MFA to recovery admin accounts
  - gcp_secret_update.sh - reads a value from a command line argument or non-echo prompt and updates a given GCP Secrets Manager secret. Useful for uploading a password without exposing it on your screen
  - gcp_secret_get.sh - finds the latest version of a given GCP Secret Manager secret and returns its value. Used by adjacent scripts
  - gcp_secret_label_k8s.sh - labels a given existing GCP secret with the current kubectl cluster name and namespace for later use by gcp_secrets_to_kubernetes.sh
  - gcp_secrets_to_kubernetes.sh - loads GCP secrets to Kubernetes secrets in a 1-to-1 mapping. Can specify a list of secrets or auto-loads all GCP secrets with labels kubernetes-cluster and kubernetes-namespace matching the current kubectl context ( kcd to the right namespace first, see .bash.d/kubernetes ). See also kubernetes_get_secret_values.sh to debug the actual values that got loaded. See also Sealed Secrets / External Secrets in my Kubernetes repo
  - gcp_secrets_to_kubernetes_multipart.sh - creates a Kubernetes secret from multiple GCP secrets (used to put private.pem and public.pem into the same secret to appear as files on volume mounts for apps in pods to use). See also Sealed Secrets / External Secrets in my Kubernetes repo
  - gcp_secrets_labels.sh - lists GCP Secrets and their labels, one per line suitable for quick views or shell pipelines
  - gcp_secrets_update_lable.sh - updates all GCP secrets in current project matching label key=value with a new label value
  - gcp_service_account_credential_to_secret.sh - creates GCP service account and exports a credential key to GCP Secret Manager (useful to stage or combine with gcp_secrets_to_kubernetes.sh )
- gke_*.sh - Google Kubernetes Engine scripts
  - gke_kube_creds.sh - auto-loads all GKE clusters credentials in the current / given / all projects so your kubectl is ready to rock on GCP
  - gke_kubectl.sh - runs kubectl commands safely fixed to a given GKE cluster using config isolation to avoid concurrency race conditions
  - gke_firewall_rule_cert_manager.sh - creates a GCP firewall rule for a given GKE cluster's masters to access Cert Manager admission webhook (auto-determines the master cidr, network and target tags)
  - gke_firewall_rule_kubeseal.sh - creates a GCP firewall rule for a given GKE cluster's masters to access Sealed Secrets controller for kubeseal to work (auto-determines the master cidr, network and target tags)
  - gke_nodepool_nodes.sh - lists all nodes in a given nodepool on the current GKE cluster via kubectl labels (fast)
  - gke_nodepool_nodes2.sh - same as above via GCloud SDK (slow, iterates instance groups)
  - gke_nodepool_taint.sh - taints/untaints all nodes in a given GKE nodepool on the current cluster (see kubectl_node_taints.sh for a quick way to see taints)
  - gke_nodepool_drain.sh - drains all nodes in a given nodepool (to decommission or rebuild the node pool, for example with different taints)
  - gke_persistent_volumes_disk_mappings.sh - lists GKE kubernetes persistent volumes to GCP persistent disk names, along with PVC and namespace, useful when investigating, resizing PVs etc.
- gcr_*.sh - Google Container Registry scripts:
  - gcr_list_tags.sh - lists all the tags for a given GCR docker image
  - gcr_newest_image_tags.sh - lists the tags for the given GCR docker image with the newest creation date (can use this to determine which image version to tag as latest )
  - gcr_alternate_tags.sh - lists all the tags for a given GCR docker image:tag (use arg <image>:latest to see what version / build hashref / date tag has been tagged as latest )
  - gcr_tag_latest.sh - tags a given GCR docker image:tag as latest without pulling or pushing the docker image
  - gcr_tag_branch.sh - tags a given GCR docker image:tag with the current Git branch without pulling or pushing the docker image
  - gcr_tag_datetime.sh - tags a given GCR docker image with its creation date and UTC timestamp (when it was uploaded or created by Google Cloud Build) without pulling or pushing the docker image
  - gcr_tag_newest_image_as_latest.sh - finds and tags the newest build of a given GCR docker image as latest without pulling or pushing the docker image
  - gcr_tags_timestamps.sh - lists all the tags and their timestamps for a given GCR docker image
  - gcr_tags_old.sh - lists tags older than N days for a given GCR docker image
  - gcr_delete_old_tags.sh - deletes tags older than N days for a given GCR docker image. Lists the image:tags to be deleted and prompts for confirmation safety
  - see also cloudbuild.yaml in the Templates repo
- CI/CD on GCP - trigger Google Cloud Build and GKE Kubernetes deployments from orthogonal CI/CD systems like Jenkins / TeamCity:
  - gcp_ci_build.sh - script template for CI/CD to trigger Google Cloud Build to build docker container image with extra datetime and latest tagging
  - gcp_ci_deploy_k8s.sh - script template for CI/CD to deploy GCR docker image to GKE Kubernetes using Kustomize
- gce_*.sh - Google Compute Engine scripts:
  - gce_foreach_vm.sh - run a command for each GCP VM instance matching the given name/ip regex in the current GCP project
  - gce_host_ips.sh - prints the IPs and hostnames of all or a regex match of GCE VMs for use in /etc/hosts
  - gce_ssh.sh - Runs gcloud compute ssh to a VM while auto-determining its zone first to override any inherited zone config and make it easier to script iterating through VMs
  - gcs_ssh_keyscan.sh - SSH keyscans all the GCE VMs returned from the above gce_host_ips.sh script and adds them to ~/.ssh/known_hosts
  - gce_meta.sh - simple script to query the GCE metadata API from within Virtual Machines
  - gce_when_preempted.sh - GCE VM preemption latch script - can be executed any time to set one or more commands to execute upon preemption
  - gce_is_preempted.sh - GCE VM return true/false if preempted, callable from other scripts
  - gce_instance_service_accounts.sh - lists GCE VM instance names and their service accounts
- gcp_firewall_disable_default_rules.sh - disables those lax GCP default network "allow all" firewall rules
- gcp_firewall_risky_rules.sh - lists risky GCP firewall rules that are enabled and allow traffic from 0.0.0.0/0
- gcp_sql_*.sh - Cloud SQL scripts:
  - gcp_sql_backup.sh - creates Cloud SQL backups
  - gcp_sql_export.sh - creates Cloud SQL exports to GCS
  - gcp_sql_enable_automated_backups.sh - enable automated daily Cloud SQL backups
  - gcp_sql_enable_point_in_time_recovery.sh - enable point-in-time recovery with write-ahead logs
  - gcp_sql_proxy.sh - boots a Cloud SQL Proxy to all Cloud SQL instances for fast convenient direct psql / mysql access via local sockets. Installs Cloud SQL Proxy if necessary
  - gcp_sql_running_primaries.sh - lists primary running Cloud SQL instances
  - gcp_sql_service_accounts.sh - lists Cloud SQL instance service accounts. Useful for copying to IAM to grant permissions (eg. Storage Object Creator for SQL export backups to GCS)
  - gcp_sql_create_readonly_service_account.sh - creates a service account with read-only permissions to Cloud SQL eg. to run export backups to GCS
  - gcp_sql_grant_instances_gcs_object_creator.sh - grants minimal GCS objectCreator permission on a bucket to primary Cloud SQL instances for exports
- gcp_cloud_schedule_sql_exports.sh - creates Google Cloud Scheduler jobs to trigger a Cloud Function via PubSub to run Cloud SQL exports to GCS for all Cloud SQL instances in the current GCP project
  - the Python GCF function is in the DevOps Python tools repo
- bigquery_*.sh - BigQuery scripts:
  - bigquery_list_datasets.sh - lists BigQuery datasets in the current GCP project
  - bigquery_list_tables.sh - lists BigQuery tables in a given dataset
  - bigquery_list_tables_all_datasets.sh - lists tables for all datasets in the current GCP project
  - bigquery_foreach_dataset.sh - executes a templated command for each dataset
  - bigquery_foreach_table.sh - executes a templated command for each table in a given dataset
  - bigquery_foreach_table_all_datasets.sh - executes a templated command for each table in each dataset in the current GCP project
  - bigquery_table_row_count.sh - gets the row count for a given table
  - bigquery_tables_row_counts.sh - gets the row counts for all tables in a given dataset
  - bigquery_tables_row_counts_all_datasets.sh - gets the row counts for all tables in all datasets in the current GCP project
  - bigquery_generate_query_biggest_tables_across_datasets_by_row_count.sh - generates a BigQuery SQL query to find the top 10 biggest tables by row count
  - bigquery_generate_query_biggest_tables_across_datasets_by_size.sh - generates a BigQuery SQL query to find the top 10 biggest tables by size
  - see also the SQL Scripts repo for many more straight BigQuery SQL scripts
- GCP IAM scripts:
  - gcp_service_account*.sh :
    - gcp_service_account_credential_to_secret.sh - creates GCP service account and exports a credential key to GCP Secret Manager (useful to stage or combine with gcp_secrets_to_kubernetes.sh )
    - gcp_service_accounts_credential_keys.sh - lists all service account credential keys and expiry dates, can grep 9999-12-31T23:59:59Z to find non-expiring keys
    - gcp_service_accounts_credential_keys_age.sh - lists all service account credential keys age in days
    - gcp_service_accounts_credential_keys_expired.sh - lists expired service account credential keys that should be removed and recreated if needed
    - gcp_service_account_members.sh - lists all members and roles authorized to use any service accounts. Useful for finding GKE Workload Identity mappings
  - gcp_iam_*.sh :
    - gcp_iam_roles_in_use.sh - lists GCP IAM roles in use in the current or all projects
    - gcp_iam_identities_in_use.sh - lists GCP IAM identities (users/groups/serviceAccounts) in use in the current or all projects
    - gcp_iam_roles_granted_to_identity.sh - lists GCP IAM roles granted to identities matching the regex (users/groups/serviceAccounts) in the current or all projects
    - gcp_iam_roles_granted_too_widely.sh - lists GCP IAM roles which have been granted to allAuthenticatedUsers or even worse allUsers (unauthenticated) in one or all projects
    - gcp_iam_roles_with_direct_user_grants.sh - lists GCP IAM roles which have been granted directly to users in violation of best-practice group-based management
    - gcp_iam_serviceaccount_members.sh - lists members with permissions to use each GCP service account
    - gcp_iam_serviceaccounts_without_permissions.sh - finds service accounts without IAM permissionns, useful to detect obsolete service accounts after a 90 day unused permissions clean out
    - gcp_iam_workload_identities.sh - lists GKE Workload Identity integrations, uses gcp_iam_serviceaccount_members.sh
    - gcp_iam_users_granted_directly.sh - lists GCP IAM users which have been granted roles directly in violation of best-practice group-based management
- gcs_bucket_project.sh - finds the GCP project that a given bucket belongs to using the GCP Storage API
- gcs_curl_file.sh - retrieves a GCS file's contents from a given bucket and path using the GCP Storage API. Useful for starting shell pipelines or being called from other scripts

Kubernetes

kubernetes/ directory:

.envrc-kubernetes - copy to .envrc for direnv to auto-load the right Kubernetes kubectl context isolated to current shell to prevent race conditions between shells and scripts caused by otherwise naively changing the global ~/.kube/config context
aws/eksctl_cluster.sh - quickly spins up an AWS EKS cluster using eksctl with some sensible defaults
kubernetes_info.sh - huge Kubernetes inventory listing of deployed resources across all namespaces in the current cluster / kube context:
- cluster-info
- master component statuses
- 노드
- namespaces
- deployments, replicasets, replication controllers, statefulsets, daemonsets, horizontal pod autoscalers
- storage classes, persistent volumes, persistent volume claims
- service accounts, resource quotas, network policies, pod security policies
- container images running
- container images running counts descending
- pods (might be too much detail if you have high replica counts, so done last, comment if you're sure nobody has deployed pods outside deployments)
kubectl.sh - runs kubectl commands safely fixed to a given context using config isolation to avoid concurrency race conditions
kubectl_diff_apply.sh - generates a kubectl diff and prompts to apply
kustomize_diff_apply.sh - runs Kustomize build, precreates any namespaces, shows a kubectl diff of the proposed changes, and prompts to apply
kustomize_diff_branch.sh - runs Kustomize build against the current and target base branch for current or all given directories, then shows the diff for each directory. Useful to detect differences when refactoring, such as switching to tagged bases
kubectl_create_namespaces.sh - creates any namespaces in yaml files or stdin, a prerequisite for a diff on a blank install, used by adjacent scripts for safety
kubernetes_check_objects_namespaced.sh - checks Kubernetes yaml(s) for objects which aren't explicitly namespaced, which can easily result in deployments to the wrong namespace. Reads the API resources from your current Kubernetes cluster and if successful excludes cluster-wide objects
kustomize_check_objects_namespaced.sh - checks Kustomize build yaml output for objects which aren't explicitly namespaced (uses above script)
kubectl_deployment_pods.sh - gets the pod names with their unpredictable suffixes for a given deployment by querying the deployment's selector labels and then querying pods that match those labels
kubectl_get_all.sh - finds all namespaced Kubernetes objects and requests them for the current or given namespace. Useful because kubectl get all misses a lof of object types
kubectl_get_annotation.sh - find a type of object with a given annotation
kubectl_restart.sh - restarts all or filtered deployments/statefulsets in the current or given namespace. Useful when debugging or clearing application problems
kubectl_logs.sh - tails all containers in all pods or filtered pods in the current or given namespace. Useful when debugging a distributed set of pods in live testing
kubectl_kv_to_secret.sh - creates a Kuberbetes secret from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
kubectl_secret_values.sh - prints the keys and base64 decoded values within a given Kubernetes secret for quick debugging of Kubernetes secrets. See also: gcp_secrets_to_kubernetes.sh
kubectl_secrets_download.sh - downloads all secrets in current or given namespace to local files of the same name, useful as a backup before migrating to Sealed Secrets
kubernetes_secrets_compare_gcp_secret_manager.sh - compares each Kubernetes secret to the corresponding secret in GCP Secret Manager. Useful to safety check GCP Secret Manager values align before enabling External Secrets to replace them
kubernetes_secret_to_external_secret.sh - generates an External Secret from an existing Kubernetes secret
kubernetes_secrets_to_external_secrets.sh - generates External Secrets from all existing Kubernetes secrets found in the current or given namespace
kubernetes_secret_to_sealed_secret.sh - generates a Bitnami Sealed Secret from an existing Kubernetes secret
kubernetes_secrets_to_sealed_secrets.sh - generates Bitnami Sealed Secrets from all existing Kubernetes secrets found in the current or given namespace
kubectl_secrets_annotate_to_be_sealed.sh - annotates secrets in current or given namespace to allow being overwritten by Sealed Secrets (useful to sync ArgoCD health)
kubectl_secrets_not_sealed.sh - finds secrets with no SealedSecret ownerReferences
kubectl_secrets_to_be_sealed.sh - finds secrets pending overwrite by Sealed Secrets with the managed annotation
kubernetes_foreach_context.sh - executes a command across all kubectl contexts, replacing {context} in each iteration (skips lab contexts docker / minikube / minishift to avoid hangs since they're often offline)
kubernetes_foreach_namespace.sh - executes a command across all kubernetes namespaces in the current cluster context, replacing {namespace} in each iteration
- Can be chained with kubernetes_foreach_context.sh and useful when combined with gcp_secrets_to_kubernetes.sh to load all secrets from GCP to Kubernetes for the current cluster, or combined with gke_kube_creds.sh and kubernetes_foreach_context.sh for all clusters!
kubernetes_api.sh - finds Kubernetes API and runs your curl arguments against it, auto-getting authorization token and auto-populating OAuth authentication header
kubernetes_autoscaler_release.sh - finds the latest Kubernetes Autoscaler release that matches your local Kubernetes cluster version using kubectl and the GitHub API. Useful for quickly finding the image override version for eks-cluster-autoscaler-kustomization.yaml in the Kubernetes configs repo
kubernetes_etcd_backup.sh - creates a timestamped backup of the Kubernetes Etcd database for a kubeadm cluster
kubernetes_delete_stuck_namespace.sh - to forcibly delete those pesky kubernetes namespaces of 3rd party apps like Knative that get stuck and hang indefinitely on the finalizers during deletion
kubeadm_join_cmd.sh - outputs kubeadm join command (generates new token) to join an existing Kubernetes cluster (used in vagrant kubernetes provisioning scripts)
kubeadm_join_cmd2.sh - outputs kubeadm join command manually (calculates cert hash + generates new token) to join an existing Kubernetes cluster
kubernetes_nodes_ssh_dump_logs.sh - fetch logs from Kubernetes nodes (eg. for support debug requests by vendors)
kubectl_exec.sh - finds and execs to the first Kubernetes pod matching the given name regex, optionally specifying the container name regex to exec to, and shows the full generated kubectl exec command line for clarity
kubectl_exec2.sh - finds and execs to the first Kubernetes pod matching given pod filters, optionally specifying the container to exec to, and shows the full generated kubectl exec command line for clarity
kubectl_pods_per_node.sh - lists number of pods per node sorted descending
kubectl_pods_important.sh - lists important pods and their nodes to check on scheduling
kubectl_pods_colocated.sh - lists pods from deployments/statefulsets that are colocated on the same node
kubectl_node_labels.sh - lists nodes and their labels, one per line, easier to read visually or pipe in scripting
kubectl_pods_running_with_labels.sh - lists running pods with labels matching key=value pair arguments
kubectl_node_taints.sh - lists nodes and their taints
kubectl_jobs_stuck.sh - finds Kubernetes jobs stuck for hours or days with no completions
kubectl_jobs_delete_stuck.sh - prompts for confirmation to delete stuck Kubernetes jobs found by script above
kubectl_images.sh - lists Kubernetes container images running on the current cluster
kubectl_image_counts.sh - lists Kubernetes container images running counts sorted descending
kubectl_image_deployments.sh - lists which deployments, statefulsets or daemonsets container images belong to. Useful to find which deployment, statefulset or daemonset to upgrade to replace a container image eg. when replacing deprecated the k8s.gcr.io registry with registry.k8s.io
kubectl_pod_count.sh - lists Kubernetes pods total running count
kubectl_pod_labels.sh - lists Kubernetes pods and their labels, one label per line for easier shell script piping for further actions
kubectl_pod_ips.sh - lists Kubernetes pods and their pod IP addresses
kubectl_container_count.sh - lists Kubernetes containers total running count
kubectl_container_counts.sh - lists Kubernetes containers running counts by name sorted descending
kubectl_pods_dump_*.sh - dump stats / logs / jstacks from all pods matching a given regex and namespace to txt files for support debugging
- kubectl_pods_dump_stats.sh - dump stats
- kubectl_pods_dump_logs.sh - dump logs
- kubectl_pods_dump_jstacks.sh - dump Java jstacks
- kubectl_pods_dump_all.sh - calls the above kubectl_pods_dump_*.sh scripts for N iterations with a given interval
kubectl_empty_namespaces.sh - finds namespaces without any of the usual objects using kubectl get all
kubectl_delete_empty_namespaces.sh - removes empty namespaces, uses kubectl_empty_namespaces.sh
kubectl_<image>.sh - quick launch one-off pods for interactive debuggging in Kubernetes
- kubectl_alpine.sh
- kubectl_busybox.sh
- kubectl_curl.sh
- kubectl_dnsutils.sh
- kubectl_gcloud_sdk.sh
- kubectl_run_sa.sh - launch a quick pod with the given service account to test private repo pull & other permissions
kubectl_port_forward.sh - launches kubectl port-forward to a given pod's port with an optional label or name filter. If more than one pod is found, prompts with an interactive dialogue to choose one. Optionally automatically opens the forwarded localhost URL in the default browser
- kubectl_port_forward_spark.sh - does the above for Spark UI
helm_template.sh - templates a Helm chart for Kustomize deployments
kustomize_parse_helm_charts.sh - parses the Helm charts from one or more kustomization.yaml files into TSV format for further shell pipe processing
kustomize_install_helm_charts.sh - installs the Helm charts from one or more kustomization.yaml files the old fashioned Helm CLI way so that tools like Nova can be used to detect outdated charts (used in Kubernetes-configs repo's CI)
kustomize_update_helm_chart_versions.sh - updates one or more kustomization.yaml files to the latest versions of any charts they contain
kustomize_materialize.sh - recursively materializes all kustomization.yaml to kustomization.materialized.yaml in the same directories for scanning with tools like Pluto to detect deprecated API objects inherited from embedded Helm charts. Parallelized for performance
ArgoCD:
- argocd_auto_sync.sh - toggle Auto-sync on/off to allow repairs and maintenance operation for a given app and also disables / re-enables the App-of-Apps base apps to stop then re-enabling the app
- argocd_apps_sync.sh - sync's all ArgoCD apps matching an optional ERE regex filter on their names using the ArgoCD CLI
- argocd_apps_wait_sync.sh - sync's all ArgoCD apps matching an optional ERE regex filter on their names using the ArgoCD CLI's while also checking their health and operation
- argocd_generate_resource_whitelist.sh - generates a yaml cluster and namespace resource whitelist for ArgoCD project config. If given an existing yaml, will merge in its original whitelists, dedupe, and write them back into the file using an in-place edit. Useful because ArgoCD 2.2+ doesn't show resources that aren't explicitly allowed, such as ReplicaSets and Pods
명왕성:
- pluto_detect_helm_materialize.sh - recursively materializes all helm Chart.yaml and runs Pluto on each directory to work around this issue
- pluto_detect_kustomize_materialize.sh - recursively materializes all kustomization.yaml and runs Pluto on each directory to work around this issue
- pluto_detect_kubectl_dump_objects.sh - dumps all live Kubernetes objects to /tmp all can run Pluto to detect deprecated API objects on the cluster from any source
목장 경영자:
- rancher_api.sh - queries the Rancher API with authentication
- rancher_kube_creds.sh - downloads all Rancher clusters credentials into subdirectories matching cluster names, with .envrc in each, so a quick cd into one and your kubectl is ready to rock
see also Google Kubernetes Engine scripts in the GCP - Google Cloud Platform section above
see also the Kubernetes configs repo

See also Knowledge Base notes for Kubernetes.

도커

docker/ directory:

docker_*.sh / dockerhub_*.sh - Docker / DockerHub API scripts:
- dockerhub_api.sh - queries DockerHub API v2 with or without authentication ( $DOCKERHUB_USER & $DOCKERHUB_PASSWORD / $DOCKERHUB_TOKEN )
- docker_api.sh - queries a Docker Registry with optional basic authentication if $DOCKER_USER & $DOCKER_PASSWORD are set
- docker_build_hashref.sh - runs docker build and auto-generates docker image name and tag from relative Git path and commit short SHA hashref and a dirty sha suffix if git contents are modified. Useful to compare docker image sizes between your clean and modified versions of Dockerfile or contents
- docker_package_check.sh - runs package installs on major versions of a docker image to check given packages are available before adding them and breaking builds across linux distro versions
- docker_registry_list_images.sh - lists images in a given private Docker Registry
- docker_registry_list_tags.sh - lists tags for a given image in a private Docker Registry
- docker_registry_get_image_manifest.sh - gets a given image:tag manifest from a private Docker Registry
- docker_registry_tag_image.sh - tags a given image with a new tag in a private Docker Registry via the API without pulling and pushing the image data (must faster and more efficient)
- dockerhub_list_tags.sh - lists tags for a given DockerHub repo. See also dockerhub_show_tags.py in the DevOps Python tools repo.
- dockerhub_list_tags_by_last_updated.sh - lists tags for a given DockerHub repo sorted by last updated timestamp descending
- dockerhub_search.sh - searches with a configurable number of returned items (older docker cli was limited to 25 results)
- clean_caches.sh - cleans out OS package and programming language caches, call near end of Dockerfile to reduce Docker image size
- see also the Dockerfiles repo
quay_api.sh - queries the Quay.io API with OAuth2 authentication token $QUAY_TOKEN

See also Knowledge Base notes for Docker.

데이터

data/ directory:

avro_tools.sh - runs Avro Tools jar, downloading it if not already present (determines latest version when downloading)
parquet_tools.sh - runs Parquet Tools jar, downloading it if not already present (determines latest version when downloading)
csv_header_indices.sh - list CSV headers with their zero indexed numbers, useful reference when coding against column positions
ini_config_add_if_missing.sh - reads INI config blocks from stdin and appends them to the specified file if the section is not found. Used by aws_profile_config_add_if_missing.sh
ini_config_duplicate_sections.sh - lists duplicate INI config sections that are using the same value for a given key in the given .ini file
ini_config_duplicate_section_names.sh - lists duplicate INI config section names that are using the same value for a given key in the given .ini file
ini_grep_section.sh - prints the named section from a given .ini file to stdout
wordcount.sh - counts and ranks words by their frequency in file(s) or stdin
Data format validation validate_*.py from DevOps Python Tools repo:
- CSV
- JSON
- Avro
- 쪽매 세공
- INI / Properties files (Java)
- LDAP LDIF
- XML
- 얌
json2yaml.sh - converts JSON to YAML
yaml2json.sh - converts YAML to JSON - needed for some APIs like GitLab CI linting (see Gitlab section above)

Big Data & NoSQL

bigdata/ and kafka/ directories:

kafka_*.sh - scripts to make Kafka CLI usage easier including auto-setting Kerberos to source TGT from environment and auto-populating broker and zookeeper addresses. These are auto-added to the $PATH when .bashrc is sourced. For something similar for Solr, see solr_cli.pl in the DevOps Perl Tools repo.
zookeeper*.sh - Apache ZooKeeper scripts:
- zookeeper_client.sh - shortens zookeeper-client command by auto-populating the zookeeper quorum from the environment variable $ZOOKEEPERS or else parsing the zookeeper quorum from /etc/**/*-site.xml to make it faster and easier to connect
- zookeeper_shell.sh - shortens Kafka's zookeeper-shell command by auto-populating the zookeeper quorum from the environment variable $KAFKA_ZOOKEEPERS and optionally $KAFKA_ZOOKEEPER_ROOT to make it faster and easier to connect
hive_*.sh / beeline*.sh - Apache Hive scripts:
- beeline.sh - shortens beeline command to connect to HiveServer2 by auto-populating Kerberos and SSL settings, zookeepers for HiveServer2 HA discovery if the environment variable $HIVE_HA is set or using the $HIVESERVER_HOST environment variable so you can connect with no arguments (prompts for HiveServer2 address if you haven't set $HIVESERVER_HOST or $HIVE_HA )
  - beeline_zk.sh - same as above for HiveServer2 HA by auto-populating SSL and ZooKeeper service discovery settings (specify $HIVE_ZOOKEEPERS environment variable to override). Automatically called by beeline.sh if either $HIVE_ZOOKEEPERS or $HIVE_HA is set (the latter parses hive-site.xml for the ZooKeeper addresses)
- hive_foreach_table.sh - executes a SQL query against every table, replacing {db} and {table} in each iteration eg. select count(*) from {table}
- hive_list_databases.sh - list Hive databases, one per line, suitable for scripting pipelines
- hive_list_tables.sh - list Hive tables, one per line, suitable for scripting pipelines
- hive_tables_metadata.sh - lists a given DDL metadata field for each Hive table (to compare tables)
- hive_tables_location.sh - lists the data location per Hive table (eg. compare external table locations)
- hive_tables_row_counts.sh - lists the row count per Hive table
- hive_tables_column_counts.sh - lists the column count per Hive table
impala*.sh - Apache Impala scripts:
- impala_shell.sh - shortens impala-shell command to connect to Impala by parsing the Hadoop topology map and selecting a random datanode to connect to its Impalad, acting as a cheap CLI load balancer. For a real load balancer see HAProxy config for Impala (and many other Big Data & NoSQL technologies). Optional environment variables $IMPALA_HOST (eg. point to an explicit node or an HAProxy load balancer) and IMPALA_SSL=1 (or use regular impala-shell --ssl argument pass through)
- impala_foreach_table.sh - executes a SQL query against every table, replacing {db} and {table} in each iteration eg. select count(*) from {table}
- impala_list_databases.sh - list Impala databases, one per line, suitable for scripting pipelines
- impala_list_tables.sh - list Impala tables, one per line, suitable for scripting pipelines
- impala_tables_metadata.sh - lists a given DDL metadata field for each Impala table (to compare tables)
- impala_tables_location.sh - lists the data location per Impala table (eg. compare external table locations)
- impala_tables_row_counts.sh - lists the row count per Impala table
- impala_tables_column_counts.sh - lists the column count per Impala table
hdfs_*.sh - Hadoop HDFS scripts:
- hdfs_checksum*.sh - walks an HDFS directory tree and outputs HDFS native checksums (faster) or portable externally comparable CRC32, in serial or in parallel to save time
- hdfs_find_replication_factor_1.sh / hdfs_set_replication_factor_3.sh - finds HDFS files with replication factor 1 / sets HDFS files with replication factor <=2 to replication factor 3 to repair replication safety and avoid no replica alarms during maintenance operations (see also Python API version in the DevOps Python Tools repo)
- hdfs_file_size.sh / hdfs_file_size_including_replicas.sh - quickly differentiate HDFS files raw size vs total replicated size
- hadoop_random_node.sh - picks a random Hadoop cluster worker node, like a cheap CLI load balancer, useful in scripts when you want to connect to any worker etc. See also the read HAProxy Load Balancer configurations which focuses on master nodes
cloudera_*.sh - Cloudera scripts:
- cloudera_manager_api.sh - script to simplify querying Cloudera Manager API using environment variables, prompts, authentication and sensible defaults. Built on top of curl_auth.sh
- cloudera_manager_impala_queries*.sh - queries Cloudera Manager for recent Impala queries, failed queries, exceptions, DDL statements, metadata stale errors, metadata refresh calls etc. Built on top of cloudera_manager_api.sh
- cloudera_manager_yarn_apps.sh - queries Cloudera Manager for recent Yarn apps. Built on top of cloudera_manager_api.sh
- cloudera_navigator_api.sh - script to simplify querying Cloudera Navigator API using environment variables, prompts, authentication and sensible defaults. Built on top of curl_auth.sh
- cloudera_navigator_audit_logs.sh - fetches Cloudera Navigator audit logs for given service eg. hive/impala/hdfs via the API, simplifying date handling, authentication and common settings. Built on top of cloudera_navigator_api.sh
- cloudera_navigator_audit_logs_download.sh - downloads Cloudera Navigator audit logs for each service by year. Skips existing logs, deletes partially downloaded logs on failure, generally retry safe (while true, Control-C, not kill -9 obviously). Built on top of cloudera_navigator_audit_logs.sh

See also Knowledge Base notes for Hadoop.

Git - GitHub, GitLab, Bitbucket, Azure DevOps

git/ , github/ , gitlab/ , bitbucket/ and azure_devops/ directories:

git/*.sh - Git scripts:
- precommit_run_changed_files.sh - runs pre-commit on all files changed on the current branch vs the default branch. Useful to reproduce pre-commit checks that are failing in pull requests to get your PRs to pass
- git_diff_commit.sh - quickly commits added or updated files to Git, showing a diff and easy enter prompt for each file. Super convenient for fast commits on the command line, and in vim and IDEs via hotkeys
- git_review_push.sh - shows diff of what would be pushed upstream and prompts to push. Convenient for fast reviewed pushes via vim or IDEs hotkeys
- git_branch_delete_squash_merged.sh - carefully detects if a squash merged branch you want to delete has no changes with the default trunk branch before deleting it. See Squash Merges in knowledge-base about why this is necessary.
- git_tag_release.sh - creates a Git tag, auto-incrementing a .N suffix on the year/month/day date format if no exact version given
- git_foreach_branch.sh - executes a command on all branches (useful in heavily version branched repos like in my Dockerfiles repo)
- git_foreach_repo.sh - executes a command against all adjacent repos from a given repolist (used heavily by many adjacent scripts)
- git_foreach_modified.sh - executes a command against each file with git modified status
- git_foreach_repo_replace_readme_actions.sh - updates the README.md badges for GitHub Actions to match the local repo name. Useful to bulk fix copied badges quickly and easily
- git_foreach_repo_update_readme.sh - git-diff-commits the README.md for each Git repo checkout using adjacent git_foreach_repo.sh and git_diff_commit.sh scripts. Useful to quickly bulk update README.md in all your projects, such as when references need updating
- git_merge_all.sh / git_merge_master.sh / git_merge_master_pull.sh - merges updates from master branch to all other branches to avoid drift on longer lived feature branches / version branches (eg. Dockerfiles repo)
- git_remotes_add_origin_providers.sh - auto-creates remotes for the 4 major public repositories (GitHub/GitLab/Bitbucket/Azure DevOps), useful for git pull -all to fetch and merge updates from all providers in one command
- git_remotes_set_multi_origin.sh - sets up multi-remote origin for unified push to automatically keep the 4 major public repositories in sync (especially useful for Bitbucket and Azure DevOps which don't have GitLab's auto-mirroring from GitHub feature)
- git_remotes_set_https_to_ssh.sh - converts local repo's remote URLs from https to ssh (more convenient with SSH keys instead of https auth tokens, especially since Azure DevOps expires personal access tokens every year)
- git_remotes_set_ssh_to_https.sh - converts local repo's remote URLs from ssh to https (to get through corporate firewalls or hotels if you travel a lot)
- git_remotes_set_https_creds_helpers.sh - adds Git credential helpers configuration to the local git repo to use http API tokens dynamically from environment variables if they're set
- git_repos_pull.sh - pull multiple repos based on a source file mapping list - useful for easily sync'ing lots of Git repos among computers
- git_repos_update.sh - same as above but also runs the make update build to install the latest dependencies, leverages the above script
- git_grep_env_vars.sh - find environment variables in the current git repo's code base in the format SOME_VAR (useful to find undocumented environment variables in internal or open source projects such as ArgoCD eg. argoproj/argocd-cd #8680)
- git_log_empty_commits.sh - find empty commits in git history (eg. if a git filter-branch was run but --prune-empty was forgotten, leaking metadata like subjects containing file names or other sensitive info)
- git_graph_commit_history_gnuplot.sh - generates GNUplot graphs of Git commits per year and per month for the entire history of the local Git repo checkout
- git_graph_commit_history_mermaidjs.sh - generates MermaidJS graphs of Git commits per year and per month for the entire history of the local Git repo checkout
- git_graph_commit_times_gnuplot.sh - generates a GNUplot graph of Git commit times from the current Git repo checkout's git log
- git_graph_commit_times_mermaidjs.sh - generates a MermaidJS graph of Git commit times from the current Git repo checkout's git log
- git_graph_commit_times_gnuplot_all_repos.sh - generates GNUplot graph of the GitHub commit times from all local adjacent Git repo checkouts listed in setup/repos.txt using Git log in each checkout
- git_graph_commit_times_mermaidjs_all_repos.sh - generates MermaidJS graph of the GitHub commit times from all local adjacent Git repo checkouts listed in setup/repos.txt using Git log in each checkout
- git_revert_line.sh - reverts the first line that matches a given regex from the Git head commit's version of the same line number. Useful to revert some changes caused by over zealous sed'ing scripts, where you want to cherry-pick revert a single line change
- git_files_in_history.sh - finds all filename / file paths in the git log history, useful for prepping for git filter-branch
- git_filter_branch_fix_author.sh - rewrites Git history to replace author/committer name & email references (useful to replace default account commits). Powerful, read --help and man git-filter-branch carefully. Should only be used by Git Experts
- git_filter_repo_replace_text.sh - rewrites Git history to replace a given text to scrub a credential or other sensitive token from history. Refuses to operate on tokens less than 8 chars for safety
- git_submodules_update_repos.sh - updates submodules (pulls and commits latest upstream github repo submodules) - used to cascade submodule updates throughout all my repos
- git_askpass.sh - credential helper script to use environment variables for git authentication
- markdown_generate_index.sh - generates a markdown index list from the headings in a given markdown file such as README.md
- markdown_replace_index.sh - replaces a markdown index section in a given markdown file using markdown_generate_index.sh
github/*.sh - GitHub API / CLI scripts:
- github_api.sh - queries the GitHub API. Can infer GitHub user, repo and authentication token from local checkout or environment ( $GITHUB_USER , $GITHUB_TOKEN )
- github_install_binary.sh - installs a binary from GitHub releases into $HOME/bin or /usr/local/bin. Auto-determines the latest release if no version specified, detects and unpacks any tarball or zip files
- github_foreach_repo.sh - executes a templated command for each non-fork GitHub repo, replacing the {owner} / {name} or {repo} placeholders in each iteration
- github_graph_commit_times_gnuplot.sh - generates GNUplot graph of GitHub commit times from all public GitHub repos for a given user. Fetches the commit data via the GitHub API
- github_graph_commit_times_mermaidjs.sh - generates MermaidJS graph of the GitHub commit times from all public GitHub repos for a given user. Fetches the commit data via the GitHub API
- github_clone_or_pull_all_repos.sh - git clones or pulls all repos for a user or organization into directories of the same name under the current directory
- github_download_release_file.sh - downloads a file from GitHub Releases, optionally determining the latest version, uses bin/download_url_file.sh
- github_download_release_jar.sh - downloads a JAR file from GitHub Releases (used by install/download_*_jar.sh for things like JDBC drivers or Java decompilers), optionally determines latest version to download, and finally validates the downloaded file's format
- github_invitations.sh - lists / accepts repo invitations. Useful to accept a large number of invites to repos generated by automation
- github_mirror_repos_to_gitlab.sh - creates/syncs GitHub repos to GitLab for migrations or to cron fast free Disaster Recovery, including all branches and tags, plus the repo descriptions. Note this doesn't include PRs/wikis/releases
- github_mirror_repos_to_bitbucket.sh - creates/syncs GitHub repos to BitBucket for migrations or to cron fast free Disaster Recovery, including all branches and tags, plus the repo descriptions. Note this doesn't include PRs/wikis/releases
- github_mirror_repos_to_aws_codecommit.sh - creates/syncs GitHub repos to AWS CodeCommit for migrations or to cron fast almost free Disaster Recovery (close to $0 compared to $100-400+ per month for Rewind BackHub), including all branches and tags, plus the repo descriptions. Note this doesn't include PRs/wikis/releases
- github_mirror_repos_to_gcp_source_repos.sh - creates/syncs GitHub repos to GCP Source Repos for migrations or to cron fast almost free Disaster Recovery (close to $0 compared to $100-400+ per month for Rewind BackHub), including all branches and tags. Note this doesn't include repo description/PRs/wikis/releases
- github_pull_request_create.sh - creates a Pull Request idempotently by first checking for an existing PR between the branches, and also checking if there are the necessary commits between the branches, to avoid common errors from blindly raising PRs. Useful to automate code promotion across environment branches. Also works across repo forks and is used by github_repo_fork_update.sh . Even populates github pull request template and does Jira ticket number replacement from branch prefix
- github_pull_request_preview.sh - opens a GitHub Pull Request preview page from the current local branch to the given or default branch
- github_push_pr_preview.sh - pushes to GitHub origin, sets upstream branch, then open a Pull Request preview from current branch to the given or default trunk branch in your browser
- github_push_pr.sh - pushes to GitHub origin, sets upstream branch, then idemopotently creates a Pull Request from current branch to the given or default trunk branch and opens the generated PR in your browser for review
- github_merge_branch.sh - merges one branch into another branch via a Pull Request for full audit tracking all changes. Useful to automate feature PRs, code promotion across environment branches, or backport hotfixes from Production or Staging to trunk branches such as master, main, dev or develop
- github_remote_set_upstream.sh - in a forked GitHub repo's checkout, determine the origin of the fork using GitHub CLI and configure a git remote to the upstream. Useful to be able to easily pull updates from the original source repo
- github_pull_merge_trunk.sh - pulls the origin or fork upstream repo's trunk branch and merges it into the local branch, In a forked GitHub repo's checkout, determines the origin of the fork using GitHub CLI, configures a git remote to the upstream, pulls the default branch and if on a branch other than the default then merges the default branch to the local current branch. Simplifies and automates keeping your checkout or forked repo up to date with the original source repo to quickly resolve merge conflicts locally and submit updated Pull Requests
- github_forked_add_remote.sh - quickly adds a forked repo as a remote from an interactive men list of forked repos
- github_forked_checkout_branch.sh - quickly check out a forked repo's branch from an interactive menu lists of forked repos and their branches
- github_tag_hashref.sh - Returns the GitHub commit hashref for a given GitHub Actions owner/repo@tag or https://github.com/owner/repo@tag . Useful for pinning 3rd party GitHub Actions to hashref instead of tag to follow GitHub Actions Best Practices
- github_actions_foreach_workflow.sh - executes a templated command for each workflow in a given GitHub repo, replacing {name} , {id} and {state} in each iteration
- github_actions_aws_create_load_credential.sh - creates an AWS user with group/policy, generates and downloads access keys, and uploads them to the given repo
- github_actions_in_use.sh - lists GitHub Actions directly referenced in the .github/workflows in the current local repo checkout
- github_actions_in_use_repo.sh - lists GitHub Actions for a given repo via the API, including following imported reusable workflows
- github_actions_in_use_across_repos.sh - lists GitHub Actions in use across all your repos
- github_actions_repos_lockdown.sh - secures GitHub Actions settings across all user repos to only GitHub, verified partners and selected 3rd party actions
- github_actions_repo_set_secret.sh - sets a secret in the given repo from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- github_actions_repo_env_set_secret.sh - sets a secret in the given repo and environment from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- github_actions_repo_secrets_overriding_org.sh - finds any secrets for a repo that are overriding organization level secrets. Useful to combine with github_foreach_repo.sh for auditing
- github_actions_repo_restrict_actions.sh - restricts GitHub Actions in the given repo to only running actions from GitHub and verfied partner companies (.eg AWS, Docker)
- github_actions_repo_actions_allow.sh - allows select 3rd party GitHub Actions in the given repo
- github_actions_runner.sh - generates a GitHub Actions self-hosted runner token for a given Repo or Organization via the GitHub API and then runs a dockerized GitHub Actions runner with the appropriate configuration
- github_actions_runner_local.sh - downloads, configures and runs a local GitHub Actions Runner for Linux or Mac
- github_actions_runner_token.sh - generates a GitHub Actions runner token to register a new self-hosted runner
- github_actions_runners.sh - lists GitHub Actions self-hosted runners for a given Repo or Organization
- github_actions_delete_offline_runners.sh - deletes offline GitHub Actions self-hosted runners. Useful to clean up short-lived runners eg. Docker, Kubernetes
- github_actions_workflows.sh - lists GitHub Actions workflows for a given repo (or auto-infers local repository)
- github_actions_workflow_runs.sh - lists GitHub Actions workflow runs for a given workflow id or name
- github_actions_workflows_status.sh - lists all GitHub Actions workflows and their statuses for a given repo
- github_actions_workflows_state.sh - lists GitHub Actions workflows enabled/disabled states (GitHub now disables workflows after 6 months without a commit)
- github_actions_workflows_disabled.sh - lists GitHub Actions workflows that are disabled. Combine with github_foreach_repo.sh to scan all repos to find disabled workflows
- github_actions_workflow_enable.sh - enables a given GitHub Actions workflow
- github_actions_workflows_enable_all.sh - enables all GitHub Actions workflows in a given repo. Useful to undo GitHub disabling all workflows in a repo after 6 months without a commit
- github_actions_workflows_trigger_all.sh - triggers all workflows for the given repo
- github_actions_workflows_cancel_all_runs.sh - cancels all workflow runs for the given repo
- github_actions_workflows_cancel_waiting_runs.sh - cancels workflow runs that are in waiting state, eg. waiting for old deployment approvals
- github_ssh_get_user_public_keys.sh - fetches a given GitHub user's public SSH keys via the API for piping to ~/.ssh/authorized_keys or adjacent tools
- github_ssh_get_public_keys.sh - fetches the currently authenticated GitHub user's public SSH keys via the API, similar to above but authenticated to get identifying key comments
- github_ssh_add_public_keys.sh - uploads SSH keys from local files or standard input to the currently authenticated GitHub account. Specify pubkey files (default: ~/.ssh/id_rsa.pub ) or read from standard input for piping from adjacent tools
- github_ssh_delete_public_keys.sh - deletes given SSH keys from the currently authenticated GitHub account by key id or title regex match
- github_gpg_get_user_public_keys.sh - fetches a given GitHub user's public GPG keys via the API
- github_generate_status_page.sh - generates a STATUS.md page by merging all the README.md headers for all of a user's non-forked GitHub repos or a given list of any repos etc.
- github_purge_camo_cache.sh - send HTTP Purge requests to all camo urls (badge caches) for the current or given GitHub repo's landing/README.md page
- github_ip_ranges.sh - returns GitHub's IP ranges, either all by default or for a select given service such as hooks or actions
- github_sync_repo_descriptions.sh - syncs GitHub repo descriptions to GitLab & BitBucket repos
- github_release.sh - creates a GitHub Release, auto-incrementing a .N suffix on the year/month/day date format if no exact version given
- github_repo_check_pat_token.sh - checks the given PAT token can access the given GitHub repo. Useful to test a PAT token used for integrations like ArgoCD
- github_repo_description.sh - fetches the given repo's description (used by github_sync_repo_descriptions.sh )
- github_repo_find_files.sh - finds files matching a regex in the current or given GitHub repo via the GitHub API
- github_repo_latest_release.sh - returns the latest release tag for a given GitHub repo via the GitHub API
- github_repo_latest_release_filter.sh - returns the latest release tag matching a given regex filter for a given GitHub repo via the GitHub API. Useful for getting the latest version of things like Kustomize which has other releases for kyaml
- github_repo_stars.sh - fetches the stars, forks and watcher counts for a given repo
- github_repo_teams.sh - fetches the GitHub Enterprise teams and their role permisions for a given repo. Combine with github_foreach_repo.sh to audit your all your personal or GitHub organization's repos
- github_repo_collaborators.sh - fetches a repo's granted users and outside invited collaborators as well as their role permisions for a given repo. Combine with github_foreach_repo.sh to audit your all your personal or GitHub organization's repos
- github_repo_protect_branches.sh - enables branch protections on the given repo. Can specify one or more branches to protect, otherwise finds and applies to any of master , main , develop , dev , staging , production
- github_repos_find_files.sh - finds files matching a regex across all repos in the current GitHub organization or user account
- github_repo_fork_sync.sh - sync's current or given fork, then runs github_repo_fork_update.sh to cascade changes to major branches via Pull Requests for auditability
- github_repo_fork_update.sh - updates a forked repo by creating pull requests for full audit tracking and auto-merges PRs for non-production branches
- github_repos_public.sh - lists public repos for a user or organization. Useful to periodically scan and account for any public repos
- github_repos_disable_wiki.sh - disables the Wiki on one or more given repos to prevent documentation fragmentation and make people use the centralized documentation tool eg. Confluence or Slite
- github_repos_with_few_users.sh - finds repos with few or no users (default: 1), which in Enterprises is a sign that a user has created a repo without assigning team privileges
- github_repos_with_few_teams.sh - finds repos with few or no teams (default: 0), which in Enterprises is a sign that a user has created a repo without assigning team privileges
- github_repos_without_branch_protections.sh - finds repos without any branch protection rules (use github_repo_protect_branches.sh on such repos)
- github_repos_not_in_terraform.sh - finds all non-fork repos for current or given user/organization which are not found in $PWD/*.tf Terraform code
- github_teams_not_in_terraform.sh - finds all teams for given organization which are not found in $PWD/*.tf Terraform code
- github_repos_sync_status.sh - determines whether each GitHub repo's mirrors on GitLab / BitBucket / Azure DevOps are up to date with the latest commits, by querying all 3 APIs and comparing master branch hashrefs
- github_teams_not_idp_synced.sh - finds GitHub teams that aren't sync'd from an IdP like Azure AD. These should usually be migrated or removed
- github_user_repos_stars.sh - fetches the total number of stars for all original source public repos for a given user
- github_user_repos_forks.sh - fetches the total number of forks for all original source public repos for a given user
- github_user_repos_count.sh - fetches the total number of original source public repos for a given username
- github_user_followers.sh - fetches the number of followers for a given username
- github_url_clipboard.sh - copies a GitHub URL file's contents to the clipboard, converting the URL to a raw GitHub content URL where necessary
gitlab/*.sh - GitLab API scripts:
- gitlab_api.sh - queries the GitLab API. Can infer GitLab user, repo and authentication token from local checkout or environment ( $GITLAB_USER , $GITLAB_TOKEN )
- gitlab_install_binary.sh - installs a binary from GitLab releases into $HOME/bin or /usr/local/bin. Auto-determines the latest release if no version specified, detects and unpacks any tarball or zip files
- gitlab_push_mr_preview.sh - pushes to GitLab origin, sets upstream branch, then open a Merge Request preview from current to default branch
- github_push_mr.sh - pushes to GitLab origin, sets upstream branch, then idemopotently creates a Merge Request from current branch to the given or default trunk branch and opens the generated MR in your browser for review
- gitlab_foreach_repo.sh - executes a templated command for each GitLab project/repo, replacing the {user} and {project} in each iteration
- gitlab_project_latest_release.sh - returns the latest release tag for a given GitLab project (repo) via the GitLab API
- gitlab_project_set_description.sh - sets the description for one or more projects using the GitLab API
- gitlab_project_set_env_vars.sh - adds / updates GitLab project-level environment variable(s) via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- gitlab_group_set_env_vars.sh - adds / updates GitLab group-level environment variable(s) via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- gitlab_project_create_import.sh - creates a GitLab repo as an import from a given URL, and mirrors if on GitLab Premium (can only manually configure for public repos on free tier, API doesn't support configuring even public repos on free)
- gitlab_project_protect_branches.sh - enables branch protections on the given project. Can specify one or more branches to protect, otherwise finds and applies to any of master , main , develop , dev , staging , production
- gitlab_project_mirrors.sh - lists each GitLab repo and whether it is a mirror or not
- gitlab_pull_mirror.sh - trigger a GitLab pull mirroring for a given project's repo, or auto-infers project name from the local git repo
- gitlab_ssh_get_user_public_keys.sh - fetches a given GitLab user's public SSH keys via the API, with identifying comments, for piping to ~/.ssh/authorized_keys or adjacent tools
- gitlab_ssh_get_public_keys.sh - fetches the currently authenticated GitLab user's public SSH keys via the API
- gitlab_ssh_add_public_keys.sh - uploads SSH keys from local files or standard input to the currently authenticated GitLab account. Specify pubkey files (default: ~/.ssh/id_rsa.pub ) or read from standard input for piping from adjacent tools
- gitlab_ssh_delete_public_keys.sh - deletes given SSH keys from the currently authenticated GitLab account by key id or title regex match
- gitlab_validate_ci_yaml.sh - validates a .gitlab-ci.yml file via the GitLab API
bitbucket/*.sh - BitBucket API scripts:
- bitbucket_api.sh - queries the BitBucket API. Can infer BitBucket user, repo and authentication token from local checkout or environment ( $BITBUCKET_USER , $BITBUCKET_TOKEN )
- bitbucket_foreach_repo.sh - executes a templated command for each BitBucket repo, replacing the {user} and {repo} in each iteration
- bitbucket_workspace_set_env_vars.sh - adds / updates Bitbucket workspace-level environment variable(s) via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- bitbucket_repo_set_env_vars.sh - adds / updates Bitbucket repo-level environment variable(s) via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- bitbucket_repo_set_description.sh - sets the description for one or more repos using the BitBucket API
- bitbucket_enable_pipelines.sh - enables the CI/CD pipelines for all repos
- bitbucket_disable_pipelines.sh - disables the CI/CD pipelines for all repos
- bitbucket_repo_enable_pipeline.sh - enables the CI/CD pipeline for a given repo
- bitbucket_repo_disable_pipeline.sh - disables the CI/CD pipeline for a given repo
- bitbucket_ssh_get_public_keys.sh - fetches the currently authenticated BitBucket user's public SSH keys via the API for piping to ~/.ssh/authorized_keys or adjacent tools
- bitbucket_ssh_add_public_keys.sh - uploads SSH keys from local files or standard input to the currently authenticated BitBucket account. Specify pubkey files (default: ~/.ssh/id_rsa.pub ) or read from standard input for piping from adjacent tools
- bitbucket_ssh_delete_public_keys.sh - uploads SSH keys from local files or standard input to the currently authenticated BitBucket account. Specify pubkey files (default: ~/.ssh/id_rsa.pub ) or read from standard input for piping from adjacent tools

CI/CD - Continuous Integration / Continuous Deployment

jenkins/ , terraform/ , teamcity/ , buildkite/ , circlci/ , travis/ , azure_devops/ , ..., cicd/ directories:

appveyor_api.sh - queries AppVeyor's API with authentication
azure_devops/*.sh - Azure DevOps scripts:
- azure_devops_api.sh - queries Azure DevOps's API with authentication
- azure_devops_foreach_repo.sh - executes a templated command for each Azure DevOps repo, replacing {user} , {org} , {project} and {repo} in each iteration
- azure_devops_to_github_migration.sh - migrates one or all Azure DevOps git repos to GitHub, including all branches and sets the default branch to match via the APIs to maintain the same checkout behaviour
- azure_devops_disable_repos.sh - disables one or more given Azure DevOps repos (to prevent further pushes to them after migration to GitHub)
circleci/*.sh - CircleCI scripts:
- circleci_api.sh - queries CircleCI's API with authentication
- circleci_project_set_env_vars.sh - adds / updates CircleCI project-level environment variable(s) via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- circleci_context_set_env_vars.sh - adds / updates CircleCI context-level environment variable(s) via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- circleci_project_delete_env_vars.sh - deletes CircleCI project-level environment variable(s) via the API
- circleci_context_delete_env_vars.sh - deletes CircleCI context-level environment variable(s) via the API
- circleci_local_execute.sh - installs CircleCI CLI and executes .circleci/config.yml locally
- circleci_public_ips.sh - lists CircleCI public IP addresses via dnsjson.com
codeship_api.sh - queries CodeShip's API with authentication
drone_api.sh - queries Drone.io's API with authentication
shippable_api.sh - queries Shippable's API with authentication
wercker_app_api.sh - queries Wercker's Applications API with authentication
gocd_api.sh - queries GoCD's API
gocd.sh - one-touch GoCD CI:
- launches in Docker
- (re)creates config repo ( $PWD/setup/gocd_config_repo.json ) from which to source pipeline(s) ( .gocd.yml )
- detects and enables agent(s) to start building
- call from any repo top level directory with a .gocd.yml config (all mine have it), mimicking structure of fully managed CI systems
concourse.sh - one-touch Concourse CI:
- launches in Docker
- configures pipeline from $PWD/.concourse.yml
- triggers build
- tails results in terminal
- prints recent build statuses at end
- call from any repo top level directory with a .concourse.yml config (all mine have it), mimicking structure of fully managed CI systems
fly.sh - shortens Concourse fly command to not have to specify target all the time
jenkins/*.sh - Jenkins CI scripts:
- jenkins.sh - one-touch Jenkins CI:
  - launches Docker container
  - installs plugins
  - validates Jenkinsfile
  - configures job from $PWD/setup/jenkins-job.xml
  - sets Pipeline to git remote origin's Jenkinsfile
  - triggers build
  - tails results in terminal
  - call from any repo top level directory with a Jenkinsfile pipeline and setup/jenkins-job.xml (all mine have it)
- jenkins_api.sh - queries the Jenkins Rest API, handles authentication, pre-fetches CSFR protection token crumb, supports many environment variables such as $JENKINS_URL for ease of use
  - jenkins_jobs.sh - lists Jenkins jobs (pipelines)
  - jenkins_foreach_job.sh - runs a templated command for each Jenkins job
  - jenkins_jobs_download_configs.sh - downloads all Jenkins job configs to xml files of the same name
  - jenkins_job_config.sh - gets or sets a Jenkins job's config
  - jenkins_job_description.sh - gets or sets a Jenkins job's description
  - jenkins_job_enable.sh - enables a Jenkins job by name
  - jenkins_job_disable.sh - disables a Jenkins job by name
  - jenkins_job_trigger.sh - triggers a Jenkins job by name
  - jenkins_job_trigger_with_params.sh - triggers a Jenkins job with parameters which can be passed as --data KEY=VALUE
  - jenkins_jobs_enable.sh - enables all Jenkins jobs/pipelines with names matching a given regex
  - jenkins_jobs_disable.sh - disables all Jenkins jobs/pipelines with names matching a given regex
  - jenkins_builds.sh - lists Jenkins latest builds for every job
  - jenkins_cred_add_cert.sh - creates a Jenkins certificate credential from a PKCS#12 keystore
  - jenkins_cred_add_kubernetes_sa.sh - creates a Jenkins Kubernetes service account credential
  - jenkins_cred_add_secret_file.sh - creates a Jenkins secret file credential from a file
  - jenkins_cred_add_secret_text.sh - creates a Jenkins secret string credential from a string or a file
  - jenkins_cred_add_ssh_key.sh - creates a Jenkins SSH key credential from a string or an SSH private key file
  - jenkins_cred_add_user_pass.sh - creates a Jenkins username/password credential
  - jenkins_cred_delete.sh - deletes a given Jenkins credential by id
  - jenkins_cred_list.sh - lists Jenkins credentials IDs and Names
  - jenkins_cred_update_cert.sh - updates a Jenkins certificate credential from a PKCS#12 keystore
  - jenkins_cred_update_kubernetes_sa.sh - updates a Jenkins Kubernetes service account credential
  - jenkins_cred_update_secret_file.sh - updates a Jenkins secret file credential from a file
  - jenkins_cred_update_secret_text.sh - updates a Jenkins secret string credential from a string or a file
  - jenkins_cred_update_ssh_key.sh - updates a Jenkins SSH key credential from a string or an SSH private key file
  - jenkins_cred_update_user_pass.sh - updates a Jenkins username/password credential
  - jenkins_cred_set_cert.sh - creates or updates a Jenkins certificate credential from a PKCS#12 keystore
  - jenkins_cred_set_kubernetes_sa.sh - creates or updates a Jenkins Kubernetes service account credential
  - jenkins_cred_set_secret_file.sh - creates or updates a Jenkins secret file credential from a file
  - jenkins_cred_set_secret_text.sh - creates or updates a Jenkins secret string credential from a string or a file
  - jenkins_cred_set_ssh_key.sh - creates or updates a Jenkins SSH key credential from a string or an SSH private key file
  - jenkins_cred_set_user_pass.sh - creates or updates a Jenkins username/password credential
- jenkins_cli.sh - shortens jenkins-cli.jar command by auto-inferring basic configuations, auto-downloading the CLI if absent, inferrings a bunch of Jenkins related variables like $JENKINS_URL , $JENKINS_CLI_ARGS and authentication using $JENKINS_USER / $JENKINS_PASSWORD , or finds admin password from inside local docker container. Used heavily by jenkins.sh one-shot setup and the following scripts:
  - jenkins_foreach_job_cli.sh - runs a templated command for each Jenkins job
  - jenkins_create_job_parallel_test_runs.sh - creates a freestyle parameterized test sleep job and launches N parallel runs of it to test scaling and parallelization of Jenkins on Kubernetes agents
  - jenkins_create_job_check_gcp_serviceaccount.sh - creates a freestyle test job which runs a GCP Metadata query to determine the GCP serviceaccount the agent pod is operating under to check GKE Workload Identity integration
  - jenkins_jobs_download_configs_cli.sh - downloads all Jenkins job configs to xml files of the same name
  - jenkins_cred_cli_add_cert.sh - creates a Jenkins certificate credential from a PKCS#12 keystore
  - jenkins_cred_cli_add_kubernetes_sa.sh - creates a Jenkins Kubernetes service account credential
  - jenkins_cred_cli_add_secret_file.sh - creates a Jenkins secret file credential from a file
  - jenkins_cred_cli_add_secret_text.sh - creates a Jenkins secret string credential from a string or a file
  - jenkins_cred_cli_add_ssh_key.sh - creates a Jenkins SSH key credential from a string or an SSH private key file
  - jenkins_cred_cli_add_user_pass.sh - creates a Jenkins username/password credential
  - jenkins_cred_cli_delete.sh - deletes a given Jenkins credential by id
  - jenkins_cred_cli_list.sh - lists Jenkins credentials IDs and Names
  - jenkins_cred_cli_update_cert.sh - updates a Jenkins certificate credential from a PKCS#12 keystore
  - jenkins_cred_cli_update_kubernetes_sa.sh - updates a Jenkins Kubernetes service account credential
  - jenkins_cred_cli_update_secret_file.sh - updates a Jenkins secret file credential from a file
  - jenkins_cred_cli_update_secret_text.sh - updates a Jenkins secret string credential from a string or a file
  - jenkins_cred_cli_update_ssh_key.sh - updates a Jenkins SSH key credential from a string or an SSH private key file
  - jenkins_cred_cli_update_user_pass.sh - updates a Jenkins username/password credential
  - jenkins_cred_cli_set_cert.sh - creates or updates a Jenkins certificate credential from a PKCS#12 keystore
  - jenkins_cred_cli_set_kubernetes_sa.sh - creates or updates a Jenkins Kubernetes service account credential
  - jenkins_cred_cli_set_secret_file.sh - creates or updates a Jenkins secret file credential from a file
  - jenkins_cred_cli_set_secret_text.sh - creates or updates a Jenkins secret string credential from a string or a file
  - jenkins_cred_cli_set_ssh_key.sh - creates or updates a Jenkins SSH key credential from a string or an SSH private key file
  - jenkins_cred_cli_set_user_pass.sh - creates or updates a Jenkins username/password credential
- jenkins_password.sh - gets Jenkins admin password from local docker container. Used by jenkins_cli.sh
- jenkins_plugins_latest_versions.sh - finds the latest versions of given Jenkins plugins. Useful to programmatically upgrade your Jenkins on Kubernetes plugins defined in values.yaml
- check_jenkinsfiles.sh - validates all *Jenkinsfile* files in the given directory trees using the online Jenkins validator
- See also Knowledge Base notes for Jenkins.
teamcity/*.sh - TeamCity CI scripts:
- teamcity.sh - one-touch TeamCity CI cluster:
  - launches Docker containers with server and 1 agent
  - click proceed and accept the EULA
  - waits for server to initialize
  - waits for agent to register
  - authorizes agent
  - creates a VCS Root if $PWD has a .teamcity.vcs.json / .teamcity.vcs.ssh.json / .teamcity.vcs.oauth.json and corresponding $TEAMCITY_SSH_KEY or $TEAMCITY_GITHUB_CLIENT_ID + $TEAMCITY_GITHUB_CLIENT_SECRET environment variables
  - creates a Project and imports all settings and builds from the VCS Root
  - creates an admin user and an API token for you
  - see also: TeamCity CI config repo for importing pipelines
- teamcity_api.sh - queries TeamCity's API, auto-handling authentication and other quirks of the API
- teamcity_create_project.sh - creates a TeamCity project using the API
- teamcity_create_github_oauth_connection.sh - creates a TeamCity GitHub OAuth VCS connection in the Root project, useful for bootstrapping projects from VCS configs
- teamcity_create_vcs_root.sh - creates a TeamCity VCS root from a save configuration (XML or JSON), as downloaded by teamcity_export_vcs_roots.sh
- teamcity_upload_ssh_key.sh - uploads an SSH private key to a TeamCity project (for use in VCS root connections)
- teamcity_agents.sh - lists TeamCity agents, their connected state, authorized state, whether enabled and up to date
- teamcity_builds.sh - lists the last 100 TeamCity builds along with the their state (eg. finished ) and status (eg. SUCCESS / FAILURE )
- teamcity_buildtypes.sh - lists TeamCity buildTypes (pipelines) along with the their project and IDs
- teamcity_buildtype_create.sh - creates a TeamCity buildType from a local JSON configuration (see teamcity_buildtypes_download.sh )
- teamcity_buildtype_set_description_from_github.sh - sync's a TeamCity buildType's description from its Github repo description
- teamcity_buildtypes_set_description_from_github.sh - sync's all TeamCity buildType descriptions from their GitHub repos where available
- teamcity_export.sh - downloads TeamCity configs to local JSON files in per-project directories mimicking native TeamCity directory structure and file naming
- teamcity_export_project_config.sh - downloads TeamCity project config to local JSON files
- teamcity_export_buildtypes.sh - downloads TeamCity buildType config to local JSON files
- teamcity_export_vcs_roots.sh - downloads TeamCity VCS root config to local JSON files
- teamcity_projects.sh - lists TeamCity project IDs and Names
- teamcity_project_set_versioned_settings.sh - configures a project to track all changes to a VCS (eg. GitHub)
- teamcity_project_vcs_versioning.sh - quickly toggle VCS versioning on/off for a given TeamCity project (useful for testing without auto-committing)
- teamcity_vcs_roots.sh - lists TeamCity VCS root IDs and Names
travis/*.sh - Travis CI API scripts (one of my all-time favourite CI systems):
- travis_api.sh - queries the Travis CI API with authentication using $TRAVIS_TOKEN
- travis_repos.sh - lists Travis CI repos
- travis_foreach_repo.sh - executes a templated command against all Travis CI repos
- travis_repo_build.sh - triggers a build for the given repo
- travis_repo_caches.sh - lists caches for a given repo
- travis_repo_crons.sh - lists crons for a given repo
- travis_repo_env_vars.sh - lists environment variables for a given repo
- travis_repo_settings.sh - lists settings for a given repo
- travis_repo_create_cron.sh - creates a cron for a given repo and branch
- travis_repo_delete_crons.sh - deletes all crons for a given repo
- travis_repo_delete_caches.sh - deletes all caches for a given repo (sometimes clears build problems)
- travis_delete_cron.sh - deletes a Travis CI cron by ID
- travis_repos_settings.sh - lists settings for all repos
- travis_repos_caches.sh - lists caches for all repos
- travis_repos_crons.sh - lists crons for all repos
- travis_repos_create_cron.sh - creates a cron for all repos
- travis_repos_delete_crons.sh - deletes all crons for all repos
- travis_repos_delete_caches.sh - deletes all caches for all repos
- travis_lint.sh - lints a given .travis.yml using the API
buildkite/*.sh - BuildKite API scripts:
- buildkite_api.sh - queries the BuildKite API, handling authentication using $BUILDKITE_TOKEN
- buildkite_pipelines.sh - list buildkite pipelines for your $BUILDKITE_ORGANIZATION / $BUILDKITE_USER
- buildkite_foreach_pipeline.sh - executes a templated command for each Buildkite pipeline, replacing the {user} and {pipeline} in each iteration
- buildkite_agent.sh - runs a buildkite agent locally on Linux or Mac, or in Docker with choice of Linux distros
- buildkite_agents.sh - lists the Buildkite agents connected along with their hostname, IP, started dated and agent details
- buildkite_pipelines.sh - lists Buildkite pipelines
- buildkite_create_pipeline.sh - create a Buildkite pipeline from a JSON configuration (like from buildkite_get_pipeline.sh or buildkite_save_pipelines.sh )
- buildkite_get_pipeline.sh - gets details for a specific Buildkite pipeline in JSON format
- buildkite_update_pipeline.sh - updates a BuildKite pipeline from a configuration provided via stdin or from a file saved via buildkite_get_pipeline.sh
- buildkite_patch_pipeline.sh - updates a BuildKite pipeline from a partial configuration provided as an arg, via stdin, or from a file saved via buildkite_get_pipeline.sh
- buildkite_pipeline_skip_settings.sh - lists the skip intermediate build settings for one or more given BuildKite pipelines
- buildkite_pipeline_set_skip_settings.sh - configures given or all BuildKite pipelines to skip intermediate builds and cancel running builds in favour of latest build
- buildkite_cancel_scheduled_builds.sh - cancels BuildKite scheduled builds (to clear a backlog due to offline agents and just focus on new builds)
- buildkite_cancel_running_builds.sh - cancels BuildKite running builds (to clear them and restart new later eg. after agent / environment change / fix)
- buildkite_pipeline_disable_forked_pull_requests.sh - disables forked pull request builds on a BuildKite pipeline to protect your build environment from arbitrary code execution security vulnerabilities
- buildkite_pipelines_vulnerable_forked_pull_requests.sh - prints the status of each pipeline, should all return false, otherwise run the above script to close the vulnerability
- buildkite_rebuild_cancelled_builds.sh - triggers rebuilds of last N cancelled builds in current pipeline
- buildkite_rebuild_failed_builds.sh - triggers rebuilds of last N failed builds in current pipeline (eg. after agent restart / environment change / fix)
- buildkite_rebuild_all_pipelines_last_cancelled.sh - triggers rebuilds of the last cancelled build in each pipeline in the organization
- buildkite_rebuild_all_pipelines_last_failed.sh - triggers rebuilds of the last failed build in each pipeline in the organization
- buildkite_retry_jobs_dead_agents.sh - triggers job retries where jobs failed due to killed agents, continuing builds from that point and replacing their false negative failed status with the real final status, slightly better than rebuilding entire jobs which happen under a new build
- buildkite_recreate_pipeline.sh - recreates a pipeline to wipe out all stats (see url and badge caveats in --help )
- buildkite_running_builds.sh - lists running builds and the agent they're running on
- buildkite_save_pipelines.sh - saves all BuildKite pipelines in your $BUILDKITE_ORGANIZATION to local JSON files in $PWD/.buildkite-pipelines/
- buildkite_set_pipeline_description.sh - sets the description of one or more pipelines using the BuildKite API
- buildkite_set_pipeline_description_from_github.sh - sets a Buildkite pipeline description to match its source GitHub repo
- buildkite_sync_pipeline_descriptions_from_github.sh - for all BuildKite pipelines sets each description to match its source GitHub repo
- buildkite_trigger.sh - triggers BuildKite build job for a given pipeline
- buildkite_trigger_all.sh - same as above but for all pipelines
terraform_cloud_*.sh - Terraform Cloud API scripts:
- terraform_cloud_api.sh - queries the Cloudflare API, handling authentication using $TERRAFORM_TOKEN
- terraform_cloud_ip_ranges.sh - returns the list of IP ranges for Terraform Cloud
- terraform_cloud_organizations.sh - lists Terraform Cloud organizations
- terraform_cloud_workspaces.sh - lists Terraform Cloud workspaces
- terraform_cloud_workspace_vars.sh - lists Terraform Cloud workspace variables
- terraform_cloud_workspace_set_vars.sh - adds / updates Terraform workspace-level sensitive environment/terraform variable(s) via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- terraform_cloud_workspace_delete_vars.sh - deletes one or more Terraform workspace-level variables
- terraform_cloud_varsets.sh - lists Terraform Cloud variable sets
- terraform_cloud_varset_vars.sh - lists Terraform Cloud variables in on or all variables sets for the given organization
- terraform_cloud_varset_set_vars.sh - adds / updates Terraform sensitive environment/terraform variable(s) in a given variable set via the API from key=value or shell export format, as args or via stdin (eg. piped from aws_csv_creds.sh )
- terraform_cloud_varset_delete_vars.sh - deletes one or more Terraform variables in a given variable set
terraform_*.sh - Terraform scripts:
- terraform_gcs_backend_version.sh - determines the Terraform state version from the tfstate file in a GCS bucket found in a local given backend.tf
- terraform_gitlab_download_backend_variable.sh - downloads backend.tf from a GitLab CI/CD variable to be able to quickly iterate plans locally
- terraform_import.sh - finds given resource type in ./*.tf code or Terraform plan output that are not in Terraform state and imports them
- terraform_import_aws_iam_users.sh - parses Terraform plan output to import new aws_iam_user additions into Terraform state
- terraform_import_aws_iam_groups.sh - parses Terraform plan output to import new aws_iam_group additions into Terraform state
- terraform_import_aws_iam_policies.sh - parses Terraform plan output to import new aws_iam_policies additions, resolves their ARNs and imports them into Terraform state
- terraform_import_aws_sso_permission_sets.sh - finds all aws_ssoadmin_permission_set in ./*.tf code, resolves the ARNs and imports them to Terraform state
- terraform_import_aws_sso_account_assignments.sh - parses Terraform plan output to import new aws_ssoadmin_account_assignment additions into Terraform state
- terraform_import_aws_sso_managed_policy_attachments.sh - parses Terraform plan output to import new aws_ssoadmin_account_assignment additions into Terraform state
- terraform_import_aws_sso_permission_set_inline_policies.sh - parses Terraform plan output to import new aws_ssoadmin_permission_set_inline_policy additions into Terraform state
- terraform_import_github_repos.sh - finds all github_repository in ./*.tf code or Terraform plan output that are not in Terraform state and imports them. See also github_repos_not_in_terraform.sh
- terraform_import_github_team.sh - imports a given GitHub team into a given Terraform state resource, by first querying the GitHub API for the team ID needed to import into Terraform
- terraform_import_github_teams.sh - finds all github_team in ./*.tf code or Terraform plan output that are not in Terraform state, then queries the GitHub API for their IDs and imports them. See also github_teams_not_in_terraform.sh
- terraform_import_github_team_repos.sh - finds all github_team_repository in Terraform plan that would be added, then queries the GitHub API for the repos and team IDs and if they both exist then imports them to Terraform state
- terraform_resources.sh - external program to get all resource ids and attribute for a given resource type to work around Terraform splat expression limitation (#19931)
- terraform_managed_resource_types.sh - quick parse of what Terraform resource types are found in *.tf files under the current or given directory tree. Useful to give you a quick glance of what services you are managing
- terraform_registry_url_extract.sh - extracts the Terraform Registry URL in either tfr:// or https://registry.terraform.io/ format from a given string, file or standard input. Useful to fast load Terraform Module documentation via editor/IDE hotkeys (see .vimrc). Based on urlextract.sh above
- terraform_registry_url_to_https.sh - converts one or more Terraform Registry URLs from tfr:// to https://registry.terraform.io/ format
- terraform_registry_url_open.sh - opens the Terraform Registry URL given as a string arg, file or standard input in either tfr:// or https://registry.terraform.io/ format
- See also Knowledge Base notes for Terraform.
checkov_resource_*.sh - Checkov resource counts - useful to estimate Bridgecrew Cloud costs which are charged per resource:
- checkov_resource_count.sh - counts the number of resources Checkov is scanning in the current or given directory
- checkov_resource_count_all.sh - counts the total number of resources Checkov is scanning across all given repo checkouts
octopus_api.sh - queries the Octopus Deploy API

See also Knowledge Base notes for CI/CD.

AI & IPaaS

ai/ and ipaas/ directories:

openai_api.sh - queries the OpenAI (ChatGPT) API with authentication
make_api.sh - queries the Make.com API with authentication

Internet Services

internet/ , cloudflare/ , pingdom/ , terraform/ directories:

Pastebins - uploads files and copies the resulting URL to your clipboard:
- code / text only - prompts to approve text / code before upload for safety:
  - pastebin.sh - uploads a file to https://pastebin.com, script auto-determines which syntax highlighting to add since API doesn't auto infer
  - dpaste.sh - uploads a file to https://dpaste.com, script auto-determines which syntax highlighting to add since API doesn't auto infer
  - termbin.sh - uploads a file to https://termbin.com (site has no syntax highlighting)
- all files, multimedia or text / code - prompts to approve text / code before upload for safety:
  - 0x0.sh - uploads a file to https://0x0.st (fast)
  - imgur.sh - uploads an image file to https://imgur.com
  - file.io.sh - uploads a file to https://file.io with 2 weeks, single download retention
  - catbox.sh - uploads a file to https://catbox.moe/ with permanent retention (slow)
  - litterbox.sh - uploads a file to https://litterbox.catbox.moe/ with temporary retention (slow)
digital_ocean_api.sh / doapi.sh - queries the Digital Ocean API with authentication
- see also the Digital Ocean CLI doctl ( install/install_doctl.sh )
atlassian_ip_ranges.sh - lists Atlassian's IPv4 and/or IPv6 cidr ranges via its API
circleci_public_ips.sh - lists CircleCI public IP addresses via dnsjson.com
cloudflare_*.sh - Cloudflare API queries and reports:
- cloudflare_api.sh - queries the Cloudflare API with authentication
- cloudflare_ip_ranges.sh - lists Cloudflare's IPv4 and/or IPv6 cidr ranges via its API
- cloudflare_custom_certificates.sh - lists any custom SSL certificates in a given Cloudflare zone along with their status and expiry date
- cloudflare_dns_records.sh - lists any Cloudflare DNS records for a zone, including the type and ttl
- cloudflare_dns_records_all_zones.sh - same as above but for all zones
- cloudflare_dns_record_create.sh - creates a DNS record in the given domain
- cloudflare_dns_record_update.sh - updates a DNS record in the given domain
- cloudflare_dns_record_delete.sh - deletes a DNS record in the given domain
- cloudflare_dns_record_details.sh - lists the details for a DNS record in the given domain in JSON format for further pipe processing
- cloudflare_dnssec.sh - lists the Cloudflare DNSSec status for all zones
- cloudflare_firewall_rules.sh - lists Cloudflare Firewall rules, optionally with filter expression
- cloudflare_firewall_access_rules.sh - lists Cloudflare Firewall Access rules, optionally with filter expression
- cloudflare_foreach_account.sh - executes a templated command for each Cloudflare account, replacing the {account_id} and {account_name} in each iteration (useful for chaining with cloudflare_api.sh )
- cloudflare_foreach_zone.sh - executes a templated command for each Cloudflare zone, replacing the {zone_id} and {zone_name} in each iteration (useful for chaining with cloudflare_api.sh , used by adjacent cloudflare_*_all_zones.sh scripts)
- cloudflare_purge_cache.sh - purges the entire Cloudflare cache
- cloudflare_ssl_verified.sh - gets the Cloudflare zone SSL verification status for a given zone
- cloudflare_ssl_verified_all_zones.sh - same as above for all zones
- cloudflare_zones.sh - lists Cloudflare zone names and IDs (needed for writing Terraform Cloudflare code)
datadog_api.sh - queries the DataDog API with authentication
dnsjson.sh - queries dnsjson.com for DNS records
gitguardian_api.sh - queries the GitGuardian API with authentication
jira_api.sh - queries Jira API with authentication
kong_api.sh - queries the Kong API Gateway's Admin API, handling authentication if enabled
traefik_api.sh - queries the Traefik API, handling authentication if enabled
ngrok_api.sh - queries the NGrok API with authentication
pingdom_*.sh - Pingdom API queries and reports for status, latency, average response times, latency averages by hour, SMS credits, outages periods and durations over the last year etc.
- pingdom_api.sh - queries the Solarwinds Pingdom API with authentication
- pingdom_foreach_check.sh - executes a templated command against each Pingdom check, replacing the {check_id} and {check_name} in each iteration
- pingdom_checks.sh - show all Pingdom checks, status and latencies
- pingdom_checks_outages.sh / pingdom_checks_outages.sh - show one or all Pingdom checks outage histories for the last year
- pingdom_checks_average_response_times.sh - shows the average response times for all Pingdom checks for the last week
- pingdom_check_latency_by_hour.sh / pingdom_checks_latency_by_hour.sh - shows the average latency for one or all Pingdom checks broken down by hour of the day, over the last week
- pingdom_sms_credits.sh - gets the remaining number of Pingdom SMS credits
terraform_cloud_api.sh - queries Terraform Cloud API with authentication
terraform_cloud_ip_ranges.sh - returns the list of IP ranges for Terraform Cloud via the API, or optionally one or more of the ranges used by different functions
wordpress.sh - boots Wordpress in docker with a MySQL backend, and increases the upload_max_filesize to be able to restore a real world sized export backup
wordpress_api.sh - queries the Wordpress API with authentication
wordpress_posts_without_category_tags.sh - checks posts (articles) for categories without corresponding tags and prints the posts and their missing tags

자바

java/ directory:

java_show_classpath.sh - shows Java classpaths, one per line, of currently running Java programs
jvm_heaps*.sh - show all your Java heap sizes for all running Java processes, and their total MB (for performance tuning and sizing)
Java Decompilers:
- java_decompile_jar.sh - decompiles a Java JAR in /tmp, finds the main class and runs a Java decompiler on its main .class file using jd_gui.sh
- jd_gui.sh - runs Java Decompiler JD GUI, downloading its jar the first time if it's not already present
- bytecode_viwer.sh - runs Bytecode-Viewer GUI Java decompiler, downloading its jar the first time if it's not already present
- cfr.sh - runs CFR command line Java decompiler, downloading its jar the first time if it's not already present
- procyon.sh - runs Procyon command line Java decompiler, downloading its jar the first time if it's not already present

See also Knowledge Base notes for Java and JVM Performance Tuning.

파이썬

python/ directory:

python_compile.sh - byte-compiles Python scripts and libraries into .pyo optimized files
python_pip_install.sh - bulk installs PyPI modules from mix of arguments / file lists / stdin, accounting for User vs System installs, root vs user sudo, VirtualEnvs / Anaconda / GitHub Workflows/ Google Cloud Shell, Mac vs Linux library paths, and ignore failure option
python_pip_install_if_absent.sh - installs PyPI modules not already in Python libary path (OS or pip installed) for faster installations only where OS packages are already providing some of the modules, reducing time and failure rates in CI builds
python_pip_install_for_script.sh - installs PyPI modules for given script(s) if not already installed. Used for dynamic individual script dependency installation in the DevOps Python tools repo
python_pip_reinstall_all_modules.sh - reinstalls all PyPI modules which can fix some issues
pythonpath.sh - prints all Python libary search paths, one per line
python_find_library_path.sh - finds directory where a PyPI module is installed - without args finds the Python library base
python_find_library_executable.sh - finds directory where a PyPI module's CLI program is installed (system vs user, useful when it gets installed to a place that isn't in your $PATH , where which won't help)
python_find_unused_pip_modules.sh - finds PyPI modules that aren't used by any programs in the current directory tree
python_find_duplicate_pip_requirements.sh - finds duplicate PyPI modules listed for install under the directory tree (useful for deduping module installs in a project and across submodules)
python_translate_import_module.sh - converts Python import modules to PyPI module names, used by python_pip_install_for_script.sh
python_translate_module_to_import.sh - converts PyPI module names to Python import names, used by python_pip_install_if_absent.sh and python_find_unused_pip_modules.sh
python_pyinstaller.sh - creates PyInstaller self-contained Python programs with Python interpreter and all PyPI modules included
python_pypi_versions.sh - prints all available versions of a given PyPi module using the API

See also Knowledge Base notes for Python.

펄

perl/ directory:

perl_cpanm_install.sh - bulk installs CPAN modules from mix of arguments / file lists / stdin, accounting for User vs System installs, root vs user sudo, Perlbrew / Google Cloud Shell environments, Mac vs Linux library paths, ignore failure option, auto finds and reads build failure log for quicker debugging showing root cause error in CI builds logs etc
perl_cpanm_install_if_absent.sh - installs CPAN modules not already in Perl libary path (OS or CPAN installed) for faster installations only where OS packages are already providing some of the modules, reducing time and failure rates in CI builds
perl_cpanm_reinstall_all.sh - re-installs all CPAN modules. Useful for trying to recompile XS modules on Macs after migration assistant from an Intel Mac to an ARM Silicon Mac leaves your home XS libraries broken as they're built for the wrong architecture
perlpath.sh - prints all Perl libary search paths, one per line
perl_find_library_path.sh - finds directory where a CPAN module is installed - without args finds the Perl library base
perl_find_library_executable.sh - finds directory where a CPAN module's CLI program is installed (system vs user, useful when it gets installed to a place that isn't in your $PATH , where which won't help)
perl_find_unused_cpan_modules.sh - finds CPAN modules that aren't used by any programs in the current directory tree
perl_find_duplicate_cpan_requirements.sh - finds duplicate CPAN modules listed for install more than once under the directory tree (useful for deduping module installs in a project and across submodules)
perl_generate_fatpacks.sh - creates Fatpacks - self-contained Perl programs with all CPAN modules built-in

Golang

packages/ directory:

golang_install.sh - bulk installs Golang modules from mix of arguments / file lists / stdin
golang_install_if_absent.sh - same as above but only if the package binary isn't already available in $PATH
golang_rm_binaries.sh - deletes binaries of the same name adjacent to .go files. Doesn't delete your bin/ etc as these are often real deployed applications rather than development binaries

메디아

media/ directory:

이미지

image_join_vertical.sh - joins two images top and bottom after matching their widths so they align correctly
image_join_horizontal.sh - joins two images left and right after matching their heights so they align correctly
imageopen.sh - opens the given image file using whatever available tool is found on Linux or Mac
svg_to_png.sh - convert an SVG image to PNG to be usable on websites that don't support SVG images like LinkedIn, Medium or Reddit
avif_to_png.sh - convert an Avif image to PNG to be usable on websites that don't support Webp images like LinkedIn
webp_to_png.sh - convert a Webp image to PNG to be usable on websites that don't support Webp images like Medium

오디오

mp3_set_artist.sh / mp3_set_album.sh - set the artist / album tag for all mp3 files under given directories. Useful for grouping artists/albums and audiobook author/books (eg. for correct importing into Mac's Books.app)
mp3_set_track_name.sh - set the track name metadata for mp3 files under given directories to follow their filenames. Useful for correctly displaying audiobook progress / chapters etc.
mp3_set_track_order.sh - set the track order metadata for mp3 files under given directories to follow the lexical file naming order. Useful for correctly ordering album songs and audiobook chapters (eg. for Mac's Books.app). Especially useful for enforcing global ordering on multi-CD audiobooks after grouping into a single audiobook using mp3_set_album.sh (otherwise default track numbers in each CD interleave in Mac's Books.app)

동영상

avi_to_mp4.sh - convert avi files to mp4 using ffmpeg. Useful to be able to play videos on devices like smart TVs that may not recognize newer codecs otherwise
mkv_to_mp4.sh - convert mkv files to mp4 using ffmpeg. Same use case as above
youtube_download_channel.sh - downloads all videos from a given YouTube channel URL

See also Knowledge Base notes for MultiMedia.

Spotify

40+ Spotify API scripts (used extensively to manage my Spotify-Playlists repo).

spotify/ directory:

spotify_playlists*.sh - list playlists in either <id> <name> or JSON format
spotify_playlist_tracks*.sh - gets playlist contents as track URIs / Artists - Track / CSV format - useful for backups or exports between music systems
spotify_backup.sh - backup all Spotify playlists as well as the ordered list of playlists
spotify_backup_playlist*.sh - backup Spotify playlists to local files in both human readable Artist - Track format and Spotify URI format for easy restores or adding to new playlists
spotify_search*.sh - search Spotify's library for tracks / albums / artists getting results in human readable format, JSON, or URI formats for easy loading to Spotify playlists
spotify_release_year.sh - searches for a given track or album and finds the original release year
spotify_uri_to_name.sh - convert Spotify track / album / artist URIs to human readable Artist - Track / CSV format. Takes Spotify URIs, URL links or just IDs. Reads URIs from files or standard input
spotify_create_playlist.sh - creates a Spotify playlist, either public or private
spotify_rename_playlist.sh - renames a Spotify playlist
spotify_set_playlists_public.sh / spotify_set_playlists_private.sh - sets one or more given Spotify playlists to public / private
spotify_add_to_playlist.sh - adds tracks to a given playlist. Takes a playlist name or ID and Spotify URIs in any form from files or standard input. Can be combined with many other tools listed here which output Spotify URIs, or appended from other playlists. Can also be used to restore a spotify playlist from backups
spotify_delete_from_playlist.sh - deletes tracks from a given playlist. Takes a playlist name or ID and Spotify URIs in any form from files or standard input, optionally prefixed with a track position to remove only specific occurrences (useful for removing duplicates from playlists)
spotify_delete_from_playlist_if_in_other_playlists.sh - deletes tracks from a given playlist if their URIs are found in the subsequently given playlists
spotify_delete_from_playlist_if_track_in_other_playlists.sh - deletes tracks from a given playlist if their 'Artist - Track' name match are found in the subsequently given playlists (less accurate than exact URI deletion above)
spotify_duplicate_uri_in_playlist.sh - finds duplicate Spotify URIs in a given playlist (these are guaranteed exact duplicate matches), returns all but the first occurrence and optionally their track positions (zero-indexed to align with the Spotify API for easy chaining with other tools)
spotify_duplicate_tracks_in_playlist.sh - finds duplicate Spotify tracks in a given playlist (these are idential Artist - Track name matches, which may be from different albums / singles)
spotify_delete_duplicates_in_playlist.sh - deletes duplicate Spotify URI tracks (identical) in a given playlist using spotify_duplicate_uri_in_playlist.sh and spotify_delete_from_playlist.sh
spotify_delete_duplicate_tracks_in_playlist.sh - deletes duplicate Spotify tracks (name matched) in a given playlist using spotify_duplicate_tracks_in_playlist.sh and spotify_delete_from_playlist.sh
spotify_delete_any_duplicates_in_playlist.sh - calls both of the above scripts to first get rid of duplicate URIs and then remove any other duplicates by track name matches
spotify_playlist_tracks_uri_in_year.sh - finds track URIs in a playlist where their original release date is in a given year or decade (by regex match)
spotify_playlist_uri_offset.sh - finds the offset of a given track URI in a given playlist, useful to find positions to resume processing a large playlist
spotify_top_artists*.sh - lists your top artists in URI or human readable format
spotify_top_tracks*.sh - lists top tracks in URI or human readable format
spotify_liked_tracks*.sh - lists your Liked Songs in URI or human readable formats
spotify_liked_artists*.sh - list artists from Liked Songs in URI or human readable formats
spotify_artists_followed*.sh - lists all followed artists in URI or human readable formats
spotify_artist_tracks.sh - gets all track URIs for a given artist, from both albums and single for chain loading to playlists
spotify_follow_artists.sh - follows artists for the given URIs from files or standard input
spotify_follow_top_artists.sh - follows all artists in your current Spotify top artists list
spotify_follow_liked_artists.sh - follows artists with N or more tracks in your Liked Songs
spotify_set_tracks_uri_to_liked.sh - sets a list of spotify track URIs to 'Liked' so they appear in the Liked Songs playlist. Useful for marking all the tracks in your best playlists as favourite tracks, or for porting historical Starred tracks to the newer Liked Songs
spotify_foreach_playlist.sh - executes a templated command against all playlists, replacing {playlist} and {playlist_id} in each iteration
spotify_playlist_name_to_id.sh / spotify_playlist_id_to_name.sh - convert playlist names <=> IDs
spotify_api_token.sh - gets a Spotify authentication token using either Client Credentials or Authorization Code authentication flows, the latter being able to read/modify private user data, automatically used by spotify_api.sh
spotify_api.sh - query any Spotify API endpoint with authentication, used by adjacent spotify scripts

More Linux & Mac

bin/ , install/ , packages/ , setup/ directories:

Linux / Mac systems administration scripts:
- install/ - installation scripts for various OS packages (RPM, Deb, Apk) for various Linux distros (Redhat RHEL / CentOS / Fedora, Debian / Ubuntu, Alpine)
- install if absent scripts for Python, Perl, Ruby, NodeJS and Golang packages - good for minimizing the number of source code installs by first running the OS install scripts and then only building modules which aren't already detected as installed (provided by system packages), speeding up builds and reducing the likelihood of compile failures
- install scripts for tarballs, Golang binaries, random 3rd party installers, Jython and build tools like Gradle and SBT for when Linux distros don't provide packaged versions or where the packaged versions are too old
- packages/ - OS / Distro Package Management:
  - install_packages.sh - installs package lists from arguments, files or stdin on major linux distros and Mac, detecting the package manager and invoking the right install commands, with sudo if not root. Works on RHEL / CentOS / Fedora, Debian / Ubuntu, Alpine, and Mac Homebrew. Leverages and supports all features of the distro / OS specific install scripts listed below
  - install_packages_if_absent.sh - installs package lists if they're not already installed, saving time and minimizing install logs / CI logs, same support list as above
  - Redhat RHEL / CentOS:
    - yum_install_packages.sh / yum_remove_packages.sh - installs RPM lists from arguments, files or stdin. Handles Yum + Dnf behavioural differences, calls sudo if not root, auto-attempts variations of python/python2/python3 package names. Avoids yum slowness by checking if rpm is installed before attempting to install it, accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across RHEL/CentOS/Fedora versions)
    - yum_install_packages_if_absent.sh - installs RPMs only if not already installed and not a metapackage provided by other packages (eg. vim metapackage provided by vim-enhanced ), saving time and minimizing install logs / CI logs, plus all the features of yum_install_packages.sh above
    - rpms_filter_installed.sh / rpms_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
  - Debian / Ubuntu:
    - apt_install_packages.sh / apt_remove_packages.sh - installs Deb package lists from arguments, files or stdin. Auto calls sudo if not root, accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across Debian/Ubuntu distros/versions)
    - apt_install_packages_if_absent.sh - installs Deb packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of apt_install_packages.sh above
    - apt_wait.sh - blocking wait on concurrent apt locks to avoid failures and continue when available, mimicking yum's waiting behaviour rather than error'ing out
    - debs_filter_installed.sh / debs_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
  - Alpine:
    - apk_install_packages.sh / apk_remove_packages.sh - installs Alpine apk package lists from arguments, files or stdin. Auto calls sudo if not root, accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across Alpine versions)
    - apk_install_packages_if_absent.sh - installs Alpine apk packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of apk_install_packages.sh above
    - apk_filter_installed.sh / apk_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
  - 스코틀랜드 사람:
    - brew_install_packages.sh / brew_remove_packages.sh - installs Mac Hombrew package lists from arguments, files or stdin. Accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across versions)
    - brew_install_packages_if_absent.sh - installs Mac Homebrew packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of brew_install_packages.sh above
    - brew_filter_installed.sh / brew_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
    - brew_package_owns.sh - finds which brew package owns a given filename argument
all builds across all my GitHub repos now make system-packages before make pip / make cpan to shorten how many packages need installing, reducing chances of build failures

Builds, Languages & Linting

bin/ , checks/ , cicd/ or language specific directories:

lint.sh - lints one or more files, auto-determines the file types, parses lint headers and calls appropriate scripts and tools. Integrated with my custom .vimrc
run.sh - runs one or more files, auto-determines the file types, any run or arg headers and executes each file using the appropriate script or CLI tool. Integrated with my custom .vimrc
check_*.sh - extensive collection of generalized tests - these run against all my GitHub repos via CI. 몇 가지 예 :
- Programming language linting:
  - Python (syntax, pep8, byte-compiling, reliance on asserts which can be disabled at runtime, except/pass etc.)
  - 펄
  - 자바
  - 스칼라
  - 루비
  - Bash / Shell
  - Misc (whitespace, custom code checks etc.)
- Build System, Docker & CI linting:
  - 만들다
  - Maven
  - SBT
  - Gradle
  - Travis CI
  - Circle CI
  - GitLab CI
  - Concourse CI
  - Codefresh CI
  - Dockerfiles
  - Docker Compose
  - Vagrantfiles

Individual Setup Parts

Optional, only if you don't do the full make install .

Install only OS system package dependencies and AWS CLI via Python Pip (doesn't symlink anything to $HOME ):

make

Adds sourcing to .bashrc and .bash_profile and symlinks dot config files to $HOME (doesn't install OS system package dependencies):

make link

undo via

make unlink

Install only OS system package dependencies (doesn't include AWS CLI or Python packages):

make system-packages

Install AWS CLI:

make aws

Install Azure CLI:

make azure

Install GCP GCloud SDK (includes CLI):

make gcp

Install GCP GCloud Shell environment (sets up persistent OS packages and all home directory configs):

make gcp-shell

Install generically useful Python CLI tools and modules (includes AWS CLI, autopep8 etc):

make python

Full Help

 > make help

 Usage:

  Common Options:

    make help                   show this message
    make build                  installs all dependencies - OS packages and any language libraries via native tools eg. pip, cpanm, gem, go etc that are not available via OS packages
    make build-retry            retries ' make build ' x 3 until success to try to mitigate temporary upstream repo failures triggering false alerts in CI systems
    make ci                     prints env, then runs ' build-retry ' for more resilient CI builds with debugging
    make printenv               prints environment variables, CPU cores, OS release, $PWD , Git branch, hashref etc. Useful for CI debugging
    make system-packages        installs OS packages only (detects OS via whichever package manager is available)
    make test                   run tests
    make clean                  removes compiled / generated files, downloaded tarballs, temporary files etc.

    make submodules             initialize and update submodules to the right release (done automatically by build / system-packages)
    make init                   same as above, often useful to do in CI systems to get access to additional submodule provided targets such as ' make ci '

    make cpan                   install any modules listed in any cpan-requirements.txt files if not already installed

    make pip                    install any modules listed in any requirements.txt files if not already installed

    make python-compile         compile any python files found in the current directory and 1 level of subdirectory
    make pycompile

    make github                 open browser at github project
    make readme                 open browser at github ' s README
    make github-url             print github url and copy to clipboard
    make status                 open browser at Github CI Builds overview Status page for all projects

    make ls                     print list of code files in project
    make wc                     show counts of files and lines

  Repo specific options:

    make install                builds all script dependencies, installs AWS CLI, symlinks all config files to $HOME and adds sourcing of bash profile

    make link                   symlinks all config files to $HOME and adds sourcing of bash profile
    make unlink                 removes all symlinks pointing to this repo ' s config files and removes the sourcing lines from .bashrc and .bash_profile

    make python-desktop         installs all Python Pip packages for desktop workstation listed in setup/pip-packages-desktop.txt
    make perl-desktop           installs all Perl CPAN packages for desktop workstation listed in setup/cpan-packages-desktop.txt
    make ruby-desktop           installs all Ruby Gem packages for desktop workstation listed in setup/gem-packages-desktop.txt
    make golang-desktop         installs all Golang packages for desktop workstation listed in setup/go-packages-desktop.txt
    make nodejs-desktop         installs all NodeJS packages for desktop workstation listed in setup/npm-packages-desktop.txt

    make desktop                installs all of the above + many desktop OS packages listed in setup/

    make mac-desktop            all of the above + installs a bunch of major common workstation software packages like Ansible, Terraform, MiniKube, MiniShift, SDKman, Travis CI, CCMenu, Parquet tools etc.
    make linux-desktop

    make ls-scripts             print list of scripts in this project, ignoring code libraries in lib/ and .bash.d/

    make github-cli             installs GitHub CLI
    make kubernetes             installs Kubernetes kubectl and kustomize to ~ /bin/
    make terraform              installs Terraform to ~ /bin/
    make vim                    installs Vundle and plugins
    make tmux                   installs TMUX TPM and plugin for kubernetes context
    make ccmenu                 installs and (re)configures CCMenu to watch this and all other major HariSekhon GitHub repos
    make status                 open the Github Status page of all my repos build statuses across all CI platforms

    make aws                    installs AWS CLI tools
    make azure                  installs Azure CLI
    make gcp                    installs Google Cloud SDK
    make digital-ocean          installs Digital Ocean CLI

    make aws-shell              sets up AWS Cloud Shell: installs core packages and links configs
                                (maintains itself across future Cloud Shells via .aws_customize_environment hook)
    make gcp-shell              sets up GCP Cloud Shell: installs core packages and links configs
                                (maintains itself across future Cloud Shells via .customize_environment hook)
    make azure-shell            sets up Azure Cloud Shell (limited compared to gcp-shell, doesn ' t install OS packages since there is no sudo)

Now exiting usage help with status code 3 to explicitly prevent silent build failures from stray ' help ' arguments
make: *** [help] Error 3

( make help exits with error code 3 like most of my programs to differentiate from build success to make sure a stray help argument doesn't cause silent build failure with exit code 0)

DevOps Bash tools