SSL/TLS Certificates: Complete Understanding¶

Table of Contents¶

1. The Fundamental Problem
2. What is an SSL/TLS Certificate?
3. Types of Certificates
4. Certificate Files Explained
5. Certificate Authorities
6. All Methods to Get Certificates
7. Manual Certificate Generation
8. Let's Encrypt vs ZeroSSL vs Commercial CAs
9. Using Certificates in Different Services
10. Understanding the Certificate Chain

The Fundamental Problem¶

Without HTTPS (Plain HTTP)¶

Your Browser → http://jenkins.ibtisam-iq.com
    ↓
Data travels in PLAIN TEXT
    ↓
Anyone in the middle can:
- Read your passwords
- See all data
- Modify requests

Problem: Data travels unencrypted - anyone can intercept and read it!

With HTTPS (HTTP + SSL/TLS)¶

Your Browser → https://jenkins.ibtisam-iq.com
    ↓
Data is ENCRYPTED
    ↓
Middle person sees: ���1j#@k!x8%^
Cannot read or modify!

Solution: Data is encrypted end-to-end - only browser and server can decrypt it!

What is an SSL/TLS Certificate?¶

Real-World Analogy¶

Certificate = Passport or Government ID

Your Passport contains:
- Your name
- Your photo
- Issued by: Government
- Expiry date
- Government's signature

SSL Certificate contains:
- Domain name (jenkins.ibtisam-iq.com)
- Public key
- Issued by: Certificate Authority
- Expiry date
- CA's digital signature

Key Point: Just like a passport proves your identity, an SSL certificate proves a website's identity.

What Certificate Does¶

Two main functions:

1. Encryption - Encrypts data so no one can read it in transit
2. Authentication - Proves you're talking to the legitimate website, not an imposter

Types of Certificates¶

1. Self-Signed Certificate¶

Who signs it? You sign it yourself

Analogy: Creating your own ID card without government verification - anyone can claim to be anyone!

How it works:

You create certificate
You sign it with your own private key
You say: "Trust me, I am jenkins.ibtisam-iq.com"

Browser's response:

⚠️ "Not Secure"
⚠️ "Your connection is not private"
⚠️ NET::ERR_CERT_AUTHORITY_INVALID

Why browser doesn't trust it? - Anyone can create self-signed certificates - No third-party verification - Could be an attacker pretending to be the legitimate site - No way to verify authenticity

When to use: - ✅ Development and testing environments - ✅ Internal company networks (where you control all clients) - ✅ localhost testing - ✅ Learning and experimentation - ❌ NEVER for public-facing websites

Pros: - Free - No external dependencies - Full control - Instant creation

Cons: - Browser warnings scare users - No trust from third parties - Manual trust configuration needed on all clients - Not suitable for production

2. CA-Signed Certificate (Trusted)¶

Who signs it? Trusted Certificate Authority (CA)

Analogy: Government-issued passport - universally trusted because a recognized authority verified your identity!

How it works:

You generate Certificate Signing Request (CSR)
Send CSR to CA (Let's Encrypt, ZeroSSL, etc.)
CA verifies: "Yes, you control jenkins.ibtisam-iq.com"
CA signs your certificate with THEIR private key
Browser trusts CA → Browser trusts your certificate

Browser's response:

✅ 🔒 Secure
✅ No warnings
✅ Green padlock icon

Why browser trusts it?

• CA is pre-trusted by browser (built into trust store)
• CA performed domain ownership verification
• CA's digital signature acts as proof of legitimacy
• Chain of trust leads to trusted root CA

Types based on validation level:

DV Certificate:

• Verifies you control the domain
• No company verification
• Fastest to get
• Good for most use cases

OV Certificate:

• Verifies domain ownership
• Verifies company is real and registered
• Shows organization name in certificate
• Better for corporate sites

EV Certificate:

• Most rigorous verification
• Legal entity verification required
• Physical address verification
• Historically showed company name in green bar (removed in modern browsers)
• Highest trust level for financial/sensitive sites

Certificate Files Explained¶

When working with SSL certificates, you'll encounter various files. Understanding each one is critical.

The Files You'll See¶

1. Private Key (.key, privkey.pem)¶

What is it?

Secret key used for decryption
NEVER share with anyone
Used to decrypt incoming encrypted data
If compromised, attacker can decrypt all traffic

File format:

-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC7...
(Base64 encoded binary data - appears as random characters)
...
-----END PRIVATE KEY-----

Critical security points:

• Keep this file absolutely secret
• Store in restricted directory (permissions 600 or 400)
• Never commit to version control
• Never send via email or unsecured channels
• If lost, you must regenerate entire certificate
• If stolen, attacker can:
• Decrypt your HTTPS traffic
• Impersonate your website
• Sign malicious content

Analogy: Your house's master key - anyone with it can unlock everything!

Typical location:

• /etc/ssl/private/ (Linux)
• /etc/letsencrypt/live/domain/privkey.pem (Let's Encrypt)

2. Certificate Signing Request (.csr)¶

What is it?

Request file you submit to Certificate Authority
Contains domain name, organization info, and public key
Does NOT contain private key (safe to share)
One-time use - discard after receiving signed certificate

File format:

-----BEGIN CERTIFICATE REQUEST-----
MIICvDCCAaQCAQAwdzELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFV0...
-----END CERTIFICATE REQUEST-----

What it contains:

• Common Name (CN): jenkins.ibtisam-iq.com
• Organization (O): Your Company Name
• Country (C): PK
• State/Province (ST): Punjab
• Locality (L): Rawalpindi
• Public Key: Cryptographic public key
• Your Signature: Proves you generated this request

Analogy: Passport application form you submit to government office - contains your information but not your actual passport yet.

When you need it:

• Manual certificate requests to commercial CAs
• ZeroSSL web interface
• Corporate certificate management
• Not needed for automated tools like Certbot

3. Public Certificate (.crt, .cert, cert.pem)¶

What is it?

Your actual signed certificate
Contains your domain and public key
Signed by Certificate Authority
Safe to share publicly (given to all clients)

File format:

-----BEGIN CERTIFICATE-----
MIIFjTCCBHWgAwIBAgISA1234567890abcdefghijklmnopqr...
(Certificate data - readable with OpenSSL tools)
...
-----END CERTIFICATE-----

What it contains:

• Subject: jenkins.ibtisam-iq.com (your domain)
• Issuer: Let's Encrypt Authority X3 (who signed it)
• Public Key: Used for encryption
• Validity Period: Not Before / Not After dates
• Serial Number: Unique identifier
• Signature Algorithm: RSA, ECDSA, etc.
• CA's Digital Signature: Proof of authenticity

Analogy: Your actual issued passport - proves your identity to others.

View certificate contents:

openssl x509 -in cert.pem -text -noout

4. Certificate Chain (chain.pem, fullchain.pem)¶

What is it?

Complete trust chain from your certificate to root CA
Your certificate + all intermediate certificates
Required for browsers to verify trust path

Why needed? Browsers need complete path to verify trust:

Your Certificate
    ↓ signed by
Intermediate Certificate 1
    ↓ signed by
Intermediate Certificate 2
    ↓ signed by
Root Certificate (pre-installed in browser)

File format:

-----BEGIN CERTIFICATE-----
(Your certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate 1)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate 2)
-----END CERTIFICATE-----

fullchain.pem vs chain.pem:

• fullchain.pem = Your certificate + intermediates (use this for Nginx)
• chain.pem = Only intermediate certificates (without your cert)
• cert.pem = Only your certificate (without intermediates)

Common mistake:

# Wrong - only your certificate
ssl_certificate /path/to/cert.pem;

# Correct - certificate + intermediates
ssl_certificate /path/to/fullchain.pem;

Result of wrong configuration:

• Works in some browsers (Chrome caches intermediates)
• Fails in others (Firefox, curl, mobile browsers)
• Error: "Unable to verify certificate chain"

File Extensions Explained¶

Different extensions, often same content:

PEM Format (most common):

• Text-based format
• Base64 encoded binary data
• Can open with any text editor
• Has BEGIN/END markers
• Can contain multiple items (cert, key, chain)

DER Format:

• Binary format
• Cannot read in text editor
• More compact than PEM
• Common in Windows and Java environments

PKCS#12 Format (.p12, .pfx):

• Password-protected bundle
• Contains private key + certificate + chain
• Binary format
• Used by Jenkins, Tomcat, Windows IIS

Converting between formats:

# PEM to DER
openssl x509 -in cert.pem -outform DER -out cert.der

# DER to PEM
openssl x509 -in cert.der -inform DER -out cert.pem

# PEM to PKCS#12 (for Jenkins)
openssl pkcs12 -export -in cert.pem -inkey privkey.pem -out cert.p12

# PKCS#12 to PEM
openssl pkcs12 -in cert.p12 -out cert.pem -nodes

File Organization Best Practices¶

Typical Let's Encrypt structure:

/etc/letsencrypt/
├── live/
│   └── jenkins.ibtisam-iq.com/
│       ├── privkey.pem       # Private key
│       ├── fullchain.pem     # Certificate + chain
│       ├── cert.pem          # Certificate only
│       └── chain.pem         # Intermediate certs only
└── archive/
    └── jenkins.ibtisam-iq.com/
        ├── privkey1.pem      # Actual private key
        ├── fullchain1.pem    # Actual full chain
        └── ...               # Numbered versions

Note: Files in live/ are symlinks to actual files in archive/

Manual certificate organization:

/etc/ssl/
├── private/              # Private keys (permissions 700)
│   └── jenkins.key       # chmod 600
├── certs/                # Public certificates
│   ├── jenkins.crt
│   └── jenkins-fullchain.crt
└── csr/                  # Certificate requests
    └── jenkins.csr

Permissions:

# Private keys - only root can read
chmod 600 /etc/ssl/private/*.key
chown root:root /etc/ssl/private/*.key

# Certificates - readable by all
chmod 644 /etc/ssl/certs/*.crt

# Directories
chmod 700 /etc/ssl/private
chmod 755 /etc/ssl/certs

Certificate Authorities¶

What is a Certificate Authority (CA)?¶

Definition: A trusted third-party organization that verifies domain ownership and issues signed certificates.

Analogy: Government passport office - everyone trusts their verification process and signature.

What CAs do:

1. Verify you own the domain
2. Sign your certificate with their private key
3. Maintain certificate revocation lists
4. Provide validation infrastructure
5. Ensure certificate standards compliance

How trust works:

• Browsers ship with pre-installed list of trusted root CAs
• Operating systems maintain trust stores
• When you visit HTTPS site, browser checks certificate chain
• If chain leads to trusted root CA, connection is trusted
• If chain is broken or leads to unknown CA, browser warns user

Popular Certificate Authorities¶

1. Let's Encrypt (Free, Automated)¶

Overview:

• Non-profit Certificate Authority
• Launched in 2016
• Completely free forever
• Automated certificate issuance and renewal
• Backed by major tech companies (Mozilla, Cisco, Google, AWS)

Features:

• ✅ 100% free
• ✅ Automated via ACME protocol
• ✅ Domain Validated (DV) certificates only
• ✅ Wildcard certificates supported
• ✅ 90-day validity (intentionally short for security)
• ✅ Automatic renewal via Certbot
• ❌ No Organization Validated (OV) certificates
• ❌ No Extended Validation (EV) certificates
• ❌ No commercial support (community only)
• ❌ Rate limits apply

Rate limits:

• 50 certificates per registered domain per week
• 5 duplicate certificates per week
• 300 new orders per account per 3 hours
• Usually not an issue for normal use

Best for:

• Personal websites and blogs
• Startups and small businesses
• Automated deployments
• Developers comfortable with CLI tools
• Any scenario where free is important

Not suitable for:

• Organizations requiring OV/EV validation
• Enterprises needing commercial support
• Situations requiring 1+ year validity

2. ZeroSSL (Free + Paid Tiers)¶

Overview:

• Commercial CA owned by Sectigo
• Offers free tier similar to Let's Encrypt
• Also has paid enterprise tiers
• Web-based dashboard available

Features:

• ✅ Free tier available (90-day DV certificates)
• ✅ Web dashboard for easier management
• ✅ Multiple verification methods (HTTP, DNS, CNAME, Email)
• ✅ REST API for automation
• ✅ Can generate certificates from browser
• ✅ Paid plans include Organization Validation
• ✅ Commercial support on paid tiers
• ⚠️ Free certificates require manual renewal
• ⚠️ Free tier has certificate limits
• ⚠️ API rate limits on free tier

Free vs Paid:

Best for:

• Users who prefer web interface over CLI
• Small teams managing multiple certificates manually
• Businesses that might need OV validation later
• Organizations wanting commercial support option

3. Commercial Certificate Authorities (Paid)¶

Major providers:

• DigiCert
• Sectigo (formerly Comodo)
• GlobalSign
• GoDaddy
• Entrust
• Thawte

Typical costs:

• DV: \(50-\)150/year
• OV: \(100-\)300/year
• EV: \(200-\)1,500/year
• Wildcard: \(200-\)500/year

Features:

• ✅ Organization Validation (OV)
• ✅ Extended Validation (EV)
• ✅ Longer validity periods (up to 1 year, limited by CAB Forum)
• ✅ Warranty/insurance (up to $1.75M coverage)
• ✅ 24/7 phone and email support
• ✅ Compliance certifications (SOC 2, WebTrust)
• ✅ Better for regulatory compliance (PCI-DSS, HIPAA)
• ✅ Account management tools
• ✅ Bulk purchase discounts
• ❌ Expensive
• ❌ Manual processes often involved
• ❌ Overkill for most use cases

When commercial CAs make sense:

• Enterprise environments with budget
• Organizations requiring OV/EV validation
• Compliance requirements (banking, healthcare, government)
• Need for warranty/insurance
• 24/7 support requirement
• Corporate policies mandate paid certificates

4. Internal/Private CAs¶

What: Run your own Certificate Authority

Use cases:

• Internal company networks
• Kubernetes clusters
• Microservices authentication
• Development environments
• IoT device management

Tools:

• OpenSSL (manual)
• CFSSL (CloudFlare's toolkit)
• Easy-RSA (used by OpenVPN)
• cert-manager (Kubernetes)

Pros:

• ✅ Full control
• ✅ No external dependencies
• ✅ No rate limits
• ✅ Custom certificate policies
• ✅ Any validity period you want

Cons:

• ❌ Not trusted by default (must distribute root CA to all clients)
• ❌ You manage entire infrastructure
• ❌ Security responsibility is yours
• ❌ Cannot use for public websites

Browser Trust Stores¶

How browsers know which CAs to trust:

Every browser/OS maintains a list of trusted root certificates:

Firefox:

• Uses own trust store (Mozilla Root Program)
• ~140 trusted root certificates
• Independent of OS

Chrome/Edge:

• Uses OS trust store on Windows/Mac
• Own store on Linux
• ~100-150 trusted roots

Safari:

• Uses macOS trust store
• Apple Root Certificate Program

Viewing trusted CAs:

# Linux - list system trusted CAs
ls /etc/ssl/certs/

# View specific CA
openssl x509 -in /etc/ssl/certs/ca-cert.pem -text -noout

# macOS - open Keychain Access
# Windows - certmgr.msc

CA must prove:

• Financial stability
• Technical competence
• Compliance with CAB Forum baseline requirements
• Proper security controls
• Annual audits (WebTrust)

CA can be removed if:

• Security incident (private key compromised)
• Mis-issuance of certificates
• Violation of baseline requirements
• Failed audits

Example: Symantec CA was distrusted by browsers in 2018 due to mis-issuance violations.

All Methods to Get Certificates¶

Method 1: Certbot (Let's Encrypt) - Fully Automated¶

What: Official ACME client for Let's Encrypt Automation Level: Fully automated Best For: Linux servers with public IP and root access

Installation:

# Ubuntu/Debian
sudo apt update
sudo apt install certbot python3-certbot-nginx -y

# CentOS/RHEL
sudo yum install certbot python3-certbot-nginx -y

# Using Snap (recommended)
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

Get certificate - Nginx (fully automated):

sudo certbot --nginx -d jenkins.ibtisam-iq.com

What Certbot does automatically:

1. Reads your Nginx configuration
2. Finds server block matching domain
3. Adds temporary location block for verification
4. Contacts Let's Encrypt ACME API
5. Let's Encrypt sends HTTP-01 challenge
6. Certbot responds to challenge
7. Let's Encrypt verifies domain ownership
8. Downloads signed certificate
9. Modifies Nginx config (adds SSL directives)
10. Reloads Nginx
11. Sets up systemd timer for auto-renewal

Get certificate - Manual (certonly):

# Get certificate without modifying server config
sudo certbot certonly --nginx -d jenkins.ibtisam-iq.com

Get certificate - Standalone (no web server):

# Certbot starts temporary web server on port 80
sudo certbot certonly --standalone -d jenkins.ibtisam-iq.com

Files created:

/etc/letsencrypt/live/jenkins.ibtisam-iq.com/
├── privkey.pem       # Private key (keep secret!)
├── fullchain.pem     # Certificate + intermediates (use in Nginx)
├── cert.pem          # Certificate only
└── chain.pem         # Intermediate certificates only

Auto-renewal:

# Check renewal timer
sudo systemctl status certbot.timer

# Test renewal (dry run)
sudo certbot renew --dry-run

# Manual renewal
sudo certbot renew

# Force renewal (even if not expiring)
sudo certbot renew --force-renewal

Advantages:

• Completely automated
• Free forever
• Auto-renewal built-in
• Widely tested and trusted
• Integrates with popular web servers

Disadvantages:

• Requires public IP address
• Requires port 80 or 443 accessible
• Command-line only (no GUI)
• 90-day validity (though auto-renewal handles this)

Method 2: ZeroSSL (Web Dashboard) - Semi-Automated¶

What: Browser-based certificate management Automation Level: Medium (manual via dashboard, API available) Best For: Non-technical users, Windows servers, manual certificate management

Process:

Step 1: Sign up

• Go to zerossl.com
• Create free account
• No credit card required for free tier

Step 2: Create certificate

1. Click "New Certificate"
2. Enter domain: jenkins.ibtisam-iq.com
3. Select 90-day validity (free)
4. Click "Next Step"

Step 3: Verify domain ownership

Choose verification method:

Option A: HTTP File Upload

1. Download verification file
2. Upload to your server: /.well-known/pki-validation/fileauth.txt
3. Make accessible via HTTP
4. Click "Verify Domain"

Option B: DNS Record

1. Get TXT record from ZeroSSL
2. Add to your DNS:

   Type: TXT
   Name: _acme-challenge.jenkins.ibtisam-iq.com
   Value: [provided by ZeroSSL]
   

3. Wait for DNS propagation
4. Click "Verify Domain"

Option C: Email Verification

1. Receive email at admin@ibtisam-iq.com
2. Click verification link
3. Confirm domain ownership

Step 4: Download certificates

After verification, download: - certificate.crt (your certificate) - ca_bundle.crt (intermediate certificates) - private.key (private key - if generated by ZeroSSL)

Step 5: Combine files

# Create full chain
cat certificate.crt ca_bundle.crt > fullchain.crt

Step 6: Install on server

# Copy files
sudo cp private.key /etc/ssl/private/jenkins.key
sudo cp fullchain.crt /etc/ssl/certs/jenkins-fullchain.crt

# Set permissions
sudo chmod 600 /etc/ssl/private/jenkins.key
sudo chmod 644 /etc/ssl/certs/jenkins-fullchain.crt

# Configure Nginx
sudo nano /etc/nginx/sites-available/jenkins.conf

Advantages:

• User-friendly web interface
• No command line required
• Multiple verification methods
• Can manage from any device
• REST API available for automation

Disadvantages:

• Free tier requires manual renewal every 90 days
• Limited to 3 active certificates on free tier
• Must log in to dashboard regularly
• No automated integration with web servers

Method 3: OpenSSL Self-Signed - Fully Manual¶

What: Generate your own self-signed certificate Automation Level: Full control, manual Best For: Development, testing, internal networks

One-command generation:

openssl req -x509 -newkey rsa:4096 \
  -keyout jenkins.key \
  -out jenkins.crt \
  -days 365 \
  -nodes \
  -subj "/CN=jenkins.ibtisam-iq.com/O=MyOrg/C=PK"

Flag explanation:

• -x509 = Output self-signed certificate (not CSR)
• -newkey rsa:4096 = Generate new 4096-bit RSA private key
• -keyout jenkins.key = Save private key to this file
• -out jenkins.crt = Save certificate to this file
• -days 365 = Valid for 365 days
• -nodes = Don't encrypt private key (no password protection)
• -subj = Certificate subject info (avoids interactive prompts)

Step-by-step generation:

# Step 1: Generate private key
openssl genrsa -out jenkins.key 4096

# Step 2: Create self-signed certificate
openssl req -x509 -new -key jenkins.key -out jenkins.crt -days 365

# You'll be prompted:
# Country Name (2 letter code): PK
# State or Province: Punjab
# Locality (city): Islamabad
# Organization: My Company
# Organizational Unit: IT
# Common Name: jenkins.ibtisam-iq.com  <- MUST match domain!
# Email: ...@ibtisam-iq.com

Files created:

jenkins.key  # Private key (4096-bit RSA)
jenkins.crt  # Self-signed certificate

View certificate:

openssl x509 -in jenkins.crt -text -noout

Advantages:

• Instant creation (no external dependencies)
• Free
• Full control over all parameters
• No rate limits
• Any validity period
• Works offline

Disadvantages:

• Browser warnings ("Not Secure")
• Not trusted by clients
• Manual distribution required for internal use
• No third-party verification
• Not suitable for production public sites

Suppressing browser warnings (NOT recommended for production):

Users must manually trust your certificate:

Chrome:

1. Click "Advanced"
2. Click "Proceed to jenkins.ibtisam-iq.com (unsafe)"

Firefox:

1. Click "Advanced"
2. Click "Accept the Risk and Continue"

For testing/internal use - add to trust store:

# Linux
sudo cp jenkins.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# macOS
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain jenkins.crt

# Windows
# Import via certmgr.msc → Trusted Root Certification Authorities

Method 4: OpenSSL with Private CA - Advanced Manual¶

What: Create your own Certificate Authority, then sign certificates Use Case: Internal networks, Kubernetes, microservices, learning

This is the same process you use in Kubernetes for user certificates!

Step 1: Create your Certificate Authority

# Generate CA private key
openssl genrsa -out ca.key 4096

# Create CA certificate (self-signed root)
openssl req -x509 -new -nodes \
  -key ca.key \
  -sha256 \
  -days 3650 \
  -out ca.crt \
  -subj "/CN=My Company Root CA/O=MyCompany/C=PK"

Files created:

ca.key  # CA private key (keep extremely secure!)
ca.crt  # CA root certificate (distribute to all clients)

Step 2: Generate server private key

openssl genrsa -out jenkins.key 4096

Step 3: Create Certificate Signing Request (CSR)

openssl req -new \
  -key jenkins.key \
  -out jenkins.csr \
  -subj "/CN=jenkins.ibtisam-iq.com/O=MyCompany/C=PK"

Step 4: Sign the CSR with your CA

openssl x509 -req \
  -in jenkins.csr \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -out jenkins.crt \
  -days 365 \
  -sha256

Files created:

ca.key         # CA private key (keep offline/secure)
ca.crt         # CA certificate (distribute to clients)
ca.srl         # Serial number tracker
jenkins.key    # Server private key
jenkins.csr    # Certificate signing request
jenkins.crt    # Server certificate (signed by your CA)

Step 5: Deploy

Server side:

ssl_certificate /etc/ssl/certs/jenkins.crt;
ssl_certificate_key /etc/ssl/private/jenkins.key;

Client side - add CA to trust store:

# Linux
sudo cp ca.crt /usr/local/share/ca-certificates/mycompany-ca.crt
sudo update-ca-certificates

# Now all certificates signed by your CA are trusted

Advantages:

• Full control over entire PKI
• Can issue unlimited certificates
• Any validity period
• Custom certificate policies
• Ideal for internal infrastructure
• Same workflow as Kubernetes certificates

Disadvantages:

• Must manage CA infrastructure
• Must securely store CA private key
• Must distribute CA certificate to all clients
• Not trusted externally
• More complex than other methods

Real-world example - Kubernetes:

# This is exactly how Kubernetes user certificates work:

# 1. Generate user key
openssl genrsa -out user.key 2048

# 2. Create CSR for user
openssl req -new -key user.key -out user.csr -subj "/CN=user/O=developers"

# 3. Sign with Kubernetes CA
openssl x509 -req -in user.csr -CA /etc/kubernetes/pki/ca.crt \
  -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial \
  -out user.crt -days 365

# Same process, different context!

Method 5: Cloudflare Tunnel - Zero Certificate Management¶

What: Cloudflare handles all SSL/TLS Automation Level: Completely automatic Best For: No public IP, ephemeral environments, simplest possible setup

How it works:

Browser → Cloudflare (HTTPS terminated here) → Tunnel → Your Server (HTTP)

You never touch certificates!

Setup:

# Install cloudflared
curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared
chmod +x cloudflared
sudo mv cloudflared /usr/local/bin/

# Install as service (token from Cloudflare dashboard)
sudo cloudflared service install eyJhIjoiXXXXXXXXXX...

# Configure in dashboard
# Subdomain: jenkins
# Domain: ibtisam-iq.com
# Service: http://localhost:8080

Result:

• https://jenkins.ibtisam-iq.com works immediately
• Cloudflare's certificate is used
• No certificate files on your server
• No renewal needed
• Works without public IP

Behind the scenes:

• Cloudflare has wildcard certificate for *.cfargotunnel.com
• Your domain CNAMEs to Cloudflare
• Cloudflare terminates SSL/TLS
• Sends plain HTTP through tunnel to your server
• Server doesn't need HTTPS at all

Advantages:

• Zero certificate management
• No public IP required
• Free
• Automatic HTTPS
• Works behind NAT/firewall
• DDoS protection included
• Automatic certificate renewal (handled by Cloudflare)

Disadvantages:

• Dependent on Cloudflare service
• TLS termination at Cloudflare (not end-to-end)
• Must use Cloudflare DNS
• Adds latency (one extra hop)
• Less control over TLS configuration

When to use:

• iximiuz Labs or similar platforms
• Development environments
• Temporary demos
• No public IP available
• Want simplest possible setup

Method 6: AWS Certificate Manager (ACM) - AWS Cloud Only¶

What: AWS managed certificate service Automation Level: Fully automated Best For: AWS infrastructure (ALB, CloudFront, API Gateway)

Process:

Step 1: Request certificate

# Via AWS Console
1. Go to Certificate Manager
2. Click "Request a certificate"
3. Enter domain: jenkins.ibtisam-iq.com
4. Choose validation: DNS or Email

# Via AWS CLI
aws acm request-certificate \
  --domain-name jenkins.ibtisam-iq.com \
  --validation-method DNS

Step 2: Validate domain

DNS Validation (recommended):

# ACM provides CNAME record
# Add to Route 53 or your DNS:
Name: _abc123.jenkins.ibtisam-iq.com
Value: _xyz789.acm-validations.aws.

Email Validation:

# ACM sends email to:
# admin@ibtisam-iq.com
# administrator@ibtisam-iq.com
# hostmaster@ibtisam-iq.com
# postmaster@ibtisam-iq.com
# webmaster@ibtisam-iq.com

Step 3: Attach to AWS resource

# Cannot export certificate!
# Can only attach to AWS services:
# - Application Load Balancer (ALB)
# - CloudFront distribution
# - API Gateway
# - Elastic Beanstalk

Example - ALB:

aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:... \
  --protocol HTTPS \
  --port 443 \
  --certificates CertificateArn=arn:aws:acm:... \
  --default-actions Type=forward,TargetGroupArn=arn:aws:...

Advantages:

• Free (when used with AWS services)
• Automatic renewal
• AWS manages everything
• Integrates with AWS services
• No certificate files to manage
• Wildcard certificates supported

Disadvantages:

• AWS services only (cannot export)
• Cannot use on EC2 directly
• Vendor lock-in
• Must use AWS DNS validation or email
• No access to private key

When to use:

• Using AWS ALB, CloudFront, or API Gateway
• Want zero certificate management
• Already in AWS ecosystem
• Cost-conscious (free for AWS services)

Manual Certificate Generation (Complete Process)¶

Understanding the Full Certificate Lifecycle¶

The complete workflow:

1. Generate Private Key
    ↓
2. Create Certificate Signing Request (CSR)
    ↓
3. Submit CSR to Certificate Authority
    ↓
4. CA Validates Domain Ownership
    ↓
5. CA Signs Certificate
    ↓
6. Download Signed Certificate
    ↓
7. Combine with Intermediate Certificates
    ↓
8. Install on Server
    ↓
9. Configure Web Server
    ↓
10. Test and Verify

Let's go through each step in detail.

Step 1: Generate Private Key¶

What: Create cryptographic key pair

# Generate 4096-bit RSA private key
openssl genrsa -out jenkins.key 4096

# Alternative: Generate with password protection
openssl genrsa -aes256 -out jenkins.key 4096
# You'll be prompted for passphrase

Key size recommendations:

• 2048 bits: Minimum acceptable, fast
• 3072 bits: Good security/performance balance
• 4096 bits: Maximum security, slower (recommended)

Output:

Generating RSA private key, 4096 bit long modulus
....++
.......................++
e is 65537 (0x10001)

File created: jenkins.key

View key information:

openssl rsa -in jenkins.key -text -noout

Check key:

# Verify key is valid RSA
openssl rsa -in jenkins.key -check

# Get key's public modulus (for matching with certificate)
openssl rsa -in jenkins.key -modulus -noout | openssl md5

Security:

# Set restrictive permissions
chmod 600 jenkins.key

# Never commit to git
echo "*.key" >> .gitignore

# Store backup in secure location
# Consider encrypting backup
openssl rsa -aes256 -in jenkins.key -out jenkins.key.encrypted

Step 2: Create Certificate Signing Request (CSR)¶

What: Request file containing your information and public key

# Interactive mode
openssl req -new -key jenkins.key -out jenkins.csr

# Non-interactive mode (with all info in command)
openssl req -new -key jenkins.key -out jenkins.csr \
  -subj "/C=PK/ST=Punjab/L=Rawalpindi/O=My Company/OU=IT/CN=jenkins.ibtisam-iq.com/emailAddress=admin@ibtisam-iq.com"

Interactive prompts:

Country Name (2 letter code) []: PK
State or Province Name (full name) []: Punjab
Locality Name (city, town) []: Rawalpindi
Organization Name (company) []: My Company
Organizational Unit Name (department) []: IT Department
Common Name (FQDN) []: jenkins.ibtisam-iq.com
Email Address []: admin@ibtisam-iq.com
A challenge password []: [Leave empty]
An optional company name []: [Leave empty]

Critical: Common Name (CN)

• MUST exactly match your domain
• Single domain: jenkins.ibtisam-iq.com
• Wildcard: *.ibtisam-iq.com (matches jenkins.ibtisam-iq.com, api.ibtisam-iq.com, etc.)
• Multiple domains: Use Subject Alternative Name (SAN) extension

SAN (Subject Alternative Names) - Multiple domains:

# Create OpenSSL config file
cat > san.cnf <<EOF
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
countryName = PK
stateOrProvinceName = Punjab
localityName = Rawalpindi
organizationName = My Company
commonName = jenkins.ibtisam-iq.com

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = jenkins.ibtisam-iq.com
DNS.2 = www.jenkins.ibtisam-iq.com
DNS.3 = api.ibtisam-iq.com
EOF

# Generate CSR with SAN
openssl req -new -key jenkins.key -out jenkins.csr -config san.cnf

File created: jenkins.csr

View CSR:

openssl req -in jenkins.csr -text -noout

Verify CSR:

# Check CSR is valid
openssl req -in jenkins.csr -noout -verify

# Extract public key from CSR
openssl req -in jenkins.csr -pubkey -noout

# Get CSR modulus (to verify it matches private key)
openssl req -in jenkins.csr -modulus -noout | openssl md5

Step 3: Submit CSR to Certificate Authority¶

Let's Encrypt (automated):

# Certbot handles CSR creation internally
sudo certbot certonly --nginx -d jenkins.ibtisam-iq.com

ZeroSSL (manual):

1. Log in to zerossl.com
2. Click "New Certificate"
3. Click "Certificate Signing Request (CSR)"
4. Paste contents of jenkins.csr
5. Click "Next"

Commercial CA (varies):

1. Go to CA's website
2. Choose certificate type (DV/OV/EV)
3. Fill order form
4. Paste CSR when prompted
5. Complete payment
6. Proceed to validation

View CSR content:

cat jenkins.csr

Output will be:

-----BEGIN CERTIFICATE REQUEST-----
MIIEojCCAooCAQAwYTELMAkGA1UEBhMCUEsxDzANBgNVBAgMBlB1bmphYjEU...
(base64 encoded data)
...
-----END CERTIFICATE REQUEST-----

Step 4: Domain Validation¶

CA must verify you own the domain. Methods:

HTTP-01 Challenge (Most Common)¶

How it works:

1. CA gives you a token: abc123xyz
2. You create file: /.well-known/acme-challenge/abc123xyz
3. File contains: abc123xyz.YOUR_ACCOUNT_KEY
4. CA fetches: http://jenkins.ibtisam-iq.com/.well-known/acme-challenge/abc123xyz
5. CA verifies content matches
6. Domain ownership proven!

Manual setup:

# Create directory
sudo mkdir -p /var/www/html/.well-known/acme-challenge/

# Create verification file (content provided by CA)
echo "abc123xyz.ACCOUNT_KEY" | sudo tee /var/www/html/.well-known/acme-challenge/abc123xyz

# Verify accessible
curl http://jenkins.ibtisam-iq.com/.well-known/acme-challenge/abc123xyz

Nginx configuration:

server {
    listen 80;
    server_name jenkins.ibtisam-iq.com;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }
}

Certbot does this automatically!

DNS-01 Challenge¶

How it works:

1. CA gives you TXT record value: abc123xyz
2. You add DNS TXT record:
   Name: _acme-challenge.jenkins.ibtisam-iq.com
   Value: abc123xyz
3. Wait for DNS propagation
4. CA queries DNS for TXT record
5. CA verifies value matches
6. Domain ownership proven!

Add DNS record (Cloudflare example):

# In Cloudflare dashboard:
Type: TXT
Name: _acme-challenge.jenkins
Content: [value from CA]
TTL: Auto

Verify DNS propagation:

dig TXT _acme-challenge.jenkins.ibtisam-iq.com

# or
nslookup -type=TXT _acme-challenge.jenkins.ibtisam-iq.com

Advantage of DNS-01:

• Works without web server running
• Can get wildcard certificates
• Works behind firewall

Email Validation¶

How it works:

1. CA sends email to admin addresses:
   - admin@ibtisam-iq.com
   - administrator@ibtisam-iq.com
   - hostmaster@ibtisam-iq.com
   - postmaster@ibtisam-iq.com
   - webmaster@ibtisam-iq.com
2. You receive email with verification link
3. Click link to confirm
4. Domain ownership proven!

Less common for automated systems.

Step 5: Receive Signed Certificate¶

After validation, CA signs your certificate and returns:

Files you'll receive:

1. Your Certificate: jenkins.crt or certificate.crt
2. Intermediate Certificate(s): intermediate.crt or ca_bundle.crt
3. Root Certificate: Usually not needed (already in browsers)

Download from CA:

Let's Encrypt (via Certbot):

# Automatically saved to:
/etc/letsencrypt/live/jenkins.ibtisam-iq.com/
├── cert.pem        # Your certificate
├── chain.pem       # Intermediate certs
├── fullchain.pem   # cert.pem + chain.pem
└── privkey.pem     # Your private key

ZeroSSL (from dashboard):

# Download ZIP containing:
# - certificate.crt
# - ca_bundle.crt
# Extract and rename
unzip certificate.zip
mv certificate.crt jenkins.crt
mv ca_bundle.crt intermediate.crt

Commercial CA (via email or dashboard):

• Download from order details page
• Or receive via email
• May be in various formats (.crt, .pem, .cer)

Step 6: Verify Certificate¶

View certificate details:

openssl x509 -in jenkins.crt -text -noout

Output shows:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 03:1a:2b:3c:...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=R3
        Validity
            Not Before: Feb 17 02:00:00 2026 GMT
            Not After : May 18 02:00:00 2026 GMT
        Subject: CN=jenkins.ibtisam-iq.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

Verify certificate matches private key:

# Get private key modulus
openssl rsa -in jenkins.key -modulus -noout | openssl md5

# Get certificate modulus
openssl x509 -in jenkins.crt -modulus -noout | openssl md5

# Both should output same MD5 hash!
# If they match, certificate and key are a pair

Check certificate expiry:

openssl x509 -in jenkins.crt -noout -dates

# Output:
# notBefore=Feb 17 02:00:00 2026 GMT
# notAfter=May 18 02:00:00 2026 GMT

Verify certificate chain:

# Verify against CA bundle
openssl verify -CAfile intermediate.crt jenkins.crt

# Should output: jenkins.crt: OK

Step 7: Create Full Chain¶

Why: Browsers need complete chain to verify trust path

# Combine certificate + intermediate(s)
cat jenkins.crt intermediate.crt > fullchain.crt

# Or if you have multiple intermediates
cat jenkins.crt intermediate1.crt intermediate2.crt > fullchain.crt

Verify full chain:

# Check how many certificates in file
openssl crl2pkcs7 -nocrl -certfile fullchain.crt | openssl pkcs7 -print_certs -noout

# View each certificate
openssl storeutl -noout -text -certs fullchain.crt

Step 8: Install on Server¶

Organize files:

# Copy to standard locations
sudo cp jenkins.key /etc/ssl/private/
sudo cp fullchain.crt /etc/ssl/certs/jenkins-fullchain.crt

# Set permissions
sudo chmod 600 /etc/ssl/private/jenkins.key
sudo chmod 644 /etc/ssl/certs/jenkins-fullchain.crt

# Set ownership
sudo chown root:root /etc/ssl/private/jenkins.key
sudo chown root:root /etc/ssl/certs/jenkins-fullchain.crt

Step 9: Configure Web Server¶

Nginx:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name jenkins.ibtisam-iq.com;

    # Certificate files
    ssl_certificate /etc/ssl/certs/jenkins-fullchain.crt;
    ssl_certificate_key /etc/ssl/private/jenkins.key;

    # Modern SSL configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
    ssl_prefer_server_ciphers off;

    # SSL session caching
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # OCSP Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/ssl/certs/jenkins-fullchain.crt;

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    listen [::]:80;
    server_name jenkins.ibtisam-iq.com;
    return 301 https://$server_name$request_uri;
}

Test configuration:

sudo nginx -t

Reload Nginx:

sudo systemctl reload nginx

Step 10: Test and Verify¶

Check HTTPS works:

curl -I https://jenkins.ibtisam-iq.com

Test SSL configuration:

# Check certificate from command line
echo | openssl s_client -connect jenkins.ibtisam-iq.com:443 -servername jenkins.ibtisam-iq.com 2>/dev/null | openssl x509 -noout -dates

# Test SSL handshake
openssl s_client -connect jenkins.ibtisam-iq.com:443 -servername jenkins.ibtisam-iq.com

Online SSL testing tools:

• SSL Labs: https://www.ssllabs.com/ssltest/
• Check for A+ rating

Browser testing:

1. Open https://jenkins.ibtisam-iq.com
2. Check for green padlock
3. Click padlock → View certificate
4. Verify:
5. Domain matches
6. Valid date range
7. Trusted chain

Common issues:

ERR_CERT_COMMON_NAME_INVALID:

• Certificate CN doesn't match domain
• Solution: Generate new certificate with correct CN

ERR_CERT_AUTHORITY_INVALID:

• Certificate not signed by trusted CA (self-signed)
• Or missing intermediate certificates
• Solution: Include fullchain.pem in Nginx config

ERR_CERT_DATE_INVALID:

• Certificate expired or not yet valid
• Solution: Renew certificate

Let's Encrypt vs ZeroSSL vs Commercial CAs¶

Feature Comparison Matrix¶

Detailed Comparison¶

Let's Encrypt¶

Pros:

• ✅ Completely free forever
• ✅ Fully automated (Certbot, ACME)
• ✅ Trusted by all major browsers
• ✅ Wildcard certificates included
• ✅ Unlimited SANs per certificate
• ✅ Open source and transparent
• ✅ Backed by major tech companies
• ✅ Strong community support
• ✅ Well-documented

Cons:

• ❌ 90-day validity (security feature, but requires attention)
• ❌ No OV/EV certificates
• ❌ No commercial support
• ❌ Command-line only (no GUI)
• ❌ Rate limits can be hit in testing
• ❌ Requires automation for best experience

Rate Limits:

• 50 certificates per registered domain per week
• 5 duplicate certificates per week
• 300 new orders per account per 3 hours
• Rarely an issue for production use

Best use cases:

• Personal websites and blogs
• Startups and small to medium businesses
• Automated CI/CD pipelines
• Microservices and APIs
• Any scenario where automation is possible
• Developers comfortable with command line

Not suitable for:

• Organizations requiring OV/EV validation
• Environments without automation capability
• Enterprises needing SLA guarantees
• Situations requiring commercial support

ZeroSSL¶

Pros:

• ✅ Free tier available
• ✅ Web dashboard (easier for beginners)
• ✅ Multiple verification methods
• ✅ REST API for automation
• ✅ Can generate from browser
• ✅ Paid plans add features
• ✅ Commercial support available
• ✅ Compatible with ACME protocol

Cons:

• ❌ Free tier limited to 3 active certificates
• ❌ Free certificates need manual renewal
• ❌ Dashboard can be slow
• ❌ Less mature than Let's Encrypt
• ❌ API rate limits on free tier
• ❌ Some features behind paywall

Free vs Paid:

Free:

• 3 active certificates
• 90-day validity
• Manual renewal
• Community support
• Basic dashboard

Paid (from $3.99/month):

• Unlimited certificates
• Auto-renewal
• Email support
• Priority validation
• API access
• Certificate management tools

Best use cases:

• Non-technical users preferring GUI
• Small teams managing certificates manually
• Windows servers
• Projects that might need OV later
• Need for web-based management

Not suitable for:

• Large-scale automated deployments (use Let's Encrypt)
• When free tier limits are exceeded
• Enterprise environments (use commercial CA)

Commercial CAs (DigiCert, Sectigo, etc.)¶

Pros:

• ✅ Organization Validation (OV)
• ✅ Extended Validation (EV)
• ✅ 24/7 phone and email support
• ✅ Warranty/insurance coverage
• ✅ Compliance certifications
• ✅ Account manager for enterprise
• ✅ Integration assistance
• ✅ Better for audits and compliance

Cons:

• ❌ Expensive (\(50-\)1500+/year)
• ❌ Overkill for most use cases
• ❌ Manual processes often involved
• ❌ Longer issuance time for OV/EV
• ❌ Annual renewal required

Price ranges:

• DV: $50-150/year
• OV: $100-300/year
• EV: $200-1500/year
• Wildcard DV: $150-400/year
• Wildcard OV: $300-800/year
• Multi-domain (SAN): Add $50-100 per additional domain

Warranty coverage:

• DV: Up to $10,000
• OV: Up to $250,000
• EV: Up to $1,750,000

Best use cases:

• Banks and financial institutions
• E-commerce with high transaction volume
• Healthcare (HIPAA compliance)
• Government websites
• Enterprise corporate sites
• When compliance requires OV/EV
• Organizations with ample budget
• Need for SLA and guaranteed uptime

Not suitable for:

• Personal websites (use Let's Encrypt)
• Startups watching costs (use Let's Encrypt)
• Simple blogs and portfolios (use Let's Encrypt)
• Projects prioritizing automation (use Let's Encrypt)

Decision Matrix¶

Use Let's Encrypt if:

• ✅ Budget is zero
• ✅ Have automation capability
• ✅ Server has public IP and open ports
• ✅ DV certificate is sufficient
• ✅ Comfortable with command line
• ✅ Want maximum automation
• ✅ Open to 90-day renewal cycle

Use ZeroSSL if:

• ✅ Prefer web interface
• ✅ Managing < 3 certificates manually
• ✅ Might need paid features later
• ✅ Windows server environment
• ✅ Want REST API option
• ✅ Need email support (paid tier)

Use Commercial CA if:

• ✅ Need OV or EV validation
• ✅ Budget allows ($100+/year)
• ✅ Compliance requirements demand it
• ✅ Want warranty/insurance
• ✅ Need 24/7 support
• ✅ Enterprise environment
• ✅ Audit requirements

Use Cloudflare Tunnel if:

• ✅ No public IP available
• ✅ Don't want ANY certificate management
• ✅ Ephemeral environments
• ✅ Want simplest possible solution
• ✅ Already using Cloudflare

Cost Over 5 Years¶

Time investment comparison:

Let's Encrypt:

• Initial setup: 30-60 minutes
• Ongoing: 0 minutes (automatic)
• Total over 5 years: 1 hour

ZeroSSL Free:

• Initial setup: 15-30 minutes
• Manual renewal: 15 minutes every 90 days
• Total over 5 years: 15 hours (60 renewals × 15 min)

Commercial CA:

• Initial setup: 30-60 minutes
• Annual renewal: 30 minutes
• Total over 5 years: 3.5 hours

Cloudflare Tunnel:

• Initial setup: 10-15 minutes
• Ongoing: 0 minutes
• Total over 5 years: 15 minutes

Technical Differences¶

Validation Methods¶

Let's Encrypt:

• HTTP-01 (web server challenge)
• DNS-01 (DNS TXT record)
• TLS-ALPN-01 (TLS-based challenge)
• Fully automated

ZeroSSL:

• HTTP file upload
• DNS CNAME record
• Email verification
• Manual or API-driven

Commercial CAs:

• Email validation
• DNS validation
• File-based validation
• Phone verification (OV/EV)
• Business registration verification (OV/EV)
• Physical address verification (EV)

Certificate Issuance Time¶

Trust Store Presence¶

All three are trusted by:

• ✅ Chrome
• ✅ Firefox
• ✅ Safari
• ✅ Edge
• ✅ Opera
• ✅ Mobile browsers
• ✅ curl, wget, etc.

No difference in browser trust!

Using Certificates in Different Services¶

Nginx¶

Most common web server configuration:

# HTTPS server block
server {
    # Listen on HTTPS port
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name jenkins.ibtisam-iq.com;

    # Certificate and key locations
    ssl_certificate /etc/letsencrypt/live/jenkins.ibtisam-iq.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/jenkins.ibtisam-iq.com/privkey.pem;

    # Modern TLS configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
    ssl_prefer_server_ciphers off;

    # SSL session caching for performance
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_session_tickets off;

    # OCSP Stapling for faster certificate validation
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/jenkins.ibtisam-iq.com/chain.pem;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;

    # Diffie-Hellman parameter for DHE ciphersuites
    ssl_dhparam /etc/ssl/certs/dhparam.pem;

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;

    # Proxy to backend
    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    listen [::]:80;
    server_name jenkins.ibtisam-iq.com;
    return 301 https://$host$request_uri;
}

Generate DH parameters (one-time):

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096
# Takes 10-30 minutes, only need to do once

Test configuration:

sudo nginx -t

Reload Nginx:

sudo systemctl reload nginx

Files needed: - fullchain.pem - Your certificate + intermediate chain - privkey.pem - Private key

Apache¶

<VirtualHost *:443>
    ServerName jenkins.ibtisam-iq.com
    ServerAdmin admin@ibtisam-iq.com

    # Enable SSL
    SSLEngine on

    # Certificate files
    SSLCertificateFile /etc/ssl/certs/jenkins.crt
    SSLCertificateKeyFile /etc/ssl/private/jenkins.key
    SSLCertificateChainFile /etc/ssl/certs/intermediate.crt

    # Modern TLS configuration
    SSLProtocol -all +TLSv1.2 +TLSv1.3
    SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    SSLHonorCipherOrder off
    SSLCompression off
    SSLSessionTickets off

    # OCSP Stapling
    SSLUseStapling on
    SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"

    # Security headers
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
    Header always set X-Frame-Options "SAMEORIGIN"
    Header always set X-Content-Type-Options "nosniff"

    # Proxy to backend
    ProxyPreserveHost On
    ProxyPass / http://localhost:8080/
    ProxyPassReverse / http://localhost:8080/
</VirtualHost>

# Redirect HTTP to HTTPS
<VirtualHost *:80>
    ServerName jenkins.ibtisam-iq.com
    Redirect permanent / https://jenkins.ibtisam-iq.com/
</VirtualHost>

Enable required Apache modules:

sudo a2enmod ssl
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod headers
sudo systemctl restart apache2

Files needed:

• jenkins.crt - Your certificate only
• jenkins.key - Private key
• intermediate.crt - Intermediate certificates

Jenkins Direct (No Reverse Proxy)¶

Jenkins can handle HTTPS directly, but requires PKCS#12 format.

Step 1: Convert certificate to PKCS#12:

openssl pkcs12 -export \
  -in /etc/letsencrypt/live/jenkins.ibtisam-iq.com/fullchain.pem \
  -inkey /etc/letsencrypt/live/jenkins.ibtisam-iq.com/privkey.pem \
  -out /var/lib/jenkins/jenkins.p12 \
  -name jenkins \
  -passout pass:changeit

# Set ownership
sudo chown jenkins:jenkins /var/lib/jenkins/jenkins.p12
sudo chmod 600 /var/lib/jenkins/jenkins.p12

Step 2: Configure Jenkins:

sudo nano /etc/default/jenkins

Add these flags to JENKINS_ARGS:

JENKINS_ARGS="--httpsPort=8443 \
              --httpPort=-1 \
              --httpsKeyStore=/var/lib/jenkins/jenkins.p12 \
              --httpsKeyStorePassword=changeit"

Step 3: Restart Jenkins:

sudo systemctl restart jenkins

Access Jenkins:

https://jenkins.ibtisam-iq.com:8443

Note: Reverse proxy (Nginx/Apache) is still recommended for:

• Standard HTTPS port (443 vs 8443)
• Better SSL/TLS configuration options
• Easier certificate renewal
• Additional security features

Files needed: - jenkins.p12 - PKCS#12 bundle (certificate + private key)

Tomcat¶

Step 1: Convert to PKCS#12:

openssl pkcs12 -export \
  -in cert.pem \
  -inkey privkey.pem \
  -out tomcat.p12 \
  -name tomcat \
  -CAfile chain.pem \
  -caname root \
  -password pass:changeit

Step 2: Configure Tomcat:

<!-- Edit: /etc/tomcat9/server.xml -->
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
           maxThreads="150" SSLEnabled="true">
    <SSLHostConfig>
        <Certificate certificateKeystoreFile="/etc/tomcat9/tomcat.p12"
                     certificateKeystorePassword="changeit"
                     certificateKeystoreType="PKCS12"
                     type="RSA" />
    </SSLHostConfig>
</Connector>

Step 3: Restart Tomcat:

sudo systemctl restart tomcat9

Docker¶

docker-compose.yml:

version: '3.8'

services:
  nginx:
    image: nginx:latest
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt/live/jenkins.ibtisam-iq.com/fullchain.pem:/etc/nginx/ssl/cert.pem:ro
      - /etc/letsencrypt/live/jenkins.ibtisam-iq.com/privkey.pem:/etc/nginx/ssl/key.pem:ro
    depends_on:
      - jenkins

  jenkins:
    image: jenkins/jenkins:lts
    ports:
      - "8080:8080"
    volumes:
      - jenkins_home:/var/jenkins_home

volumes:
  jenkins_home:

nginx.conf:

events {
    worker_connections 1024;
}

http {
    upstream jenkins {
        server jenkins:8080;
    }

    server {
        listen 443 ssl;
        server_name jenkins.ibtisam-iq.com;

        ssl_certificate /etc/nginx/ssl/cert.pem;
        ssl_certificate_key /etc/nginx/ssl/key.pem;
        ssl_protocols TLSv1.2 TLSv1.3;

        location / {
            proxy_pass http://jenkins;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }

    server {
        listen 80;
        server_name jenkins.ibtisam-iq.com;
        return 301 https://$host$request_uri;
    }
}

Kubernetes¶

Create TLS secret:

kubectl create secret tls jenkins-tls \
  --cert=/etc/letsencrypt/live/jenkins.ibtisam-iq.com/fullchain.pem \
  --key=/etc/letsencrypt/live/jenkins.ibtisam-iq.com/privkey.pem \
  --namespace=default

Or from files:

kubectl create secret tls jenkins-tls \
  --cert=fullchain.pem \
  --key=privkey.pem

Verify secret:

kubectl describe secret jenkins-tls

Ingress with TLS:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: "letsencrypt-prod"  # If using cert-manager
spec:
  tls:
  - hosts:
    - jenkins.ibtisam-iq.com
    secretName: jenkins-tls
  rules:
  - host: jenkins.ibtisam-iq.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: jenkins
            port:
              number: 8080

Using cert-manager for automatic certificates:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ...@ibtisam-iq.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

This is exactly like creating user certificates in Kubernetes - same concept, different purpose!

Understanding the Certificate Chain¶

What is Certificate Chain?¶

Definition: Series of certificates linking your certificate to a trusted root CA.

Analogy: Chain of trust, like verifying someone's identity through mutual friends:

You meet a stranger (website) →
Stranger shows ID signed by your friend (intermediate CA) →
You trust your friend →
Your friend's ID was signed by government (root CA) →
You trust government →
Therefore, you trust stranger!

Why Certificate Chains Exist¶

Problem: Root CA private keys are incredibly valuable

If root CA compromised:

• All certificates worldwide become untrustworthy
• Browsers must remove CA from trust store
• Millions of websites affected
• Years to recover

Solution: Intermediate certificates act as buffer

Root CA (kept offline, extremely secure)
    ↓ signs
Intermediate CA (used daily for signing)
    ↓ signs
Your Certificate

If intermediate compromised:

• Only certificates from that intermediate affected
• Root CA can revoke compromised intermediate
• Issue new intermediate
• Root CA stays safe
• Recovery measured in days, not years

The Chain Structure¶

Typical Let's Encrypt chain:

ISRG Root X1 (Root CA)
    ↓ signed by
Let's Encrypt Authority X3 (Intermediate CA)
    ↓ signed by
jenkins.ibtisam-iq.com (Your Certificate)

Typical ZeroSSL chain:

USERTrust RSA Certification Authority (Root CA)
    ↓ signed by
Sectigo RSA Domain Validation Secure Server CA (Intermediate CA)
    ↓ signed by
jenkins.ibtisam-iq.com (Your Certificate)

Commercial CA chain (may have multiple intermediates):

DigiCert Global Root CA (Root CA)
    ↓ signed by
DigiCert SHA2 Secure Server CA (Intermediate CA 1)
    ↓ signed by
DigiCert TLS RSA SHA256 2020 CA1 (Intermediate CA 2)
    ↓ signed by
jenkins.ibtisam-iq.com (Your Certificate)

How Browsers Verify Chains¶

Step-by-step verification:

1. Browser receives your certificate
2. Certificate says "I'm jenkins.ibtisam-iq.com, signed by Let's Encrypt Authority X3"
3. Browser checks: "Do I have Let's Encrypt Authority X3?"
   - If in trust store → ✅ Trust
   - If not → Check intermediate certificate
4. Intermediate certificate says "I'm Let's Encrypt Authority X3, signed by ISRG Root X1"
5. Browser checks: "Do I have ISRG Root X1?"
   - If in trust store → ✅ Trust entire chain
   - If not → ⚠️ Not trusted
6. All checks pass → 🔒 Green padlock

Browser trust store:

• Pre-installed list of root CAs
• Usually 100-150 root certificates
• Updated with browser/OS updates
• Can manually add/remove roots

Missing Chain Problem¶

Common issue: Incomplete certificate chain

Symptoms:

• Works in Chrome
• Fails in Firefox
• Fails in curl
• Error: "Unable to verify certificate chain"

Why Chrome works:

• Chrome caches intermediate certificates
• If Chrome saw intermediate before, it remembers
• Creates false sense of security

Why others fail:

• Firefox doesn't cache intermediates
• curl requires complete chain
• Mobile browsers may not cache

Diagnosis:

# Test with curl
curl https://jenkins.ibtisam-iq.com

# If error:
# curl: (60) SSL certificate problem: unable to get local issuer certificate
# → Missing intermediate certificate

Fix:

# Wrong - only your certificate
ssl_certificate /path/to/cert.pem;

# Correct - certificate + intermediates
ssl_certificate /path/to/fullchain.pem;

Viewing Certificate Chain¶

OpenSSL command:

openssl s_client -connect jenkins.ibtisam-iq.com:443 -showcerts

Output shows full chain:

depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R3
verify return:1
depth=0 CN=jenkins.ibtisam-iq.com
verify return:1
---
Certificate chain
 0 s:CN=jenkins.ibtisam-iq.com
   i:C=US, O=Let's Encrypt, CN=R3
-----BEGIN CERTIFICATE-----
(Your certificate)
-----END CERTIFICATE-----
 1 s:C=US, O=Let's Encrypt, CN=R3
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
-----BEGIN CERTIFICATE-----
(Intermediate certificate)
-----END CERTIFICATE-----
---

Verify chain locally:

# Verify against system trust store
openssl verify -CApath /etc/ssl/certs fullchain.pem

# Should output: fullchain.pem: OK

Certificate Chain Files¶

cert.pem (your certificate only):

-----BEGIN CERTIFICATE-----
(Your certificate for jenkins.ibtisam-iq.com)
-----END CERTIFICATE-----

chain.pem (intermediates only):

-----BEGIN CERTIFICATE-----
(Intermediate certificate 1)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate 2 - if exists)
-----END CERTIFICATE-----

fullchain.pem (complete chain):

-----BEGIN CERTIFICATE-----
(Your certificate for jenkins.ibtisam-iq.com)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate 1)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate certificate 2 - if exists)
-----END CERTIFICATE-----

Creating fullchain manually:

# Concatenate certificate + intermediates
cat cert.pem intermediate.crt > fullchain.pem

# Or with multiple intermediates
cat cert.pem intermediate1.crt intermediate2.crt > fullchain.pem

Root Certificates¶

Why not include root in chain?

Browsers already have root certificates installed!

Including root would:

• Waste bandwidth
• Provide no additional security
• Is unnecessary duplication

Browser trust store location:

Linux:

/etc/ssl/certs/ca-certificates.crt
/etc/ssl/certs/ca-bundle.crt
/usr/share/ca-certificates/

macOS:

/System/Library/Keychains/SystemRootCertificates.keychain
# GUI: Keychain Access → System Roots

Windows:

certmgr.msc → Trusted Root Certification Authorities

View system roots:

# List all trusted CAs
ls /etc/ssl/certs/

# View specific CA certificate
openssl x509 -in /etc/ssl/certs/DigiCert_Global_Root_CA.pem -text -noout

Cross-Signing¶

What: Root CA signed by another root CA

Why: Compatibility with older systems

Example: Let's Encrypt

Old chain (compatibility):
ISRG Root X1 (Let's Encrypt) → Signed by DST Root CA X3 (old, widely trusted)

New chain (after DST expiration):
ISRG Root X1 (Let's Encrypt) → Self-signed

Impact:

• Android < 7.1.1 doesn't trust ISRG Root X1 directly
• Needed DST Root CA X3 signature
• DST Root CA X3 expired September 2021
• Modern devices use new chain
• Old devices may have issues

Chain Validation Errors¶

Common errors and solutions:

ERR_CERT_AUTHORITY_INVALID:

Cause: Chain doesn't lead to trusted root
Solution: Include intermediate certificates in fullchain.pem

SSL_ERROR_BAD_CERT_DOMAIN:

Cause: Certificate domain doesn't match URL
Solution: Generate certificate with correct Common Name

ERR_CERT_COMMON_NAME_INVALID:

Cause: Accessing site with different domain than certificate
Solution: Access site using exact domain in certificate

ERR_CERT_DATE_INVALID:

Cause: Certificate expired or not yet valid
Solution: Renew certificate, check system clock

MOZILLA_PKIX_ERROR_MITM_DETECTED:

Cause: Antivirus/firewall intercepting HTTPS
Solution: Disable SSL inspection, add exception

Key Takeaways¶

Files You Must Know¶

Certificate Types¶

Self-Signed:

• You sign it yourself
• Browser warnings
• Free, instant, full control
• Only for dev/testing

CA-Signed:

• Trusted authority signs it
• No browser warnings
• Industry standard
• For production sites

Certificate Authorities¶

Let's Encrypt:

• Free, automated, 90-day validity
• Best for most use cases

ZeroSSL:

• Free tier with web dashboard
• Good for manual management

Commercial CAs:

• Expensive, OV/EV available
• For enterprises with compliance needs

Cloudflare Tunnel:

• No certificates needed!
• Best for no public IP scenarios

The Complete Process¶

Generate Key → Create CSR → Submit to CA → Domain Validation →
Receive Certificate → Create Full Chain → Install on Server →
Configure Service → Test → Renew Before Expiry

Not About Memorizing Commands¶

Understand:

• What each file contains and does
• Why certificate chains exist
• When to use which method
• How to verify everything works
• How to troubleshoot issues

Like Kubernetes:

• You know what .crt and .key files are
• You can create secrets from them
• You can configure any service that needs them
• Same knowledge, different context

Connection to Your Experience¶

Kubernetes certificates (CKA exam):

# Generate user key
openssl genrsa -out user.key 2048

# Create CSR
openssl req -new -key user.key -out user.csr -subj "/CN=user/O=group"

# Sign with cluster CA
openssl x509 -req -in user.csr \
  -CA /etc/kubernetes/pki/ca.crt \
  -CAkey /etc/kubernetes/pki/ca.key \
  -CAcreateserial -out user.crt -days 365

# Create kubeconfig
kubectl config set-credentials user --client-certificate=user.crt --client-key=user.key

Jenkins/Nginx HTTPS (same process!):

# Generate server key
openssl genrsa -out jenkins.key 4096

# Create CSR
openssl req -new -key jenkins.key -out jenkins.csr

# Submit to Let's Encrypt (or any CA)
sudo certbot --csr jenkins.csr

# Install in Nginx
sudo cp jenkins.key /etc/ssl/private/
sudo cp fullchain.pem /etc/ssl/certs/

THE PROCESS IS IDENTICAL - SAME CRYPTOGRAPHY, DIFFERENT USE CASE!