Server-side timelapse processing

Author:

Every five minutes, the Raspberry Pi system at the lake sends a short timelapse segment to Scootercam’s webserver. Here’s what happens next.

Security-Hardened Timelapse System

A PHP-based timelapse video management system that receives video segments from cameras and combines them into daily period videos with automatic 8-hour rotation and comprehensive security controls.

Overview

This system manages timelapse videos from multiple cameras with enterprise-grade security by:

  • Receiving authenticated video segments via HTTP uploads
  • Appending segments to period-based daily videos
  • Auto-rotating videos every 8 hours via cron
  • Archiving completed videos for long-term storage
  • Comprehensive security logging and monitoring
  • Rate limiting and access controls

System Architecture

Cameras: 2 (Amcrest, Reolink)
Periods: 3 per day (Night: 00:00-08:00, Day: 08:00-16:00, Evening: 16:00-24:00)
Total Videos: Maximum 6 active videos at any time
Rotation: Every 8 hours via cron (00:00, 08:00, 16:00)
Security: Two-tier API key authentication, rate limiting, comprehensive validation

Directory Structure

/home/scootercam/public_html/timelapse/
├── upload.php              # Main secure script
├── videos/                 # Active videos (max 6 files)
│   ├── amcrest_day_20250901.mp4
│   ├── amcrest_evening_20250901.mp4
│   ├── amcrest_night_20250902.mp4
│   ├── reolink_day_20250901.mp4
│   ├── reolink_evening_20250901.mp4
│   └── reolink_night_20250902.mp4
├── videos/archive/         # Archived videos
│   ├── amcrest_day_20250831_235959.mp4
│   └── reolink_evening_20250831_235959.mp4
├── temp/                   # Temporary processing files
├── security.log            # Security audit trail
└── rate_limit.json         # Rate limiting data

Security Features

🔐 Two-Tier API Authentication

  • Upload Key: Cameras can upload segments and basic operations
  • Admin Key: Full system control including rotation and status

🛡️ Comprehensive File Validation

  • File size limits (50MB maximum)
  • MIME type verification (video/mp4, video/quicktime)
  • File header validation (MP4 format check)
  • Upload method verification
  • Path traversal protection

⚡ Rate Limiting

  • Maximum 10 uploads per minute per IP address
  • Automatic blocking of excessive requests
  • Rate limit data logging and monitoring

📊 Security Monitoring

  • All access attempts logged with IP tracking
  • Suspicious activity alerts
  • Failed authentication tracking
  • File operation audit trail

🔒 Input Sanitization

  • Strict camera name validation (whitelist only)
  • Command injection prevention
  • Path traversal protection
  • Error information hiding

Setup

1. Generate API Keys

# Generate secure upload key (64 characters)
openssl rand -hex 32

# Generate secure admin key (64 characters) 
openssl rand -hex 32

2. Configure Script

Edit upload.php and replace the placeholder API keys:

private $api_keys = [
    'upload' => 'your_64_char_upload_key_here',
    'admin' => 'your_64_char_admin_key_here'
];

3. Set File Permissions

chmod 755 /home/scootercam/public_html/timelapse/upload.php
chmod 750 /home/scootercam/public_html/timelapse/videos
chmod 750 /home/scootercam/public_html/timelapse/temp
chmod 640 /home/scootercam/public_html/timelapse/security.log
chown -R scootercam:scootercam /home/scootercam/public_html/timelapse/

4. Update Cron Job

# Replace with your actual admin API key
0 0,8,16 * * * /usr/bin/curl --silent "http://scootercam.net/timelapse/upload.php?action=rotate&api_key=YOUR_ADMIN_KEY_HERE" > /dev/null 2>&1

5. Configure Web Server Security

Add to .htaccess:

# Deny access to sensitive files
<Files "security.log">
    Deny from all
</Files>

<Files "rate_limit.json">
    Deny from all
</Files>

# Optional: Basic DDoS protection
<IfModule mod_evasive24.c>
    DOSPageCount        5
    DOSPageInterval     1
    DOSBlockingPeriod   300
</IfModule>

6. Verify FFmpeg

which ffmpeg
# Should return: /bin/ffmpeg

API Endpoints

Upload Video Segment

POST upload.php?action=receive (default)

Authentication: Upload or Admin API key required

Parameters:

  • camera (POST): amcrest or reolink
  • api_key (POST): Valid upload or admin API key
  • video (FILE): MP4 video segment (max 50MB)

Response:

{
  "status": "success",
  "message": "Segment appended",
  "period": "day",
  "camera": "amcrest"
}

Example:

curl -X POST \
  -F "camera=amcrest" \
  -F "api_key=your_upload_key_here" \
  -F "video=@segment.mp4" \
  http://scootercam.net/timelapse/upload.php

Security Validations:

  • API key authentication
  • Rate limiting (10 requests/minute/IP)
  • File type validation (MP4/MOV only)
  • File size validation (≤50MB)
  • MIME type verification
  • File header validation

Manual Rotation

GET upload.php?action=rotate&api_key=ADMIN_KEY

Authentication: Admin API key required

Moves all active videos to archive directory.

Response:

{
  "status": "success",
  "message": "Rotated 6 videos"
}

Example:

curl "http://scootercam.net/timelapse/upload.php?action=rotate&api_key=your_admin_key_here"

System Status

GET upload.php?action=status&api_key=ADMIN_KEY

Authentication: Admin API key required

Response:

{
  "status": "success",
  "current_period": "day",
  "videos": [
    {
      "camera": "amcrest",
      "period": "day",
      "exists": true,
      "size": 15728640,
      "modified": "2025-09-01 14:30:15"
    }
  ]
}

Archive Cleanup

GET upload.php?action=cleanup&days=7&api_key=ADMIN_KEY

Authentication: Admin API key required

Deletes archive files older than specified days (1-30 day range enforced).

Response:

{
  "status": "success",
  "message": "Deleted 12 old archive files"
}

Example:

curl "http://scootercam.net/timelapse/upload.php?action=cleanup&days=5&api_key=your_admin_key_here"

How It Works

1. Secure Upload Process

  1. Rate Limiting: Check IP hasn’t exceeded 10 uploads/minute
  2. Authentication: Validate API key (upload or admin level)
  3. File Validation: Comprehensive security checks on uploaded file
  4. Camera Validation: Ensure camera name is in allowed whitelist
  5. Period Detection: Determine current time period (night/day/evening)
  6. Secure Processing: Append to appropriate daily video using validated paths
  7. Cleanup: Remove temporary files and log successful operation

2. Video Concatenation

Uses FFmpeg’s concat demuxer with security controls:

ffmpeg -f concat -safe 0 -i concat_list.txt -c copy -avoid_negative_ts make_zero output.mp4

3. Automatic Rotation

  • Cron triggers rotation every 8 hours with admin API key
  • All videos in /videos/ moved to /videos/archive/ with timestamp
  • Security logging of rotation operations
  • Fresh videos created as new segments arrive

4. Security Monitoring

  • All operations logged to security.log with timestamps and IP addresses
  • Rate limiting data tracked in rate_limit.json
  • Failed authentication attempts logged as warnings
  • Suspicious activity (invalid files, path traversal attempts) logged as errors

5. File Naming Convention

Active Videos: {camera}_{period}_{date}.mp4

  • amcrest_day_20250901.mp4

Archived Videos: {camera}_{period}_{date}_{time}.mp4

  • amcrest_day_20250901_160002.mp4

Camera Configuration

Update Camera Upload Scripts

Cameras must now include API keys in their upload requests:

Amcrest Camera:

#!/bin/bash
API_KEY="your_upload_key_here"
CAMERA="amcrest"
SEGMENT="/path/to/segment.mp4"

curl -X POST \
  -F "camera=${CAMERA}" \
  -F "api_key=${API_KEY}" \
  -F "video=@${SEGMENT}" \
  http://scootercam.net/timelapse/upload.php

Reolink Camera:

#!/bin/bash
API_KEY="your_upload_key_here" 
CAMERA="reolink"
SEGMENT="/path/to/segment.mp4"

curl -X POST \
  -F "camera=${CAMERA}" \
  -F "api_key=${API_KEY}" \
  -F "video=@${SEGMENT}" \
  http://scootercam.net/timelapse/upload.php

Monitoring

Security Log Monitoring

# Real-time security monitoring
tail -f /home/scootercam/public_html/timelapse/security.log

# Check for suspicious activity
grep "WARNING\|ERROR" /home/scootercam/public_html/timelapse/security.log

# Monitor rate limiting
grep "Rate limit" /home/scootercam/public_html/timelapse/security.log

System Status Monitoring

# Check current system status (requires admin key)
curl "http://scootercam.net/timelapse/upload.php?action=status&api_key=your_admin_key_here"

# Verify active video count (should be ≤6)
ls -la /home/scootercam/public_html/timelapse/videos/*.mp4 | wc -l

Disk Usage Monitoring

# Check active videos disk usage
du -sh /home/scootercam/public_html/timelapse/videos/

# Check archive disk usage
du -sh /home/scootercam/public_html/timelapse/videos/archive/

# Overall system disk usage
df -h

Performance Monitoring

# Check recent uploads (last 24 hours)
find /home/scootercam/public_html/timelapse/videos/ -name "*.mp4" -mtime -1 -ls

# Monitor temp directory (should be empty most of the time)
ls -la /home/scootercam/public_html/timelapse/temp/

Troubleshooting

Problem: Upload Rejected (401 Unauthorized)

Cause: Invalid or missing API key
Solution:

  1. Verify API key in camera configuration
  2. Check security.log for failed authentication attempts
  3. Ensure API key matches configuration in upload.php

Problem: Upload Rejected (429 Rate Limited)

Cause: Too many requests from same IP
Solution:

  1. Check rate limiting in security.log
  2. Wait for rate limit reset (1 minute)
  3. Investigate if camera is sending duplicate requests

Problem: Upload Rejected (400 Bad Request)

Cause: File validation failure
Solution:

  1. Check file size (must be ≤50MB)
  2. Verify file format (MP4/MOV only)
  3. Ensure camera name is valid (amcrest or reolink)
  4. Check security.log for specific validation errors

Problem: FFmpeg Concatenation Failures

Check:

  1. FFmpeg availability: which ffmpeg
  2. Temporary directory permissions: ls -la temp/
  3. Video codec compatibility
  4. Security log for FFmpeg errors

Problem: Too Many Active Videos (>6)

Solution: Force rotation

curl "http://scootercam.net/timelapse/upload.php?action=rotate&api_key=your_admin_key_here"

Problem: Large Archive Directory

Solution: Clean old archives

curl "http://scootercam.net/timelapse/upload.php?action=cleanup&days=5&api_key=your_admin_key_here"

Security Incident Response

Suspicious Activity Detection

Monitor security.log for these indicators:

  • Multiple failed authentication attempts from same IP
  • Invalid file upload attempts
  • Path traversal attempts
  • Unusual upload patterns or timing

Immediate Response Actions

  1. Block suspicious IPs at firewall level
  2. Rotate API keys if compromised
  3. Review and analyze security logs
  4. Verify system integrity

API Key Compromise Response

# 1. Generate new API keys
openssl rand -hex 32  # New upload key
openssl rand -hex 32  # New admin key

# 2. Update upload.php with new keys
# 3. Update camera configurations
# 4. Update cron job with new admin key
# 5. Clear rate limiting data
rm -f /home/scootercam/public_html/timelapse/rate_limit.json

Maintenance

Daily Tasks

  • Monitor security log for suspicious activity
  • Verify 6 or fewer active videos exist
  • Check cron rotation is working (00:00, 08:00, 16:00)
  • Monitor disk usage

Weekly Tasks

  • Review security log trends
  • Clean old archives (automated with API call)
  • Verify video quality and completeness
  • Update API keys if security policy requires

Monthly Tasks

  • Security audit and log analysis
  • Performance optimization review
  • Archive storage management
  • System backup verification

Emergency Procedures

Complete System Reset:

# 1. Backup all data
mkdir /home/scootercam/timelapse_emergency_backup
cp -r /home/scootercam/public_html/timelapse/* /home/scootercam/timelapse_emergency_backup/

# 2. Stop all uploads temporarily (block at firewall)
# 3. Force rotation with admin key
curl "http://scootercam.net/timelapse/upload.php?action=rotate&api_key=your_admin_key_here"

# 4. Clear rate limiting
rm -f /home/scootercam/public_html/timelapse/rate_limit.json

# 5. Restart system monitoring

Expected Performance

File Sizes (approximate):

  • 8-hour period video: 15-25 MB
  • Daily total per camera: 45-75 MB
  • Archive growth: ~100-150 MB per day

Processing Performance:

  • Segment append: <2 seconds
  • Authentication check: <0.1 seconds
  • File validation: <0.5 seconds
  • Rotation: <10 seconds for all videos
  • Rate limiting check: <0.1 seconds

Security Overhead:

  • Minimal performance impact (<5% processing overhead)
  • Comprehensive logging with efficient file I/O
  • Rate limiting with memory-efficient tracking

Compliance & Auditing

Security Audit Trail

All operations are logged with:

  • Timestamp (Y-m-d H:i:s format)
  • Client IP address (with proxy detection)
  • Action performed
  • Success/failure status
  • Error details (for failures)

Log Retention

  • Security logs: Recommend 90-day retention minimum
  • Rate limit data: Automatic cleanup (5 minutes retention)
  • Archive videos: Configurable cleanup (default 7 days)

Access Control Summary

  • Upload Operations: Upload or Admin API key required
  • System Status: Admin API key required
  • Manual Rotation: Admin API key required
  • Archive Cleanup: Admin API key required
  • Log Access: File system level permissions

Version History

v3.0 – Security-Hardened System

  • Added two-tier API key authentication
  • Implemented comprehensive file validation
  • Added rate limiting per IP address
  • Security logging and audit trail
  • Path traversal protection
  • Command injection prevention
  • Input sanitization and validation
  • Error information hiding
  • File size and type restrictions

v2.0 – Simplified System

  • Removed sunset/golden hour logic
  • Fixed multiple file creation issue
  • Improved cron-based rotation
  • Added status monitoring endpoint

v1.0 – Original System

  • Basic file upload and concatenation
  • Manual rotation support
  • Archive functionality
Categories: Timelapse

Leave a Reply

Your email address will not be published. Required fields are marked *