If it matters, systemd should be the thing starting it
The claim Every long-running process on a production Linux host should be a systemd unit. Not pm2, not a screen session someone detached from in the spring, not nohup & in a deploy...
The claim
Every long-running process on a production Linux host should be a systemd unit. Not pm2, not a screen session someone detached from in the spring, not nohup & in a deploy script. Those tools answer "how do I keep this running after I log out" and silently fail to answer the questions that matter: what starts it after a reboot, what restarts it after a crash, where do its logs go, and what stops it from taking the whole host down with it. systemd answers all four, is already installed, and costs a twenty-line file.
The unit file
[Unit]
Description=Orders API
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
Type=simple
User=orders
WorkingDirectory=/srv/orders
ExecStart=/srv/orders/bin/server
Restart=always
RestartSec=2
Environment=PORT=8080
EnvironmentFile=/etc/orders/env
MemoryMax=1500M
TasksMax=256
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/srv/orders/data
PrivateTmp=yes
[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now orders
Walk through what each block buys you, because every line replaces an incident.
Restart policy that distinguishes crash from crash-loop
Restart=always with RestartSec=2 brings a crashed process back in two seconds. Add the rate limit that separates a transient failure from a broken deploy:
StartLimitIntervalSec=60
StartLimitBurst=5
Five failures inside a minute and systemd stops retrying and marks the unit failed — which your monitoring should catch via systemctl is-failed. Without the limit, a binary that segfaults on startup restarts forever, generating gigabytes of logs and a CPU graph that looks like service.
Resource limits are the difference between one bad service and one bad host
MemoryMax=1500M means a leak in this service gets this service killed — and restarted cleanly by the policy above — instead of inviting the kernel's out-of-memory killer to choose a victim, which it does with no regard for what you consider important. On a 4 GB host running an application and a database, an uncapped application leak usually ends with the database dead. TasksMax does the same for a thread or fork explosion.
The sandboxing lines are free hardening
ProtectSystem=strict makes the entire filesystem read-only to the process except paths you list in ReadWritePaths. ProtectHome, PrivateTmp, and NoNewPrivileges close common escalation routes. This converts "attacker exploits the app" from "attacker owns the host" into "attacker can write to one data directory" — a meaningfully different Tuesday. Verify what you have:
systemd-analyze security orders.service
The score is blunt but the itemised list is genuinely useful, and most services can adopt half of it with no code changes.
Logs without a logging stack
Anything the process writes to stdout lands in the journal, timestamped, rate-limited, and rotated without configuration:
journalctl -u orders -S -1h # last hour
journalctl -u orders -p err -S today # errors only
journalctl -u orders -o json | jq . # structured, if you log JSON
For a single host or a small fleet, this replaces a log shipper entirely. Set a cap in /etc/systemd/journald.conf with SystemMaxUse=2G and stop thinking about it.
Graceful shutdown is part of the contract
On stop and on deploy, systemd sends SIGTERM, waits, then sends SIGKILL. The default wait is 90 seconds; set it deliberately with TimeoutStopSec=30 and make the application use the window — stop accepting connections, finish in-flight requests, close the database pool, exit zero. A process that ignores SIGTERM gets killed mid-transaction on every single deploy, and the resulting half-written records surface as mysterious support tickets weeks later. If the application needs longer for a final flush, ExecStop= can run an explicit drain script first. Test it the blunt way: systemctl stop orders under load, then check the logs for requests that died uncompleted.
Scheduled work belongs here too
A timer unit replaces the cron entry, and the pairing gets you the same journal, the same resource caps, and two properties cron lacks: Persistent=true runs a job that was missed while the host was down, and a service that is still running when the next tick arrives is not started twice.
[Timer]
OnCalendar=*-*-* 02:30
Persistent=true
RandomizedDelaySec=300
The randomised delay matters on a fleet: twenty hosts all hitting the database at exactly 02:30 is a self-inflicted spike.
Where the other tools still fit
pm2 remains reasonable as a development convenience, and inside a container platform the orchestrator plays this role and systemd stays out of it. The claim is narrower and firmer: on a host you operate, where the process matters, the init system you already have is the supervisor. If the answer to "what starts this after a power cycle" involves a human remembering something, that is the outage you have scheduled without picking the date.