FreeGate's product is an all-in-one "appliance" for small to medium size organizations. Called the OneGate, it rolls all of the usual suspects into a nicely integrated package with a consistent user interface. The checklist is long (and growing), but most people want our firewall, VPN, SRA and e-mail "features." We also have a web server, ftp (including anonymous), mailing lists, CGI "sandboxes" and others. I jokingly call it the product that should put system administrators out of work. At least for the run of the mill tasks anyway.
I was hired (employee #17) as Build Master / System Admin / Network Admin / whatever. The first task was to relieve the CTO from installing new PC's on the LAN and generating nightly builds. The second task was to automate the builds so a new build was ready every morning for engineers and QA to test and use. Since our product is an "embedded" version of FreeBSD, any changes to libraries or headers had to propagate out to the engineers build environment or else they might test (or hack) against an out of date library. I accomplished this by gently whacking gcc and the stock FreeBSD make files and then wrapping make in a shell script, "build," which set up the appropriate search paths. After the builds were working, it was time to beat CVS and GNATS (bug database) into shape. Check-ins to CVS had to log information to the appropriate GNATS report. QA needed a GUI front-end (webgnats) to the database, and then some extensions.
Meanwhile, back at the patch panel: FreeGate was growing from a seed round startup to a full blown hardware and software facility, but we managed to cram ourselves into 2 small offices a couple of blocks from the Faultline Brewing Company. The servers were (and still are) several Pentiums of various vintages running FreeBSD. There is one PC running NT, and acting as file server and PDC for the Windows 95 based desktops (and Xwin32 to interface with the Unix systems). We also had a Sun Sparcstation running Oracle. Over time we've added some more, but the number nodes on the net still rank in the "small" category.
After six months or so of starting at FreeGate, though, things were running along about as well as they could. We had a contractor in doing some work for the first release of the product, but the work was not going to finish on time. I offered to help, "coding is good for you, like calculus." I wrapped up a unit tested alpha in about a week. I learned getting $100/hour doesn't make you anything special any more. (That and I was being horribly underpaid.) One thing led to another and I took on more development responsibilities. Soon, I owned NTP, SNMP and creation of upgrade packages.
NTP and SNMP are really pretty straightforward: Download and port the latest code (the first from open source, the second from a vendor), and check it into the build. The second half is a little more tricky, but not too much so. Our product uses "service managers" as an init daemon on steroids. The service managers connect the UI and a configuration database together to configure and manage services. If the configuration changes, the service manager will write a new config file (or whatever), then tell the daemon to reload with the new config. We even have a library API for making it mostly a cookbook recipe.
Upgrade on the other hand, is almost impossible to describe, which makes it even harder to implement. Think of computing the transitive closure of the differences between complete releases (a full OS and all), then tar it up, encrypt it, sign it and bundle it up to an ftp server. That's the creation of the payload. (More or less.) That's how it started. Soon, QA needed packages on a regular basis, so I made the script (a command line perl program) a little more friendly with some curses based UI components. Of course a little UI can be a dangerous thing. Later, the developer who was writing the box-side components (receiving notifications, checking for applicability, downloading, reassembling, prepping for installation, and all the attendent failure modes), wanted a change of pace and to work on Java and UI features. So, since I knew more about how the beastie worked than anyone else, (I had to test the payloads somehow), I inherited the box-side components.