Is upstart already installed?

Yes Upstart should be installed on any base Ubuntu distro. You can verify by doing: "dpkg-query -l upstart"

Perhaps what I really need is an example upstart guide to follow? Step 1, create job, Step 2 create event.

Do you really need events? Or do you just want to run a job that gets started at boot and restarted when it dies? Maybe you can be more specific about what kind of events you want to handle, and the relationship/dependencies of your job to other jobs on the system (if any).

I want the service to start when the OS starts. I want the service to start if it has been stopped.

Well, at a guess I'd say you don't want it to start if you stop it manually :-). This is the sort of thing Upstart does very easily. I'll update my answer with an example. You do need to know something about how the service starts: i.e. does it just run in the foreground? Or does it fork itself into the background? Does it fork twice (this is common for some services)?

The scripts seem to have -service flags. That is all I know.

You'll have to do your own footwork. I won't research it for you. Try running it by hand. Do you get a shell prompt, while it runs in background, or does it keep running in the foreground?

The process begins running in the background. A command prompt is returned. How do I tell how many times it forks?

Unfortunately that's not so easy. You'd need to use strace. Is there a man page? Is there an option to run it in foreground? I'm going to try moving this discussion to chat, as StackExchange keeps nudging us to do.

Often, processes that background themselves will do their own monitoring/restarting if they die for some reason. Are you sure these do not?

If they do, then it's just a question of ensuring they start at boot time. Easily done. Let me know.

In fact, let's just try it. We'll just use the RC mechanism (which Upstart runs, but it's a simpler, legacy mechanism):

1. Go to /etc/init.d, and make a copy of dns-clean (rename it as you like).

2. Edit the file. Change the INIT INFO section by altering the descriptions and clearing out the Required-Start and Required-Stop fields. Make a new name for Provides.

(basically change everything but Default-Start)

3. Delete these lines:

4. Change the lines in the start) block to run your scripts. Leave the ";;" at the end.

That's it.

Reboot your system, and let me know how it goes.

I've never used chat before, and don't know how notifications work. I'll be working in another window so might not see your response, when it comes. I've enabled "desktop notification", whatever that is, so hopefully I will.

thank you for the example. I already feel a lot better. This seems fairly simple. Is the begin init info used by anything? Or is that just convention?

Sorry - forgot a step. Yes the init info is used by update-rc.d. Documentation is here: wiki.debian.org/LSBInitScripts

So once you've edited your file, say it's called foobar (probably your Provides should match the file name), you can run: update-rc.d foobar enable

That instruments your RC script into the necessary runlevels for boot time, based on the contents of the INIT INFO script

Note that these simple RC scripts will start a service. They will not restart it if it dies. If you really need to do that manually (i.e. if the service itself doesn't handle the restarting), you will have to use Upstart directly as we started to discuss. But this can be a good test to see how far you can get with the simpler approach.

Will you put that into an answer so that I can accept it?

Sure, let me write something more concise. Does this mean it's working for you?

Well, no. It is a good answer, but it doesn't solve for the service abending. What I have done is used cron to launch a bash script. The bash script looks for the process. If the process is not found, the bash script calls the service start script. The worst thing I can think of happening is that cron tries to start a service after we tried to stop it.

Lol you'd be better off with Upstart, which will start your service immediately if it terminates unexpectedly. But that'll work. If you want to finess it, you can run "strace -o /tmp/strace.out -f YOUR_SCRIPT". Then look at /tmp/strace.out and see how many fork() calls there are. Then you could craft a proper Upstart script which would immediately restart your service when it dies.

One advantage of the RC approach is that it would be portable to any Linux distro. Upstart is Ubuntu-specific. Everybody else uses Systemd (and Ubuntu will also, some day - Upstarts days are numbered).

I have updated my answer with what I've communicated to you. Would appreciate you marking it as "Answered". Cheers.