Hi all—forgive the length of this message. I post it here, including the really helpful responses from Travis, in CP tech support, because he thought there might be additional wisdom out there on this forum. If you don’t want to wade through the details, below, the gist of my question was this: Time Machine and package-based programs (i.e., Notebook, Scrivener) may have some problems playing nicely together. My question concerned the interaction between Time Machine, which runs backups periodically throughout the day, and Notebook, which will necessarily be in different states throughout the day as it’s being backed up.
Again, for anyone who’s interested, you’ll see below my own queries; Travis’s responses, and a few items from the Scrivener list, where this subject came up after I’d lost a good deal of work in Scrivener.

A few months ago I lost a good deal of work while using Scrivener, and the
> only thing that the forum and the program’s developer could come up with was
> something having to do with the package-based format / structure of
> Scrivener and the Time Machine program of Mac. I use Time Machine—have a
> drive hooked up to my computer all the time, and I’d like to be sure that
> something like that loss with Scriv. not happen with Notebook. But it
> occurs to me that Notebook, like Scrivener, shares the “many different kinds
> of files / data all living happily under one filename” quality, and I’m
> wondering a few things:
>
> 1. Does Notebook use something like the same file structure as Scrivener?

I’m not familiar with Scrivener, but the forum excerpt you copied suggests that Scrivener uses a package format, which is what NoteBook uses. You can think of it as a self contained file format.

2. Have you ever heard of any problems using Time Machine (or any other
> automated backup program) with Notebook (I see no such mentions on the forum
> for Notebook, but I’m not sure my search was complete)?

Yes, we have had issues with some backup applications, specifically how they recognize packaged files. The problem is that some backup applications recognize these package files as a folder containing other files, but because of the types of files they contain, there is information that is lost in the transfer, resulting in corruption. That being said, I have not personally had any issues restoring notebooks from my TM nor have I heard of our customers having issues specifically with Time Machine, but I don’t think posting your question on the forums is a bad idea, even if you’re satisfied with my answer :-)

3. Do you have any particular recommendations or cautions for keeping data
> safe on Notebook? With Scivener, one of the conclusions of the forum was
> that one should NOT use TM in any automated mode with Scriv, because the TM
> backup might be initiated at the wrong point in an autosave moment for
> Scriv. and cause data loss.

In cases where these backup applications have caused corruption, the solution is to compress the files before you backup. Compressing the file will capsulate it in a format that will protect it from corruption when backing up transferring. You could compress your notebooks, or make a folder containing all of your notebooks and compress the file before you backup. To do this, select the file or folder in the Finder and under the File menu, select Compress. That will create a .zip file to appear in the same window which will be safe from the corruption you’re concerned about.

But again, I haven’t heard of any issues with Time Machine. You may want to send an inquiry to Apple and see if they’ve heard of issues like this and if they have any recommendation for backing up package files. I’m inclined to say they will tell you everything should be fine, since may of the applications in OS X use the package file format.

I’ve excerpted a few of the relevant posts from my Scrivener forum
> discussion below:
>
> thanks for any help you can give me
>
> John Whittier-Ferguson
>
>
>
>
>
> Regarding using time machine and what might be happening. Best practices
> first.
>
> TM is a backup system (duh). As such it will “snapshot” the FS at a point in
> time. This snap shot is based on disk allocation via inode/block usage and
> the journal. if you watch TM in action you will see that the first thing is
> does is “prepare” the backup. I suspect the TM is doing the same things that
> its brothers do and is FREEZING the journal and copying it to a new
> location. In theory this is fast and safe (I use very $$ apps that do this
> all day). There is one condition that makes this “unsafe”. That is when a
> write intensive (think save) app start building data in the journal before
> the copy/move is complete. This is compounded by large writes like a big
> binary (think mov). Basically the journal is frozen, so the write doesn’t
> really complete. Now your file handle is pointing to a bad location and data
> is written but the sectors are not marked as associated with the first part
> of the file. Since the data is still in memory of the process it looks
> correct UNTIL you stop (quit) the process) and open the file again. Oracle
> and I spent quite a bit of time working on this with a backup vendor that is
> NOT apple, but the claim was that the technology was very similar to TM.
> Hence Oracle recommends full exports instead of “hot” backups†.

My knowledge of Time Machine is not in depth enough to know it’s internal workings on this level, but I will tell you that I used to work for Apple as a technician and I rarely had issues of customers that experienced any kind of loss whatsoever with this software and on a personal level as an Apple enthusiast, I can say this is one of the most reliable and stable Apple products that I’ve used.

As far as handling write intensive applications, Apple has plenty of those and I’ve relied on TM to recover many of their files. Final Cut, Pro Tools, and Motion to name a few, so I don’t think hot backups pose a threat with TM.

Wonderful that we understand how this might happen. How do we prevent it?
>
> We take a note from Oracle best practices and do the following:
> 1. DO NOT ENABLE TM OVER WIRELESS.
> 2. Run TM on a direct attached drive.
> 3. DO NOT ATTACH DRIVE WHILE YOU ARE WORKING ON PROJECTS.
> 4. Quit scriv (and all other package based apps) before attaching your TM
> drive.
> 5. Attach TM drive.
> 6. Once TM completes reopen project and get back to work.
>
> and this is from the creator of Scrivener:
> The cause seems to have been synchronisation – that can be different from
> backing up. By far the best way of backing up is to use Scrivener’s File >
> Backup To feature and choose to zip up the backup. This ensures that the
> search info is updated etc before creating the backup. Of course, to be
> doubly sure, unzip the backup and check it opens. That way you know you have
> a working back up.
>
> The problem with synchronising a .scriv file is that it is prone to the same
> problems that synchronising a folder is prone to. I would need to know a
> little more about your exact synchronisation set up, though. I have never
> used ChronoSync so would need to look at that, but if you could tell me
> exactly what you did, that would help. Were you synchronising two ways (that
> is, trying to update an existing version of a project with an updated
> version) or one way (just copying a project), for instance?
>
> Incidentally, given the problems mentioned above, you are probably wondering
> why Scrivener uses a package-based format, then, rather than a flat one. The
> reason for this is that Scrivener files need to pack a lot of information
> and data, as you can import movie files, PDF files and so forth. “Flat”
> files tend to get read into memory in their totality. A Pages or Word file
> is unlikely to get so big that reading it all into memory at any one time is
> going to cause a problem; and besides, you want to see the entirety of these
> files in the interface at once. But reading a Scrivener project containing
> 100MB movies, hundreds of PDF files and so forth all into memory at once,
> and keeping it there while you work, could slow everything to a crawl. And
> really there is no need to have everything in memory all the time anyway, as
> you generally only ever look at a couple or three files at a time. The
> package-based format allows programs such as Scrivener to load into memory
> only what it needs. It is ideal for the job, but unfortunately the pay-off
> is that it does not play well with synchronisation.

That’s exactly why NoteBook uses this file type, and there are synchronization issues with it, but as I said, haven’t heard of any with TM. Aside from the precaution I mentioned earlier, you can never have too many backups of your important files :-)

Regards,

Travis
Circus Ponies Customer Support