FileManager.replaceItemAt(_:withItemAt:) fails sporadically on ubiquitous items

I’m encountering a strange, sporadic error in FileManager.replaceItemAt(_:withItemAt:) when trying to update files that happen to be stored in cloud containers such as iCloud Drive or Dropbox. Here’s my setup:

  • I have an NSDocument-based app which uses a zip file format (although the error can be reproduced using any kind of file).

  • In my NSDocument.writeToURL: implementation, I do the following:

  1. Create a temp folder using FileManager.url(for: .itemReplacementDirectory, in: .userDomainMask, appropriateFor: fileURL, create: true).

  2. Copy the original zip file into the temp directory.

  3. Update the zip file in the temp directory.

  4. Move the updated zip file into place by moving it from the temp directory to the original location using FileManager.replaceItemAt(_:withItemAt:).

This all works perfectly - most of the time. However, very occasionally I receive a save error caused by replaceItemAt(_withItemAt:) failing. Saving can work fine for hundreds of times, but then, once in a while, I’ll receive an “operation not permitted” error in replaceItemAt.

I have narrowed the issue down and found that it only occurs when the original file is in a cloud container - when FileManager.isUbiquitousItem(at:) returns true for the original fileURL I am trying to replace. (e.g. Because the user has placed the file in iCloud Drive.) Although strangely, the permissions issue seems to be with the temp file rather than with the original (if I try copying or deleting the temp file after this error occurs, I’m not allowed; I am allowed to delete the original though - not that I’d want to of course).

Here’s an example of the error thrown by replaceItemAt:

Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission to save the file “test-file.txt” in the folder “Dropbox”." UserInfo={NSFileBackupItemLeftBehindLocationKey=file:///var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSFileOriginalItemLocationKey=file:///var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSURL=file:///Users/username/Library/CloudStorage/Dropbox/test-file.txt, NSFileNewItemLocationKey=file:///Users/username/Library/CloudStorage/Dropbox/test-file.txt, NSUnderlyingError=0xb1e22ff90 {Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission to save the file “test-file.txt” in the folder “NSIRD_TempFolderBug_y3UvzP”." UserInfo={NSURL=file:///var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSFilePath=/var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSUnderlyingError=0xb1e22ffc0 {Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"}}}}

And here’s some very simple sample code that reproduces the issue in a test app:

    // Ask user to choose this via a save panel.
    var savingURL: URL? {
        didSet {
            setUpSpamSave()
        }
    }
    
    var spamSaveTimer: Timer?
    
    // Set up a timer to save the file every 0.2 seconds so that we can see the sporadic save problem quickly.
    func setUpSpamSave() {
        spamSaveTimer?.invalidate()
        let timer = Timer(fire: Date(), interval: 0.2, repeats: true) { [weak self] _ in
            self?.spamSave()
        }
        spamSaveTimer = timer
        RunLoop.main.add(timer, forMode: .default)
    }
    
    func spamSave() {
        guard let savingURL else { return }
        
        let fileManager = FileManager.default
        
        // Create a new file in a temp folder.
        guard let replacementDirURL = try? fileManager.url(for: .itemReplacementDirectory, in: .userDomainMask, appropriateFor: savingURL, create: true) else {
            return
        }
        let tempURL = replacementDirURL.appendingPathComponent(savingURL.lastPathComponent)
        guard (try? "Dummy text".write(to: tempURL, atomically: false, encoding: .utf8)) != nil else {
            return
        }
        
        do {
            // Use replaceItemAt to safely move the new file into place.
            _ = try fileManager.replaceItemAt(savingURL, withItemAt: tempURL)
            print("save succeeded!")
            
            try? fileManager.removeItem(at: replacementDirURL) // Clean up.
            
        } catch {
            print("save failed with error: \(error)")
            // Note: if we try to remove replaceDirURL here or do anything with tempURL we will be refused permission.
            NSAlert(error: error).runModal()
        }
    }

If you run this code and set savingURL to a location in a non-cloud container such as your ~/Documents directory, it will run forever, resaving the file over and over again without any problems.

But if you run the code and set savingURL to a location in a cloud container, such as in an iCloud Drive folder, it will work fine for a while, but after a few minutes - after maybe 100 saves, maybe 500 - it will throw a permissions error in replaceItemAt.

(Note that my real app has all the save code wrapped in file coordination via NSDocument methods, so I don’t believe file coordination to be the problem.)

What am I doing wrong here? How do I avoid this error? Thanks in advance for any suggestions.

An update on this weird behaviour:

I have discovered that when replaceItem fails in this circumstance, the temp file has in fact been moved into place correctly and has replaced the original file. But when I get the error, the old original file has taken the place of the old temp file and it's that which cannot be removed.

I have tested this by checking both the content and the fileResourceIdentifier of the original file and the temp file, and logging them before and after the error. After the error they are swapped.

I’m encountering a strange, sporadic error in FileManager.replaceItemAt(_:withItemAt:) when trying to update files that happen to be stored in cloud containers such as iCloud Drive or Dropbox.

Have you filed a bug on this and, if so, what's the bug number? As part of that bug, I'd suggest installing the "iCloud Drive" profile, reproducing the issue a few times, then uploading a sysdiagnose of the failure. See the profile installation instructions for the full details of that process.

And here’s some very simple sample code that reproduces the issue in a test app:

Thank you for that. I got your code up and running in a test app and was able to replicate the problem fairly easily. As to WHY it's happening, that's unclear. From the console log, it appears that the entire replace sequence worked fine but the sandbox then rejected access to the temporary file as the kernel was trying to cleanup post-swap. Weirdly, it doesn't appear to be blocking actual access to the file (continuing after the failure worked fine), so I think the issue is at least partially tied to the very specific circumstances the swap creates.

What am I doing wrong here?

I'm not sure you're doing anything wrong, as I think this is a bug.

How do I avoid this error?

Have you tried retrying the save? That appears to work in my testing, though it may not be a workable solution in your case. Beyond that, I'd need a better understanding of exactly how you're interacting with the files and what your full requirements are.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Many thanks for the reply and information, Kevin, much appreciated.

Have you filed a bug on this and, if so, what's the bug number? As part of that bug, I'd suggest installing the "iCloud Drive" profile, reproducing the issue a few times, then uploading a sysdiagnose of the failure. See the profile installation instructions for the full details of that process.

I hadn’t filed a bug report yet because I had assumed it was something I was doing wrong given that using replaceItem and a temporary folder is presumably a common pattern. I’ll file a report tomorrow - I’m following the iCloud Drive profile instructions you linked to and am now waiting the 24 hours they say I need to wait before I can get the sysdiagnose. Once I have that I’ll file the report along with a sample project.

Have you tried retrying the save? That appears to work in my testing, though it may not be a workable solution in your case. Beyond that, I'd need a better understanding of exactly how you're interacting with the files and what your full requirements are.

With a bit of refactoring I probably could retry the save. In my app this is all done inside my NSDocument’s writeToURL method. I use my own drop-in replacement for FileWrapper (you helped me with some of the finer points of FileWrapper a few years ago) that incrementally writes changes to a zip file using Libzip, which supports incremental saves on copy-on-write systems such as APFS.

A potential problem with the re-save approach is that my save usually works by copying the zip file at the original location to a temporary location, updating it there, and then moving it into place using replaceItemAt. After this particular replaceItemAt error, however, the original file has in fact been updated despite the error (the error being on the old version of the file which is now in the temporary directory). So if I re-save by making a copy of that and try updating again, I could potentially mess up the file by trying to save into it stuff that has actually already been done. (However, I do keep a snapshot of the older archive around in case of problems, so I might be able to work around this problem using that.)

I wonder, though - given that the original file has in fact been replaced by the temp file despite the error, can I not just check for this and ignore the error if the file seems to have been replaced after all? E.g.:

  1. Before replacement, record the file resource ID of the temp file.

  2. Use replaceItemAt(originalURL, withItemAt: tempURL).

  3. If there’s an error, get the file resource ID for the file at the intended saving location and compare it against the ID I recorded in (1). If they are the same, I know the replacement has succeeded despite the error. In this case, I can just try to delete the temporary folder and move on.

  4. If the file IDs of the current user file and the temp file from before replace don’t match or couldn’t be got, attempt a re-save.

Is there something wrong with this approach? (I’ve attached some sample code below demonstrating how this might work.)

Many thanks, Keith

// Get a temporary folder appropriate for creating the new file in.
let replacementDirURL = try fileManager.url(for: .itemReplacementDirectory, in: .userDomainMask, appropriateFor: savingURL, create: true)

// Create the new file at the temporary location.
let tempURL = replacementDirURL.appendingPathComponent(savingURL.lastPathComponent)
try createNewContentAt(url: tempURL)

// Record the file resource ID of the temp file we created.
let tempFileID = (try? tempURL.resourceValues(forKeys: [.fileResourceIdentifierKey]))?.fileResourceIdentifier

// Now try to move the file into place.
do {
    // Use replaceItemAt to safely replace the original file with the updated file we created at the temp location.
    _ = try fileManager.replaceItemAt(savingURL, withItemAt: tempURL)

    // Clean up.
    try? fileManager.removeItem(at: replacementDirURL)
    
} catch {
    // Check to see if the original file was in fact replaced despite the error.
    if let tempFileID,
       let savingFileID = (try? savingURL.resourceValues(forKeys: [.fileResourceIdentifierKey]))?.fileResourceIdentifier,
       tempFileID.isEqual(savingFileID) {
        
        // If so, just try to remove the temp dir and move on.
        try? fileManager.removeItem(at: replacementDirURL)
                
    } else {
        // If we got here, replace really did fail and we need to handle it.
                
        // We should do some more work and try to resave here before throwing an error.
                
        throw error
    }
}
FileManager.replaceItemAt(_:withItemAt:) fails sporadically on ubiquitous items
 
 
Q