Saturday, March 26, 2011

Working remotely over a high-latency connection

Does anyone have any tips for working remotely with high network latency? My project is Linux/Windows only, so I work remotely on a beastly machine at CSAIL. It works great when I'm at CSAIL, but terribly from Palo Alto. My two solutions thus far have been:

Use vi over ssh. This sucks because vi then occupies my terminal, and all my keystrokes are buffered over ssh so typing is slow.

Use MacVim and OpenAFS (or any other network filesystem here, including sshfs). This works better for me, but when switching files it takes *forever* (> 15 seconds) to tab-complete filenames or open or save a file. I have to believe this is due to inefficiencies in AFS, because I can scp files faster than MacVim can write to AFS.

I think the ideal solution might look something like Dropbox, where I have a full copy of my source code on both machines, and I only wait for the network when I want to go run the tests. However, I'm doing the initial sync now, and it's taking forever, so I'm not sure this is a real solution.

I haven't tried using bcvi or vim's scp:// file protocol. I'm not enthusiastic about them because I tend to use vim like people use emacs, meaning I keep lots of buffers open and open new files from the same vim instance. I want to be able to use tab completion to open new files, and I don't think vim's scp protocol will let me do that, and if it does, it would suffer from the same latency issues.



  1. I've never used it but perhaps is worth looking at.

  2. sshfs might be what you're looking for.

  3. Ironically I believe rsync was written for exactly this purpose to make working on code that needed to be kept in sync between Australia and North America nicer at the time...

  4. Don't: remote filesystems encounter a truly frustrating number of edge cases - even simple things like running ls on a directory cause a number of stat() calls which can't be cached without breaking POSIX and/or losing data. There are a ton of different filesystems, tuneables for implementations like SSHFS, etc. - none of which actually work over more than across-town latency because the real fix would require everything to be rewritten to use non-blocking I/O, which hasn't really gone mainstream outside of web browsers.

    After fighting this for years, I switched to a DVCS model where I use a local Git repo and push changes out to the remote system for testing - the main advantage being that it only sends deltas and is even reasonably efficient for large binary files which get append-only changes. I believe Dropbox uses similar delta-handling algorithms so your initial painful sync should be much worse than normal but I haven't seen how well it would handle simultaneous changes on both sides.

  5. Two options:

    1. Use NX ( This only works on Linux (X11), but it's how I did most of my PhD work and it lets you run a real graphical editor window (or 5) with latency acceptable enough to work over a modem or cellular data connection if necessary. If you find SSH too high-latency, this may still be too obnoxious for you though.

    2. Just bite the bullet, run a Linux/Windows VM locally to prototype, and use a DVCS or rsync to run things for real. That's what I'm doing right now; I have a Windows VM running in the background and have its drive mounted on my Mac so I can use editors, the command line, etc. on the Mac side wherever possible.

  6. You could still use bcvi to support your preferred way of working. By default it runs 'gvim scp:...' but you could add vim's --remote option to have it open the file in an existing editor process.

    Of course the bcvi workflow assumes that you have a shell open on the remote server. Using the SSH 'ControlMaster' config option to reuse an existing SSH connection (see link) will significantly reduce latency. If you don't have an existing shell connection open then there's not likely to be any solution to your latency problem.

  7. very nice thanks for sharing

    hey friend see snow on google
    Type “Let It Snow” on @Google If you click and drag you can wipe the snow away. It is great. source: