bbftpPRO, a Professional Data Transfer Solution
bbftpPRO is a data transfer program - as opposed to ordinary file transfer programs, capable of transferring arbitrary data over LAN/WANs at parallel speed.
bbftpPRO has started (at the Particle Physics Dept. of Weizmann Institute of Science) as an enhancement of bbftp but it is not supported by the bbftp group at IN2P3.
Please contact the author, Dan Schrager, for all bbftpPRO support.
To download bbftpPRO click here »
New: multipath support is implemented (for unix/piped modes) in version 9.
New: bbftpPRO version lite+ has been released, and it's supposedly lightning fast. But, hey, no backward compatibility with previous versions, and yet same known usage.
A description of bbftpPRO's backward compatibility and upgrades to bbftp could be found here »
bbftpPRO Pipe Command Examples
bbftpPRO implements a so called long and large pipe between two Unix commands that (usually) do not run on the same host, ensuring I/O data streaming at parallel speed with its "pipe" command (syntax: pipe ‹remote-command›), where the local command is piped to/from the bbftp client command.
Here are examples of pipe commands:
- bbftp -V -e "pipe cat foo.bar" | cat >foobar
(remote file ~/foo.bar is transferred as local file ./foobar)
- bbftp -V -e "pipe bbftp -V -e \"pipe cat foo.bar\"" | cat >foobar
(a relayed data transfer via an intermediary host)
- cat foo.bar | bbftp -V -e "pipe cat >/tmp/foo.bar"
(local file ./foo.bar is transferred as remote file /tmp/foo.bar)
- cat foo.bar | bbftp -V -e "pipe bbftp -V -e \"pipe cat >/tmp/foo.bar \""
(a relayed data transfer via an intermediary host)
- bbftp -V -e "pipe cat foo.bar" | bbftp -V -e "pipe cat >/tmp/foobar"
(a third-party relayed transfer, where the client/current host actually relays the data between the two remote hosts - could be faster if the current host has better bandwidth connectivity to each remote host than the direct connection between remotes)
- bbftp -V -e "pipe cat /dev/zero" | cat >/dev/null --or--
cat /dev/zero | bbftp -V -e "pipe cat >/dev/null"
(raw networking rate estimation)
- bbftp -V -e "pipe cat /dev/zero | pmr -l 10MiB" | pmr -l 10MiB | cat >/dev/null --or--
cat /dev/zero | pmr -l 10MiB | bbftp -V -e "pipe pmr -l 10MiB | cat >/dev/null"
(capping transfer rate - eventually at both ends - at 10 MiB/s using the pmr command line tool)
Any pair of commands can be piped as long as they are meaningful.
(e.g. local/remote data compression and remote/local storage with paired "tar cj" and "cat").
Extra levels of piping are possible (with extra escaped (\) sequences).
Note that the remote user and host have been left out in the above examples (they default to the local user and localhost respectively).
bbftpPRO Wrapping Scripts
The pipe mode is suitable for scripting.
There are available a few wrapping scripts that implement simple get, reget -NEW, put, reput -NEW file transfers, based on "dd".
Fast rsync style transfers can be made with the xsync -NEW, and xsync.perl -NEW/recommendable scripts, based on "tar".
The file name designation is user@host:path, either user or/and host can be omitted (e.g. reget foo.bar).
Parameters relevant to bbftpPRO could be appended (e.g. xsync.perl u1@h1:foo u2@h2:bar -s -V -p 4).
reget and reput are world premiers for the parallel streaming mode.
xsync is capable of transfers between two remote hosts.
Note: an obsolete built-in (parallel) streamed tar transfer mode is available too. Use commands get/put together with "setoption (no)unix" to control this feature. An RFIO-enabled tar is also required for CASTOR users. It should be renamed/symlinked as "rftar" after building from this source »
Tip: Pre lite+ versions, in ssh mode and only in correlation with switch -D (i.e. passive client as data connection listener), use the bbftp command line switch -a ‹IP-name-or-address› to specify client's public name or IP address if behind a NAT router or multihomed.
Tip: In version lite+, switch -a has been removed from client and has been replaced with
switch -n ‹NAT-name-or-address›, at either the client or/and the server command, to help the data
connection listening side to specify its (public) address.
Tip: Use command ssh-keygen -t rsa -m PEM to generate RSA keys in the legacy PEM format, accepted by bbftpPRO as well as ssh.
A bbftpPRO Practical Networking Guide is here »
Last update: September 18, 2022