I spent an hour or so figuring out how different programs merge folders/subfolders with the same name. I did this by resetting the folder before running each command. Here are the results: Nested...
I spent an hour or so figuring out how different programs merge folders/subfolders with the same name. I did this by resetting the folder before running each command.
I thought trailing slash mattered more, but it actually only matters in the "merged destination" instance.
Out of all of these, I think rclone provides the least surprising result (the actual destination is the specified destination)--but rclone is a lot slower than mv in many scenarios and it should be noted that you can't rename files with rclone like you can with mv.
I don't really find any of these cases confusing, or even terribly inconsistent. The way Docker and some other tools handle it is far worse. Having /* is a glom copying the contents of the...
I don't really find any of these cases confusing, or even terribly inconsistent. The way Docker and some other tools handle it is far worse.
Having /* is a glom copying the contents of the directory that gets silently expanded behind the scenes, not the directory itself. The trailing slash is otherwise always ignored. That does seem to be the source of much of the author's confusion.
Perhaps some confusion derives from that rclone and rsync are intended to work primarily with directories while mv and cp are just as likely to be for files themselves, especially cp. There's a reason you need an extra flag to copy a directory with cp.
These feel intuitive to me now too, but I’m sure there was a time when they didn’t. The trailing slash is ignored by GNU cp but not by BSD cp, and rsync behaves like the BSD version. As far as I...
These feel intuitive to me now too, but I’m sure there was a time when they didn’t.
The trailing slash is ignored by GNU cp but not by BSD cp, and rsync behaves like the BSD version.
As far as I know, best practice for interoperability between flavors of cp is to append a /. if your goal is to have the contents of sourceDir copied into destinationDir because the behavior of the BSD and GNU version of cp are the same for:
cp -r sourceDir/. destinationDir
Using dir/. instead of dir/* copies all files in a directory including hidden files - which I think we’re calling dot files these days - as well.
Used thusly, cp and rsync have the same source and destination parameters in all three tables.
Thanks for this comment! I wasn't aware that rsync will always create a subfolder if there is no trailing slash. That helps me understand it a bit more. You inspired me to do some more experiments...
Thanks for this comment!
I wasn't aware that rsync will always create a subfolder if there is no trailing slash. That helps me understand it a bit more.
You inspired me to do some more experiments and add this to the document:
Glad I could I inspire you. I always check the behavior of / before running rsync manually and always check the destination before I run dd. Too many bad experiences in my formative years from...
Glad I could I inspire you.
I always check the behavior of / before running rsync manually and always check the destination before I run dd. Too many bad experiences in my formative years from carelessness with those two in particular.
This one has a relatively simple reason, although pardon if my wording is bad. I have the mental model but am unsure if I've articulated clearly. I might have also reached a semi-correct...
This one has a relatively simple reason, although pardon if my wording is bad. I have the mental model but am unsure if I've articulated clearly. I might have also reached a semi-correct conclusion with some weird incorrect jumps on the way. If anyone can correct me I'd be thrilled.
# mv one/. three
mv: cannot move 'one/.' to 'three/.': Device or resource busy
one/. is a reference to the directory one, but from within the directory itself. So if you have a directory structure like this.
one/a.txt
one/b.txt
two/one/
You could "expand" the mv command sort of like this to get an idea of how that works:
cd one
mv ../one ../three
mv: cannot move '../one' to a subdirectory of itself, 'three'
This is a bit weird likely because .. is a hardlink (like a file) to the parent directory. But that is its own can of worms. Anyway...
The reason you get the specific Device or resource busy when doing the original command is the same reason you get this (relies on having a device to mount):
# mount /dev/zram0 ./one
# cd one
# umount ../one
umount: /home/vord/tmp/one: target is busy.
You're trying to remove the device while you're using the device.
Do you or @xk3 mind taking one for the team and trying a mv dir_1/. dir_2 when dir_2 is on a different file system. I’m curious as to whether the fallback in that situation for mv to be equivalent...
Do you or @xk3 mind taking one for the team and trying a mv dir_1/. dir_2 when dir_2 is on a different file system.
I’m curious as to whether the fallback in that situation for mv to be equivalent to a cp -a + rm will throw the same error.
After skimming the source for coreutils mv - acknowledging I never messed with c and my c++ is 15 years rusty - I’m optimistic that it might work and would suggest the limitation is somewhere deeper like rename(). That’s just a guess though, really.
On OpenSUSE Tumbleweed: $ mv tmp/. /srv/storage mv: inter-device move failed: 'tmp/.' to '/srv/storage/.'; unable to remove target: Invalid argument (tmp is not /tmp)
On OpenSUSE Tumbleweed:
$ mv tmp/. /srv/storage
mv: inter-device move failed: 'tmp/.' to '/srv/storage/.'; unable to remove target: Invalid argument
Interesting. I know rm has exceptions / should fail for stuff like rm dir/. so maybe the behavior across file systems is the result of a clean up routine when the “rm“ fails. I don’t know but I’m...
Interesting. I know rm has exceptions / should fail for stuff like rm dir/. so maybe the behavior across file systems is the result of a clean up routine when the “rm“ fails. I don’t know but I’m certainly curious about sorting it out.
I mean…realpath, cp, and rsync have no problem with it and with mv the error differs between same file system and across file systems.
My typical rsync options are -avz. The options are thoroughly explained here. Short explanation, Archive, Verbose, Compress. I find this to be a quick way of getting everything I want 99% of the...
My typical rsync options are -avz. The options are thoroughly explained here. Short explanation, Archive, Verbose, Compress. I find this to be a quick way of getting everything I want 99% of the time. I also do a dry run -n first, so -avzn, just to make sure I didn't mess up my folders. Occasionally you have to apply other options, like --size-only, if your timestamps are a mess. Always happy to hear other opinions to better my workflow.
I spent an hour or so figuring out how different programs merge folders/subfolders with the same name. I did this by resetting the folder before running each command.
Here are the results:
Nested subfolder (actual destination
three/one/one):mv one three/oneonethree/onecp -r one three/one && rm -rf oneonethree/onersync -auh --remove-source-files one three/oneonethree/onerclone move -q --no-traverse one three/one/oneonethree/one/oneMerged subfolder (actual destination
three/one):mv one/* three/oneone/*three/onecp -r one three && rm -rf oneonethreersync -auh --remove-source-files one threeonethreerclone move -q --no-traverse one three/oneonethree/oneMerged destination (actual destination
three):mv one/* threeone/*threecp -r one/* three && rm -rf oneone/*threersync -auh --remove-source-files one/ threeone/threerclone move -q --no-traverse one threeonethreeI thought trailing slash mattered more, but it actually only matters in the "merged destination" instance.
Out of all of these, I think rclone provides the least surprising result (the actual destination is the specified destination)--but rclone is a lot slower than
mvin many scenarios and it should be noted that you can't rename files withrclonelike you can withmv.I don't really find any of these cases confusing, or even terribly inconsistent. The way Docker and some other tools handle it is far worse.
Having
/*is a glom copying the contents of the directory that gets silently expanded behind the scenes, not the directory itself. The trailing slash is otherwise always ignored. That does seem to be the source of much of the author's confusion.Perhaps some confusion derives from that
rcloneandrsyncare intended to work primarily with directories whilemvandcpare just as likely to be for files themselves, especiallycp. There's a reason you need an extra flag to copy a directory withcp.These feel intuitive to me now too, but I’m sure there was a time when they didn’t.
The trailing slash is ignored by GNU
cpbut not by BSDcp, andrsyncbehaves like the BSD version.As far as I know, best practice for interoperability between flavors of
cpis to append a/.if your goal is to have the contents ofsourceDircopied intodestinationDirbecause the behavior of the BSD and GNU version ofcpare the same for:Using
dir/.instead ofdir/*copies all files in a directory including hidden files - which I think we’re calling dot files these days - as well.Used thusly,
cpandrsynchave the same source and destination parameters in all three tables.I guess this is true. I've mostly used GNU tooling outside a very brief stint with FreeBSD. Mac users would probably see mostly the reverse.
If anyone is curious, Section 2.1 of the
rsyncarch wiki covers the trailing slash behavior of rsync and its origin (BSDcpvs GNUcp).Thanks for this comment!
I wasn't aware that rsync will always create a subfolder if there is no trailing slash. That helps me understand it a bit more.
You inspired me to do some more experiments and add this to the document:
When the destination doesn't exist:
mv one twoonetwomv one/* twoone/*rclone move -q --no-traverse one twoonetwocp -r one two && rm -rf oneonetwocp -r one/. two && rm -rf oneone/.twocp -r one/* two && rm -rf oneone/*rsync -auh --remove-source-files one/ twoone/tworsync -auh --remove-source-files one twoonetwo/oneWhen the destination is an empty folder:
mv one twoonetwo/onemv one/* twoone/*tworclone move -q --no-traverse one twoonetwocp -r one two && rm -rf oneonetwo/onecp -r one/. two && rm -rf oneone/.twocp -r one/* two && rm -rf oneone/*tworsync -auh --remove-source-files one/ twoone/tworsync -auh --remove-source-files one twoonetwo/oneGlad I could I inspire you.
I always check the behavior of
/before runningrsyncmanually and always check the destination before I rundd. Too many bad experiences in my formative years from carelessness with those two in particular.This one has a relatively simple reason, although pardon if my wording is bad. I have the mental model but am unsure if I've articulated clearly. I might have also reached a semi-correct conclusion with some weird incorrect jumps on the way. If anyone can correct me I'd be thrilled.
one/.is a reference to the directoryone, but from within the directory itself. So if you have a directory structure like this.You could "expand" the
mvcommand sort of like this to get an idea of how that works:This is a bit weird likely because
..is a hardlink (like a file) to the parent directory. But that is its own can of worms. Anyway...The reason you get the specific
Device or resource busywhen doing the original command is the same reason you get this (relies on having a device to mount):You're trying to remove the device while you're using the device.
Do you or @xk3 mind taking one for the team and trying a
mv dir_1/. dir_2whendir_2is on a different file system.I’m curious as to whether the fallback in that situation for
mvto be equivalent to acp -a+rmwill throw the same error.After skimming the source for coreutils
mv- acknowledging I never messed with c and my c++ is 15 years rusty - I’m optimistic that it might work and would suggest the limitation is somewhere deeper likerename(). That’s just a guess though, really.Also on mac OS the error is “Invalid Argument”
On OpenSUSE Tumbleweed:
(tmp is not /tmp)
Silly question but just to confirm: contents of
tmp/aren’t in/srv/storage?Nope. I just had a blank file from a touch in
tmp.Interesting. I know
rmhas exceptions / should fail for stuff likerm dir/.so maybe the behavior across file systems is the result of a clean up routine when the “rm“ fails. I don’t know but I’m certainly curious about sorting it out.I mean…
realpath,cp, andrsynchave no problem with it and withmvthe error differs between same file system and across file systems.Interesting, quite interesting.
yeah I also get "Invalid argument"
My typical
rsyncoptions are-avz. The options are thoroughly explained here. Short explanation, Archive, Verbose, Compress. I find this to be a quick way of getting everything I want 99% of the time. I also do a dry run-nfirst, so-avzn, just to make sure I didn't mess up my folders. Occasionally you have to apply other options, like--size-only, if your timestamps are a mess. Always happy to hear other opinions to better my workflow.