Advice on poor Linux performance vs Windows
Hello!
I recently reworked my setup such that I have my drive which holds the operating systems (dual boot of Windows 11 and Linux Mint Cinnamon) and another drive which holds all my actual data. This is my first dive into trying to move to Linux as my daily driver but I'm noticing some performance issues.
The first thing I noticed is that transfer speeds are much less than expected. Copying a large file within my data drive I get about 300MB/s on Linux, which is pretty slow for M.2 drives- I get about 10x that speed doing the exact same operation on Windows. I could be okay with this but I also noticed that some video files, like the 4K mp4s off my phone, are virtually unplayable. They'll run at maybe 1fps and/or bog down the UI so much I cant even use the seek. This is the case in both the default media player and VLC. These same files play with no issue on Windows.
I suspect the reason for all this stems from my data drive being NTFS, though my file system and Linux knowledge is pretty weak so this is just a theory. Any ideas / best practices that might help me here?
Are you sure it isn’t confusion between MBps and Mbps? 300MBps is 2400 Mbps. The fact that you say the difference is about 10x matches this, since the B to b difference is 8x.
You’ll get a decent few GB/s (capital B) sequential read out of a good NVMe drive, though, so I would be a bit suspicious of only a couple of hundred MB/s when dealing with large files.
Yup! it's a Samsung 970 evo, should have plenty in the tank for those kinds of speeds
Im pretty sure windows is all measured in bytes. Just anecdotally the thing that kicked this investigation off was a 60ish GB file copy which was several minutes in Linux which felt slow hence retesting in Windows which was probably only like 15 seconds. I'll confirm tomorrow though.
Could be a difference in write cache... but it is probably the NTFS driver
https://unix.stackexchange.com/questions/30286/can-i-configure-my-linux-system-for-more-aggressive-file-system-caching
Is the video issue all files or just specific ones? I can get on board with the NTFS driver theory for the transfer speed in general, but even 4K video is far less than 300MB/s (as long as you’re not pulling it straight off a cinema camera), so I think the video is a separate problem.
If there are some files that work and others that don’t, it’d be good to know if they’re different codecs and/or bitrates - it sounds like the player might be sending them to a device that can’t decode them properly, where it should be falling back to straightforward CPU decoding in that scenario instead.
This is true I'll experiment with some different files at various quality levels. The CPU comment is curious because I have a Nvidia gfx card so now I'm wondering if this is something where it's trying and failing to process it on the card which causing some issue. I did pull down the Nvidia driver's when I set up initially though.
This idea sounds right to me. Rather than experiment with different files, the first thing you should do is verify that your Nvidia card is actually using the drivers you think it is. There's probably a way to do that in the Mint settings in something like "System Information", or you could check the output of lshw, though that's a little more esoteric. If you get stuck hit me up later, I can help.
Does Mint come with
inxipre-installed these days? If it doesinxi -Gmight be easier on the eyes thanlshw -c display.I'd try looking if you actually run Nvidia proprietary drivers. I think glxinfo (command) can tell you more - if you don't have it, don't install it. Or if you have Steam installed, you can find infk about your hardware there - if the driver is MESA, you are not running Nvidia proprietary drivers and that could mean you are not decoding on GPU but actually on CPU. Or you can try command nvidia-smi, which will tell you some basic info about Nvidia GPU (or end up with an error if you don't run proprietary driver).
You can find more information about your video files by using command ffprobe (from ffmpeg package), that way you can narrow down the differences between videos that play alright and the ones that don't.
My first thought with the choppy 4k video was that you're not using hardware decoding, meaning the video is being decoded on the cpu instead of the gpu. Some media players like mpv have hardware acceleration disabled by default. Not sure about VLC, but you should check. It's the only way a lot of lower-end computers are able to play high def video at all.
The tricky bit is that this can go either way, too: a decently modern desktop CPU will handle 4K video in software no problem, whereas hardware acceleration could be hitting a driver issue while trying to pass it through to NVDEC (among other possibilities). Or, as you say, a lower end CPU might be struggling to decode in software just because that’s the default, when passing off to the GPU could take the load off entirely assuming the configuration is correct.
As with so much in tech, it probably comes down to “eh, try both - you’ll either learn something from how they differ, or learn something from the fact they don’t”!
Depending on which NTFS implementation your distro uses that could very well be the case. In fact, the state of NTFS support on Linux is such that one of the kernel devs has started on a clean implementation.
I believe that driver has been released but isn't entirely complete. Mind you, I am just going of recent news articles as well.
There aren't that many options available if you want to dualboot. The common "wisdom" is to use a NAS for data, but that is a bit much for a quick fix.
I have heard people using exFAT for data drives rather than NTFS. But, that really is purely for data and I am not entirely sure how well that performs as it is more or less something I heard "in the hallways" and have no experience with myself.
This is interesting. I'm not sure why I didn't think to try this before but I can try pulling the files into the Linux os partition and retest there to see if this is the case. I'm open to reformatting the drive so long as it remains accessible in both OSs and can handle large files.
If you’re looking for easy use across operating systems - Mac x Linux or Windows x Linux or Windows x Mac - exFAT is the least bothersome filesystem. It’s missing some important features like journaling and support for symlinks, but it’s the only option you can mount easily on any modern OS.
I would not use exFAT for long term storage. After years of use, I found many corrupted media files on my external hdd formatted as exFAT that I was using across macOS and Windows. Maybe a coincidence, maybe not. The thing is, people on the net have been warning against the use of exFAT if you care about your data.
So I don't use NTFS drives from Linux, but a cursory review of kernel news suggests NTFS is enough of a mess to warrant (yet another) a new NTFS driver
However, reading throughput is suggested to be identical as the old driver. So unless you're doing heavy reading and writing at the same time, it shouldn't be cripplingly slow to read.
Assuming you're using the ntfs-3g driver (it's the current, not the new one, and if you aren't, start there), they offer a slew of things to check under Performance -> WHY IS THE DRIVER SO SLOW
I will check tomorrow, thank you
Long time Linux user here, currently on Mint Cinnamon, who started out doing almost exactly what you are doing (NTFS formatted data drive to keep windows compatibility).
While I will preface by saying it could be something else, I think the most likely explanation is that your data drive is NTFS formatted.
After I realized I could do everything I wanted for my home desktop on Linux, I went in all the way, cross-loaded all my data onto other drives, and formatted my data drive in ext4, then put my data back on it.
It worked much better afterwards. Lower resource usage and much better transfer speeds all around.
I second this as I had an identical issue. I couldn't even install Steam games on that drive until I formatted it to ext4.
Glad you found success with it! For now I want to keep it accessible on both OSs but am open to reformatting the drive if it improves things
I seem to remember there was a software package for windows that allowed it to read ext, but it’s been about a decade since I had to worry about such things.
You are correct, an ext4 driver for Windows. Most recent signed release
And the code.
There are unsupported ext4 features, so using ext3 for compatability is probably prudent.
I mostly just use exfat for USB sticks that need compatibility. You would think "we <3 open source" Microsoft would be open to releasing a new NTFS driver for linux or a ext4/btrfs/xfs driver for windows.
open a terminal and type
cpupower frequency-infoOften there's a powersetting that is set to a lower draw setting which can impact performance, it's typically one of the first things I do to set it to performance or balanced.
This is what I got. Is it good, or is it bad?
powersave mode is good for when you are on a battery but this might be part of the cause of the choppy 4K decoding. You could also try with
mpvinstead of VLCLike others have said, are you sure you are not reading MBps instead of Mbps or vice versa?
Side note. I recently switched to Linux Mint entirely from windows, and I have gotten used to things as well. I actually copy large files from various drive types and across different networking limitations/locally as well.
All of this is to say, I'm happy to hop on a call with you or something and debug any bottlenecks or dispel any misunderstandings. Feel free to privately DM me.
That's a very kind offer! I may just take you up on that if I can't get it working right soon.
Regardless, if you do figure it out let us know! It's good to knowledge share!
Use
hdparm -tT /dev/your-driveto check read speeds without FS.Use
lshwto check bus situation.You could check
sudo dmesg --human --nopager --decode --level emerg,alert,crit,err,warnto see if there are any interesting messages relating to your hardware.It may be worth checking for firmware updates for your M.2 drives--a friend of mine had some trouble recently with a WD Black SN770 and installing an updated firmware fixed his Linux performance issues and also made his computer stop freezing in Windows a few times per day. Just a thought!
I will check, thank you!
My Linux system is faster across the board than on Windows with one exception, like you, disk I/O. It took me a while to figure it out, and it was that I have my drive encrypted with LUKS. When I upgraded my SSD to a newer/larger one, I decided to try installing without LUKS to see if it fixed the performance issue (it did). I ended up going back to using encryption after testing, but this could be your issue.