To me, the real problem is not the dotfiles, but the regular (i.e. non-hidden) files that applications surreptitiously create in my home folder. (Which the author does mention as being particularly bad.) dotfiles in the home folder are not a new thing on Linux (e.g. .bashrc, .profile, .emacs, etc.), even if it would be better if more applications used the XDG folders. And if I don't see them when I "ls", they don't bother me.
But nowadays, lots of applications seem to think it's OK to put normal files in my home folder without asking first. This is just outright barbarism. In particular, Ubuntu now seems like it's all-in on snaps, and if you use snaps, they create a non-hidden "snap" folder in the user's home folder. Madness! And when people complain, the devs have the gall to suggest that this is all fine:
I like the idea of snap/flatpak in the abstract, but until they get their act together, and stop creating non-hidden files in my home folder without asking first, I am never going to use snaps.
And of course, all this is on top of the new fashion for the OS to "helpfully" create Documents, Music, Video, etc folders in my home folder, and set the XDG environment variables to point to them. Noooooope.
But at least in that case the user can change the environment variables and delete the stupid folders. No such luck with ~/snap.
This is particularly weird, since click (snap's "predecessor" at Canonical/Ubuntu) was fully XDG compliant, even encouraging developers to separate data from cache etc. (unlike flatpak which just dumps everything in ~/.local/share/[...], though that is still infinitely better than using ~/snap).
I personally don't like xdg. I have bin files from pip and other stuff in ~/.local/bin.
I have huge amounts of cache files in both .cache and .config. It's supposed to make my life easier by being able to backup .config and dump everything. but in reality this doesnt happen.
Backing up .config for just the user defined configs is a massive PITA. you have to put so many gitignore exceptions and I'm not even talking about the gigs of data all the chromium derivatives put into that folder.
I think xdg has an issue of being obtuse when I googled it a while ago, it was very confusing, and I didn't find any real definitions. However I didn't look that hard.
You can try. I mount .cache as tmpfs. You need at least 500mb for your browser alone.
Browsers tend to work fine. Some apps just fail badly like rofi (command history stored there oof)
For what it's worth I also tmpfs+asd .nvm, .npm, and */node_modules, without any major headaches. I did have to fix a couple minor pathing dramas but nothing too drastic.
> unlike flatpak which just dumps everything in ~/.local/share/[...]
Here you are referring to separating the config and data?
Is this actually a problem? I can see how for things like games it would be valuable to have a special location for thing like saves games, like how you can move the roaming save for minecraft from a computer to another; but I never understood the net gain from splitting the application files in config/data
Curated configuration generally outlives data. If storage space becomes an issue, there is value in recovering storage space but config is generally small and much more important in the long term.
Yeah it is crazy when I noticed Ubuntu created a ~/snap folder that is required when anything is installed via snap. People defend placing application data/config folders in ~/ because they're old and it's always been like that, but Snap is brand new and they still neglected to follow the modern standard.
Other "new" softwares that fail to follow best-practices by putting top level files into ~/ just on my laptop are: npm, Ansible, pylint, docker, steam, rustup+cargo, vscode, IDEA IDE, Zoom, AWS CLI. These applications have no excuse.
Also I agree on the Documents, Music, Video, etc. I hate upper case letters in my folders, so when I do a reinstall I delete those and restore my backups from the lower-case versions.
Windows actually has the same dotfile infestation problem these days. I mean, you still get stuff in Documents that shouldn't be there, but on top of that you get apps - often Unix ports - that believe that %USERPROFILE% (i.e. one folder above Documents) is the appropriate place for their configs and such (it's not - that should also go into AppData\Local or \Roaming as needed). And they usually use a dot-prefixed name, just because.
I just looked at my Win10 profile dir, and it has 36 dot-prefixed folders, and 7 more files on top level. Offenders include .gitconfig, .rustup, and .vscode, just to name a few.
On Windows, you could move the configs elsewhere, make a symlink to them in the old place, and mark the symlink as hidden so you don't see it in explorer. Too bad this can't work on Linux because the hidden-state is encoded in the filename.
Well, the hidden-bit on Windows is really semantically equivalent to dot-prefix on Unix, and people set up their file managers to show such files for similar reasons (I don't even use Explorer, mostly). In Win10, it's even exposed as a setting in Settings | For Developers
For actual file management, and generally working in the terminal, I'm firmly in the "two blue panes" club, so it's Far Manager on Windows (and Midnight Commander on Linux). Far is probably the most advanced text-mode file manager in general right now.
Compared to 15 years ago I think Explorer is much better now, it could be even called usable.
Nevertheless there are alternatives. One liked by many are Norton Commander alike file managers like Total Commander or even Far. I am not too much in favor of them though.
I like to use Directory Opus. It is powerful and flexible. Costs money though.
It's fair to store documents and saved games in My Documents.
Much better than storing them in C:\Windows or C:\Program Files, as was the case a decade ago, before windows introduced UAC and blocked write access to these.
It's senseless to store files that don't have user-facing meaning in My Documents. I know games that use it varyingly for web caches, shader caches, binary config files, debug logs, Lua scripts, downloaded mods, and even executables. And while arguably most gamers understand what save files are, they can't be double-clicked to open them, so I don't consider them documents (Maybe this is pedantic).
All this kind of stuff is what Appdata\Roaming and Appdata\Local are supposed to be for.
I understand not backing up AppData/Local, but everything in AppData/Roaming is supposedly important enough to sync it over the network (if you use domains or another mechanism to give you the same Windows account on multiple computers). Any default policy which doesn't back up AppData/Roaming seems ill advised.
Nope, Witcher 1 saves are just really large (for some reason in addition to all the flags they also contain some locale data and such). 10GiB is actually pretty reasonable for a Witcher playthrough.
It's similar in the Witcher 2, 3 and Skyrim. The saves start small and they grow to 10MB as you progress in the game. I guess there is a lot of world state and items to hold in an RPG. Count a few hundred saves, manual or autosave and that's a lot of space.
In the witcher 1 there is also an uncompressed BMP thumbnail with each save that is about a megabyte. It's PNG compressed in the witcher 2. They learned from their mistake.
My AppData/Local is 14GB, mostly caches from Chrome, Firefox and Spotify. My AppData/Roaming is 3.3GB, mostly my emails (with my Browser's localstorage a distant second).
Chrome's cache is in AppData/Local/Google/Chrome/User Data and Firefox's cache is in AppData/Local/Mozilla/Firefox/Profiles/. The only big offender on my system is npm with a cache in AppData/Roaming/npm-cache
Thunderbird infamously places the local copy of IMAP mails (i.e. Effectively a cache) into Appdata\Roaming. It's the single largest directory I have there. Everything else is quite reasonable in size.
You can't do it as a runtime check. You have to compile the software with the flags to import and link to the latest windows library. The program will hard crash if run on an older version.
I just tried to use the API. I literally couldn't find a working example on the internet in 2 hours so I guess noone figured it out in a decade.
You won't have the chance to call anything. If the executable is statistically linked to the latest API, the executable won't be able to load on older systems.
I'm surprised that these new IsWindows...() functions claim compatibility down to Windows 2000. Doubt microsoft knew how they would name their next release 20 years ago. A Google search indicates that they are in fact broken.
You won't be able to compile with the function imported using the modern lib (because it doesn't exist on XP) but you can conditionally link it at runtime with LoadLibrary/GetProcAddress.
If IsWindows... doesn't work then you can use the VerifyVersionInfo function directly.
Why did they have to narrow it down to games? Could just go with "Saved config (and data)", so other apps could just dump small config files which can be effectively backed up.
A game's exe dir is a long-lived but potentially replaceable thing, like a car. Saved games may be persisted across game versions, so you don't necessarily want them to be in the same directory. Admittedly not the best analogy, but was just observing that it's strange and incongruent to store the 2 together.
Which is sort of an improvement from the past, where user-level config data was thrown into the actual app directory in "program files." So if you ever wanted to back-up those settings, you needed to track down every single unique location for each app.
If it's the entire game, it might be to allow kids to install the game without administrator privilege (read: parental knowledge/consent), though I agree that letting people choose would be best.
What? I'm not sure how that connects to my comment. I'm saying that some games install to "My Documents" to avoid requiring administrator privilege for users who might not have that privilege.
I have a slightly different pet peeve. I don't like applications that default to saving files to my home folder. I wish they would look for e.g. /home/user/Documents or whatever else seems appropriate and even semi-standardized given the type of file.
But just dropping stuff in my home directory--look, if I'm in a hurry and my defenses are down, you basically just encouraged me to clutter things up.
The last software I found that does this is a "download helper"...even though I have a Downloads folder that my distro already set up, that Chrome and Firefox know to use, this download helper thinks I want those downloads dumped in my home folder. I hope some generous person changes this behavior, meanwhile I forgot to charge it manually for the last batch and I have a bunch of cleanup to do.
That’s what the article talks about. How applications should use $XDG_DATA_HOME, $XDG_CONFIG_HOME and $XDG_CACHE_HOME and some defaults if they’re not set.
I can't find "official" XDG documentation of this, but the Arch Wiki describes, for instance, the XDG_DOCUMENTS_DIR and XDG_VIDEOS_DIR environment variables:
You should be able to change the default settings for these directories by changing /etc/xdg/user-dirs.defaults (system-wide) and ~/.config/user-dirs.dirs (users override)
I strongly prefer apps default to the previously used directory, and the home directory otherwise. IME apps are wrong 99% of the time when they guess their own save location.
I have great difficulty understanding why the snap developers think the excuse “it’s hard to change!” flies.
If it’s hard to change, you should’ve either fixed your architecture from the start, or thought a bit harder about the implications of magically creating a new directory in $HOME.
It shouldn't be hard to change unless the string "/snap" is hardcoded all over the codebase. There should be a function like "get_app_dir" or "get_install_dir" or some other config string lookup that doesn't use a static string.
On Windows Golang defaults to install to C:\. Had no problems running it from Program Files instead. It's crazy that a new programming language cannot follow decades-old conventions!
Python did this until very recently too. Windows has a problem, because people expect to use those programs in the command line and the %PATH% has a size limit and is a bit difficult to set-up properly.
The sum of the length of all environment variables has to be below 32767 characters. That doesn't seem like a good excuse to save 14 characters by putting your program's folder in C:/ instead of C:/Program Files/
If you do deeply nested paths (as npm used to do) then you can run against the old MAX_PATH limit that limits paths to 260 characters, but the correct solution is to either choose a flatter directory structure or use the newer APIs that allow paths with 32,767 characters.
Steam is doing just fine writing into C:\Program Files (x86)\Steam\steamapps from an unpriviledged process. It just gives all users full rights over that folder. (whether that's a great idea is a different question, but UAC isn't a problem if you just set your access rights correctly)
Steam has definitely been handling Program Files wrong for as long as Program Files has existed. It's a marvel that there isn't more historic Malware intentionally targeting Steam install paths, though it looks like that storm is brewing on the horizon as I've seen references to such malware in the process of becoming a problem.
Steam should have worked to figure out a better approach to UAC years ago. At this point it is one of several big tech debt issues with Steam that has my confidence in Valve at an all time low as a consumer.
If the user never had admin priviledges then you can't install in C:\ either and the program should just install in AppData\Local (as my sibling poster correctly stated)
The problem there is "how lazy is VendorX"? With \Program Files\ Microsoft has a ton of top level ACLs to lock down access to the directories that everything just inherits by default. How much work does the OS need to do to allow something like "folders named `data` under \Programs\ have ACLs XData and `code` have ACLs YCode"? Does it need to watch for silly mistakes like Devs that localize `code` (`Código` for instance) or misspell it (`ocde` or `coode`)?
Also would make it easier to backup the program and its data or copy them to a new system.
But it seems because of the idea that some people may want to back up data and config separately, we end up with stuff scattered everywhere, with some even hidden away in the Windows Registry. Makes backups ridiculously complex.
I always thought it was because that space in “Program Files” was troublesome for a lot of programs, particularly command-line ones. IIRC, older versions of MinGW for instance would simply not work at all if installed in a path containing a space, because some of their startup scripts didn’t properly quote all parts of the path.
Default gopath was just a way to let newbies get up and running quickly; it originally required you to specify where to put it. I imagine they put it at ~/go so it would be glaringly obvious. Hopefully modules will render it all moot anyway.
They put it at ~/go because that's what every official document used as the example $GOPATH, before it had a default value, so it's largely just to minimize change from what they previously recommended.
That said, for something like source code, it makes sense to go into $HOME, so I don't understand this particular complaint the other poster made.
Other people's code shouldn't be in my $HOME if I didn't put it there explicitly. If it's part of the dependencies of a project, it should be either in a cache directory of that particular project, or in the cache directory of the build tool (that preferably conforms to XDG), like all sane build tools do. GOPATH is an incomplete (thus incorrect) solution to a problem that was being solved correctly and completely in other languages (like Ruby).
It's weird that the Go developers were working in such isolation that they didn't pick up on this. All programming languages that I know of that work on modern tooling (Haskell, Rust, Node.JS, even C#!) adopted the Ruby style dependency management when it turned out to be correct years ago.
In my comment I say that it's better than the way Go did it, and Go is switching to the Ruby style, so maybe you could add some kind of argument why it's actually not better than that in any other language?
I can't remember the last time I had a version management problem in Ruby, though the ecosystem is so stable right now not many backwards incompatible things happen any more.
Lots of programming languages and IDEs seem to do this. Rosegarden (the sequencer) and Steam (last time I tried anyway) both create folders right under my home folder as well. Barbarism is right.
> And when people complain, the devs have the gall to suggest that this is all fine
From my impression the answer was more "we set up things in a hurry because of deadlines and now we are stuck with this until we implement the epoch system"
> And of course, all this is on top of the new fashion for the OS to "helpfully" create Documents, Music, Video, etc folders in my home folder, and set the XDG environment variables to point to them. Noooooope.
Yuuuup. It baffles me how Linux users can simultaneously complain that 'its never the year of the Linux desktop' and then at the same time rile against basic usability tenets. The average users wants/needs Documents, Pictures, Videos, etc folders, and it wants the OS to be aware and use these as such.
Now I'm okay with being able to set a specific flag that doesn't make these folders during install, but by default they absolutely should be there, unless you're doing a server install.
Well, I know that I hate them. And I know a lot of other people that hate them. And I'm pretty suspicious of 'features' that allegedly make life easier for naive users by definitely making life harder for experienced users.
And I guess I would call it a "basic usability tenet" that the computer does what I tell it to. Documents, Music, and Videos folders violate this, because if I do the seemingly-obvious thing of just deleting them, the applications that want to use them will just recreate them unless I edit ~/.config/user-dirs.dirs. So, if developers really think they need to 'help' their 'average users' by creating these folders by default, they should finish the job and honor my desire to not use them, which I've expressed by deleting the damn things.
Also, if I want to rename them, the system should honor that. E.g. if I rename "Music" to "music", applications should not then go and re-create a "Music" folder.
Or, you know, they could just simplify my life and theirs by not creating these 'special' folders in the first place.
Damn straight. Where does the system get off trying to tell me how I should organize my stuff? Keep system stuff in a folder for system stuff and the rest of the file hierarchy is mine, because it is my goddamned computer. The original Mac got this right and no one else has since.
Snaps sound good in principle and aren't a horrible solution when they work, but despite the tagline ('Snaps are universal Linux packages') it's not clear that they work well outside of Ubuntu: they aren't sandboxed unless Apparmor is present, and despite their touted 'universality' have a hard (and unnecessary) dependency on systemd.
At this point, Guix and Nix seem the most promising for cross-distro packages.
Do you just have a /snap directory and no $HOME/snap? Normally, snap creates /snap for the binaries (shared among all users) and $HOME/snap for the per-user configuration/data files.
I'm not surprised, I left ubuntu a decade ago because their way of doing things was causing me headaches and they did not seem to care, all that seemed to matter was bug 1 : becoming microsoft in place of microsoft.
I don't understand the whole Snap thing. Not a single snap has ever worked in any of my computers in the past 5 years (I've tried maybe 5 or 6), and they always say they'll add themselves to the system. Add where? Where are these beasts hiding?
I just «snap install pycharm» and it shows up ready for work. Never had an issue with a snap. Can’t recall where it’s stored, but that’s easy to determine with «which». This is on ubuntu 1{6,8}.04 fwiw.
Oh, I was talking about AppImage, not Snap. Sorry.
I actually tried the Snap thing too, I don't remember how it went but I would say it was bad. I think I couldn't find a way to search for packages from the CLI.
If you're speaking of AppImage, what do you mean by "where are they hiding"? AppImages are quite literally exactly where you put them and nowhere else.
Probably that was appimaged, the daemon that handles some convenience things like menu integration and attempting (but usually failing) to show appimage files with icons. appimaged is entirely optional.
>And when people complain, the devs have the gall to suggest that this is all fine.
You're complaining on a really high level and nobody suggested "this is fine". They suggested "wishlist" is fine according to their definition of "if it's not immediately broken and a fix is not trivial"
If you have an issue with default folders that are quite normal in a modern desktop environment I suggest you move back to Arch or wherever you came from and install i3 again?
Also, the bug was first reported in 2016, and Mark Shuttleworth himself has commented on it. And yet it's still not fixed. That gives me the impression that the devs don't consider it super-important. Which is odd to me, since I consider not doing this kind of thing to be 'table stakes' for any application that wants me to take it seriously.
> If you have an issue with default folders that are quite normal in a modern desktop environment I suggest you move back to Arch or wherever you came from and install i3 again?
Ummm, I came from Linux before they started having default Documents, Videos, etc folders. And, like I said, the nice thing is that, in Linux, I can change the XDG environment variables and delete the (IMHO) foolish Documents, Music, Videos, etc folders.
I've only been using Linux for around 10 years as my only desk and laptop OS and this is the first I heard of XDG user dirs! This laptop runs Arch. Thanks for the heads up.
I'll drop "/etc/papersize" in return. If you are not familiar with this and do not print to letter paper then you will find it very, very useful.
It really does make CUPS, Libre Office and all the rest believe you when you say "I want to print on A4, yes A4, no not fucking Letter, A4, end of!" Obviously, substitute your paper size of choice. It probably even lends itself to some form of scripting and perhaps could be changed in reaction to events from say NetworkManager. Whatever.
My life at home is now worth living every time the wife hits print. Such a simple idea - brilliant.
The real problem is that any app I run accesss all files I own with equal permissions. App1 and App2 shouldn’t write their data to my home, or read each other’s data, or, please, be able to write each other’s data.
A video game can read my tax forms. WTF.
Mobile & tablet OSs solve this. Everything has to follow.
Until then, basically every app should run in a docker container. Then it can do what it wants.
Mobile & tablet OSs solve this. Everything has to follow.
No no noNO!!!!
Think of all the things you can't do on a mobile device that you can do on a PC, or which require ridiculous workarounds like uploading from one app to some server on the Internet and then downloading again into another. Effortless data interchange is what makes computing great, and stops proprietary walled-garden data-silos from becoming the norm.
There is a middle ground. Universal data interchange formats but sandboxed on a per directory/app level. Imagine how easy it is to do docker run --volume=~/tax_forms. Put some UX on top of that, plus something resembling Android Intents for escape hatches without giving out full access, and you will have a fantastic solution.
Implementation really really is key when it comes to the UX here though. Windows actually already has this feature (Controlled folder access) but it's done so badly you'd almost think it's a joke. Take this as a warning of how bad things can be if you just sandbox everything without thinking. Enabling the feature accidentally is deceptively simple without any warnings on how difficult it is to use: "Protect files from unauthorized changes by unfriendly apps" is all it says, who wouldn't want that? After that havoc breaks loose. You can only have ONE group of folders protected, you must have Documents protected. Whenever a new program tries to write in a protected folder the program just fails, then 15 seconds later you get a confusing prompt saying excel_32_broker_confusing_name.exe tried to write to protected area, clicking the prompt takes you to a page where you can't simply press yes to allow, you have to manually browse to find the exact program location, good luck finding c:\program files(x86)\office\bin or whatever they decide to call the exe.
Agreed, and we’re now in serious danger of “Okay button fatigue” or whatever you’d like to call it.
If every normal app has to ask for permission to use your disk, you’re just going to click “Okay” when malware asks. Then the whole system was for naught.
Maybe just having some groups, like Firefox's containers.
It's less secure, but a simple setup would drastically reduce the number of programs with access to important information.
So just don't do that. The application needs an "Open" dialog already so you can tell it what files to open, just have that dialog be a process run by the system instead of the application, treat the permission as implicit since the user is the one selecting the file, and pass a handle back to the application.
What can you do wrt file-management on your laptop that you can't on your phone? I have a file manager on my phone that can send files to whatever app needs it; why should a random game have access to arbitrary files?
Exactly! I remember the days of proprietary data formats, when people would literally have to print out data from one program to type it back into another. Let's not regress to that.
You can create another user and sudo / gksudo to it when working on sensitive data. It never occurred to me before reading your comment. I could start doing it myself.
Indeed the capability exists (unix/linux pioneered it?), but it relies on manual containment by the user. It should be automated, with prompts at install or execution time, like on mobile OSs.
An idea is each file should have its own user, basically, with both relative and absolute file creation/modification permissions. If you run an arbitrary executable as a user there's great security risk since it can do anything you can (without sudo of course).
It seems like the natural progression from
Run everything as root ->
Separate root and day-to-day user ->
Don't use root and achieve things with sudo ->
Each file has its own permissions (i.e. essentially its own user) set by the parent user
Permissions should be both for individual actions and permanent classes of actions, on the user's discretion (i.e. allow it to create this file or always allow it to create files)
This approach is even naturally hierarchical, executables could create other executables as long as they have permission to do so, and set the child permissions to at most its own. In this context an executable asking for permissions can be akin to sudo: it really is a file-user asking for its parent file-user for expanded permissions.
To give a real life allegory, consider a Technician in a company wants to make a tool purchase, so he asks the Engineer. The Engineer doesn't have permission, so he denies or asks the Manager. Finally the Manager either denies or asks the CEO which has permission over everything.
There are differences between the apps I use on my phone and the programs I use on my PC.
I don't trust many of the apps on my phone (all of Google, FB, messaging, etc). I trust the open source apps (example K9 email) and I'm replacing proprietary apps with open source ones.
I trust almost all programs I use on my PC, which I install with apt-get from Ubuntu or other open source PPAs (no snap, no flatpack). There are a few proprietary programs I might doubt of, for example NVidia driver, Skype, TeamViewer, Telegram. I understand that one is enough to compromise all the PC.
I'm ok with separating untrusted apps/programs but I don't want to get permissions in the way of programs I trust. Software development would be a nightmare if emacs and vim and all GNU userland and interpreters, compilers, etc would run in different environments.
That's because you have to. I trust all my programs too, because I wouldn't install them if I didn't trust them. We've spent decades telling people only to install applications they trust, and not to click on suspicious executables. The result is that they are very resistant to install new programs, unless the programs are from organizations they already know and trust.
That's partly why web applications are so popular: it requires less trust to use a website than to run a native application. If native applications were less dangerous, people might be willing to use more of them.
Valid point. I thought about this, for system apps you would be able to assign lasting permission requests to read/write/modify anything., by default -- so it would work as right now. Just like in browsers, permissions (permission type prompt) could be remembered in a cache. If you're not comfortable with giving any command full reign on your system, you could simply give granular, temporary permissions as required.
A concept I find useful is that there are two kinds of security:
-- Accidental security
-- Malicious security
Permissions help with both. Not only you don't want 3rd parties to invade or disrupt your system, you don't want users to invade or disrupt your system (or their own systems) accidentally.
A classic example I believe is an user wanting to delete all files in current directory, typing
$ sudo rm ./*
and missing the dot, getting
$ sudo rm /* #(delete *all files*)
Ideally in this case you would see two different requests:
> The command rf requests root access for file deletion at /
> Accept? (y/N)
> User's administrative password:
or
> The command rf requests access for file deletion at /home/user/cache/
> Accept? (Y/n)
Note you're not prompted for password in the second case because you already have permissions for this operations (only the process 'rf' doesn't).
You may not be particularly worried about malicious security for system programs, but even professional administrators could be worried about accidental security and best practices (if they're not they could easily select persistent full permissions). I get the impression the last thing e.g. sysadmins want is mess up production systems, and wouldn't mind a few more prompts when doing things manually (which should be rare?), or assgining scripts the exact permissions needed for their job.
They're painful to set up yourself, but that's pretty much true of all current sandboxing approaches. If you just let your distro do it and follow their conventions it you barely notice.
It seems like MacOS has been headed in this direction, at least with the App Store apps. But it's a hard problem to elegantly implement without breaking everything.
I still see a lot of Mac apps that basically say "disable system protection first."
Yes, this entire discussion is part of a whole other problem: there's too many properties of programs one has to trust esp. on desktop (or alternatively be stuck configuring selinux for days).
One should/must/can not build an operating system on trust alone, as especially the mobile software space consistently continues to show with its numerous malware apps (imagine that on the desktop! "oh sure, have root/admin '50 smileys FOR FREE'", "oh sure, be able to read and write all my user's files <insert dubious clicker app>").
Mobile OSs have become vastly better (not perfect still) than this than desktop OSs.
I would like to see desktop Linux's application management adapt a minimum necessary permissions policy: Allow access to some ressource/API/permission only if the app needs it (and the user allows it of course) and offer mechanisms to allow permission management and ways to enable inter-application communication. This also modularises applications more by bundling up their connections to the rest of the platform and their effects in a central location.
For instance sandboxing applications into their own empty folder would make it simpler to operate on files these applications created (backup, migrate, autocreate <- NixOS would benefit from these especially, allowing finer grained declarative control). It would still be okay then for application developers to follow their own conventions within that sandbox.
Nowadays files are all bunched up in arbitrary folders and nobody knows where they're coming from. It's become so much harder these days to coordinate so many projects that need to create files in the home folder on Linux by trying to introduce standards versus taking that capability away from programs transparently; this should also be much more efficient man-hours-wise.
That said, this is going to be a nontrivial project with how many implicit connections between applications exist. Executing untrusted (or buggy!, I'm looking at you, Steam) software should no longer have to make me afraid my system gets messed up.
For now I think it's best to simply not run untrusted programs and creating regular backups... I wish I had the time/skills to work on something like this.
Just changing the HOME environment variable, which would be what such a hypothetical utility would boil down to, does not address what sakoht was talking about, which is Hurd-style subsetting of a user's account and files.
This is already solved by Linux, as you probably know. You can just use groups and users if you want. Also you can run Docker containers if you want, just don't force me to do it, as they're stupid.
There's a very simple solution to this: Don't run apps that you don't trust.
If you simply must, use a VM or AppArmor or SELinux. Don't inconvenience everybody because you're too lazy to be responsible for your own data.
Besides, untrustworthy apps will bypass the protections anyway. Judging by the popularity of "curl <url> | sudo bash" lately, they'll probably just ask for root directly.
I have some degree of distrust of most of my applications. The "open trust" model usually ends up getting abused to the point where we have to lock it down...like early Windows, wide open for easy of administration, leading to things like the notorious "Blaster" worm, or the open architecture of the Internet, where we now have massive spam and DNS amplification attacks and all sorts of problems based on the notion of a trustworthy network.
In general, everything in computing will keep going in the direction of "trust everything as little as possible for it to do its job" forever, I think, and probably has to.
Running untrusted programs is the default way we do computing. No one has time to audit every single program they run. The current desktop security model comes from when people mostly ran the programs that came with the OS and programs they wrote themself. Its vastly inappropriate for modern computer usage.
One thing I’ve been thinking about is that it would be nice to tag files with the program that created them. Aside from the clutter in my home directory, I have a lot of files that I have to guess as to where they originated from based on metadata like modification timestamps or permissions, but it would be nice to see something like “~/.config was created by some-tool”.
Apple's Macintosh did this in 1984. Files were tagged with 32 bit codes: one for the "creator" (the associated app) and one for the type. These were often made up out of readable characters: a so called "fourcc" (four character code). The "fourcc" persists in cross-platform media formats.
The first software I ever sold online was an app that would change the creator code of a file to whatever application you wanted. For some reason, there was no obvious built-in way to do that in Mac OS 8 (besides opening the document in the other application and saving it, or changing it in ResEdit).
I only ever sold one license, but I was 12, so I was pretty excited.
Why wouldn't it to be easy to implement? You could probably make a global LD_PRELOAD library that hooks into all open() CREAT calls and tags the inode's xattr with the program that created the file. You could use an attribute like, user.created_by.
Why such a hack? This is a solved problem already. Try Linux audit API. You can set it up for tracking various actions, and you'll get a log with all the information including a process name.
I believe the question was asking for a method of storing the metadata, I was just suggesting xattrs. This will work at the expense of running a daemon.
Can audit be configured to only log file creation, not modification or access? Last time I didn't find a way to do that and I don't want to write a log file entry for every file access.
Programs can use programs like 'touch' to create the files so you'd have to get meta into a chain of process ownership or something while exempting the user's shell.
I think touch is a generic example, but not a practical one. The idea is that program A can launch program B, and you'd have to recursively search for the parent. But it's not easy to find which parent is the true owner, maybe an exception can be made for init but shells can't have that exception as they are sometimes the owner.
MacOS does something sorta like this. It uses extended attributes to mark where a file came from in some cases, such as when an executable is downloaded from a browser, it will mark it as "unsafe" and when you attempt to run it, this causes it to fail to run and you have to whitelist it via security preferences.
Adding an interface to access the tag store won't be very hard.
It would be harder to make it so that all the tag-oblivious programs, like mv or vi, to say nothing of rsync, would preserve the attributes when moving or modifying a file.
f = open("the_file.new")
write(f, new_contents);
close(f);
rename("the_file", "the_file~");
rename("the_file.new", "the_file");
So the old file is never modified, it's renamed to the backup copy, and an entirely new file is created to take its place.
This has a number of advantages, but does not play well with any extended info the old file used to have, unless they are copied explicitly. And in a tagging-oblivious program, they won't be.
f = open( "the_file.new.$$" );
write( f, new_contents );
close( f );
rename( "the_file.new.$$", "the_file.new" )
If not implemented like this, then something that attempts to read "the_file.new" may get partial contents, normally truncated along the file system block size.
Backups often take place, too, like you mentioned, depending on editor.
If the value matches, you renamed the right file. If it does not match, you renamed a wrong file. Without holding the fd, it is impossible to know if it is the right file.
Thanks for teaching me this! I wasn't aware. So, the original xattr (and the program that created it) would be deleted and replaced by vi for example. Still technically right, but I'm unsure how you'd keep a history.
Please no. This article is literally "Dotfile madness", don't make it worse!
I have to deal already with these on file shares, specifically for Apple: .DS_Store, .Trashes and .AppleDouble, or for Windows: Thumbs.db, $RECYCLE.BIN (for some reason Windows sometimes ignores the fact I've disabled the recycle bin on a share and creates this instead) and desktop.ini. Please don't drop crap around directories where there exist a multitude of tidier alternatives.
Because when you have multiple OSs accessing the same shared folder, they all create their own crap, which is then visible to the other OSs, and fills up directories with stuff that confuses normal users.
I don't run multiple OSes, but I do use USB sticks and SD cards on other people's computers and vice versa. I'm pretty sure that is "normal user" behaviour.
It gets annoying real quickly because you see the "crap" files of all the OSes you're not using. And you delete them, you only have to insert the stick and they're back. This happens in different ways for all of Mac, Windows, Android and certain Linuxes.
Sidecar files for storing file tags make querying the tag system a huge chore; it totally kills performance. I understand the conceptual appeal but it's just not the way to go.
There's an entire research field dedicated to the information flow control which could solve just that (if it were actually used).
In my research team, we used a tainting tracing mechanism to understand the behavior of malware. Basically, we installed a malware on a clean phone and we then traced all information flow originating from the APK to processes, to files, to sockets, etc. It helped reverse-engineering the malware.
I'd love something like this on program level. Given contents of a variable, I'd like to know where they came from - which pieces of code contributed to the result. I'd also love to be able to mark a piece of data, and see what code touched it or derived new data from it. A programming equivalent of injecting radioisotopes into the body.
I’d suggest looking into taint analysis tools, though those are usually aimed more at finding things like unsanitized input ending up in a call to system.
BeFileSystem (Haiku)
HFS+ and APFS
Even NTFS has attributes.
The problem is file manager. Tracker/Finder are several steps ahead of Explorer. Under Linux, its a greater mess even though there are no technical limitations.
What's not quite so straight forward is how you query the system. The worst implementations of file tagging only implement retrieving a list of all files that have a given tag. Slightly better than this are tagging systems that will return the intersection between two or more tags. Most tag systems never go beyond this.
Going slightly further, tag exclusions are powerful and are sometimes implemented (given a set of files from some subquery, exclude all files that have a given tag.) What you rarely see are systems that allow you to exclude one subquery from another.
However what you almost never see implemented is a system for preferential tags; given a subquery, reorder the results such that files with preferred tags are raised to the top, ordered by how many of the preferred tags they have. Once you implement this, the system's UX changes dramatically because the user no longer has to make strong assumptions about how well their files have been tagged. Many files might be missing relevant tags and the user may not be sure if the file they're after is one of these incompletely tagged files. When using a system with preferential tag querying, the user will receive files in their result list that don't match all the tags listed, but most of them. This is a bigger deal than it may sound, since the main drawback of using tags for file management is incomplete tagging. By addressing this drawback, you stand to bring file tagging to the next conceptual level, which is rendering hierarchical file management obsolete.
Consider that hierarchical file management is a strict subset of file tagging. You can model file hierarchies inside a file tagging system. To demonstrate this by example, consider /home/joesixpack/Documents/seventh-novel.pdf For each level of the hierarchy, we can create a new tag, such that this document has the tags: '/home/', '/home/joesixpack/', '/home/joesixpack/Documents/'. But because we're using file tagging, we can also automatically tag that file with things like 'pdf' or maybe even 'Documents'.
Now before I go on, there is something to be said about the number of tags in the system exploding when you model a tree as a tagset. In practice this probably isn't the approach any real system should take, if only because most of those tags will be useless and because there is a great deal of redundancy in trees that wouldn't exist in a native filetagging system. Consider /home/joesixpack/Documents/ and /home/johnsmith/Documents/. We have two different Documents directories because they exist in different parts of the hierarchy. However in a file tagging system we'd ideally only have a single Documents tag and one tag per user, such that querying the intersection between 'joesixpack' and 'Documents' returns the files that would otherwise be in /home/joesixpack/Documents. Some tags, such as the user tag, could be implicit; such that if joesixpack simply queries 'Documents' the tag 'joesixpack' is automatically intersected with the results presented to him.
With that out of the way, consider how we can go further if we apply straight forward statistics to the tag system. Suppose 99% of joesixpack's 'Documents' that are 'pdf' are also 'novel'. When joesixpack creates a new file that's in the tags 'Documents' and 'pdf', what are the chances that file should also be tagged with 'novel'? We could analyze the contents of the file if the system has a semantic understanding of what 'novel' means, but let's not go there. Tag systems created by the likes of Facebook and Google do this sort of analysis, but it's heavy, tricky, goes wrong in ways that create PR disasters, etc. We can get pretty good results by ignoring the contents of the file and looking instead at merely what that file is already tagged with. If a file is tagged with 'joesixpack', 'Documents', and 'pdf', it _probably_ should be tagged with 'novel' as well. Probably, but not necessarily. So the system can expose to joesixpack the suggestion that he tag that particular file with 'novel'. By presenting suggestions like that to the user, you greatly reduce the UX friction needed to tag files well and therefore increase the usability of the tagging system, while at the same time improving the quality of the tag suggestions in the future. What we create here is a 'virtuous cycle' of sorts that gives back to the user more as it's used more. Such a tag suggester can be implemented as multi-label classification using naive bayes; it's fairly straightforward.
Going further; if we have such a tag suggester, and an appropriate caching scheme, tag suggestions can be used by queries. If joesixpack queries for 'pdf','novel', the system could return to him the intersection of 'pdf','novel', but also return to him files that are tagged with 'pdf' but not tagged with 'novel', in cases where the tag suggester indicates there is a high likelihood that the file should be tagged with 'novel'.
That last paragraph may or may not play out well, I haven't experimented with it extensively yet. But getting back to my original point: if you're going to implement file tagging in a filesystem, you should think carefully about how the users will query that system, and whether the query system will be extensible in userspace to facilitate new powerful methods of querying it. It would be a total tragedy if the system only supported basic tag intersections and the only way to extend it was to implement a new kernel module.
This would require a ton of difference from how modern desktop and mobile OSes handle files / documents. Maybe some time later, some research OS would implement it.
I've implemented the above in userspace for my own experimentation (not yet ready for public release, but maybe soon.) Nearly everything I've described above can be implemented with relatively straight forward SQL queries and SQLites performance has been everything I could have asked from it (in the neighborhood of ten thousand tags and hundreds of thousands of files.)
Getting file tagging into the kernel level to replace directory hierarchies would be a huge paradigm shift, a very dramatic departure from Unix. To be honest I'm not sure whether or not getting such a system into a kernel would be appropriate or not. Traditional hierarchical file management seems more than sufficient for "system files". But I'm really interested in replacing hierarchical file management from the users' perspective. More or less, put ~/Documents, ~/Downloads, ~/Desktop, etc under control of the tagging system but leave the rest of the system as-is. At least for the proof of concept.
I think a demonstration system could be implemented as a custom desktop environment running on regular old linux, where the open and save dialogs of GUI applications have been replaced with tagging/querying windows. Instead of the user clicking through a few directories to find a file to open or find a directory to save something, they would instead click or type in tags to add or query. The GUI file manager would likewise be replaced with the GUI frontend to the tagging system.
The story I heard was that performance sucked. Granted it could have sucked for other reasons, as other failed efforts like writing more OS components in C# were happening at the same time.
If we start sandboxing applications and give each one its own UID, like Android does, we get this feature for free. It's kind of silly to associate exactly one security context with exactly one user.
I think saagarjha's point was that an even simpler standard would be for the creating application to use file metadata to annotate the files. And a simpler standard is generally more likely to be followed, especially if it's unilateral.
This doesn't help with installation, obviously, but it helps with cleanup.
not sure if this is the exact solution to your problem, but rcm (https://github.com/thoughtbot/rcm) allows you to create dedicated dotfiles in a .dotfiles directory which are being symlinked back to files, and it has a -t option to create and restore dotfiles with tags. mkrc -t emacs .emacs.d would create a .dotfiles/tag-emacs/ folder and rcup -t emacs restores it.
>One thing I’ve been thinking about is that it would be nice to tag files with the program that created them.
Isn't what you describe basically the file extension? Not so much what program created the file, so much as what program is meant to handle/open the file.
File extensions are more nomenclature than file type indicator. More than one file type might share the same extension, more than one application might be used to open the file according to its extension. They mostly work, but file extensions != file types != file creators.
File extensions are trivial to fake, don’t really have a strong association with the application that is supposed to handle them, and in this case often doesn’t even apply because most of the files in my home directory seem to lack an extension.
The lack of useful names on the files is IMHO the most egregious issue. (OK using names that don’t start with ‘.’ is worse). I don’t ls -A my homedir as much, but what the hell are .517827.padl? Who created .android (I don’t use android and don’t develop for it) etc.
These people are arrogant shitheads and I’d like to know who they are so I can avoid their software. I assume if they do crap like this their software does other nefarious things.
> These people are arrogant shitheads and I’d like to know who they are so I can avoid their software.
Wow, this is really assuming the worst in people. It’s much more likely to be ignorance because the other developer doesn’t know where the file came from either (some compiler flag on a library accidentally enabled), or also doesn’t ls -a in their home.
> Wow, this is really assuming the worst in people.
Yes, I am. I'm fed up with programs that spam my system without asking: install plug ins, menu items, directories on the desktop, dock items etc for "convenience" -- it's convenient for them, not me. It's especially painful when I pay for an app and it sprays shrapnel through my filesystem.
> It’s much more likely to be ignorance
Indeed, but if they can't be bothered to learn basic system hygiene how can I be confident the rest of their package is safe to use?
A big driver for sandbox/containers in user systems is not protection against malign actors per se but against the lazy and ignorant.
Have I written bugs into code? Of course I have, I'm human. But at least I try to get things right up front; to be a good citizen.
(there are additional reasons for containers in server systems, hence my qualification of "user systems" above).
A "compiler flag on a library" that leads to garbage in your home directory => bad library => developer's failure to vet and research their dependencies.
Perhaps having all parts of this chain called "shithead" is extreme but they would all have responsibility and disrespect for the user.
A lot of other people have made some fantastic points here, so I won't harp on "putting random crap in ~/ bad!" any further.
However, since it has not been mentioned, I would like to shout out to the Python "appdirs" module[1], which neatly solves this issue for those writing programs in Python with minimal effort. The best part is that it uses the platform's native standards, so XDG on Linux, plus whatever macOS and Windows call their implementations.
Sadly I am not aware of comparable options for other programming languages. I think having high-quality easy to use libraries for this sort of thing would go a long way.
Many of these are named Windows-centric because of the history of the API. But where possible, they still map to something reasonable elsewhere. Looking at the mapping table, I can't think of anything it doesn't cover.
> The best part is that it uses the platform's native standards, so XDG on Linux, plus whatever macOS and Windows call their implementations.
Boo. Apps not using XDG on macOS drive me mad anyway; 1000 times more so when they do on Linux, but use Library or whatever stupid place on macOS despite the presence of XDG environment variables.
Related pet peeve: macOs '.DS_Store' files. Can someone better-informed than me please explain how to prevent heir creation? Barring that, I'd settle for a justification.
They store attributes used by the Finder for the they're within [0]. You can set a Mac to prevent their creation on mounted network volumes [1] but to my knowledge there isn't an option to prevent their creation on local volumes. For Samba shares, you can configure it to prevent their creation.
They're a bug/feature from the early days of OS X. Apple should have done the work of storing directory metadata in something like ~/Library/Preferences/com.apple.finder.plist but they were too busy getting anything at all to work.
This is the type of low hanging fruit that Apple evidently doesn't care about anymore, but would go really far towards improving user sanity. It's like when you go to a restaurant and order the cheapest thing on the menu as a way to gauge the overall quality of the food there.
> This is the type of low hanging fruit that Apple evidently doesn't care about anymore
This is an unfair comment, showing you don't know _why_ those files exist. They're an ugly tradeoff, but they serve a purpose.
Dot files and .DS_Store files are meant to replicate the movement of metadata that Classic Mac OS achieved with creator codes and whatnot, as well as the layout settings for the folder. Those files need to be in the same directory as the files/folder because Finder knows to copy those files as well.
Finder saves the layout options and settings for each individual folder, harkening back to the spatial Finder of Classic Mac OS. It's an integral feature of the system enabled by default. It would be an absolutely terrible idea for Apple to put the preferences for every single folder to which the user has navigated in one preference file, especially given that it would need to be updated whenever files/folders undergo any CRUD operation.
> but would go really far towards improving user sanity
99.9% of the time, users don't notice this at all. Those files are currently everywhere on a macOS system, but nobody sees them. The only time they really come up is when you're using ``ls -A`` or copying files to Windows; apart from then, they're truly invisible.
What Apple should do is provide an option to not copy the dotfiles and .DS_Store files to USB sticks, especially if they're FAT32 or ExFAT format. That's a better solution than one huge preference file of shame.
Leaving it next to the files means if directories are moved, copied, or renamed, the metadata is still there. So you or I might prefer it to live in a separate file under ~/ but I cannot say littering the whole tree with the metadata buys nothing or can't serve a purpose.
Because macOS doesn't know when files and directories are moved on other devices, network drives, etc.
Nor can it preserve that metadata across other devices, network drives, etc. without copying it across somehow; the easiest solution was to store that data in hidden files and let Finder handle their automatic transfer.
Failure of your imagination isn't failure on someone else's part.
Hmmm this is the first good argument I've heard for the existence of .DS_Store files. You get a point :-)
I'd be curious to test this, by zipping folders, copying them to other machines, etc. My gut feeling is that Apple probably missed something and it doesn't quite preserve the metadata like it should (most likely due to version incompatibilities or duplication of state somewhere else in the OS that overrides it). I'd be pleasantly surprised if it did work though.
It brings up another point however, that probably metadata is as important as data (and sometimes more important, if you look at how we organize data today). The earliest filesystems probably should have implemented multidimensional storage somehow, similar to the metadata in the BeOS filesystem. I'm not familiar with how it's implemented, and it probably also breaks when files are copied to another filesystem, unfortunately:
.DS_Store contains the positions of file icons, the last view mode (list, icons, etc.), and other stuff for Finder. You can avoid creating it by not using Finder, and you can probably also avoid it if you carefully avoid not dragging icons around. If it bothers you because you keep committing it into your Git repo, then you can exclude it in a global .gitignore[0].
You can use https://github.com/JK3Y/asepsis to have them all redirected to one folder. So not have them in every folder. That’s forked from the project originally being by TotalFinder’s creator as far as I know.
A couple of caveats: one, it seems to inject into other processes, which is not something you want to do with System Integrity Protection enabled, and two you need to run touch ~/.no-asepsis-os-restriction to make this work :)
This kind of just sounds like more standard crazy unix/linux "someone did the wrong thing and now everyone's doing the wrong thing and it's too late to realistically fix", for which it's easier to just come up with another subversion (waves at the three different 'user' solutions) like not just building a user home dir, but creating a dir inside that the user's usename as well, so that ~ is for all the bullshit dotfiles and ~/yourusername is actually your data. And then someone will no doubt come up with an alias that takes you to that dir instead of just the original homedir, like "cd `" instead of "cd ~" (and no, ~ doesn't make any more sense than ` would, it's just what we got used to)
This is a symptom of a bigger problem, which is that programmer respect for resources that belong to the user is at an all-time low.
* CPU: We'll just have this web page peg the user's CPU running gobs of code because our designer wants to unnecessarily animate this thinggy or our ad network wants to mine bitcoins. (Related resource: Mobile phone battery)
* Memory: We'll just use this web framework and embedded browser that uses up 1GB of the user's RAM, for a one-window chat application we're writing because we can't be bothered to learn anything but Javascript.
* Disk/Storage: We'll just cache 20GB of easily-regeneratable internal data so the app seems faster than it is, dumping it somewhere for the user to find and clean up.
* Network Bandwidth: Our shopping cart web page load is larger than the full installation of a 2000-era video game because we have to have 6 layers of framework abstraction and taking the time to optimize/minimize it wouldn't let us release fast and "crush it" in the market!
Nobody seems to care. "Oh, Xyz is cheap, programmer time is expensive!" is the rallying cry. So let's push the cost our shitty practices to the user and make her buy more hardware.
Oddly I feel completely the opposite about this topic. I think dot files in your home directory are exactly the way to go. Using the XDG standard leaves you with whatever the distro decides is the correct location for these files. While their recommendation is all in the home directory, I wouldn't put it past some of these distro developers to make some crazy /var/users/{uid}/config directory. This is important when setting up your partitions. I keep home and the system separate so that I can easily swap out the OS. Home should be the only location a user has write access.
> Using the XDG standard leaves you with whatever the distro decides is the correct location for these files.
This is not true. Even if the distribution sets global xdg variables (none that I've come across do) - the user can always override them.
> While their recommendation is all in the home directory, I wouldn't put it past some of these distro developers to make some crazy /var/users/{uid}/config directory
This sounds like a strawman. Are you aware of any existing distros doing this? Users don't have write access to system directories in the vast majority of distributions.
So rather than use a standard that allows you to override it simply by changing some environment variables, you think that hardcoding a specific path is a better solution?
I think dot files in home directory for configuration files are fine, especially if the user creates the files themself rather than the program created them (although sometimes a program may wish to create a user cache file, for example Free Hero Mesh creates a file ~/.heromeshsession by default but you can configure a different name if you wish; also some programs such as web browsers may store cookies and so on, although you should be allowed to reconfigure that, too). A program might also support a command-line switch and/or environment variable to override what file (in what directory) to use.
Requiring files in your home directory that do not have a dot at first (with a fixed name) is no good, though.
To those of you reading this: I beg you. Avoid creating files or directories of any kind in your user's $HOME directory in order to store your configuration or data.
My immediate response was pretty much "where the hell else should they put them then?"
What exactly else is the /home dir even for otherwise?
The purpose of home is for user-level persistent files. Giving it access to something else means giving random programs write access to some global part of the file tree, and I'd rather not allow that except when actually necessary. As it is I hate having stuff that manual installs in /opt, or the way too many programs on Ubuntu are configured to require sudo to work.
I'm actually totally on board with more stuff that installs entirely local only, especially if it's from outside the distro's package system. I don't like stuff mangling the global tree with no clear standardized path for removal and uninstallation. At least if it's in /home, I can just delete it myself without even needing to go sudo.
And the purpose of .dotfiles in /home is because it prevents those user configuration files and other such local data from cluttering your directory when doing a plain ls for regular user files.
This article just sounds like a misunderstanding of how the file tree is even supposed to work.
> My immediate response was pretty much "where the hell else should they put them then?"
He didn't mean to avoid $HOME as a whole. Just the root of the $HOME directory.
I too much prefer having programs write their config files to $HOME/.config/program-name instead of $HOME because it's less clutter and now I can think "ok, $HOME/.config is where all of my app's configuration is", instead of having a bunch of files littered in $HOME while other apps use $HOME/.config/file_name or $HOME/.app_name.
It bothers me that .pypirc exists in $HOME, rubygems credentials are in $HOME/.gem/credentials and Docker's credentials are in $HOME/.docker/config.json. That's 3 different location styles for an app's config file. They should all be in $HOME/.config/$app_name/relevant_file_name for whatever the config is for.
the author explicitly refer to the locations ~/.local/share and ~/.config in accordande to the XDG standard. That the request is not referring to the entire $HOME subtree is quite obvious.
I have long deplored the sheer amount of config files that clutter the top level of any project nowadays. .gitignore, package.json, webpack.config.js, babelrc, travis.yml, it’s endless. This very short list is really the tip of the iceberg. These things need to be in a dedicated config folder.
It’s so nice to see this article at the top of HN. I had long thought I was the only one who felt this way.
It's often worse on Windows, since dotfiles aren't hidden and meny dev tools assume they'll be running on *nix even when they support Windows. https://imgur.com/a/Im6G20B capture from my windows box of my $HOME.
It's just ~/Documents, which I symlink to ~/Desktop so that if any asshole developer dumps a file into it I'll be able to see it and remove it immediatelty.
Also on macOS, where [~]/Library/ApplicationSupport is the designed and recommended place to store config files, logs, preferences and all kinds of persistent data, the home directory still gets cluttered.
It's a laziness issue, it won't be solved solely by making is easier to access the right locations on each system/platform, but that would be a step in the right direction.
At this point the right solution is to just not allow applications to choose. Give each their own view of the file hierarchy but keep all the crap they spew out in a location obviously related to them from the user's view of the file hierarchy.
While this cleans up the file hierarchy, it absolutely destroys any interoperability between programs. If a program cannot output files that other programs can see, then it cannot output files that other programs can use as input. Between having a messier file hierarchy on the one hand, and not being able to open a Microsoft Word document outside of Microsoft Word, the former is by far the lesser evil.
For GUI apps, this is trivially solved by requiring the standard Open File/Folder dialog to be used. The system can then control the presentation, and once the user has picked a file, provide the app with a security token that encodes that permission (or even just an open file descriptor). So apps can see each other's data, but only when user explicitly directs them towards it. This is more or less how it works in Win10 UWP apps.
Using the filesystem for primitive “interoperability between programs” is increasingly recognized as a serious security problem. As someone else in this thread pointed out, a video game can access their tax returns.
I would honestly like to blacklist my home folder. Do you want to create a new file/folder? I am notified with a popup and if you ask to often I am gonna stop using that. Not really workable but one can still dream...
That's how sandboxed apps (=app store apps) on macOS work. They can only access files and directories that the user has explicitly selected (through the system file dialogs or drag and drop)
This is actually possible (on Linux, at least, probably Mac/Windows too) with something like selinux. It ends up being a huge headache though, both because selinux is kinda hard to configure, and because you will get a constant barrage of requests.
This was a bigger problem in the earlier days of OS X (I even created a "Actually My Documents" folder inside the Documents folder to work around it), but now I'm down to two offenders - Adobe and DVDFab.
Windows also has an app data folder that contains a whole mishmash of stuff. And of course you have the registry.
Part of the problem in Windows is that Program Files is write protected for security, so you can't store a local configuration file relative to the exe. So where does it go?
I assume there must be documentation from Microsoft and Apple about best practices for this... I hope?
To answer your last question: in Qt you can use QSettings and it'll write to a canonical platform-specific location. Useful for storing eg the last accessed folder in a GUI, or things which users don't reallty need to change regularly.
> Part of the problem in Windows is that Program Files is write protected for security, so you can't store a local configuration file relative to the exe. So where does it go?
This hasn't been a problem since the first version of Windows that had a filesystem with such protections - that would be Windows NT 3.1.
The correct location for such stuff is %USERPROFILE%\AppData\Roaming or ...\Local, depending on whether those settings are machine-specific, or should be synced across all devices for that user accounts (in environments where this is appropriate, such as domain networks). This particular location is in use since Vista - before, the names were slightly different ("Application Data"), but in any case a well-behaved app is supposed to use Win32 API to retrieve the correct path, rather than hard-coding it - which also worked on Win9x.
> I assume there must be documentation from Microsoft and Apple about best practices for this... I hope?
Hence why I suggested that Qt has the right approach. It also has http://doc.qt.io/qt-5/qstandardpaths.html which abstracts out standard paths for cross-platform applications.
> The fact it happens so often suggests that the documentation isn't good enough.
Any library or framework author knows that documentation is never enough. ~ On Windows, though, there's also historical reasons - back in Win9x days, there was no practical reason for apps to bother if they didn't anticipate running on NT, which is why it was so common to just dump configs next to the .exe (and even to this day, Windows uses FS virtualization to make this work for old apps). When 2K and especially XP became popular, and everyone was rapidly porting their stuff, people often didn't bother looking up docs and all, and just used the "least significant change" approach to fixing things. And that kind of set the background for later - why bother being nice, if nobody else does?
> This page describes pre-Vista, deprecated APIs. Unless you need to target XP, a desktop app should use the known folder mechanism.
Actually the current documentation refers you to the CSIDL list if you want to know what any of the GUIDs actually mean. The new docs just provide a list of IDs with minimal information.
Respectful Windows apps should call SHGetKnownFolderPath() with FOLDERID_LocalAppData or FOLDERID_RoamingAppData.
Hardcoding paths is a bad idea, since they can change - indeed, they have already changed once in Vista, and the paths that you suggest are obsolete on Vista and above - it's %UserProfile%\AppData now (although it still provides symlinks for compatibility).
It seems to me that distros are in the right position to nudge this in the right direction. For example, Fedora did the /usr merge, moved to systemd, wayland, etc. The great dotfile cleanup would probably work by patching all packaged software to do config files the right way, plus sending the patches upstream. Then in a few release cycles this would be more or less solved.
The XDG directories are by default under your home dir only, so you can keep the same backupability properties. Better ones, even, since the files of things supporting it are in one place.
Slight tangent, but I wish there was a standardized way to tell running processes to re-read their dotfiles.
If I change my .zshrc, I have to re-source it in each running instance of zsh. Sure, I could automate this. But I'd also like the same done in vim. And in weechat. And in mpv. And in everything using GNU Readline... you get the picture.
This is probably at the top of my list "biggest complaints that won't be resolved for numerous reasons." It sure would be nice though.
I don't get the problem. Why bother where dotfiles are written to? GUI file managers and userland tools hide them by default. What would I gain by having them written outside of $HOME?
Easier to version your configs if they're all in one place. Easier to find a config file, less guess-work. Generally helps keep the home directory tidy of cruft. Not a huge deal, obviously, but programs should be well-behaved programs.
The whole point of .cache is that it's supposed to be that one folder that you know you shouldn't back up. As opposed to a dozen different app-specific ones that we have today.
Yes, it's a great idea in principle -- and would be even better if its location were configurable, per the XDG spec.
My comment wasn't meant as a criticism of the basic idea of a standard directory scheme; it was a reply to someone asking why anyone would ever care about files you can't see.
Point of order: Some of us really want ~/.local backed up or otherwise persisted; I've got the equivalent of a second /usr in there, and I only don't worry about "backing it up" because ~/.local/etc is in version control, and the rest is installed by my setup scripts.
My partner has to manage having a 500MB quota on their home directory at work. A couple of times a month something decides to dump 300+ MB of data in there and break everything.
Apps use dotfiles or dotdirs for caching data. Think browser caches. That's the point of the standard... All the non essential caching data lives in one place and so is easy to prune when needed.
Preferences files tend to be pretty small; I'd be rather surprised even if a 128MB thumb drive couldn't store them all. The bulk of the space will almost certainly be consumed by other things.
You cannot and should never assume that dotfiles are just preference files. They might be directories, and those directories might be chock full of big files — or worse, temp files that should really be in /tmp.
I think it makes a lot of sense to have all apps run in a sandbox or container by default. Everything is segregated and easy to monitor and track what its doing.
On my fedora ec2 created with a standard ami, docker doesn't work without sudo (didn't try to reconfigure, so it may if I hack), so that behavior is understandable, though not desirable.
I definitely have looked at my home directory in awe at the number of dot files, but I think the article is a bit hyperbolic. I get that it's "clutter" but I think the simplicity of storing configuration in $HOME might be worth that trade-off.
Even the $XDG_ env variables still require the user to set that up, or the OS maintainer needs to set it up. If the OS maintainer sets it up, the user probably won't know where to find configuration.
The reason $HOME is so commonly used is because the developer knows with almost 100% certainty that the unprivileged user will 1) be able to read/write to it without superuser privileges and 2) it keeps all configuration data contained in the user's home directory for portability, instead of throwing it all over the file system in locations nobody will know.
That portability will come in handy when you're backing up or restoring from backups, which I just did yesterday. It means restoring is as simple as copying your entire home directory back. I don't need to tell each app where my configuration is, it just knows it will be in $HOME.
And that assumption will be correct for almost any *NIX-based operating system.
I do agree that non-dotfiles should not be created in $HOME, that is a nuisance.
.files are very useful. I like the consistent idiom for where your "global" settings go for virtualenv, npm, tslint, flake8, git, whatever.
But it would have been so nice just to move it into a ~/cfgs or fine... even a ~/.cfgs or whatever.
I'm guessing engineers went, "ewww, I'd have to walk down the tree to find the next most relevant config file to use and then go UP into a directory when I hit home."
I tend to carry $HOME around with me when I change systems. When I ls -Alr it was like opening up a time capsule, or a hope chest. The first 4 pages on a 23 line terminal had file and directory names for content old enough to buy liquor in all 50 states, old friends like .cshrc, .addressbook, .X11Startup, .nofinger, amd.home, .zirconrc, dot.signature, aub.home, and so on.
I just keep my home folder in /etc/username and use this construct:
(just an example, not actual)
environment.etc = {
"qemu/bridge.conf".text = ''allow br0'';
"username/.inputrc".text = ''
\$include /etc/inputrc
"\e[A": history-search-backward
"\e[B": history-search-forward
set show-all-if-ambiguous on
set completion-ignore-case on
'';
};
to write configs into /etc proper and my home folder.
Some of us would like to manage our systems, and processes like backups, transfer to a new system, etc., in something resembling an orderly process.
The very concept that architects, designers, &/or programmers think it's OK to just litter our systems with their stuff wherever happens to be convenient to them to code up this afternoon is no better than people just deciding to crap wherever they feel. When I see software doing this, I automatically downgrade them in my assessment of what I want to use going forward.
The simple concept is that there are a few categories of storage used -- these should each be kept separate & independent, under the USER's control.
Why is this a difficult concept?
Program code, system-config, per-user-config, user data. A simple table of the default and user-overridden location for each.
If you as a coder have a problem with this, I really have to wonder what other complexity you will handle badly.
If this simple system is a mess, why would I expect other parts of your system to be any good?
I agree. If I hit ‘Ctrl-H’ my lovely, clean Nautilus File Manager becomes a heinous, ugly clutter which is impossible to navigate. I’ve tried to delete some or move others to no avail. Please follow this standard. It’s clean, organized, and beautiful. Once done, we can then tackle the next problem (I’m looking at you & Snaps Shuttleworth!).
I have been fighting this but I gave up: people just don't care or are somewhat opposed, even when presented with a backword compatible patch.
Now I just mount the home directory with rewritefs and try make the mess more or less compliant to the spec. It's not ideal: it's a hack and it causes a performance hit but I can be at peace in my ~.
I think this is a silly complaint. These are preferences storage. The home directory is absolutely the most logical place for these.
I actually like that there are applications that actually still have preferences. My biggest pet peeves are applications WITHOUT preferences. Careful what you wish for.
I agree that the home directory is the most logical place for files like these, but it would still be nice if they were in a consistent place. That way, if you wanted to delete your preferences for a specific application, you know exactly where to go without searching around. Personally, I think this should be enforced by the operating system instead of leaving it up to the developers.
In a long battle to be more intentional about what I keep around, I wanted ~/ to function both as my desktop and downloads directory, on both macOS and nixos. Using it like this would force me to deal with cruft quickly, and avoid crap silently piling up out of sight until its origin is forgotten (though desktops will hide the .dotfiles; those could still accumulate unnoticed).
It was easy to set this up on nixos, but I found the handful of apps dropping non-hidden files there grating. Magic folder behavior on macOS didn't work as well with this.
I ended up replacing ~/Desktop and ~/Documents with links to ~/Downloads. The naming isn't sensible, but I'm happy with how this works on both systems so far.
All of this is a leftover from the days when *nix was a multiuser OS and used as such. It completely ignores the way we use machines now, with only one owner, and expectations of only one user. In fact, I can’t imagine loaning my work laptop to anyone, for security reasons, nor my personal laptop, because that’s just gross AND a security violation.
Computing power is so cheap now, that we don’t need to share machines when working in the cloud, either. We can, and do, log into our own VMs, work on data and software and commit back to an intermediary before final merge into a shared repository.
Perhaps it’s time to return to some version of single user OSes and clean up the relics left behind by no-loner-needed multiuser environments.
You're never going to be able to stop developers dumping their shit in ~/. The best you can do is create your own directory like ~/Data to put your stuff in. Since programs don't already know about it, it will stay clean.
> Avoid creating files or directories of any kind in your user's $HOME directory in order to store your configuration or data. This practice is bizarre at best
I believe the article itself is an existence proof that it is in fact, not bizarre.
Crazy idea: what if we were to just nest a personal directory within ~?
i.e., New standard directory `~/personal`, new environment variable, $PERSONAL.
If we could come up with a default `~` like character for globbing to `$PERSONAL` like `~` does to `$HOME`, say, `@` (obviously that isn't the char to go with and I'm not a BASH expert to be able to pick one not already occupied). All your personal dotfiles are in `~`, all your personal files are in `@`. Default `cd` without args to `$PERSONAL`. No worries about tons of files that are in `~` because you live in `@`.
Same mess on Windows with the My Documents folder. I simply abandoned it, treating the folder as "My Spam From App Programmers", and keep the meaningful stuff in a totally different place.
I have the opposite perspective. Dotfiles are fine, and I find the proliferation of XDG directories confusing, mostly because they're misused. Why should a game's save files be under .config? Making dotfiles under $HOME is a simple approach that's worked for a long time and that doesn't cause serious problems, and I see no need to adopt the XDG approach, which I see as overengineered.
> I find the proliferation of XDG directories confusing, mostly because they're misused. Why should a game's save files be under .config?
A game's save should be $XDG_DATA_HOME, misuse is exactly that, misuse. I have hundreds of games installed on my linux machine and not a single one has put it's save files into my $XDG_CONFIG_HOME folder. On the other hand many DO outright ignore XDG and dump all their garbage into .game_name/.dev_name folders (looking at you Factorio). I think you are exaggerating the issue.
> Making dotfiles under $HOME is a simple approach that's worked for a long time and that doesn't cause serious problems, and I see no need to adopt the XDG approach, which I see as overengineered.
What exactly is over engineered here? It's just a set of environmental variables, developers ALREADY use $HOME to spam up my home directory. The only real effort a developer needs to put into this is making sure they separate their config, data and cache folders which is something most already do within their cute .program_name folders.
while maybe not "serious" I certainly do find it do be quite the problem that all user-specific program data is randomly dumped into poorly named hidden folders within my home directory. *Nix systems at a root level already divide config files into /etc, cached files into /var and /tmp, and data files into /usr. Why can't there be at least a little symmetry with how the home folder is handled?
The XDG approach is over-engineered: it adds complexity and provides no benefit. What is the specific concrete advantage of putting some parts of foo's state in ~/.config/foo, some in ~/.local, and so on? All this approach does is scatter what's conceptually a single state blob across different parts of the filesystem. I dealt with this stuff in Windows for years, and we don't need it in the Unix world.
The conventional dotfile approach is just fine: there's an application and it puts state under a directory named after itself.
Warning, its Windows support is all theorycraft so far (Travis CI + Windows is totally broken for Node projects right now). PRs welcome if anyone wants to give it a try.
I'll take dotfile madness over registry madness. I have been trying to figure out where in the registry Visual Studio 2015 puts its install location so I can remove that registry value when I reinstall it to put it in it's default location. So far I've wasted a ton of time trying to accomplish this. A single file .VS2015 would be preferable.
There is a program for windows that does this if I recall correctly. It records every file and registry entry created during installation and allows you to compare snapshots of "before" and "after" installation.
I personally detest the whole XDG thing but lots of non-XDG applications are guilty of littering the home directory. For example, every time I use the Less pager, it creates a .lesshst file (fixable by setting "LESSHISTFILE" to "-"). Vim creates .viminfo files so I had to create an alias to it where vim equals "vim -i NONE".
I prefer to use `ls -lah` which always shows the dot files. It sure would be nice if these were placed in a different folder. Maybe ~/.../ to put all the things. Sure, cat ~/.bash_history would need to be cat ~/.../.bash_history which isn't as convenient.
But, I sure do agree with the frustration. My work machine has nearly a hundred hidden things. My Chrome usually downloads things to ~/Downloads/ but Firefox often likes to store the file in ~ which then is often hard to find when I jump over to a console.
I've made an effort to clean things up. Trying to set firefox to download to the Downloads folder, deleting everything personal out of the home folder. But, of course, then that just goes to prove that I do not have control of my ~ folder. I'm doing everything I can to keep my own stuff out of it. Which is sad.
I realised that keeping a clean user home directory was futile long ago... this is not even unique to linux or *nix.
Just don't put anything you care about in home directories: in the modern world people own computers not user accounts, you likely have ultimate control over your system, therefore designate your own "home" directory, don't give the other "developers" the choice to polute it by making it non-standard.
I like to be able to nuke my user-home directory whenever I like, I achieve this without pain by additionally placing my hand picked application configurations that I care about into a read only XDG config directory.
That's why I never use $HOME for personal files. All of my files are in the Dropbox folder or another custom folder on my work PC. It's nonstandard, so no one litters in it besides me.
Yeah, great, rather than just going to the .foo directory for the foo program to find user specific state for that program I have to first figure out where the particular distribution likes to put things by default and then hunt through something like 5 different hierarchies for the foo directory.
The "old" way forced program developers to put things in a very specific place. The new way allows then to interpret the XDG standard however they want and scatter everything all over the place.
node_modules and yarn.lock etc. in the location where the tool was run is a feature, not a bug. Ask any python user digging to find where pip decided to drop it this time.
I was going to say, I've been actively working with nodejs for a few years and I've never had node_modules, package-lock.json and yarn.lock in my home dir, on either Linux or macOS. If you run npm/yarn commands in your home dir for some reasons, then you should expect those files there. On the less than handful of times where I ran npm in my home dir accidentally, I make sure to immediately go and delete those files.
It's also possible that the author runs a lot of scripts from various projects from github and sometimes those scripts' authors are not careful about running npm/yarn within their projects. I guess I don't play around with OSS enough. :)
I think most of that can be chalked up to someone simply mistakenly running those commands in the wrong directory (root of $HOME). I've never heard of a JS app just randomly storing a global node_modules or yarn.lock to the root of $HOME.
On Windows I've never put anything in C:/, yet I have a shitload of install.res.xxxx.dll files, shitload of eula.xxxx.txt files, globdata.ini, vcredist.bmp, install.ini, VC_RED.msi. In my home folder I've .QtWebEngineProcess folder (never installed qt), .mongorc.js (which is empty), .v8flags.xxxx.json, .dbshell, and so on and so on. I'm not delving into insanity which is /Program Data, Windows/Temp, or [user]/AppData
Gonna say I expect to see the dot files and don't really care.
I think to be upset over that is just looking for a fight.
It works across all of my OS's, I know its config specific to the app, and if I want to jump in and get some info it is just a ~/ away...the overhead is so low I've spent more time thinking about it in this statement vs all the time ever wrestling through them or battling tribal knowledge.
I guess I just don't see a strong enough argument here.
And to Linux developers who've made the jump to macOS development or are making a cross-platform port, please use NSFileManager to find the paths for this or that, don't hardcode everything into Library/Application Support. Not only that, but a lot of stuff ported from Linux could stand to put things in .app bundles rather than _every bloody where_.
TIL I'm using ~ in the wrong way. For me ~ was always the default place for user-level configurations and time taught me to do so: .X* .bash* .vimrc and others.
Than I create my personal folders ~/{src, docs, bin, porn, anime etc}. ~ is for temporary files I'll delete after use like windows users do with their Desktop folder.
Also my ls isn't an alias with -a flag...I'm alone here? :|
I think an issue I have with it is if I want to move the configuration to another place, the only way I can do it is via a symlink. If applications used the variables above, you would be able to easily organize things differently if you choose to.
This would also allow you to test if something there is causing issues or if you wanted to format and reset things. Just delete that directory or set the variable to another path.
I don’t set ls to ls -a, but sometimes I create hidden folders and files on ~/ so it’s not distracting. Scanning through a lot of things because everything is dumped there is a mess.
Not sure what the problem is -- dotfiles or other application files need to be somewhere, whether in the home directory or somewhere else (that the OS doesn't enforce but just requires by convention) makes no difference. What would really help is what another comment mentioned -- the ability to tell which modules owns a particular file -- that would be really cool.
One of the advantages, though, is per-user config for the same program, e.g. the .bash files. It's pretty common to have separate /home and / partitions too, and less common but doable to dual-boot different distros but share the same /home partition. Which is itself actually a reason to be very careful with back-compatability of dotfiles
Homebrew just started offering Linux support but they would put stuff under /home/linuxbrew/ but wtf is this about? Won't be using until this is fixed.
I haven't used Linuxbrew, but Homebrew is kind of finicky if you try to install anything to a non-standard folder. I remember a while back they used to say "you can change the default folder" but then later said that this was not recommended.
Yep, I tried to run mine in ~/.homebrew but gave up. I can't remember why, but stuff was breaking and it was clear Homebrew didn't want you to do it.
I've spent some of my career building packages from source and I know the assumptions you get stuck with because the app developer hard-codes certain things. So it makes sense they just go for what is most standard and expected.
The title seems misleading. The problem here isn't dotfiles. The problem is that developers are using $HOME instead of a standard directory for storing user-level config files. I had never even heard of $XDG_DATA_HOME before reading this article.
Why should non-desktop applications follow the XDG standard? Even for applications designed for "desktop" use, the standard is questionable IMO. After all, the "Desktop" folder that OP is complaining about is an XDG recommendation.
Why do we even need hidden files? Why not just put all the configuration files/directories under a single directory specifically dedicated for this and use file system attributes for all the directory-local metadata?
Wow I though this was going to be about graphing dot files and I was like what could this madness possibly be about. But hidden dot files, eg “.fileXYX” is spot on in the problems of their proliferation.
So, what's gonna happen on Wayland when these X-variables aren't defined? Am I going to lose config and have to copy directories from XDG to .config/ because that's new default?
By "X-variables" you mean $XDG_CONFIG_DIRS, $XDG_DATA_DIRS etc.? They aren't related to the X server. The "XDG" is for the old name of the Freedesktop.org project (X Desktop Group) [1]. Freedesktop.org is the host for the XDG Base Directory Specification [2]. Apart from that they are also involved in Wayland. So I see no reason for worry.
> It is beyond me why my home directory ended up up containing a node_modules directory, package-lock.json, a yarn.lock file (I have never even consciously used yarn!),
macOS has Application Support, Caches, and Preferences folders under /System/Library, /Library, and ~/Library. Windows has %ProgramData%, %AppData%, and %LocalAppData%.
This smacks of somebody looking for a problem. If it's not a problem when you don't look, and you have to go out of your way to look, how is it a problem?
Anything where your own way of seeing it creates the problem, and you can't describe the problem any other way, you're not ready to pitch it as a problem to other people. Find some harm independent of your perception, and then you've got a reason for people to listen.
They've explained why it's a problem. Having an alternative viewpoint on "why programs are randomly polluting my home folder, a sensitive location with my personal data, with unwanted files without leaving any trace of their origin" isn't going to help.
I developed a utility for my own use and found an answer to your question.
When the program starts up, it needs to know what its configuration is. This is not stored in the code. So it has to find its configuration data somewhere. Ideally, you'd let the user configure where they want the config file. But then the program has to read the config file to find out where the config file is, and it doesn't know where to look.
(So it seems like you pretty much have to put config data in the same place on every computer, or use environment variables.)
If you don't know about the XDG Specification in the article (I didn't), a $HOME/.myconfig file is by far the simplest and most robust solution.
In order to have per user settings you have to have one dir per user. For practical purposes and security it ought to be somewhere only user has access it. Optionally it would be nice if you could back up all user settings and files in one place.
The logical conclusion is that it ought to live somewhere in home which meets the above criteria. Its obviously cleaner to store data in $XDG_* but keep in mind that some of the apps creating said files are actually older than linux and certainly older than such standards.
From dev and user perspective it would be nice to have them updated to follow the standard but in practice the practical effect is nil so lots of stuff doesn't bother.
That's an intellectually lazy thought. If everyone had that attitude we wouldn't have modern society at all (grow your food, don't expect someone else to grow it for you).
Nobody is forcing you to interact with those "new users", if you don't like them, ignore them.
Frankly, if everyone had your attitude, your Linux community will be dead and gone in a few short years as time progresses, and your old users pass away.
I've been a Linux user for a long time, I and many other long time Linux users do not share this persons opinions. New Linux users, keep coming, the vast majority of us are not elitist.
Except isn't there a specific directory for such things now in $XDG_CONFIG_HOME? Or is this not standard at all? Either way, $HOME/.config makes perfect sense to me!
It’s a LAZY place to store user data that works across platforms.
FTFY.
If you don’t want to follow the spec, keep your program’s data in your program’s directory. If another program needs the data it can follow the $PATH, or ask the user.
Even better, ask the user at first run. You know, when asking people to accept some Byzantine EULA, license, and permission to track their grandmother.
This is the consequence of enterprise practices of hiring el-cheapo, hit-and-run contractors or outsourcing development to middlemen in “cost-efficient” regions.
End developer is inexperienced, temporary, not valued and hence his deliverables
The reason I like it is it provides directories to
use across platforms (edit: Not sure about Windows), relative to the user. Since the
applications download a file (UNIHAN database [1]) to bootstrap,
we want to store it in the user's files, not the systems.
The other thing I do that's helpful is using sqlite (which is
often built with Python).
This combination gives a database, and a place to store
configuration, downloaded/built files, in an organized
way across systems.
The nice things XDG "standardizes" which I'm using:
$XDG_DATA_HOME defines the base directory relative to which user specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.
$XDG_CONFIG_HOME defines the base directory relative to which user specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.
$XDG_CACHE_HOME defines the base directory relative to which user specific non-essential data files should be stored. If $XDG_CACHE_HOME is either not set or empty, a default equal to $HOME/.cache should be used.
Why the implementation is helpful:
It defines a common directory for cache/data/config files, with applications inside them, e.g. ~/.config/i3, ~/.config/tmuxp, rather than adding more stuff to ~/.i3, ~/.tmuxp. When standards aren't being followed, programs end up storing their configurations / data / etc all over.
Another thing is it defines directories relative to the user themselves and for the system files. Dotfiles may imply configuration / artifacts in the users home directory, it doesn't have. The thing I like is I can check for configuration in user files (for instance ~/.gtk-3.0/), then fall back on checking the system files (/etc/gtk-3.0).
The benefit of user files is they're writable out of the box, so if there's going to be any file generated that the user needs to keep (like an unzipped version of UNIHAN) that can be cached so they don't have to download it again.
".local": It's in my home directory, but local to it, and not somewhere else? This doesn't make a lot of sense, but er, Okay.
"shared": For Christ's sake, what is this? It's in my home, and local within it, but also shared? With whom? How? Why? I give up.
Why would anyone want to do this? What concrete benefit does this path give us? What specific thing does it let you do that ~/.myprogram doesn't? The article is bad, and the XDG standard is bad. The traditional approach is fine.
Some things XDG is trying to do that's not readily apparent:
~/.local:
It's intended to be /usr/local in the user's home directory. There's a bit of a backstory here though: Linux distros don't follow `/usr/local` as well as BSD's, not much may be there.
~/.local/share:
Same idea, for /usr/local/share. But on many linux distros, this is `/usr/share`. XDG is trying to do is mirror the system directories in the user's filespace.
Let's assume there's going to be a lot of applications: they won't agree on where cache, config, and data storage is kept at within ~/.myprogram. With XDG, if someone wants to clean out all their cache files? rm -rf ~/.cache.
The other thing is: Typically the user won't be looking at ~/.local/share/myprogram - it's for data the application itself is going to use. It doesn't really count against it. What I think we will be looking at though: configuration files. That's where I think XDG is kind of original - but it's really close to what /etc is.
As for ~/.myprogram: I'd be okay with that, if it were purely configuration. But XDG serves the article writer's purpose fine: Keeping $HOME clean. I kind of think splitting between ~/.config, ~/.local/share, and ~/.cache ends up feeling more consistent with how linux/bsd organizes files.
To name and shame some culprits (the one's who have shame, I won't name java because I won't expect them to care)
1) Mozilla
2) Gnome
3) Gimp
4) GPG
5) Thunderbird (Why is there a sep directory and not under .mozilla)
6) mplayer
I guess one reason why this occurs is that it's easier for to be carelessly cross-platform this way. Except you'd expect better from mozilla, specially because they store Windows stuff very well inside %localappdata%
Why not _shame_ openssh? Or vim? Or emacs? I would advise not calling for shame on any of these projects because they have a pattern to support that has been in place longer than the XDG standard has been in place. You can change gnupg's home dir, but it would break a lot of currently functional setups if it just moved.
I'm personally willing to give SSH a partial pass because I feel like having it separate makes it easier to reason about from a security perspective (this would also apply to GPG).
For vim... yeah, AFAIK "vim" does that, but neovim uses ~/.config/nvim :)
And to your general point... yes, many of these programs predate XDG. I happily support keeping them backwards compatible (which should be easy; checking multiple places in order is a time-honored tradition). But I honestly don't think this was a good idea even decades ago, and "we've always done it this way" is a terrible reason to not even support a newer, cleaner approach. Again, there's no reason to break existing systems; just support an additional location, probably make it the default, and everyone will be happy.
I guess I was being nice to old programs. Also, these are programs that a lot of external tools interact with. Nobody cares about where firefox stores its data- except the now almost dead flash plugin.
Yes, very good point I do think that the older programs/projects have to plan a migration which really can be challenging. I think there are some that might care for example selenium might have the mozilla directory as a default and some might hard code it. The orchestration of such a change is difficult because I don't think too many people read changelog or really keep up with upstream modifications.
> You can change gnupg's home dir, but it would break a lot of currently functional setups if it just moved.
You can just change the default: if .gnupg exists, write everything to it. If not, write to $XDG_CONFIG_HOME/gnupg/ and $XDG_DATA_HOME/gnupg/ as appropriate (deciding how to carry out the split is not obvious, but can be done).
I agree, though, that calling it "shaming" is neither justified, nor useful.
> (deciding how to carry out the split is not obvious, but can be done).
For this reason I like better the windows distinction, that (if I understand in correctly) distinguishes between machine-dependent files and machine independent files. for something like gnupg I would not really say what is data and what is configuration
Speaking of Windows, their situation is worse. I remember installing a couple of games and suddenly my "My Documents" folder had "EA", "EA Games", "Electronic arts", "Rockstar games" folders.
Games do that by design so that they can be cleanly uninstalled without removing saved games and preferences. Also you should be able to backup your documents folder and not your programs folder and be safe. A good uninstaller should ask if you want to save them and remove them if you don’t.
I am not saying it is the best way. But I don’t think it is malicious or lazy.
^ This. And the amazing thing is it's probably the least common place for games to put save data. I think I've played exactly one game that ever used it.
It’s probably because Windows makes the actual home folder somewhat difficult for the average non-technical user to see. They effectively made the My Documents folder the home folder for most of their userbase.
Which is especially sad as supporting the standard described in my post is a non breaking change if you are still going to read the old location of the configuration and data files as well.
Files in the Linux ecosystem are a mess. Files scattered everywhere and each instance of Linux uses some different arbitrary directory scheme. We really need a reboot. New micro kernel based OS with clear directory logic and dedicated directories for applications to keep stuff.
Files everywhere are a mess. Windows, Linux, MacOS, Solaris, HP-UX, VMS, CP/CMS. Every corporate network share I’ve ever seen has been a dumpster fire.
There are actually consultants who advise organisations on cleaning up their file systems.
> decided to hijack the primary location designed as a storage for my personal files.
that's the basic misunderstanding, the user folder is for everything that's not system wide on a multiuser system; it's not there for your convenience, it's there to allow programs a writable space that doesn't conflict with everyone's else.
instead of subdirectoring everything else pretending programs to follow some weird never agreed upon specification, just add a "document" folder and put your user generated content there, problem solved.
Edit: apparently even on "hacker news" the mentality of forcing everyone else into obeying instead of solving minor inconveniences the only logical way has taken hold. Not even on widows with their nice buckets and environment variables could solve this, resulting in an exposion of games, my games, three app datas, documents/pictures/videos with apps doing whatever (i.e. shadow play putting screenshots under videos) but no, Rando comes over recycling three environmental variables that are specific of a program and everyone sucks up to it ignoring all the technical, social and other issues to the approach, while shitting upon the only sensible action that can and will work.
From a user standpoint, a lot of of unrecognizable things show up if you pass that -l flag.
In my opinion, having a conventional place to look for user configuration in plain text is a good thing, not a bad thing, and of course I would want it to be in the directory on my system where I'm the sole owner. As somebody else mentioned, this means backing up my personal settings and files is braindead simple, too.
Asking programmers to respect $XDG_HOME_CONFIG is not a bad idea, but it's no replacement for home directory hygiene. I believe that ultimately, cleanliness is a personal responsibility.
Edit: reduced smugness quotient while trying to preserve my argument. I apologize for the pre-edit tone issues with this post.
That's my point exactly. Most of these programs have command line flags you can pass to indicate the configuration file that should be used. Doing that, and then by default putting my configuration in the most reasonable place for it--a dotfile-- is, in my opinion, the most respectful way a programmer can treat someone's machine.
Not to mention that most of these are free and open source, so building according to convention is the best way to keep their userbase happy (as fiddlerwoaroof pointed out, $HOME was the original conventional dotfile location pre-2003.)
So if your program doesn't respect my machine (meaning allow me to configure it to prevent unwanted behavior) then yes absolutely, into the bin with it. And rm -rf its dotfiles after.
you could check if ~/.local/share and ~/.config exist. also there is a difference between files that the user is supposed to interact with and inner configuration of random apps. ~/.ssh is fine, I need to look at it for many .emacs.d is fine as is something you are expected to edit manually, but I don't even know what some of these are:
.aspell.en.prepl .dmrc .dropbox .dropbox-dist .gtkrc-2.0 .joe_state .oracle_jre_usage .pdfsam .pylint.d snap .texlive2016 .wget-hsts .vscode .xdvirc
This is from a fresh install 3 months ago, do I need 2 dot-folders for dropbox?
I’ve been using Linux long enough that things moving to the freedesktop specification disrupts my expectations. Plus, it’s not obvious that having to traverse the extra directory is worth it: it’s nice to be able to do vim ~/.prog<tab> and save a couple keystrokes when editing commonly edited config files.
Plus, the backwards compatibility issues of changing standardized pathnames has a pretty high cost in terms of updating existing shell scripts and such.
But nowadays, lots of applications seem to think it's OK to put normal files in my home folder without asking first. This is just outright barbarism. In particular, Ubuntu now seems like it's all-in on snaps, and if you use snaps, they create a non-hidden "snap" folder in the user's home folder. Madness! And when people complain, the devs have the gall to suggest that this is all fine:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1575053
I like the idea of snap/flatpak in the abstract, but until they get their act together, and stop creating non-hidden files in my home folder without asking first, I am never going to use snaps.
And of course, all this is on top of the new fashion for the OS to "helpfully" create Documents, Music, Video, etc folders in my home folder, and set the XDG environment variables to point to them. Noooooope.
But at least in that case the user can change the environment variables and delete the stupid folders. No such luck with ~/snap.