From gsr.b3d@infernal-iceberg.com Thu Feb 19 23:22:42 2009 Date: Thu, 19 Feb 2009 23:22:42 +0100 From: GSR To: Ton Roosendaal Subject: Blender config layout (was Re: [17594] branches/blender2.5/blender/source /blender: Patch to allow pre-2.50 Blenders to read newer files.) Message-ID: <20090219222242.GA22087@crossbow.battleship> References: <20081127160100.C654F398C5@proog.blender.org> <20081127181811.GA5199@crossbow.battleship> <74489a82473ef3a606facb445f9962ce@blender.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <74489a82473ef3a606facb445f9962ce@blender.org> Status: RO Content-Length: 7916 Lines: 215 Hi, ton@blender.org (2008-11-28 at 1158.58 +0100): > Hi, > > Send me notes, we can come back to that when install procedure is on > todo. :) Well, it is more about locations where to read and save things than just install. One of the things is to keep "unpack is install". And as such, it would link more into the part in charge of looking for disk data (wherever that goes now with the 2.50 rewrite), that installing anything. It took more time than planed as the original clean up became a search of info about other OSes and then added another topic, file handling (kicked off by the reports about security issues). The Blender Files part just needs checks and some fixes, the File Handling is a rough sketch of what and why it should be done. ##################### ### BLENDER FILES ### ##################### The following is mostly in UNIX terms and taking into account the the advantages of keeping things packed thus allowing direct "unpack and run" (Freedesktop spec XDG Base Directory Specification spreads different things over multiple places which do not map Blender nicely, B.blend is config and data, eg). Non UNIX OSes require some review, but should easily map into the concepts. Any file not covered is probably an oversight, thinking it was really old cruft. There're 2 basic places: D1. user config (for an exact version). D2. global for machine/install (exact version again). Each base place contains version subdirectories, named simply 250, 251, etc. User config also has a subdirectory named "defaults". Loading config (or any other data) comes from the first place where it's found so newest user version overrides older ones and user files always override global files. No lookup of versioned files in global case, only for user files. Saving goes to the proper subdir in D1 as per executable version, always by save to temporary name, delete old, rename. Never direct overwrite, old data better than useless data. See the other section after this one, then the coder does not have to care about it, but just use the Blender calls instead of pure fopen, etc. Symlinks should be respected, so advanced users can share file-history, eg. FIXME: Carefully try to use realpath() to figure what file has to be really removed before renaming? Or just remove the symlink? Hardlinks are a non issue (the rename step breaks the link to the old data, while keeping it), but symlinks are. No (leading) dots for any file (or subdir) (avoids fake extension issues) and no files should directly go into ~/ ever again (probably no files in ~/.blender/ either, just subdirectories). - D1 family of values: UNIX -> ~/.blender/VERSION/ (~/.blender/253/ eg) OSX -> ~/Library/Application Support/Blender/VERSION/ MSWindows -> Documents and Settings\USERNAME\Application Data\Blender Fundation\Blender\Configuration\VERSION\ or Users\USERNAME\AppData\Roaming\Blender Fundation\Blender\Configuration\VERSION\ (varies with OS version) Users can create the special version "defaults" (~/.blender/defaults/ eg) if they want, that will be used if no numerical VERSION matches, before resorting to D2. - D2 values (compile time defines): Any "unpack and run" package -> UNPACKDIR/data/ UNIX -> /usr/share/blender/VERSION/ [*1] OSX -> /Library/Application Support/Blender/VERSION/ (FIXME/CHECK) MSWindows -> (FIXME) There is no "defaults" subdir. *1: Distributions are free to use it as base and split things via symlinks (for example to allow plugins of different archs, using /usr/lib/blender/ etc). # Tree structure: For simplicity the same for all (D1 & D2). icons/ plugins/ plugins/sequence/ plugins/texture/ bpython/ bpython/data/ (old bpydata... still used? maybe more correctly named tmp/?) bpython/modules/ bpython/registry/ (old bpydata/config/) bpython/scripts/ Example of complete path: ~/.blender/258/plugins/texture/ # Important files: Leading dots aren't used as they aren't needed anymore (and solves fake extension issues). If nothing specified, they appear in the base dir (~/.blender/250/B.blend eg). B.blend directories (old .Bfs) file-history (old .Blog) font (old .Bfont... is this needed anymore?) bpython/menus (old Bpymenus) last-session (dir name where last session quit.blend and other temp files could be located) FIXME: Blanguages, final name and where (still needed?). # Changing Blender versions: On start up, a check for current version dir D1 (~/.blender/250/ eg) must be performed, and if missing, created. Reads are performed to the newest (number, not date) version avaliable, and saves always to the current version subdirectory. This way data migration will be done from latest avaliable version when there is a need for saving anything, without wasting disk space. Users can clean old configs, or keep the parts they want in case they want to run multiple versions in parallel. They should be informed in documentation about the detail that if they delete 251/B.blend instead of moving it to 252/B.blend when their version is 2.52, they lose the config. ################################## ### FILE HANDLING (IN GENERAL) ### ################################## An API to open, close, read and write files. Currently some places use raw fopen and related calls, with obvious issues, like security and data loss. Wrapping fopen, read, write and other calls with some own calls allows encapsulating all the differences and what is more, provide extra functions freeing the coder of the responsability of doing them. The calls will follow POSIX ones, with extra parameters to declare the type of intended usage and compression. Requisite for the system is that on every start up, Blender will obtain a temporary directory (see mkdtemp) and store the path in ~/.blender/VERSION/last-session. Example /tmp/blender-session.GmdSAA. # Special handling of r, r+, w, etc: Pure read operations (mode r) are done on the original file. Coder must be aware that in some cases that means he gets the globally installed data. If he wants to write at a later point, he has to close and open again to get a new file descriptor. The rest are done in a new file created with a temporary name (see mkstemp), with empty size (w, w+) or by copying the original data (r+, a, a+). Until the file writing is not finished, original data still exists on disk. On the close call, the file is moved/renamed to the right name (the one requested in the open call) by the API implementation. # Extra parameters: The usage param will cover the following usages: - Normal: filename is handled to expand // if needed, but special handling is still performed. Example, "open for append (mode a) //foo.png" will copy //foo.png to //foo.png.suftsX and set the stream at end of this new file. Closing will rename //foo.png.suftsX to //foo.png. - Configuration: relative file name is mapped to the right place as per previous section. Coder does not provide any version info (nor will ever know), the API takes care of looking up the data from the avaliable D1s or D2. Example, "open for read (mode r) B.blend" when user is running 2.56 will look up for the newest B.blend possible by looking in D1/256/B.blend .. D1/250/B.blend (D1/defaults/B.blend) and finally D2/256/B.blend. - Temporary: relative filename is mapped to the temp directory for this work session. Example "open for write (mode w+) bake.0001.gz" will create /tmp/blender-session.GmdSAA/bake.0001.gz.QsipkhaX and on close will rename to file to get rid of the extra random extension .QsipkhaX. - Raw: bypass all extra measures except // expansion. To be used under coder responsability and with a good reason. The compression parameter: - None: wrap fopen, read, write... - Gzip: wrap gzopen, gzread, gzwrite... mode parameter still has to expecify the desired compression options with a digit and extra letters (see gzopen). ########### ### END ### ########### GSR