Index: ps/trunk/source/lib/status.h =================================================================== --- ps/trunk/source/lib/status.h (revision 9960) +++ ps/trunk/source/lib/status.h (revision 9961) @@ -1,450 +1,451 @@ /* Copyright (c) 2011 Wildfire Games * * Permission is hereby granted, free of charge, to any person obtaining * a copy of this software and associated documentation files (the * "Software"), to deal in the Software without restriction, including * without limitation the rights to use, copy, modify, merge, publish, * distribute, sublicense, and/or sell copies of the Software, and to * permit persons to whom the Software is furnished to do so, subject to * the following conditions: * * The above copyright notice and this permission notice shall be included * in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ /* * error handling system: defines status codes, translates them to/from * other schemes (e.g. errno), associates them with descriptive text, * simplifies propagating errors / checking if functions failed. */ /** Error handling system Why Error Codes? ---------------- To convey information about what failed, the alternatives are unique integral codes and direct pointers to descriptive text. Both occupy the same amount of space, but codes are easier to internationalize. Method of Propagating Errors ---------------------------- When a low-level function has failed, this must be conveyed to the higher-level application logic across several functions on the call stack. There are two alternatives: 1) check at each call site whether a function failed; if so, return to the caller. 2) throw an exception. We will discuss the advantages and disadvantages of exceptions, which are the opposites of call site checking. - performance: they shouldn't be used in time-critical code. - predictability: exceptions can come up almost anywhere, so it is hard to say what execution path will be taken. - interoperability: not compatible with other languages. + readability: cleans up code by separating application logic and error handling. however, this is also a disadvantage because it may be difficult to see at a glance if a piece of code does error checking at all. + visibility: errors are more likely to be seen than relying on callers to check return codes; less reliant on discipline. Both have their place. Our recommendation is to throw error code exceptions when checking call sites and propagating errors becomes tedious. However, inter-module boundaries should always return error codes for interoperability with other languages. Simplifying Call-Site Checking ------------------------------ As mentioned above, this approach requires discipline. We provide "enforcer" macros to simplify this task by propagating errors to the calling function. Consider the following example: Status status = doWork(); if(status != INFO::OK) return status; This can be replaced by: RETURN_STATUS_IF_ERR(doWork()); This provides a visible sign that the code handles errors but reduces clutter. When to warn the user? ---------------------- When a function fails, there are 2 places we can raise a warning: as soon as the error condition is known, or higher on the call stack. We prefer the former because it is easier to ensure that all possible return paths have been covered: search for all "return ERR::*" or "return StatusFrom*" that are not followed by a "// NOWARN" comment. The latter approach also risks multiple warnings along the call stack for the same error. Note the special case of "validator" functions that e.g. verify the state of an object: we now discuss pros/cons of just returning errors without warning, and having their callers take care of that. + they typically have many return paths (-> increased code size) - this is balanced by validators that have many call sites. - we want all return statements wrapped for consistency and easily checking if any were forgotten - adding // NOWARN to each validator return statement would be tedious. - there is no advantage to checking at the call site; the call stack indicates which caller of the validator failed anyway. Validator functions should therefore also use WARN_RETURN. Numbering Scheme ---------------- Each module header defines its own error codes to avoid a full rebuild whenever a new code is added. Error codes start at -100000 (warnings are positive, but the corresponding negative value should not be used to avoid confusion). This scheme avoids collisions with all other known error codes. Each header gets 100 possible values; the tens value may be used to denote groups within that header. The subsystem is denoted by the ten-thousands digit: 0 general 1 file 2 res (resource management) 3 sysdep (system-dependent) 4 win (Windows-specific) To summarize: +/-1SHHCC (S=subsystem, HH=header, CC=code number) 10 general 00CC misc 03CC path 04CC debug 05CC debug_stl 06CC secure_crt 07CC wchar 11 file 01CC vfs 03CC file 04CC archive 12 res + 00CC h_mgr 01CC tex 02CC ogl_shader 13 sysdep 00CC cpu 01CC os_cpu 14 win 00CC whrt **/ #ifndef INCLUDED_STATUS #define INCLUDED_STATUS #include "lib/lib_api.h" // an integral type allows defining error codes in separate headers, // but is not as type-safe as an enum. use Lint's 'strong type' checking // to catch errors such as Status Func() { return 1; }. // this must be i64 because some functions may multiplex Status with // file offsets/sizes in their return value. typedef i64 Status; // associates a status code with a description [and errno_equivalent]. struct StatusDefinition // POD { Status status; // typically a string literal; must remain valid until end of program. const wchar_t* description; // omit initializer (or initialize to 0) if there is no errno equivalent. int errno_equivalent; }; // retrieving description and errno_equivalent requires going through all // StatusDefinition instances. we avoid dynamic memory allocation (which // is problematic because status codes may be needed before _cinit) by // organizing them into a linked list, with nodes residing in static storage. // since modules may introduce many status codes, they are stored in an // array, aka "bucket", which includes a link to the next bucket. // initialized via STATUS_ADD_DEFINITIONS; opaque. struct StatusDefinitionBucket // POD { const StatusDefinition* definitions; size_t numDefinitions; StatusDefinitionBucket* next; }; /** * (called via STATUS_ADD_DEFINITIONS) * * @param bucket is being added; its definitions and numDefinitions must * already be initialized. * @return previous bucket in list, suitable for initializing bucket->next. * * (this function must be callable as a static initializer; initializing * next avoids the need for a separate dummy variable) **/ LIB_API StatusDefinitionBucket* StatusAddDefinitions(StatusDefinitionBucket* bucket); /** * add a module's array of StatusDefinition to the list. * typically invoked at file scope. * @param definitions name (identifier) of the array **/ #define STATUS_ADD_DEFINITIONS(definitions) static StatusDefinitionBucket definitions##_bucket = { definitions, ARRAY_SIZE(definitions), StatusAddDefinitions(&definitions##_bucket) } /** * generate textual description of a Status. * * @param buf destination buffer (allows generating strings with * the code's numerical value if no definition is found) * @param max_chars size of buffer [characters] * @return buf (allows using this function in expressions) **/ LIB_API wchar_t* StatusDescription(Status status, wchar_t* buf, size_t max_chars); /** * @return the errno equivalent of a Status. * * used in wposix - underlying functions return Status but must be * translated to errno at e.g. the mmap interface level. higher-level code * that calls mmap will in turn convert back to Status. **/ extern int ErrnoFromStatus(Status status); /** * @return Status equivalent of errno, or ERR::FAIL if there's no equivalent. * * NB: reset errno to 0 before calling POSIX functions to avoid confusion * with previous errors. **/ extern Status StatusFromErrno(); // note: other conversion routines (e.g. to/from Win32) are implemented in // the corresponding modules to keep this header portable. //----------------------------------------------------------------------------- // propagation macros // warn and return a status. use when an error is first detected to // begin propagating it to callers. #define WARN_RETURN(status)\ do\ {\ DEBUG_WARN_ERR(status);\ return status;\ }\ while(0) // warn if expression is negative, i.e. an error. // (this macro is more convenient than ENSURE) #define WARN_IF_ERR(expression)\ do\ {\ const Status status_ = (expression);\ if(status_ < 0)\ DEBUG_WARN_ERR(status_);\ }\ while(0) // return expression if it is negative, i.e. pass on errors to // the caller. use when failures are common/expected. #define RETURN_STATUS_IF_ERR(expression)\ do\ {\ const Status status_ = (expression);\ if(status_ < 0)\ return status_;\ }\ while(0) // warn and return expression if it is negative. // use if a function doesn't raise warnings when it returns errors. #define WARN_RETURN_STATUS_IF_ERR(expression)\ do\ {\ const Status status_ = (expression);\ if(status_ < 0)\ {\ DEBUG_WARN_ERR(status_);\ return status_;\ }\ }\ while(0) // warn and throw expression if it is negative. use to propagate // errors from within constructors. #define THROW_STATUS_IF_ERR(expression)\ do\ {\ const Status status_ = (expression);\ if(status_ < 0)\ {\ DEBUG_WARN_ERR(status_);\ throw status_;\ }\ }\ while(0) // if expression (typically the invocation of a callback) evaluates to: // - INFO::OK, do nothing; // - INFO::ALL_COMPLETE, return INFO::OK; // - anything else, return that. #define RETURN_STATUS_FROM_CALLBACK(expression)\ do\ {\ const Status status_ = (expression);\ if(status_ == INFO::ALL_COMPLETE)\ return INFO::OK;\ else if(status_ != INFO::OK)\ return status_;\ }\ while(0) // return 0 if expression is negative. use in functions that return pointers. #define RETURN_0_IF_ERR(expression)\ do\ {\ const Status status_ = (expression);\ if(status_ < 0)\ return 0;\ }\ while(0) // warn if expression is false, i.e. zero. #define WARN_IF_FALSE(expression)\ do\ {\ if(!(expression))\ debug_warn(L"FYI: WARN_IF_FALSE reports that a function failed. Feel free to ignore or suppress this warning.");\ }\ while(0) // warn and return 0 if expression is false, i.e. zero. #define WARN_RETURN_0_IF_FALSE(expression)\ do\ {\ if(!(expression))\ {\ debug_warn(L"FYI: WARN_RETURN_0_IF_FALSE reports that a function failed. Feel free to ignore or suppress this warning.");\ return 0;\ }\ }\ while(0) //----------------------------------------------------------------------------- // shared status code definitions namespace INFO { const Status OK = 0; // note: these values are > 100 to allow multiplexing them with // coroutine return values, which return completion percentage. // notify caller that nothing was done. const Status SKIPPED = +100001; // function is incapable of doing the requested task with the given inputs. // this implies SKIPPED, but also conveys a bit more information. const Status CANNOT_HANDLE = +100002; // function is meant to be called repeatedly, and now indicates that // all jobs are complete. const Status ALL_COMPLETE = +100003; } // namespace INFO namespace ERR { const Status FAIL = -1; // unknown failure // general const Status LOGIC = -100010; const Status EXCEPTION = -100011; const Status TIMED_OUT = -100012; const Status REENTERED = -100013; const Status CORRUPTED = -100014; const Status ABORTED = -100015; // invalid values (usually function arguments) const Status INVALID_ALIGNMENT = -100020; const Status INVALID_OFFSET = -100021; const Status INVALID_HANDLE = -100022; const Status INVALID_POINTER = -100023; const Status INVALID_SIZE = -100024; const Status INVALID_FLAG = -100025; const Status INVALID_PARAM = -100026; const Status INVALID_VERSION = -100027; // system limitations const Status AGAIN = -100030; const Status LIMIT = -100031; const Status NOT_SUPPORTED = -100032; const Status NO_MEM = -100033; // these are for cases where we just want a distinct value to display and // a symbolic name + string would be overkill (e.g. the various // test cases in a validate() call). they are shared between multiple // functions; when something fails, the stack trace will show in which // one it was => these errors are unambiguous. // there are 3 tiers - 1..9 are used in most functions, 11..19 are // used in a function that calls another validator and 21..29 are // for for functions that call 2 other validators (this avoids // ambiguity as to which error actually happened where) const Status _1 = -100101; const Status _2 = -100102; const Status _3 = -100103; const Status _4 = -100104; const Status _5 = -100105; const Status _6 = -100106; const Status _7 = -100107; const Status _8 = -100108; const Status _9 = -100109; const Status _11 = -100111; const Status _12 = -100112; const Status _13 = -100113; const Status _14 = -100114; const Status _15 = -100115; const Status _16 = -100116; const Status _17 = -100117; const Status _18 = -100118; const Status _19 = -100119; const Status _21 = -100121; const Status _22 = -100122; const Status _23 = -100123; const Status _24 = -100124; const Status _25 = -100125; const Status _26 = -100126; const Status _27 = -100127; const Status _28 = -100128; const Status _29 = -100129; } // namespace ERR #endif // #ifndef INCLUDED_STATUS Index: ps/trunk/source/lib/res/graphics/ogl_tex.cpp =================================================================== --- ps/trunk/source/lib/res/graphics/ogl_tex.cpp (revision 9960) +++ ps/trunk/source/lib/res/graphics/ogl_tex.cpp (revision 9961) @@ -1,1071 +1,1071 @@ /* Copyright (c) 2010 Wildfire Games * * Permission is hereby granted, free of charge, to any person obtaining * a copy of this software and associated documentation files (the * "Software"), to deal in the Software without restriction, including * without limitation the rights to use, copy, modify, merge, publish, * distribute, sublicense, and/or sell copies of the Software, and to * permit persons to whom the Software is furnished to do so, subject to * the following conditions: * * The above copyright notice and this permission notice shall be included * in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ /* * wrapper for all OpenGL texturing calls. provides caching, hotloading * and lifetime management. */ #include "precompiled.h" #include "ogl_tex.h" #include #include "lib/app_hooks.h" #include "lib/ogl.h" #include "lib/bits.h" #include "lib/sysdep/gfx.h" #include "lib/tex/tex.h" #include "lib/res/h_mgr.h" #include "lib/fnv_hash.h" //---------------------------------------------------------------------------- // OpenGL helper routines //---------------------------------------------------------------------------- static bool filter_valid(GLint filter) { switch(filter) { case GL_NEAREST: case GL_LINEAR: case GL_NEAREST_MIPMAP_NEAREST: case GL_LINEAR_MIPMAP_NEAREST: case GL_NEAREST_MIPMAP_LINEAR: case GL_LINEAR_MIPMAP_LINEAR: return true; default: return false; } } static bool wrap_valid(GLint wrap) { switch(wrap) { case GL_CLAMP: case GL_CLAMP_TO_EDGE: case GL_CLAMP_TO_BORDER: case GL_REPEAT: case GL_MIRRORED_REPEAT: return true; default: return false; } } static bool filter_uses_mipmaps(GLint filter) { switch(filter) { case GL_NEAREST_MIPMAP_NEAREST: case GL_LINEAR_MIPMAP_NEAREST: case GL_NEAREST_MIPMAP_LINEAR: case GL_LINEAR_MIPMAP_LINEAR: return true; default: return false; } } static bool fmt_is_s3tc(GLenum fmt) { switch(fmt) { case GL_COMPRESSED_RGB_S3TC_DXT1_EXT: case GL_COMPRESSED_RGBA_S3TC_DXT1_EXT: case GL_COMPRESSED_RGBA_S3TC_DXT3_EXT: case GL_COMPRESSED_RGBA_S3TC_DXT5_EXT: return true; default: return false; } } // determine OpenGL texture format, given and Tex . static GLint choose_fmt(size_t bpp, size_t flags) { const bool alpha = (flags & TEX_ALPHA) != 0; const bool bgr = (flags & TEX_BGR ) != 0; const bool grey = (flags & TEX_GREY ) != 0; const size_t dxt = flags & TEX_DXT; // S3TC if(dxt != 0) { switch(dxt) { case DXT1A: return GL_COMPRESSED_RGBA_S3TC_DXT1_EXT; case 1: return GL_COMPRESSED_RGB_S3TC_DXT1_EXT; case 3: return GL_COMPRESSED_RGBA_S3TC_DXT3_EXT; case 5: return GL_COMPRESSED_RGBA_S3TC_DXT5_EXT; default: DEBUG_WARN_ERR(ERR::LOGIC); // invalid DXT value return 0; } } // uncompressed switch(bpp) { case 8: ENSURE(grey); return GL_LUMINANCE; case 16: return GL_LUMINANCE_ALPHA; case 24: ENSURE(!alpha); return bgr? GL_BGR : GL_RGB; case 32: ENSURE(alpha); return bgr? GL_BGRA : GL_RGBA; default: DEBUG_WARN_ERR(ERR::LOGIC); // invalid bpp return 0; } UNREACHABLE; } //---------------------------------------------------------------------------- // quality mechanism //---------------------------------------------------------------------------- static GLint default_filter = GL_LINEAR; // one of the GL *minify* filters static int default_q_flags = OGL_TEX_FULL_QUALITY; // OglTexQualityFlags static bool q_flags_valid(int q_flags) { const size_t bits = OGL_TEX_FULL_QUALITY|OGL_TEX_HALF_BPP|OGL_TEX_HALF_RES; // unrecognized bits are set - invalid if((q_flags & ~bits) != 0) return false; // "full quality" but other reduction bits are set - invalid if(q_flags & OGL_TEX_FULL_QUALITY && q_flags & ~OGL_TEX_FULL_QUALITY) return false; return true; } // change default settings - these affect performance vs. quality. // may be overridden for individual textures via parameter to // ogl_tex_upload or ogl_tex_set_filter, respectively. // // pass 0 to keep the current setting; defaults and legal values are: // - q_flags: OGL_TEX_FULL_QUALITY; combination of OglTexQualityFlags // - filter: GL_LINEAR; any valid OpenGL minification filter void ogl_tex_set_defaults(int q_flags, GLint filter) { if(q_flags) { ENSURE(q_flags_valid(q_flags)); default_q_flags = q_flags; } if(filter) { ENSURE(filter_valid(filter)); default_filter = filter; } } // choose an internal format for based on the given q_flags. static GLint choose_int_fmt(GLenum fmt, int q_flags) { // true => 4 bits per component; otherwise, 8 const bool half_bpp = (q_flags & OGL_TEX_HALF_BPP) != 0; // early-out for S3TC textures: they don't need an internal format // (because upload is via glCompressedTexImage2DARB), but we must avoid // triggering the default case below. we might as well return a // meaningful value (i.e. int_fmt = fmt). if(fmt_is_s3tc(fmt)) return fmt; switch(fmt) { // 8bpp case GL_LUMINANCE: return half_bpp? GL_LUMINANCE4 : GL_LUMINANCE8; case GL_INTENSITY: return half_bpp? GL_INTENSITY4 : GL_INTENSITY8; case GL_ALPHA: return half_bpp? GL_ALPHA4 : GL_ALPHA8; // 16bpp case GL_LUMINANCE_ALPHA: return half_bpp? GL_LUMINANCE4_ALPHA4 : GL_LUMINANCE8_ALPHA8; // 24bpp case GL_RGB: case GL_BGR: // note: BGR can't be used as internal format return half_bpp? GL_RGB4 : GL_RGB8; // 32bpp case GL_RGBA: case GL_BGRA: // note: BGRA can't be used as internal format return half_bpp? GL_RGBA4 : GL_RGBA8; default: { wchar_t buf[100]; swprintf_s(buf, ARRAY_SIZE(buf), L"choose_int_fmt: fmt 0x%x isn't covered! please add it", fmt); DEBUG_DISPLAY_ERROR(buf); DEBUG_WARN_ERR(ERR::LOGIC); // given fmt isn't covered! please add it. // fall back to a reasonable default return half_bpp? GL_RGB4 : GL_RGB8; } } UNREACHABLE; } //---------------------------------------------------------------------------- // texture state to allow seamless reload //---------------------------------------------------------------------------- // see "Texture Parameters" in docs. // all GL state tied to the texture that must be reapplied after reload. // (this mustn't get too big, as it's stored in the already sizeable OglTex) struct OglTexState { // glTexParameter // note: there are more options, but they do not look to // be important and will not be applied after a reload! // in particular, LOD_BIAS isn't needed because that is set for // the entire texturing unit via glTexEnv. // .. texture filter // note: this is the minification filter value; magnification filter // is GL_NEAREST if it's GL_NEAREST, otherwise GL_LINEAR. // we don't store mag_filter explicitly because it // doesn't appear useful - either apps can tolerate LINEAR, or // mipmaps aren't called for and filter could be NEAREST anyway). GLint filter; // .. wrap mode GLint wrap_s; GLint wrap_t; // .. anisotropy // note: ignored unless EXT_texture_filter_anisotropic is supported. GLfloat anisotropy; }; // fill the given state object with default values. static void state_set_to_defaults(OglTexState* ots) { ots->filter = default_filter; ots->wrap_s = GL_REPEAT; ots->wrap_t = GL_REPEAT; ots->anisotropy = 1.0f; } // send all state to OpenGL (actually the currently bound texture). // called from ogl_tex_upload. static void state_latch(OglTexState* ots) { // filter const GLint filter = ots->filter; glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, filter); const GLint mag_filter = (filter == GL_NEAREST)? GL_NEAREST : GL_LINEAR; glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, mag_filter); // wrap const GLint wrap_s = ots->wrap_s; const GLint wrap_t = ots->wrap_t; glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, wrap_s); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, wrap_t); // .. only CLAMP and REPEAT are guaranteed to be available. // if we're using one of the others, we squelch the error that // may have resulted if this GL implementation is old. if((wrap_s != GL_CLAMP && wrap_s != GL_REPEAT) || (wrap_t != GL_CLAMP && wrap_t != GL_REPEAT)) ogl_SquelchError(GL_INVALID_ENUM); // anisotropy const GLfloat anisotropy = ots->anisotropy; if (anisotropy != 1.0f && ogl_tex_has_anisotropy()) { glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAX_ANISOTROPY_EXT, anisotropy); } } //---------------------------------------------------------------------------- // texture resource object //---------------------------------------------------------------------------- // ideally we would split OglTex into data and state objects as in // SndData / VSrc. this gives us the benefits of caching while still // leaving each "instance" (state object, which owns a data reference) // free to change its state. however, unlike in OpenAL, there is no state // independent of the data object - all parameters are directly tied to the // GL texture object. therefore, splitting them up is impossible. // (we shouldn't even keep the texel data in memory since that's already // covered by the FS cache). // // given that multiple "instances" share the state stored here, we conclude: // - a refcount is necessary to prevent ogl_tex_upload from freeing // as long as other instances are active. // - concurrent use risks cross-talk (if the 2nd "instance" changes state and // the first is reloaded, its state may change to that of the 2nd) // // as bad as it sounds, the latter issue isn't a problem: we do not expect // multiple instances of the same texture where someone changes its filter. // even if it is reloaded, the differing state is not critical. // the alternative is even worse: disabling *all* caching/reuse would // really hurt performance and h_mgr doesn't support only disallowing // reuse of active objects (this would break the index lookup code, since // multiple instances may then exist). // note: make sure these values fit inside OglTex.flags (only 16 bits) enum OglTexFlags { // "the texture is currently uploaded"; reset in dtor. OT_IS_UPLOADED = 1, // "the enclosed Tex object is valid and holds a texture"; // reset in dtor and after ogl_tex_upload's tex_free. OT_TEX_VALID = 2, //size_t tex_valid : 1; // "reload() should automatically re-upload the texture" (because // it had been uploaded before the reload); never reset. OT_NEED_AUTO_UPLOAD = 4, // (used for validating flags) OT_ALL_FLAGS = OT_IS_UPLOADED|OT_TEX_VALID|OT_NEED_AUTO_UPLOAD }; struct OglTex { Tex t; // allocated by OglTex_reload; indicates the texture is currently uploaded. GLuint id; // ogl_tex_upload calls choose_fmt to determine these from . // however, its caller may override those values via parameters. // note: these are stored here to allow retrieving via ogl_tex_get_format; // they are only used within ogl_tex_upload. GLenum fmt; GLint int_fmt; OglTexState state; // OglTexQualityFlags u8 q_flags; // to which Texture Mapping Unit was this bound? u8 tmu; u16 flags; }; H_TYPE_DEFINE(OglTex); static void OglTex_init(OglTex* ot, va_list args) { Tex* wrapped_tex = va_arg(args, Tex*); if(wrapped_tex) { ot->t = *wrapped_tex; // indicate ot->t is now valid, thus skipping loading from file. // note: ogl_tex_wrap prevents actual reloads from happening. ot->flags |= OT_TEX_VALID; } state_set_to_defaults(&ot->state); ot->q_flags = default_q_flags; } static void OglTex_dtor(OglTex* ot) { if(ot->flags & OT_TEX_VALID) { tex_free(&ot->t); ot->flags &= ~OT_TEX_VALID; } // note: do not check if OT_IS_UPLOADED is set, because we allocate // OglTex.id without necessarily having done an upload. glDeleteTextures(1, &ot->id); ot->id = 0; ot->flags &= ~OT_IS_UPLOADED; } static Status OglTex_reload(OglTex* ot, const PIVFS& vfs, const VfsPath& pathname, Handle h) { // we're reusing a freed but still in-memory OglTex object if(ot->flags & OT_IS_UPLOADED) return INFO::OK; // if we don't already have the texture in memory (*), load from file. // * this happens if the texture is "wrapped". if(!(ot->flags & OT_TEX_VALID)) { shared_ptr file; size_t fileSize; RETURN_STATUS_IF_ERR(vfs->LoadFile(pathname, file, fileSize)); if(tex_decode(file, fileSize, &ot->t) >= 0) ot->flags |= OT_TEX_VALID; } glGenTextures(1, &ot->id); // if it had already been uploaded before this reload, // re-upload it (this also does state_latch). if(ot->flags & OT_NEED_AUTO_UPLOAD) (void)ogl_tex_upload(h); return INFO::OK; } static Status OglTex_validate(const OglTex* ot) { if(ot->flags & OT_TEX_VALID) { RETURN_STATUS_IF_ERR(tex_validate(&ot->t)); // width, height // (note: this is done here because tex.cpp doesn't impose any // restrictions on dimensions, while OpenGL does). size_t w = ot->t.w; size_t h = ot->t.h; // .. == 0; texture file probably not loaded successfully. if(w == 0 || h == 0) WARN_RETURN(ERR::_11); // .. greater than max supported tex dimension. // no-op if ogl_Init not yet called if(w > (size_t)ogl_max_tex_size || h > (size_t)ogl_max_tex_size) WARN_RETURN(ERR::_12); // .. not power-of-2. // note: we can't work around this because both NV_texture_rectangle // and subtexture require work for the client (changing tex coords). // TODO: ARB_texture_non_power_of_two if(!is_pow2(w) || !is_pow2(h)) WARN_RETURN(ERR::_13); } // texture state if(!filter_valid(ot->state.filter)) WARN_RETURN(ERR::_14); if(!wrap_valid(ot->state.wrap_s)) WARN_RETURN(ERR::_15); if(!wrap_valid(ot->state.wrap_t)) WARN_RETURN(ERR::_16); // misc if(!q_flags_valid(ot->q_flags)) WARN_RETURN(ERR::_17); if(ot->tmu >= 128) // unexpected that there will ever be this many WARN_RETURN(ERR::_18); if(ot->flags > OT_ALL_FLAGS) WARN_RETURN(ERR::_19); // .. note: don't check ot->fmt and ot->int_fmt - they aren't set // until during ogl_tex_upload. return INFO::OK; } static Status OglTex_to_string(const OglTex* ot, wchar_t* buf) { swprintf_s(buf, H_STRING_LEN, L"OglTex id=%d flags=%x", ot->id, ot->flags); return INFO::OK; } // load and return a handle to the texture given in . // for a list of supported formats, see tex.h's tex_load. Handle ogl_tex_load(const PIVFS& vfs, const VfsPath& pathname, size_t flags) { Tex* wrapped_tex = 0; // we're loading from file return h_alloc(H_OglTex, vfs, pathname, flags, wrapped_tex); } // return Handle to an existing object, if it has been loaded and // is still in memory; otherwise, a negative error code. Handle ogl_tex_find(const VfsPath& pathname) { const uintptr_t key = fnv_hash(pathname.string().c_str(), pathname.string().length()*sizeof(pathname.string()[0])); return h_find(H_OglTex, key); } // make the given Tex object ready for use as an OpenGL texture // and return a handle to it. this will be as if its contents // had been loaded by ogl_tex_load. // // we need only add bookkeeping information and "wrap" it in // a resource object (accessed via Handle), hence the name. // // isn't strictly needed but should describe the texture so that // h_filename will return a meaningful comment for debug purposes. // note: because we cannot guarantee that callers will pass distinct // "filenames", caching is disabled for the created object. this avoids // mistakenly reusing previous objects that share the same comment. Handle ogl_tex_wrap(Tex* t, const PIVFS& vfs, const VfsPath& pathname, size_t flags) { // this object may not be backed by a file ("may", because // someone could do tex_load and then ogl_tex_wrap). // if h_mgr asks for a reload, the dtor will be called but // we won't be able to reconstruct it. therefore, disallow reloads. // (they are improbable anyway since caller is supposed to pass a // 'descriptive comment' instead of filename, but don't rely on that) // also disable caching as explained above. flags |= RES_DISALLOW_RELOAD|RES_NO_CACHE; return h_alloc(H_OglTex, vfs, pathname, flags, t); } // free all resources associated with the texture and make further // use of it impossible. (subject to refcount) Status ogl_tex_free(Handle& ht) { return h_free(ht, H_OglTex); } //---------------------------------------------------------------------------- // state setters (see "Texture Parameters" in docs) //---------------------------------------------------------------------------- // we require the below functions be called before uploading; this avoids // potentially redundant glTexParameter calls (we'd otherwise need to always // set defaults because we don't know if an override is forthcoming). // raise a debug warning if the texture has already been uploaded // (except in the few cases where this is allowed; see below). // this is so that you will notice incorrect usage - only one instance of a // texture should be active at a time, because otherwise they vie for // control of one shared OglTexState. static void warn_if_uploaded(Handle ht, const OglTex* ot) { #ifndef NDEBUG // we do not require users of this module to remember if they've // already uploaded a texture (inconvenient). since they also can't // tell if the texture was newly loaded (due to h_alloc interface), // we have to squelch this warning in 2 cases: // - it's ogl_tex_loaded several times (i.e. refcount > 1) and the // caller (typically a higher-level LoadTexture) is setting filter etc. // - caller is using our Handle as a caching mechanism, and calls // ogl_tex_set_* before every use of the texture. note: this // need not fall under the above check, e.g. if freed but cached. // workaround is that ogl_tex_set_* won't call us if the // same state values are being set (harmless anyway). - int refs = h_get_refcnt(ht); - if(refs > 1) + intptr_t refs = h_get_refcnt(ht); + if(intptr_t > 1) return; // don't complain if(ot->flags & OT_IS_UPLOADED) DEBUG_WARN_ERR(ERR::LOGIC); // ogl_tex_set_*: texture already uploaded and shouldn't be changed #else // (prevent warnings; the alternative of wrapping all call sites in // #ifndef is worse) UNUSED2(ht); UNUSED2(ot); #endif } // override default filter (as set above) for this texture. // must be called before uploading (raises a warning if called afterwards). // filter is as defined by OpenGL; it is applied for both minification and // magnification (for rationale and details, see OglTexState) Status ogl_tex_set_filter(Handle ht, GLint filter) { H_DEREF(ht, OglTex, ot); if(!filter_valid(filter)) WARN_RETURN(ERR::INVALID_PARAM); if(ot->state.filter != filter) { warn_if_uploaded(ht, ot); ot->state.filter = filter; } return INFO::OK; } // override default wrap mode (GL_REPEAT) for this texture. // must be called before uploading (raises a warning if called afterwards). // wrap is as defined by OpenGL. Status ogl_tex_set_wrap(Handle ht, GLint wrap_s, GLint wrap_t) { H_DEREF(ht, OglTex, ot); if(!wrap_valid(wrap_s)) WARN_RETURN(ERR::INVALID_PARAM); if(!wrap_valid(wrap_t)) WARN_RETURN(ERR::INVALID_PARAM); if(ot->state.wrap_s != wrap_s || ot->state.wrap_t != wrap_t) { warn_if_uploaded(ht, ot); ot->state.wrap_s = wrap_s; ot->state.wrap_t = wrap_t; } return INFO::OK; } // override default anisotropy for this texture. // must be called before uploading (raises a warning if called afterwards). Status ogl_tex_set_anisotropy(Handle ht, GLfloat anisotropy) { H_DEREF(ht, OglTex, ot); if(anisotropy < 1.0f) WARN_RETURN(ERR::INVALID_PARAM); if(ot->state.anisotropy != anisotropy) { warn_if_uploaded(ht, ot); ot->state.anisotropy = anisotropy; } return INFO::OK; } //---------------------------------------------------------------------------- // upload //---------------------------------------------------------------------------- // OpenGL has several features that are helpful for uploading but not // available in all implementations. we check for their presence but // provide for user override (in case they don't work on a card/driver // combo we didn't test). // tristate; -1 is undecided static int have_auto_mipmap_gen = -1; static int have_s3tc = -1; static int have_anistropy = -1; // override the default decision and force/disallow use of the // given feature. should be called from ah_override_gl_upload_caps. void ogl_tex_override(OglTexOverrides what, OglTexAllow allow) { ENSURE(allow == OGL_TEX_ENABLE || allow == OGL_TEX_DISABLE); const bool enable = (allow == OGL_TEX_ENABLE); switch(what) { case OGL_TEX_S3TC: have_s3tc = enable; break; case OGL_TEX_AUTO_MIPMAP_GEN: have_auto_mipmap_gen = enable; break; case OGL_TEX_ANISOTROPY: have_anistropy = enable; break; default: DEBUG_WARN_ERR(ERR::LOGIC); // invalid break; } } // detect caps (via OpenGL extension list) and give an app_hook the chance to // override this (e.g. via list of card/driver combos on which S3TC breaks). // called once from the first ogl_tex_upload. static void detect_gl_upload_caps() { // note: gfx_card will be empty if running in quickstart mode; // in that case, we won't be able to check for known faulty cards. // detect features, but only change the variables if they were at // "undecided" (if overrides were set before this, they must remain). if(have_auto_mipmap_gen == -1) { have_auto_mipmap_gen = ogl_HaveExtension("GL_SGIS_generate_mipmap"); } if(have_s3tc == -1) { // note: we don't bother checking for GL_S3_s3tc - it is incompatible // and irrelevant (was never widespread). have_s3tc = ogl_HaveExtensions(0, "GL_ARB_texture_compression", "GL_EXT_texture_compression_s3tc", NULL) == 0; } if(have_anistropy == -1) { have_anistropy = ogl_HaveExtension("GL_EXT_texture_filter_anisotropic"); } // allow app hook to make ogl_tex_override calls if(AH_IS_DEFINED(override_gl_upload_caps)) { ah_override_gl_upload_caps(); } // no app hook defined - have our own crack at blacklisting some hardware. else { const std::wstring cardName = gfx::CardName(); // rationale: janwas's laptop's S3 card blows up if S3TC is used // (oh, the irony). it'd be annoying to have to share this between all // projects, hence this default implementation here. if(cardName == L"S3 SuperSavage/IXC 1014") ogl_tex_override(OGL_TEX_S3TC, OGL_TEX_DISABLE); } } // take care of mipmaps. if they are called for by , either // arrange for OpenGL to create them, or see to it that the Tex object // contains them (if need be, creating them in software). // sets *plevels_to_skip to influence upload behavior (depending on // whether mipmaps are needed and the quality settings). // returns 0 to indicate success; otherwise, caller must disable // mipmapping by switching filter to e.g. GL_LINEAR. static Status get_mipmaps(Tex* t, GLint filter, int q_flags, int* plevels_to_skip) { // decisions: // .. does filter call for uploading mipmaps? const bool need_mipmaps = filter_uses_mipmaps(filter); // .. does the image data include mipmaps? (stored as separate // images after the regular texels) const bool includes_mipmaps = (t->flags & TEX_MIPMAPS) != 0; // .. is this texture in S3TC format? (more generally, "compressed") const bool is_s3tc = (t->flags & TEX_DXT) != 0; *plevels_to_skip = TEX_BASE_LEVEL_ONLY; if(!need_mipmaps) return INFO::OK; // image already contains pregenerated mipmaps; we need do nothing. // this is the nicest case, because they are fastest to load // (no extra processing needed) and typically filtered better than // if automatically generated. if(includes_mipmaps) *plevels_to_skip = 0; // t contains mipmaps // OpenGL supports automatic generation; we need only // activate that and upload the base image. else if(have_auto_mipmap_gen) { // note: we assume GL_GENERATE_MIPMAP and GL_GENERATE_MIPMAP_SGIS // have the same values - it's heavily implied by the spec // governing 'promoted' ARB extensions and just plain makes sense. glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP, GL_TRUE); } // image is S3TC-compressed and the previous 2 alternatives weren't // available; we're going to cheat and just disable mipmapping. // rationale: having tex_transform add mipmaps would be slow (since // all<->all transforms aren't implemented, it'd have to decompress // from S3TC first), and DDS images ought to include mipmaps! else if(is_s3tc) return ERR::FAIL; // NOWARN // image is uncompressed and we're on an old OpenGL implementation; // we will generate mipmaps in software. else { RETURN_STATUS_IF_ERR(tex_transform_to(t, t->flags|TEX_MIPMAPS)); *plevels_to_skip = 0; // t contains mipmaps } // t contains mipmaps; we can apply our resolution reduction trick: if(*plevels_to_skip == 0) { // this saves texture memory by skipping some of the lower // (high-resolution) mip levels. // // note: we don't just use GL_TEXTURE_BASE_LEVEL because it would // require uploading unused levels, which is wasteful. // .. can be expanded to reduce to 1/4, 1/8 by encoding factor in q_flags. const size_t reduce = (q_flags & OGL_TEX_HALF_RES)? 2 : 1; *plevels_to_skip = (int)ceil_log2(reduce); } return INFO::OK; } // tex_util_foreach_mipmap callbacks: upload the given level to OpenGL. struct UploadParams { GLenum fmt; GLint int_fmt; }; static void upload_level(size_t level, size_t level_w, size_t level_h, const u8* RESTRICT level_data, size_t UNUSED(level_data_size), void* RESTRICT cbData) { const UploadParams* up = (const UploadParams*)cbData; glTexImage2D(GL_TEXTURE_2D, (GLint)level, up->int_fmt, (GLsizei)level_w, (GLsizei)level_h, 0, up->fmt, GL_UNSIGNED_BYTE, level_data); } static void upload_compressed_level(size_t level, size_t level_w, size_t level_h, const u8* RESTRICT level_data, size_t level_data_size, void* RESTRICT cbData) { const UploadParams* up = (const UploadParams*)cbData; pglCompressedTexImage2DARB(GL_TEXTURE_2D, (GLint)level, up->fmt, (GLsizei)level_w, (GLsizei)level_h, 0, (GLsizei)level_data_size, level_data); } // upload the texture in the specified (internal) format. // split out of ogl_tex_upload because it was too big. // // pre: is valid for OpenGL use; texture is bound. static void upload_impl(Tex* t, GLenum fmt, GLint int_fmt, int levels_to_skip) { const GLsizei w = (GLsizei)t->w; const GLsizei h = (GLsizei)t->h; const size_t bpp = t->bpp; const u8* data = (const u8*)tex_get_data(t); const UploadParams up = { fmt, int_fmt }; if(t->flags & TEX_DXT) tex_util_foreach_mipmap(w, h, bpp, data, levels_to_skip, 4, upload_compressed_level, (void*)&up); else tex_util_foreach_mipmap(w, h, bpp, data, levels_to_skip, 1, upload_level, (void*)&up); } // upload the texture to OpenGL. // if not 0, parameters override the following: // fmt_ovr : OpenGL format (e.g. GL_RGB) decided from bpp / Tex flags; // q_flags_ovr : global default "quality vs. performance" flags; // int_fmt_ovr : internal format (e.g. GL_RGB8) decided from fmt / q_flags. // // side effects: // - enables texturing on TMU 0 and binds the texture to it; // - frees the texel data! see ogl_tex_get_data. Status ogl_tex_upload(const Handle ht, GLenum fmt_ovr, int q_flags_ovr, GLint int_fmt_ovr) { ONCE(detect_gl_upload_caps()); H_DEREF(ht, OglTex, ot); Tex* t = &ot->t; ENSURE(q_flags_valid(q_flags_ovr)); // we don't bother verifying *fmt_ovr - there are too many values // upload already happened; no work to do. // (this also happens if a cached texture is "loaded") if(ot->flags & OT_IS_UPLOADED) return INFO::OK; if(ot->flags & OT_TEX_VALID) { // decompress S3TC if that's not supported by OpenGL. if((t->flags & TEX_DXT) && !have_s3tc) (void)tex_transform_to(t, t->flags & ~TEX_DXT); // determine fmt and int_fmt, allowing for user override. ot->fmt = choose_fmt(t->bpp, t->flags); if(fmt_ovr) ot->fmt = fmt_ovr; if(q_flags_ovr) ot->q_flags = q_flags_ovr; ot->int_fmt = choose_int_fmt(ot->fmt, ot->q_flags); if(int_fmt_ovr) ot->int_fmt = int_fmt_ovr; // now actually send to OpenGL: ogl_WarnIfError(); { // (note: we know ht is valid due to H_DEREF, but ogl_tex_bind can // fail in debug builds if OglTex.id isn't a valid texture name) RETURN_STATUS_IF_ERR(ogl_tex_bind(ht, ot->tmu)); int levels_to_skip; if(get_mipmaps(t, ot->state.filter, ot->q_flags, &levels_to_skip) < 0) // error => disable mipmapping ot->state.filter = GL_LINEAR; // (note: if first time, applies our defaults/previous overrides; // otherwise, replays all state changes) state_latch(&ot->state); upload_impl(t, ot->fmt, ot->int_fmt, levels_to_skip); } ogl_WarnIfError(); ot->flags |= OT_IS_UPLOADED; // see rationale for at declaration of OglTex. // note: tex_free is safe even if this OglTex was wrapped - // the Tex contains a mem handle. - int refs = h_get_refcnt(ht); + intptr_t refs = h_get_refcnt(ht); if(refs == 1) { // note: we verify above that OT_TEX_VALID is set tex_free(t); ot->flags &= ~OT_TEX_VALID; } } ot->flags |= OT_NEED_AUTO_UPLOAD; return INFO::OK; } //---------------------------------------------------------------------------- // getters //---------------------------------------------------------------------------- // retrieve texture dimensions and bits per pixel. // all params are optional and filled if non-NULL. Status ogl_tex_get_size(Handle ht, size_t* w, size_t* h, size_t* bpp) { H_DEREF(ht, OglTex, ot); if(w) *w = ot->t.w; if(h) *h = ot->t.h; if(bpp) *bpp = ot->t.bpp; return INFO::OK; } // retrieve TexFlags and the corresponding OpenGL format. // the latter is determined during ogl_tex_upload and is 0 before that. // all params are optional and filled if non-NULL. Status ogl_tex_get_format(Handle ht, size_t* flags, GLenum* fmt) { H_DEREF(ht, OglTex, ot); if(flags) *flags = ot->t.flags; if(fmt) { ENSURE(ot->flags & OT_IS_UPLOADED); *fmt = ot->fmt; } return INFO::OK; } // retrieve pointer to texel data. // // note: this memory is freed after a successful ogl_tex_upload for // this texture. after that, the pointer we retrieve is NULL but // the function doesn't fail (negative return value) by design. // if you still need to get at the data, add a reference before // uploading it or read directly from OpenGL (discouraged). Status ogl_tex_get_data(Handle ht, u8** p) { H_DEREF(ht, OglTex, ot); *p = tex_get_data(&ot->t); return INFO::OK; } // retrieve colour of 1x1 mipmap level extern Status ogl_tex_get_average_colour(Handle ht, u32* p) { H_DEREF(ht, OglTex, ot); warn_if_uploaded(ht, ot); *p = tex_get_average_colour(&ot->t); return INFO::OK; } //---------------------------------------------------------------------------- // misc API //---------------------------------------------------------------------------- // bind the texture to the specified unit [number] in preparation for // using it in rendering. if is 0, texturing is disabled instead. // // side effects: // - changes the active texture unit; // - (if return value is 0:) texturing was enabled/disabled on that unit. // // notes: // - assumes multitexturing is available. // - not necessary before calling ogl_tex_upload! // - on error, the unit's texture state is unchanged; see implementation. Status ogl_tex_bind(Handle ht, size_t unit) { // note: there are many call sites of glActiveTextureARB, so caching // those and ignoring redundant sets isn't feasible. pglActiveTextureARB((int)(GL_TEXTURE0+unit)); // special case: disable texturing if(ht == 0) { glDisable(GL_TEXTURE_2D); return INFO::OK; } // if this fails, the texture unit's state remains unchanged. // we don't bother catching that and disabling texturing because a // debug warning is raised anyway, and it's quite unlikely. H_DEREF(ht, OglTex, ot); ot->tmu = (u8)unit; // if 0, there's a problem in the OglTex reload/dtor logic. // binding it results in whiteness, which can have many causes; // we therefore complain so this one can be ruled out. ENSURE(ot->id != 0); glEnable(GL_TEXTURE_2D); glBindTexture(GL_TEXTURE_2D, ot->id); return INFO::OK; } // apply the specified transforms (as in tex_transform) to the image. // must be called before uploading (raises a warning if called afterwards). Status ogl_tex_transform(Handle ht, size_t transforms) { H_DEREF(ht, OglTex, ot); Status ret = tex_transform(&ot->t, transforms); return ret; } // change the pixel format to that specified by . // (note: this is equivalent to ogl_tex_transform(ht, ht_flags^new_flags). Status ogl_tex_transform_to(Handle ht, size_t new_flags) { H_DEREF(ht, OglTex, ot); Status ret = tex_transform_to(&ot->t, new_flags); return ret; } // return whether native S3TC support is available bool ogl_tex_has_s3tc() { // ogl_tex_upload must be called before this ENSURE(have_s3tc != -1); return (have_s3tc != 0); } // return whether anisotropic filtering support is available bool ogl_tex_has_anisotropy() { // ogl_tex_upload must be called before this ENSURE(have_anistropy != -1); return (have_anistropy != 0); } Index: ps/trunk/source/lib/res/h_mgr.h =================================================================== --- ps/trunk/source/lib/res/h_mgr.h (revision 9960) +++ ps/trunk/source/lib/res/h_mgr.h (revision 9961) @@ -1,442 +1,428 @@ /* Copyright (c) 2010 Wildfire Games * * Permission is hereby granted, free of charge, to any person obtaining * a copy of this software and associated documentation files (the * "Software"), to deal in the Software without restriction, including * without limitation the rights to use, copy, modify, merge, publish, * distribute, sublicense, and/or sell copies of the Software, and to * permit persons to whom the Software is furnished to do so, subject to * the following conditions: * * The above copyright notice and this permission notice shall be included * in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ /* * handle manager for resources. */ /* [KEEP IN SYNC WITH WIKI] introduction ------------ a resource is an instance of a specific type of game data (e.g. texture), described by a control block (example fields: format, pointer to tex data). this module allocates storage for the control blocks, which are accessed via handle. it also provides support for transparently reloading resources from disk (allows in-game editing of data), and caches resource data. finally, it frees all resources at exit, preventing leaks. handles ------- handles are an indirection layer between client code and resources (represented by their control blocks, which contains/points to its data). they allow an important check not possible with a direct pointer: guaranteeing the handle references a given resource /instance/. problem: code C1 allocates a resource, and receives a pointer p to its control block. C1 passes p on to C2, and later frees it. now other code allocates a resource, and happens to reuse the free slot pointed to by p (also possible if simply allocating from the heap). when C2 accesses p, the pointer is valid, but we cannot tell that it is referring to a resource that had already been freed. big trouble. solution: each allocation receives a unique tag (a global counter that is large enough to never overflow). Handles include this tag, as well as a reference (array index) to the control block, which isn't directly accessible. when dereferencing the handle, we check if the handle's tag matches the copy stored in the control block. this protects against stale handle reuse, double-free, and accidentally referencing other resources. type: each handle has an associated type. these must be checked to prevent using textures as sounds, for example. with the manual vtbl scheme, this type is actually a pointer to the resource object's vtbl, and is set up via H_TYPE_DEFINE. this means that types are private to the module that declared the handle; knowledge of the type ensures the caller actually declared, and owns the resource. guide to defining and using resources ------------------------------------- 1) choose a name for the resource, used to represent all resources -of this type. we will call ours "Res1"; all below occurences of this +of this type. we will call ours "Res1"; all below occurrences of this must be replaced with the actual name (exact spelling). why? the vtbl builder defines its functions as e.g. Res1_reload; your actual definition must match. 2) declare its control block: struct Res1 { void* data; // data loaded from file size_t flags; // set when resource is created }; Note that all control blocks are stored in fixed-size slots (HDATA_USER_SIZE bytes), so squeezing the size of your data doesn't -necessarily help unless yours is the largest. However, if the filename -passed to h_alloc fits within the remaining space, it is stored there -(thus saving time+memory). Therefore, do not be extravagant with space. +help unless yours is the largest. 3) build its vtbl: H_TYPE_DEFINE(Res1); this defines the symbol H_Res1, which is used whenever the handle manager needs its type. it is only accessible to this module (file scope). note that it is actually a pointer to the vtbl. this must come before uses of H_Res1, and after the CB definition; there are no restrictions WRT functions, because the macro forward-declares what it needs. 4) implement all 'virtual' functions from the resource interface. note that inheritance isn't really possible with this approach - all functions must be defined, even if not needed. -- init: one-time init of the control block. called from h_alloc. precondition: control block is initialized to 0. static void Type_init(Res1* r, va_list args) { r->flags = va_arg(args, int); } if the caller of h_alloc passed additional args, they are available in args. if init references more args than were passed, big trouble. however, this is a bug in your code, and cannot be triggered maliciously. only your code knows the resource type, and it is the only call site of h_alloc. there is no provision for indicating failure. if one-time init fails (rare, but one example might be failure to allocate memory that is for the lifetime of the resource, instead of in reload), it will have to set the control block state such that reload will fail. -- reload: does all initialization of the resource that requires its source file. called after init; also after dtor every time the file is reloaded. static Status Type_reload(Res1* r, const VfsPath& pathname, Handle); { // already loaded; done if(r->data) return 0; r->data = malloc(100); if(!r->data) WARN_RETURN(ERR::NO_MEM); // (read contents of into r->data) return 0; } reload must abort if the control block data indicates the resource has already been loaded! example: if texture's reload is called first, it loads itself from file (triggering file.reload); afterwards, file.reload will be called again. we can't avoid this, because the handle manager doesn't know anything about dependencies (here, texture -> file). return value: 0 if successful (includes 'already loaded'), negative error code otherwise. if this fails, the resource is freed (=> dtor is called!). note that any subsequent changes to the resource state must be stored in the control block and 'replayed' when reloading. example: when uploading a texture, store the upload parameters (filter, internal format); when reloading, upload again accordingly. -- dtor: frees all data allocated by init and reload. called after reload has indicated failure, before reloading a resource, after h_free, or at exit (if the resource is still extant). except when reloading, the control block will be zeroed afterwards. static void Type_dtor(Res1* r); { free(r->data); } again no provision for reporting errors - there's no one to act on it if called at exit. you can ENSURE or log the error, though. be careful to correctly handle the different cases in which this routine can be called! some flags should persist across reloads (e.g. choices made during resource init time that must remain valid), while everything else *should be zeroed manually* (to behave correctly when reloading). be advised that this interface may change; a "prepare for reload" method or "compact/free extraneous resources" may be added. -- validate: makes sure the resource control block is in a valid state. returns 0 if all is well, or a negative error code. called automatically when the Handle is dereferenced or freed. static Status Type_validate(const Res1* r); { const int permissible_flags = 0x01; if(debug_IsPointerBogus(r->data)) WARN_RETURN(ERR::_1); if(r->flags & ~permissible_flags) WARN_RETURN(ERR::_2); return 0; } 5) provide your layer on top of the handle manager: Handle res1_load(const VfsPath& pathname, int my_flags) { // passes my_flags to init return h_alloc(H_Res1, pathname, 0, my_flags); } Status res1_free(Handle& h) { // control block is automatically zeroed after this. return h_free(h, H_Res1); } (this layer allows a res_load interface on top of all the loaders, and is necessary because your module is the only one that knows H_Res1). 6) done. the resource will be freed at exit (if not done already). here's how to access the control block, given a : a) H_DEREF(h, Res1, r); creates a variable r of type Res1*, which points to the control block of the resource referenced by h. returns "invalid handle" (a negative error code) on failure. b) Res1* r = h_user_data(h, H_Res1); if(!r) ; // bail useful if H_DEREF's error return (of type signed integer) isn't acceptable. otherwise, prefer a) - this is pretty clunky, and we could switch H_DEREF to throwing an exception on error. */ #ifndef INCLUDED_H_MGR #define INCLUDED_H_MGR // do not include from public header files! // handle.h declares type Handle, and avoids making // everything dependent on this (rather often updated) header. #include // type init routines get va_list of args #ifndef INCLUDED_HANDLE #include "handle.h" #endif #include "lib/file/vfs/vfs.h" extern void h_mgr_init(); extern void h_mgr_shutdown(); // handle type (for 'type safety' - can't use a texture handle as a sound) // registering extension for each module is bad - some may use many // (e.g. texture - many formats). // handle manager shouldn't know about handle types /* ///xxx advantage of manual vtbl: no boilerplate init, h_alloc calls ctor directly, make sure it fits in the memory slot vtbl contains sizeof resource data, and name! but- has to handle variable params, a bit ugly */ // 'manual vtbl' type id // handles have a type, to prevent using e.g. texture handles as a sound. // // alternatives: // - enum of all handle types (smaller, have to pass all methods to h_alloc) // - class (difficult to compare type, handle manager needs to know of all users) // // checked in h_alloc: // - user_size must fit in what the handle manager provides // - name must not be 0 // // init: user data is initially zeroed // dtor: user data is zeroed afterwards // reload: if this resource type is opened by another resource's reload, // our reload routine MUST check if already opened! This is relevant when // a file is reloaded: if e.g. a sound object opens a file, the handle // manager calls the reload routines for the 2 handles in unspecified order. // ensuring the order would require a tag field that can't overflow - // not really guaranteed with 32-bit handles. it'd also be more work // to sort the handles by creation time, or account for several layers of // dependencies. struct H_VTbl { void (*init)(void* user, va_list); Status (*reload)(void* user, const PIVFS& vfs, const VfsPath& pathname, Handle); void (*dtor)(void* user); Status (*validate)(const void* user); Status (*to_string)(const void* user, wchar_t* buf); size_t user_size; const wchar_t* name; }; typedef H_VTbl* H_Type; #define H_TYPE_DEFINE(type)\ /* forward decls */\ static void type##_init(type*, va_list);\ static Status type##_reload(type*, const PIVFS&, const VfsPath&, Handle);\ static void type##_dtor(type*);\ static Status type##_validate(const type*);\ static Status type##_to_string(const type*, wchar_t* buf);\ static H_VTbl V_##type =\ {\ (void (*)(void*, va_list))type##_init,\ (Status (*)(void*, const PIVFS&, const VfsPath&, Handle))type##_reload,\ (void (*)(void*))type##_dtor,\ (Status (*)(const void*))type##_validate,\ (Status (*)(const void*, wchar_t*))type##_to_string,\ sizeof(type), /* control block size */\ WIDEN(#type) /* name */\ };\ static H_Type H_##type = &V_##type // note: we cast to void* pointers so the functions can be declared to // take the control block pointers, instead of requiring a cast in each. // the forward decls ensure the function signatures are correct. // convenience macro for h_user_data: // casts its return value to the control block type. // use if H_DEREF's returning a negative error code isn't acceptable. #define H_USER_DATA(h, type) (type*)h_user_data(h, H_##type) // even more convenient wrapper for h_user_data: // declares a pointer (), assigns it H_USER_DATA, and has // the user's function return a negative error code on failure. // // note: don't use STMT - var decl must be visible to "caller" #define H_DEREF(h, type, var)\ /* h already indicates an error - return immediately to pass back*/\ /* that specific error, rather than only ERR::INVALID_HANDLE*/\ if(h < 0)\ WARN_RETURN((Status)h);\ type* const var = H_USER_DATA(h, type);\ if(!var)\ WARN_RETURN(ERR::INVALID_HANDLE); // all functions check the passed tag (part of the handle) and type against // the internal values. if they differ, an error is returned. -// resource scope -// used together with flags (e.g. in mem), so no separate type -/* -enum -{ - RES_TEMP = 1, - RES_LEVEL = 2, - RES_STATIC = 4 -}; - -#define RES_SCOPE_MASK 7 -*/ // h_alloc flags enum { // alias for RES_TEMP scope. the handle will not be kept open. RES_NO_CACHE = 0x01, // not cached, and will never reuse a previous instance RES_UNIQUE = RES_NO_CACHE|0x10, // object is requesting it never be reloaded (e.g. because it's not // backed by a file) RES_DISALLOW_RELOAD = 0x20 }; const size_t H_STRING_LEN = 256; // allocate a new handle. // if key is 0, or a (key, type) handle doesn't exist, -// the first free entry is used. +// some free entry is used. // otherwise, a handle to the existing object is returned, // and HDATA.size != 0. //// user_size is checked to make sure the user data fits in the handle data space. // dtor is associated with type and called when the object is freed. // handle data is initialized to 0; optionally, a pointer to it is returned. extern Handle h_alloc(H_Type type, const PIVFS& vfs, const VfsPath& pathname, size_t flags = 0, ...); extern Status h_free(Handle& h, H_Type type); // find and return a handle by key (typically filename hash) // currently O(log n). // // HACK: currently can't find RES_UNIQUE handles, because there // may be multiple instances of them, breaking the lookup data structure. extern Handle h_find(H_Type type, uintptr_t key); // returns a void* pointer to the control block of the resource , // or 0 on error (i.e. h is invalid or of the wrong type). // prefer using H_DEREF or H_USER_DATA. extern void* h_user_data(Handle h, H_Type type); extern VfsPath h_filename(Handle h); extern Status h_reload(const PIVFS& vfs, const VfsPath& pathname); // force the resource to be freed immediately, even if cached. // tag is not checked - this allows the first Handle returned // (whose tag will change after being 'freed', but remaining in memory) // to later close the object. // this is used when reinitializing the sound engine - // at that point, all (cached) OpenAL resources must be freed. extern Status h_force_free(Handle h, H_Type type); // increment Handle 's reference count. // only meant to be used for objects that free a Handle in their dtor, // so that they are copy-equivalent and can be stored in a STL container. // do not use this to implement refcounting on top of the Handle scheme, // e.g. loading a Handle once and then passing it around. instead, have each // user load the resource; refcounting is done under the hood. extern void h_add_ref(Handle h); // retrieve the internal reference count or a negative error code. // background: since h_alloc has no way of indicating whether it // allocated a new handle or reused an existing one, counting references // within resource control blocks is impossible. since that is sometimes // necessary (always wrapping objects in Handles is excessive), we // provide access to the internal reference count. -extern int h_get_refcnt(Handle h); +extern intptr_t h_get_refcnt(Handle h); #endif // #ifndef INCLUDED_H_MGR Index: ps/trunk/source/lib/res/h_mgr.cpp =================================================================== --- ps/trunk/source/lib/res/h_mgr.cpp (revision 9960) +++ ps/trunk/source/lib/res/h_mgr.cpp (revision 9961) @@ -1,863 +1,762 @@ /* Copyright (c) 2010 Wildfire Games * * Permission is hereby granted, free of charge, to any person obtaining * a copy of this software and associated documentation files (the * "Software"), to deal in the Software without restriction, including * without limitation the rights to use, copy, modify, merge, publish, * distribute, sublicense, and/or sell copies of the Software, and to * permit persons to whom the Software is furnished to do so, subject to * the following conditions: * * The above copyright notice and this permission notice shall be included * in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ /* * handle manager for resources. */ #include "precompiled.h" #include "h_mgr.h" #include #include // CHAR_BIT #include #include #include // std::bad_alloc #include "lib/fnv_hash.h" #include "lib/allocators/overrun_protector.h" +#include "lib/allocators/pool.h" #include "lib/module_init.h" +#include "lib/sysdep/cpu.h" // cpu_CAS64 + + +namespace ERR { +static const Status H_IDX_INVALID = -120000; // totally invalid +static const Status H_IDX_UNUSED = -120001; // beyond current cap +static const Status H_TAG_MISMATCH = -120003; +static const Status H_TYPE_MISMATCH = -120004; +} +static const StatusDefinition hStatusDefinitions[] = { + { ERR::H_IDX_INVALID, L"Handle index completely out of bounds" }, + { ERR::H_IDX_UNUSED, L"Handle index exceeds high-water mark" }, + { ERR::H_TAG_MISMATCH, L"Handle tag mismatch (stale reference?)" }, + { ERR::H_TYPE_MISMATCH, L"Handle type mismatch" } +}; +STATUS_ADD_DEFINITIONS(hStatusDefinitions); -static const size_t MAX_EXTANT_HANDLES = 10000; // rationale // // why fixed size control blocks, instead of just allocating dynamically? // it is expected that resources be created and freed often. this way is // much nicer to the memory manager. defining control blocks larger than // the allotted space is caught by h_alloc (made possible by the vtbl builder // storing control block size). it is also efficient to have all CBs in an // more or less contiguous array (see below). // // why a manager, instead of a simple pool allocator? // we need a central list of resources for freeing at exit, checking if a // resource has already been loaded (for caching), and when reloading. // may as well keep them in an array, rather than add a list and index. // // handle // // 0 = invalid handle value // < 0 is an error code (we assume < 0 <==> MSB is set - // true for 1s and 2s complement and sign-magnitude systems) // fields: // (shift value = # bits between LSB and field LSB. // may be larger than the field type - only shift Handle vars!) -// - tag (1-based) ensures the handle references a certain resource instance. -// (field width determines maximum unambiguous resource allocs) -#define TAG_BITS 32 -const size_t TAG_SHIFT = 0; -const u32 TAG_MASK = 0xFFFFFFFF; // safer than (1 << 32) - 1 - // - index (0-based) of control block in our array. // (field width determines maximum currently open handles) #define IDX_BITS 16 -const size_t IDX_SHIFT = 32; -const u32 IDX_MASK = (1l << IDX_BITS) - 1; +static const u64 IDX_MASK = (1l << IDX_BITS) - 1; + +// - tag (1-based) ensures the handle references a certain resource instance. +// (field width determines maximum unambiguous resource allocs) +typedef i64 Tag; // matches cpu_CAS64 type +#define TAG_BITS 48 +static const u64 TAG_MASK = 0xFFFFFFFF; // safer than (1 << 32) - 1 // make sure both fields fit within a Handle variable cassert(IDX_BITS + TAG_BITS <= sizeof(Handle)*CHAR_BIT); // return the handle's index field (always non-negative). // no error checking! -static inline u32 h_idx(const Handle h) +static inline size_t h_idx(const Handle h) { - return (u32)((h >> IDX_SHIFT) & IDX_MASK) - 1; + return (size_t)(h & IDX_MASK) - 1; } // return the handle's tag field. // no error checking! -static inline u32 h_tag(const Handle h) +static inline Tag h_tag(Handle h) { - return (u32)((h >> TAG_SHIFT) & TAG_MASK); + return h >> IDX_BITS; } // build a handle from index and tag. // can't fail. -static inline Handle handle(const u32 _idx, const u32 tag) +static inline Handle handle(size_t idx, u64 tag) { - const u32 idx = _idx+1; - ENSURE(idx <= IDX_MASK && tag <= TAG_MASK && "handle: idx or tag too big"); - // somewhat clunky, but be careful with the shift: - // *_SHIFT may be larger than its field's type. - Handle h_idx = idx & IDX_MASK; h_idx <<= IDX_SHIFT; - Handle h_tag = tag & TAG_MASK; h_tag <<= TAG_SHIFT; - Handle h = h_idx | h_tag; + const size_t idxPlusOne = idx+1; + ENSURE(idxPlusOne <= IDX_MASK); + ENSURE((tag & IDX_MASK) == 0); + Handle h = tag | idxPlusOne; ENSURE(h > 0); return h; } // // internal per-resource-instance data // - -// determines maximum number of references to a resource. -static const size_t REF_BITS = 16; -static const u32 REF_MAX = (1ul << REF_BITS)-1; - -static const size_t TYPE_BITS = 8; - - -// chosen so that all current resource structs are covered, -// and so sizeof(HDATA) is a power of 2 (for more efficient array access -// and array page usage). -static const size_t HDATA_USER_SIZE = 44+64; +// chosen so that all current resource structs are covered. +static const size_t HDATA_USER_SIZE = 100; struct HDATA { + // we only need the tag, because it is trivial to compute + // &HDATA from idx and vice versa. storing the entire handle + // avoids needing to extract the tag field. + Handle h; // NB: will be overwritten by pool_free + uintptr_t key; - u32 tag : TAG_BITS; + intptr_t refs; - // smaller bitfields combined into 1 - u32 refs : REF_BITS; - u32 type_idx : TYPE_BITS; + // smaller bit fields combined into 1 // .. if set, do not actually release the resource (i.e. call dtor) // when the handle is h_free-d, regardless of the refcount. // set by h_alloc; reset on exit and by housekeeping. u32 keep_open : 1; // .. HACK: prevent adding to h_find lookup index if flags & RES_UNIQUE // (because those handles might have several instances open, // which the index can't currently handle) u32 unique : 1; u32 disallow_reload : 1; H_Type type; // for statistics size_t num_derefs; // storing PIVFS here is not a good idea since this often isn't // `freed' due to caching (and there is no dtor), so // the VFS reference count would never reach zero. VfsPath pathname; u8 user[HDATA_USER_SIZE]; }; // max data array entries. compared to last_in_use => signed. -static const ssize_t hdata_cap = 1ul << IDX_BITS; +static const ssize_t hdata_cap = (1ul << IDX_BITS)/4; -// allocate entries as needed so as not to waste memory -// (hdata_cap may be large). deque-style array of pages -// to balance locality, fragmentation, and waste. -static const size_t PAGE_SIZE = 4096; -static const size_t hdata_per_page = PAGE_SIZE / sizeof(HDATA); -static const size_t num_pages = hdata_cap / hdata_per_page; -static HDATA* pages[num_pages]; - -// these must be signed, because there won't always be a valid -// first or last element. -static ssize_t first_free = -1; // don't want to scan array every h_alloc -static ssize_t last_in_use = -1; // don't search unused entries +// pool of fixed-size elements allows O(1) alloc and free; +// there is a simple mapping between HDATA address and index. +static Pool hpool; // error checking strategy: // all handles passed in go through h_data(Handle, Type) -// get a (possibly new) array entry; array is non-contiguous. +// get a (possibly new) array entry. // -// fails (returns 0) if idx is out of bounds, or if accessing a new page -// for the first time, and there's not enough memory to allocate it. -// -// also used by h_data, and alloc_idx to find a free entry. -static HDATA* h_data_from_idx(const ssize_t idx) +// fails if idx is out of bounds. +static Status h_data_from_idx(ssize_t idx, HDATA*& hd) { - // don't compare against last_in_use - this is called before allocating - // new entries, and to check if the next (but possibly not yet valid) - // entry is free. tag check protects against using unallocated entries. - if(idx < 0 || idx >= hdata_cap) - return 0; - HDATA*& page = pages[idx / hdata_per_page]; - if(!page) - { - page = (HDATA*)calloc(1, PAGE_SIZE); - if(!page) - return 0; + // don't check if idx is beyond the current high-water mark, because + // we might be allocating a new entry. subsequent tag checks protect + // against using unallocated entries. + if(size_t(idx) >= hdata_cap) // also detects negative idx + WARN_RETURN(ERR::H_IDX_INVALID); - // Initialise all the VfsPath members - for(size_t i = 0; i < hdata_per_page; ++i) - new (&page[i].pathname) VfsPath; - } - - // note: VC7.1 optimizes the divides to shift and mask. - - HDATA* hd = &page[idx % hdata_per_page]; + hd = (HDATA*)(hpool.da.base + idx*hpool.el_size); hd->num_derefs++; - return hd; + return INFO::OK; +} + +static ssize_t h_idx_from_data(HDATA* hd) +{ + if(!pool_contains(&hpool, hd)) + WARN_RETURN(ERR::INVALID_POINTER); + return (uintptr_t(hd) - uintptr_t(hpool.da.base))/hpool.el_size; } // get HDATA for the given handle. // only uses (and checks) the index field. // used by h_force_close (which must work regardless of tag). -static inline HDATA* h_data_no_tag(const Handle h) +static inline Status h_data_no_tag(const Handle h, HDATA*& hd) { ssize_t idx = (ssize_t)h_idx(h); + RETURN_STATUS_IF_ERR(h_data_from_idx(idx, hd)); // need to verify it's in range - h_data_from_idx can only verify that // it's < maximum allowable index. - if(0 > idx || idx > last_in_use) - return 0; - return h_data_from_idx(idx); + if(uintptr_t(hd) > uintptr_t(hpool.da.base)+hpool.da.pos) + WARN_RETURN(ERR::H_IDX_UNUSED); + return INFO::OK; } // get HDATA for the given handle. // also verifies the tag field. // used by functions callable for any handle type, e.g. h_filename. -static inline HDATA* h_data_tag(const Handle h) +static inline Status h_data_tag(Handle h, HDATA*& hd) { - HDATA* hd = h_data_no_tag(h); - if(!hd) - return 0; + RETURN_STATUS_IF_ERR(h_data_no_tag(h, hd)); - // note: tag = 0 marks unused entries => is invalid - u32 tag = h_tag(h); - if(tag == 0 || tag != hd->tag) - return 0; + if(h != hd->h) + { + debug_printf(L"h_mgr: expected handle %llx, got %llx\n", hd->h, h); + WARN_RETURN(ERR::H_TAG_MISMATCH); + } - return hd; + return INFO::OK; } // get HDATA for the given handle. // also verifies the type. // used by most functions accessing handle data. -static HDATA* h_data_tag_type(const Handle h, const H_Type type) +static Status h_data_tag_type(const Handle h, const H_Type type, HDATA*& hd) { - HDATA* hd = h_data_tag(h); - if(!hd) - return 0; + RETURN_STATUS_IF_ERR(h_data_tag(h, hd)); // h_alloc makes sure type isn't 0, so no need to check that here. if(hd->type != type) - return 0; - - return hd; -} - - -//----------------------------------------------------------------------------- - -// idx and hd are undefined if we fail. -// called by h_alloc only. -static Status alloc_idx(ssize_t& idx, HDATA*& hd) -{ - // we already know the first free entry - if(first_free != -1) { - idx = first_free; - hd = h_data_from_idx(idx); + debug_printf(L"h_mgr: expected type %ws, got %ws\n", hd->type->name, type->name); + WARN_RETURN(ERR::H_TYPE_MISMATCH); } - // need to look for a free entry, or alloc another - else - { - // look for an unused entry - for(idx = 0; idx <= last_in_use; idx++) - { - hd = h_data_from_idx(idx); - ENSURE(hd); // can't fail - idx is valid - - // found one - done - if(!hd->tag) - goto have_idx; - } - - // add another - // .. too many already: IDX_BITS must be increased. - if(last_in_use >= hdata_cap) - WARN_RETURN(ERR::LIMIT); - idx = last_in_use+1; // just incrementing idx would start it at 1 - hd = h_data_from_idx(idx); - if(!hd) - WARN_RETURN(ERR::NO_MEM); - // can't fail for any other reason - idx is checked above. - { // VC6 goto fix - bool is_unused = !hd->tag; - ENSURE(is_unused && "invalid last_in_use"); - } - -have_idx:; - } - - // check if next entry is free - HDATA* hd2 = h_data_from_idx(idx+1); - if(hd2 && hd2->tag == 0) - first_free = idx+1; - else - first_free = -1; - - if(idx > last_in_use) - last_in_use = idx; return INFO::OK; } -static Status free_idx(ssize_t idx) -{ - if(first_free == -1 || idx < first_free) - first_free = idx; - return INFO::OK; -} - - //----------------------------------------------------------------------------- // lookup data structure //----------------------------------------------------------------------------- // speed up h_find (called every h_alloc) // multimap, because we want to add handles of differing type but same key // (e.g. a VFile and Tex object for the same underlying filename hash key) // // store index because it's smaller and Handle can easily be reconstructed // // // note: there may be several RES_UNIQUE handles of the same type and key // (e.g. sound files - several instances of a sound definition file). // that wasn't foreseen here, so we'll just refrain from adding to the index. // that means they won't be found via h_find - no biggie. typedef boost::unordered_multimap Key2Idx; typedef Key2Idx::iterator It; static OverrunProtector key2idx_wrapper; enum KeyRemoveFlag { KEY_NOREMOVE, KEY_REMOVE }; static Handle key_find(uintptr_t key, H_Type type, KeyRemoveFlag remove_option = KEY_NOREMOVE) { Key2Idx* key2idx = key2idx_wrapper.get(); if(!key2idx) WARN_RETURN(ERR::NO_MEM); // initial return value: "not found at all, or it's of the // wrong type". the latter happens when called by h_alloc to // check if e.g. a Tex object already exists; at that time, // only the corresponding VFile exists. Handle ret = -1; std::pair range = key2idx->equal_range(key); for(It it = range.first; it != range.second; ++it) { ssize_t idx = it->second; - HDATA* hd = h_data_from_idx(idx); - // found match - if(hd && hd->type == type && hd->key == key) - { - if(remove_option == KEY_REMOVE) - key2idx->erase(it); - ret = handle(idx, hd->tag); - break; - } + HDATA* hd; + if(h_data_from_idx(idx, hd) != INFO::OK) + continue; + if(hd->type != type || hd->key != key) + continue; + + // found a match + if(remove_option == KEY_REMOVE) + key2idx->erase(it); + ret = hd->h; + break; } key2idx_wrapper.lock(); return ret; } static void key_add(uintptr_t key, Handle h) { Key2Idx* key2idx = key2idx_wrapper.get(); if(!key2idx) return; const ssize_t idx = h_idx(h); // note: MSDN documentation of stdext::hash_multimap is incorrect; // there is no overload of insert() that returns pair. (void)key2idx->insert(std::make_pair(key, idx)); key2idx_wrapper.lock(); } static void key_remove(uintptr_t key, H_Type type) { Handle ret = key_find(key, type, KEY_REMOVE); ENSURE(ret > 0); } //---------------------------------------------------------------------------- // h_alloc //---------------------------------------------------------------------------- static void warn_if_invalid(HDATA* hd) { #ifndef NDEBUG H_VTbl* vtbl = hd->type; // validate HDATA // currently nothing to do; is checked by h_alloc and // the others have no invariants we could check. // have the resource validate its user_data Status err = vtbl->validate(hd->user); ENSURE(err == INFO::OK); // make sure empty space in control block isn't touched // .. but only if we're not storing a filename there const u8* start = hd->user + vtbl->user_size; const u8* end = hd->user + HDATA_USER_SIZE; for(const u8* p = start; p < end; p++) ENSURE(*p == 0); // else: handle user data was overrun! #else UNUSED2(hd); #endif } static Status type_validate(H_Type type) { if(!type) WARN_RETURN(ERR::INVALID_PARAM); if(type->user_size > HDATA_USER_SIZE) WARN_RETURN(ERR::LIMIT); if(type->name == 0) WARN_RETURN(ERR::INVALID_PARAM); return INFO::OK; } -static u32 gen_tag() +static Tag gen_tag() { - static u32 tag; - if(++tag >= TAG_MASK) + static volatile Tag tag; + for(;;) { - debug_warn(L"h_mgr: tag overflow - allocations are no longer unique."\ - L"may not notice stale handle reuse. increase TAG_BITS."); - tag = 1; + const Tag oldTag = tag; + const Tag newTag = oldTag + (1ull << IDX_BITS); + // it's not easy to detect overflow, because compilers + // are allowed to assume it'll never happen. however, + // pow(2, 64-IDX_BITS) is "enough" anyway. + if(cpu_CAS64(&tag, oldTag, newTag)) + return newTag; } - return tag; } static Handle reuse_existing_handle(uintptr_t key, H_Type type, size_t flags) { if(flags & RES_NO_CACHE) return 0; // object of specified key and type doesn't exist yet Handle h = h_find(type, key); if(h <= 0) return 0; - HDATA* hd = h_data_tag_type(h, type); - // too many references - increase REF_BITS - if(hd->refs == REF_MAX) - WARN_RETURN(ERR::LIMIT); + HDATA* hd; + RETURN_STATUS_IF_ERR(h_data_tag_type(h, type, hd)); // h_find means this won't fail - hd->refs++; + cpu_AtomicAdd(&hd->refs, 1); // we are reactivating a closed but cached handle. // need to generate a new tag so that copies of the // previous handle can no longer access the resource. // (we don't need to reset the tag in h_free, because // use before this fails due to refs > 0 check in h_user_data). if(hd->refs == 1) { - const u32 tag = gen_tag(); - hd->tag = tag; + const Tag tag = gen_tag(); h = handle(h_idx(h), tag); // can't fail + hd->h = h; } return h; } static Status call_init_and_reload(Handle h, H_Type type, HDATA* hd, const PIVFS& vfs, const VfsPath& pathname, va_list* init_args) { Status err = INFO::OK; H_VTbl* vtbl = type; // exact same thing but for clarity // init if(vtbl->init) vtbl->init(hd->user, *init_args); // reload if(vtbl->reload) { // catch exception to simplify reload funcs - let them use new() try { err = vtbl->reload(hd->user, vfs, pathname, h); if(err == INFO::OK) warn_if_invalid(hd); } catch(std::bad_alloc) { err = ERR::NO_MEM; } } return err; } static Handle alloc_new_handle(H_Type type, const PIVFS& vfs, const VfsPath& pathname, uintptr_t key, size_t flags, va_list* init_args) { - ssize_t idx; - HDATA* hd; - RETURN_STATUS_IF_ERR(alloc_idx(idx, hd)); + HDATA* hd = (HDATA*)pool_alloc(&hpool, 0); + if(!hd) + WARN_RETURN(ERR::NO_MEM); + new(&hd->pathname) VfsPath; + + ssize_t idx = h_idx_from_data(hd); + RETURN_STATUS_IF_ERR(idx); // (don't want to do this before the add-reference exit, // so as not to waste tags for often allocated handles.) - const u32 tag = gen_tag(); + const Tag tag = gen_tag(); Handle h = handle(idx, tag); // can't fail. - hd->tag = tag; + hd->h = h; hd->key = key; hd->type = type; hd->refs = 1; if(!(flags & RES_NO_CACHE)) hd->keep_open = 1; if(flags & RES_DISALLOW_RELOAD) hd->disallow_reload = 1; hd->unique = (flags & RES_UNIQUE) != 0; hd->pathname = pathname; if(key && !hd->unique) key_add(key, h); Status err = call_init_and_reload(h, type, hd, vfs, pathname, init_args); if(err < 0) goto fail; return h; fail: // reload failed; free the handle hd->keep_open = 0; // disallow caching (since contents are invalid) (void)h_free(h, type); // (h_free already does WARN_IF_ERR) // note: since some uses will always fail (e.g. loading sounds if // g_Quickstart), do not complain here. return (Handle)err; } // any further params are passed to type's init routine Handle h_alloc(H_Type type, const PIVFS& vfs, const VfsPath& pathname, size_t flags, ...) { RETURN_STATUS_IF_ERR(type_validate(type)); const uintptr_t key = fnv_hash(pathname.string().c_str(), pathname.string().length()*sizeof(pathname.string()[0])); // see if we can reuse an existing handle Handle h = reuse_existing_handle(key, type, flags); RETURN_STATUS_IF_ERR(h); // .. successfully reused the handle; refcount increased if(h > 0) return h; // .. need to allocate a new one: va_list args; va_start(args, flags); h = alloc_new_handle(type, vfs, pathname, key, flags, &args); va_end(args); return h; // alloc_new_handle already does WARN_RETURN_STATUS_IF_ERR } //----------------------------------------------------------------------------- -// currently cannot fail. -static Status h_free_idx(ssize_t idx, HDATA* hd) +static void h_free_hd(HDATA* hd) { - // only decrement if refcount not already 0. - if(hd->refs > 0) - hd->refs--; + for(;;) + { + const intptr_t refs = hd->refs; + if(refs <= 0) // skip decrement + break; + if(cpu_CAS(&hd->refs, refs, refs-1)) // success + break; + } // still references open or caching requests it stays - do not release. if(hd->refs > 0 || hd->keep_open) - return INFO::OK; + return; // actually release the resource (call dtor, free control block). // h_alloc makes sure type != 0; if we get here, it still is H_VTbl* vtbl = hd->type; // call its destructor // note: H_TYPE_DEFINE currently always defines a dtor, but play it safe if(vtbl->dtor) vtbl->dtor(hd->user); if(hd->key && !hd->unique) key_remove(hd->key, hd->type); #ifndef NDEBUG // to_string is slow for some handles, so avoid calling it if unnecessary if(debug_filter_allows(L"H_MGR|")) { wchar_t buf[H_STRING_LEN]; if(vtbl->to_string(hd->user, buf) < 0) wcscpy_s(buf, ARRAY_SIZE(buf), L"(error)"); debug_printf(L"H_MGR| free %ls %ls accesses=%lu %ls\n", hd->type->name, hd->pathname.string().c_str(), (unsigned long)hd->num_derefs, buf); } #endif hd->pathname.~VfsPath(); // FIXME: ugly hack, but necessary to reclaim memory memset(hd, 0, sizeof(*hd)); - new (&hd->pathname) VfsPath; // FIXME too: necessary because otherwise it'll break if we reuse this page - - free_idx(idx); - - return INFO::OK; + pool_free(&hpool, hd); } Status h_free(Handle& h, H_Type type) { - ssize_t idx = h_idx(h); - HDATA* hd = h_data_tag_type(h, type); + // 0-initialized or an error code; don't complain because this + // happens often and is harmless. + if(h <= 0) + return INFO::OK; // wipe out the handle to prevent reuse but keep a copy for below. const Handle h_copy = h; h = 0; - // h was invalid - if(!hd) - { - // 0-initialized or an error code; don't complain because this - // happens often and is harmless. - if(h_copy <= 0) - return INFO::OK; - // this was a valid handle but was probably freed in the meantime. - // complain because this probably indicates a bug somewhere. - WARN_RETURN(ERR::INVALID_HANDLE); - } + HDATA* hd; + RETURN_STATUS_IF_ERR(h_data_tag_type(h_copy, type, hd)); - return h_free_idx(idx, hd); + h_free_hd(hd); + return INFO::OK; } //---------------------------------------------------------------------------- // remaining API void* h_user_data(const Handle h, const H_Type type) { - HDATA* hd = h_data_tag_type(h, type); - if(!hd) + HDATA* hd; + if(h_data_tag_type(h, type, hd) != INFO::OK) return 0; if(!hd->refs) { // note: resetting the tag is not enough (user might pass in its value) DEBUG_WARN_ERR(ERR::LOGIC); // no references to resource (it's cached, but someone is accessing it directly) return 0; } warn_if_invalid(hd); return hd->user; } VfsPath h_filename(const Handle h) { // don't require type check: should be useable for any handle, // even if the caller doesn't know its type. - HDATA* hd = h_data_tag(h); - if(!hd) - { - DEBUG_WARN_ERR(ERR::LOGIC); + HDATA* hd; + if(h_data_tag(h, hd) != INFO::OK) return VfsPath(); - } return hd->pathname; } // TODO: what if iterating through all handles is too slow? Status h_reload(const PIVFS& vfs, const VfsPath& pathname) { const u32 key = fnv_hash(pathname.string().c_str(), pathname.string().length()*sizeof(pathname.string()[0])); // destroy (note: not free!) all handles backed by this file. // do this before reloading any of them, because we don't specify reload // order (the parent resource may be reloaded first, and load the child, // whose original data would leak). - for(ssize_t i = 0; i <= last_in_use; i++) + for(HDATA* hd = (HDATA*)hpool.da.base; hd < (HDATA*)(hpool.da.base + hpool.da.pos); hd = (HDATA*)(uintptr_t(hd)+hpool.el_size)) { - HDATA* hd = h_data_from_idx(i); - if(!hd || hd->key != key || hd->disallow_reload) + if(hd->key == 0 || hd->key != key || hd->disallow_reload) continue; hd->type->dtor(hd->user); } Status ret = INFO::OK; // now reload all affected handles - for(ssize_t i = 0; i <= last_in_use; i++) + size_t i = 0; + for(HDATA* hd = (HDATA*)hpool.da.base; hd < (HDATA*)(hpool.da.base + hpool.da.pos); hd = (HDATA*)(uintptr_t(hd)+hpool.el_size), i++) { - HDATA* hd = h_data_from_idx(i); - if(!hd || hd->key != key || hd->disallow_reload) + if(hd->key == 0 || hd->key != key || hd->disallow_reload) continue; - Handle h = handle(i, hd->tag); - - Status err = hd->type->reload(hd->user, vfs, hd->pathname, h); + Status err = hd->type->reload(hd->user, vfs, hd->pathname, hd->h); // don't stop if an error is encountered - try to reload them all. if(err < 0) { - h_free(h, hd->type); + h_free(hd->h, hd->type); if(ret == 0) // don't overwrite first error ret = err; } else warn_if_invalid(hd); } return ret; } Handle h_find(H_Type type, uintptr_t key) { return key_find(key, type); } // force the resource to be freed immediately, even if cached. // tag is not checked - this allows the first Handle returned // (whose tag will change after being 'freed', but remaining in memory) // to later close the object. // this is used when reinitializing the sound engine - // at that point, all (cached) OpenAL resources must be freed. Status h_force_free(Handle h, H_Type type) { // require valid index; ignore tag; type checked below. - HDATA* hd = h_data_no_tag(h); - if(!hd || hd->type != type) - WARN_RETURN(ERR::INVALID_HANDLE); - u32 idx = h_idx(h); + HDATA* hd; + RETURN_STATUS_IF_ERR(h_data_no_tag(h, hd)); + if(hd->type != type) + WARN_RETURN(ERR::H_TYPE_MISMATCH); hd->keep_open = 0; hd->refs = 0; - return h_free_idx(idx, hd); + h_free_hd(hd); + return INFO::OK; } // increment Handle 's reference count. // only meant to be used for objects that free a Handle in their dtor, // so that they are copy-equivalent and can be stored in a STL container. // do not use this to implement refcounting on top of the Handle scheme, // e.g. loading a Handle once and then passing it around. instead, have each // user load the resource; refcounting is done under the hood. void h_add_ref(Handle h) { - HDATA* hd = h_data_tag(h); - if(!hd) - { - DEBUG_WARN_ERR(ERR::LOGIC); // invalid handle + HDATA* hd; + if(h_data_tag(h, hd) != INFO::OK) return; - } ENSURE(hd->refs); // if there are no refs, how did the caller manage to keep a Handle?! - hd->refs++; + cpu_AtomicAdd(&hd->refs, 1); } // retrieve the internal reference count or a negative error code. // background: since h_alloc has no way of indicating whether it // allocated a new handle or reused an existing one, counting references // within resource control blocks is impossible. since that is sometimes // necessary (always wrapping objects in Handles is excessive), we // provide access to the internal reference count. -int h_get_refcnt(Handle h) +intptr_t h_get_refcnt(Handle h) { - HDATA* hd = h_data_tag(h); - if(!hd) - WARN_RETURN(ERR::INVALID_HANDLE); + HDATA* hd; + RETURN_STATUS_IF_ERR(h_data_tag(h, hd)); ENSURE(hd->refs); // if there are no refs, how did the caller manage to keep a Handle?! return hd->refs; } static ModuleInitState initState; static Status Init() { + RETURN_STATUS_IF_ERR(pool_create(&hpool, hdata_cap*sizeof(HDATA), sizeof(HDATA))); return INFO::OK; } static void Shutdown() { debug_printf(L"H_MGR| shutdown. any handle frees after this are leaks!\n"); // forcibly close all open handles - for(ssize_t i = 0; i <= last_in_use; i++) + for(HDATA* hd = (HDATA*)hpool.da.base; hd < (HDATA*)(hpool.da.base + hpool.da.pos); hd = (HDATA*)(uintptr_t(hd)+hpool.el_size)) { - HDATA* hd = h_data_from_idx(i); - // can't fail - i is in bounds by definition, and - // each HDATA entry has already been allocated. - if(!hd) - { - DEBUG_WARN_ERR(ERR::LOGIC); // h_data_from_idx failed - why?! - continue; - } - // it's already been freed; don't free again so that this // doesn't look like an error. - if(!hd->tag) + if(hd->key == 0) continue; // disable caching; we need to release the resource now. hd->keep_open = 0; hd->refs = 0; - h_free_idx(i, hd); // currently cannot fail + h_free_hd(hd); } - // free HDATA array - for(size_t j = 0; j < num_pages; j++) - { - if (pages[j]) - for(size_t k = 0; k < hdata_per_page; ++k) - pages[j][k].pathname.~VfsPath(); // FIXME: ugly hack, but necessary to reclaim memory - free(pages[j]); - pages[j] = 0; - } + pool_destroy(&hpool); } void h_mgr_init() { ModuleInit(&initState, Init); } void h_mgr_shutdown() { ModuleShutdown(&initState, Shutdown); } Index: ps/trunk/source/lib/allocators/pool.cpp =================================================================== --- ps/trunk/source/lib/allocators/pool.cpp (revision 9960) +++ ps/trunk/source/lib/allocators/pool.cpp (revision 9961) @@ -1,132 +1,129 @@ /* Copyright (c) 2010 Wildfire Games * * Permission is hereby granted, free of charge, to any person obtaining * a copy of this software and associated documentation files (the * "Software"), to deal in the Software without restriction, including * without limitation the rights to use, copy, modify, merge, publish, * distribute, sublicense, and/or sell copies of the Software, and to * permit persons to whom the Software is furnished to do so, subject to * the following conditions: * * The above copyright notice and this permission notice shall be included * in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ /* * pool allocator */ #include "precompiled.h" #include "lib/allocators/pool.h" #include "lib/alignment.h" #include "lib/allocators/freelist.h" #include "lib/timer.h" TIMER_ADD_CLIENT(tc_pool_alloc); Status pool_create(Pool* p, size_t max_size, size_t el_size) { if(el_size == POOL_VARIABLE_ALLOCS) p->el_size = 0; else p->el_size = Align(el_size); p->freelist = mem_freelist_Sentinel(); RETURN_STATUS_IF_ERR(da_alloc(&p->da, max_size)); return INFO::OK; } Status pool_destroy(Pool* p) { // don't be picky and complain if the freelist isn't empty; // we don't care since it's all part of the da anyway. // however, zero it to prevent further allocs from succeeding. p->freelist = mem_freelist_Sentinel(); return da_free(&p->da); } bool pool_contains(const Pool* p, void* el) { // outside of our range if(!(p->da.base <= el && el < p->da.base+p->da.pos)) return false; // sanity check: it should be aligned (if pool has fixed-size elements) if(p->el_size) ENSURE((uintptr_t)((u8*)el - p->da.base) % p->el_size == 0); return true; } void* pool_alloc(Pool* p, size_t size) { TIMER_ACCRUE(tc_pool_alloc); // if pool allows variable sizes, go with the size parameter, // otherwise the pool el_size setting. const size_t el_size = p->el_size? p->el_size : Align(size); + ASSERT(el_size != 0); - // note: this can never happen in pools with variable-sized elements + // note: freelist is always empty in pools with variable-sized elements // because they disallow pool_free. void* el = mem_freelist_Detach(p->freelist); - if(el) - goto have_el; - - // alloc a new entry + if(!el) // freelist empty, need to allocate a new entry { // expand, if necessary if(da_reserve(&p->da, el_size) < 0) return 0; el = p->da.base + p->da.pos; p->da.pos += el_size; } -have_el: - ENSURE(pool_contains(p, el)); // paranoia + ASSERT(pool_contains(p, el)); // paranoia return el; } void pool_free(Pool* p, void* el) { // only allowed to free items if we were initialized with // fixed el_size. (this avoids having to pass el_size here and // check if requested_size matches that when allocating) if(p->el_size == 0) { DEBUG_WARN_ERR(ERR::LOGIC); // cannot free variable-size items return; } if(pool_contains(p, el)) mem_freelist_AddToFront(p->freelist, el); else DEBUG_WARN_ERR(ERR::LOGIC); // invalid pointer (not in pool) } void pool_free_all(Pool* p) { p->freelist = mem_freelist_Sentinel(); // must be reset before da_set_size or CHECK_DA will complain. p->da.pos = 0; da_set_size(&p->da, 0); } size_t pool_committed(Pool* p) { return p->da.cur_size; }