When to NOT Linearize Your Textures
Although most images are non-linear, sometimes you should leave them that way.
Knowing how and when to deal with colour spaces in your textures files is crucial to a realistic rendering workflow.
In short: since monitors are usually non-linear devices, image files are generally stored with the inverse non-linearity baked in (the general example of which is sRGB). Since rendering calculations work in a linear space, you must convert your images back to the linear space before using them otherwise your renders will not be accurate.
While it is key that you linearize your colour textures (e.g. diffuse, specular colour, etc.), you must be very thoughtful of doing the same with your data textures (e.g. displacement, bump, etc.).
For example, lets look at a displacement:
The left is a linear-step gradient (created by a Python script so that the values are guaranteed linear steps of 10%). The middle is the result of using that texture as displacement as-is; notice how it lines up perfectly with the reference plane. The right is the result of using that texture after "linearizing" it via de-sRGB; it no longer follows the reference plane, an exhibits the standard sRGB curve.
If your data textures are created with the specific values in mind (e.g. you picked valued in Photoshop numerically) then you must not perform the standard de-sRGB step to linearize those textures, as they already are linear! If you painted a data texture and judged it by eye, then (perhaps) you should de-sRGB since the perceptual differences are more important.
Unfortunately there isn't an easy rule to follow. You must always be aware of what colour space your textures are stored in, and whether their contents were created perceptually or by value.
As always, the VES "Cinematic Colour" paper is an amazing resource for learning the details of colour workflows for film and VFX. For a quick primer on this topic specifically you can Google "linear workflow