Finishing a game requires a LOT of persistence and a lot of ability to just plough through small details and so forth.
But sometimes you need to recognize when something you're working on should be just dropped. I don't mean a whole game, I mean a feature or an idea. You've already decided to add a certain feature; how do you undecide?
You can't just undecide based on difficulty. Some features may be necessary but fairly painful to implement, for example hunting down memory leaks. This could take you a few days but you can't really avoid it. For these, I think you just have to stick to it.
Other features may not be necessary but seem like a good idea. I.e., if you could get this feature in it would fix certain gameplay-related problems. In this situation, I think you normally have to follow through as well. Reason being, if you don't pay attention to certain problems with gameplay, those problems are going to ruin the whole rest of your game.
Sometimes you can comprimise. For instance, a blog post or so ago I talked about NPCs opening doors. "Properly", the NPC should determine that they are in front of a door, stop, open the door, walk through, stop, and close the door. This is a ton of game logic. Instead, I took the middle ground and had them detect when they were already on top of a door, open it (so that technically it opens late) and then once they are off of it, close it. They don't stop, and they technically pass through the door before opening it. But it's a good comprimise. Without this, the game have a rather large gaping "hole": NPCs would pass miraculously through doors. I can't really get away without fixing that.
So when do you actually decide to cut your losses and drop a feature altogether?
I would say, you "undecide" when you realize that you made bad decision, and why.
Recently, I decided to clean up the side-to-side area transitions. When you move from areas on the world map, there was a "wiping" transition that sort of wipes the screen to black, then unwipes it. Not really terrible but as someone pointed out, quite jarring. I had stopped seeing it as jarring, but after reading a comment on a video I realized, in fact, it was.
This is probably something I need to fix, one way or another. It breaks continuity in a fairly major way.
But I made a bad decision. I wanted to make the screens "slide". If you've played the original Legend of Zelda, or the SNES version Link to the Past, you know what I mean. I thought this would be a nice retro effect, and provide good transition.
But there were problems with decision:
First, the game engine was built from the very start to assume there is only one "area" live at a time. This was in order to simplify implementing many things and to remove many design decisions that would otherwise have to be made. As well, it gives the player a nice sense of "concreteness"; the game is divided into discrete areas, so they have a good sense of where they are.
Allowing for a slide transition meant I had to jerry rig the engine to have more than one area open or "live" at once. This was problematic, though I eventually did actually get it working. Not a total hack, but far from elegant. And it had (as you may expect) some unforseen consequences that would have required perhaps some painful fixing, for example there were lighting and "noise" seams between areas.
Framerate also dropped off significantly during the slide transition, because we're rendering more than one area at once. This sort of negates the benefits of sliding, since the framerate becomes chop-city.
The other problem had to do with the camera, and coordinate systems. Again, stemming from the fact that I only wanted one area open, everything is coordinated relative to the current area. Changing areas means recoordinating: the camera now needs an offset, we need to know where the player should be relative to each screen, etc. etc. Ugly!
The first part of the solution was to dump this idea, which I happily did. I needed to recognize why I had made a bad decision, and cut my losses. It's not just that it was difficult. It was difficult BECAUSE it conflicted with other core design decisions made for the game.
The second part is to come up with a replacement that will not be so disrupting. In this case, I need some element to create a sense of continuity. That element, it turns out, will be the sky.
The sky right now during the daytime shows just blue (at nighttime it shows stars, which is good). My plan, not yet implemented, is to have clouds going back and forth on the sky. I will then have a nice faux-reflective effect on water surfaces which shows this sky. The effect is overall somewhat trippy, since the sky is visually "below" you (rendered underneath the terrain, along the edges) but in reality represents something "above" you. It was a cute idea when I had it and a great one.
What I'm going to do is put animated clouds on the sky. These clouds will blow back and forth peacefully creating a nice sense of place. When I transition between areas, I will wipe not the entire screen but just the terrain/area etc, so that the sky shows beneath it. This way, we get a greater feeling of continuity without violating the core mechanic.
Later, I can update it to also show the player (a bit tricky, but it should be possible), and the effect should be complete.
Don't give up on solving problems that need to be solved, but cut your losses when you realize you've made a bad decision.