Torment the Tiefling had by far the most detailed backstory of anyone in the party. From the first session, the wild magic sorcerer was incredibly engaged in roleplay and seemed completely invested in the story (detailed here). That game showed me a flaw in the party when they brushed by a TPK. Nobody ever wanted to heal. With two paladins and an artificer, I figured they would be fine, but they always chose hitting the baddie over healing a partner. This continued into the next game, when Torment decided to persuade a pack of gnolls to stop collecting tolls.
Torment walked up alone and spoke to the hulking hyena-men. The leader of the gang spoke only broken Common, and Torment was refusing to pay. Obviously, when he tried to push past, a gnoll grabbed him. He struck back with a Booming Blade and we rolled initiative. He never stood a chance. One round later, the rest of the party arrived and Torment was on two death fails. The gnolls defended themselves from the party, but none of the PC's healed their fallen comrade. In the third round, despite being right next to Torment, who is on two fails and a success, the Paladin forgoes healing to smite an enemy. Two of the gnolls even ran, but the party chased and killed one before returning to heal Torment, who was on two fails and two saves when they arrived.
I should note, these players aren’t idiots. They had all played four sessions together at this point and knew how combat works. Some have been playing for years, even DMing. They knew their third level abilities, but chose not to use them.
So they burned the bodies and returned to the inn. That night, they discovered that a blacksmith they'd been buying from had been kidnapped and his home torched. After Torment and the artificer looted the house, they long rested and set off in the morning to follow the tracks. Many hours later they arrived at a camp, one of them subduing a giant hyena guard while the rest of the party handled the six gnolls in the camp, the seventh escaped, running north. In the fight, Torment burned down two tents. Having expended many spells and taken damage, the party chose to take a long rest. I allowed it, but made sure they understood the fact that they were long resting in a camp from which a gnoll had escaped. They understood and the warforged kept watch (one was focused on their restrained pet). After a few hours, one of them saw two giant hyenas and several gnolls rush the camp. Initiative was rolled and Torment tossed a chaos bolt. When he did, his wild magic kicked in and engulfed the party, the Hyenas, the gnolls, and the rest of the tents in a massive fireball. Over the next few rounds, the artificer and monk made short work of the remaining enemies and the paladin gained more trust from the Hyena with healing and food. After the fight, the party was bloodied and magically tapped out. They were in severe need of a long rest, most of them only having a hit die left. They were so battered they didn't even chase after the one of their assailants. Instead, they decided to try again for a rest in the now empty, ash covered field. Again, I reminded them that they were in a gnoll camp and that one had just escaped. They decided to have the monk keep watch along with the two warforged this time.
Obviously, this was dangerous. They were low on resources and health and they knew it. Even so, they chose to risk another assault. This time the leader of the gnolls, a fanatic human cultist of Yeenaghu, along with several witherlings, and six gnoll archers arrived to investigate. It was brutal. The party only survived the melee fighters because they incurred the wrath of Ed (the giant hyena). Ed fell to the archers a round later, who then focused fire on the party. All the PCs dropped prone in the dirt and ash, now muddy with blood. Their gunner, the artificer, went unconscious, as he posed a ranged threat. He got back up as the party formed a phalanx, throwing javelins, daggers, and cantrips. Then, Torment, the frontline mage, decided to take out one of the archers, who was 30 feet away. Torment chose to crawl out to the vicious gnoll, breaking from the group that was turtling their way towards the spellcaster. Using his action to firebolt, he could only crawl about 15 feet.
They had enough warning. They were experienced enough. They should have known what was coming. The archer, seeing the lone tiefling, burned, bruised, and bleeding, crawling towards the edge of the clearing dropped his bow and ran in for the kill. The first stab from the gnoll made Torment slip into darkness. The second brought him to two death saves. The paladins, each with one point of lay on hands left, chose to fight the spellcaster as I rolled Torment's death save. He rolled a 7. They knocked out the caster and the two remaining archers fled. The party rushed to Torment, but no amount of lay on hands could help. He had died.
He didn't have to die. They could have left the camp and rested in the surrounding forest after the first fight. They could have walked back to town after the second fight. They could have run away in the last fight, but they chose to stand (or crouch) and fight. It's these choices that make deaths feel earned. They knew the risks of each of their choices and they accepted them. The gritty, deadly fight was in no way a surprise to the players. A permanent death should never feel like a random event. It should be based firmly on the specific choices of the PC's.
That's what makes death different for DM's and PC's. Every death, whether it be a villain or a player, has to be contingent on the decisions of the players. The DM could kill any character at any time but must instead surrender that control to the other people at the table. A DM should make a world that can kill a PC, but it has to be a PC’s choices that actually kill them.