In a medium saucepan, whisk together the granulated sugar, flour, and salt until well combined.
Gradually whisk in the milk, ensuring there are no lumps.
In a separate bowl, lightly beat the egg yolks. Slowly add about half of the warm milk mixture to the egg yolks, stirring constantly to temper the eggs.
Pour the egg mixture back into the saucepan with the remaining milk mixture. Cook over medium heat, whisking constantly until the mixture thickens and comes to a gentle boil (about 8-10 minutes).
Remove the saucepan from heat and stir in the vanilla extract and butter until melted and smooth.
In the bottom of a 9x13-inch baking dish, layer half of the vanilla wafers. Top with half of the sliced bananas and half of the pudding mixture. Repeat with the remaining wafers, bananas, and pudding.
Cover and refrigerate the banana pudding for at least 2 hours to allow it to set.
Before serving, top with whipped cream if desired.