Preheat your oven to 350°F (175°C). Line a baking sheet with parchment paper and set aside.
In a large mixing bowl, combine the mashed sweet potatoes, granulated sugar, brown sugar, vegetable oil, eggs, and vanilla extract. Mix well until smooth using an electric mixer or whisk.
In another bowl, sift together the flour, baking powder, baking soda, cinnamon, nutmeg, and salt.
Gradually add the dry ingredient mixture to the sweet potato mixture, stirring gently until just combined. If desired, fold in the chopped pecans.
Pour the batter into the prepared baking sheet, spreading it evenly. Bake in the preheated oven for about 30 minutes, or until a toothpick inserted in the center comes out clean.
Remove the cake from the oven and allow it to cool in the pan on a cooling rack for about 15 minutes. Then, carefully remove the cake from the pan and let it cool completely on the rack.
While the cake cools, prepare the cream cheese frosting. In a mixing bowl, beat the softened cream cheese and butter together until smooth. Gradually add the powdered sugar, mixing until well combined. Add the vanilla extract and enough milk to reach your desired frosting consistency.
Once the cake is completely cool, spread the cream cheese frosting evenly over the top of the cake. Cut into squares and serve.