I've never driven directly to NYC (I usually putz around a bit before we get there), but it takes just over an hour to get home from Staten Island. I'd say it would take about 90 minutes to get to Manhattan via the Holland Tunnel.
The time depends on two factors: Where you live in Philadelphia and how bad traffic is in NYC and North Jersey. The average time to NYC from Philadelphia is a tad over 2 hours. I live at the far NE end of the city, and not too far from the NJ Turnpike, so it's pretty easy for us to go up there. By contrast, UT lives about 40 miles west of me, so I would imagine it takes him about 30 min-1 hr longer.
When Rho and I drove to Boston from DC 2 years ago, it took us about 12 hours, when it should have only taken about 8. It took us 4 hours to get from Newark, NJ to the NY/CT state line...a distance of about 40 miles. Most of that time was spent sitting on the approach to the George Washington Bridge, trying to get into Manhattan. We hit Newark around 6pm. Bad timing.
(Sycamore's suggestion of the day: When driving to points beyond NYC and you're using the NB NJ Turnpike, take I-278 E from the NJ Turnpike to I-495 E in Brooklyn, then take I-678 N in Queens across the Whitestone Bridge to I-95 N in the Bronx. It takes about 45 minutes and you spend about $10 or so on the tolls. It's worth it.
)