I poked my skin with a pencil and I think graphite is stuck inside. Should I just leave it in?

It depends. General rule of thumb: remove any foreign body from wounds. With a pencil, often one thinks there is something in the wound due to the tatooing from the graphite. Look at the pencil to see if the point broke. If the point is intact usually just cleaning the wound and using an antibacterial ointmnet is sufficient, if you suspect the point is in the wound, treat it like a splinter.