No problem I've been doing it for years, as Steve says if it's being painted over. Just make sure you rub it down flat with surrounding surface and fill and holes that usually appear after you've packed the stuff away.
Years ago in the dark old days it used to be sawdust and glue mixed then some used wall plaster or chalk mixed with glue which was hard to flatten.
Then lo and behold a there came to our land of plenty, oops! forgot I was in UK! land of less than plenty, a person who invented white and grey powder nad he called it filla and all the people in property building and mantenance work danced with glee at the filla because it is so easy to work with.the end. Sleep well..