No , it's not a silly question. Let us define a hypothetical quantity fynsnman, whose derivative is displacement. Then the area under displacement time graph will represent fynsnman. Physically it doesn’t make any sense, that’s why we don’t talk about it. On the other hand third derivative of displacement is called jerk and still has physical interpretation.