The correct option is
D were
The verb 'was' refers to past tense. The given sentence states two actions that are parallel in meaning.
I wasn't tired but my friends were tired. 'But' presents a contrast between these two actions 'I wasn't tired' but 'my friends were tired'.
(Also, 'but my friends ...' is an elliptical sentence, because the verb 'tired' is missing. A word can be omitted or implied if it has been mentioned before. The word 'tired' has been mentioned before and is hence omitted.)
Option B: 'Were' s the plural form of 'was'. The subject and verb must agree with one another in number. So when the subject 'I' is singular, the verb is singular 'was' and when the subject 'friends' is plural, the verb 'were' is also plural. This verb forms a complete meaningful sentence and hence option B is correct.
Option A: 'Do' refers to an action. But the verb 'tired' is a state of being, not an action. Also, it is not consistent with the former part of the sentence 'was tired'.
Option C: 'Could' refers to possibility.
Option D: 'Should' refers to suggestion or obligation.
The sentence does not express any of these moods (possibility or obligation).
Hence options A, C and D are wrong.