What is the simple subject of this sentence? Some of the rocks were hanging from the ceiling?

Since they they can be subject or object depending which way round you look at it, I suppose either rocks or ceiling could be called the subject.

The subject is the indefinite pronoun some.

Some [subject] of the rocks [prepositional phrase] were [linking verb] hanging from the ceiling [subject complement].

Read more about subject-verb agreement.